Upload multiple artifacts dynamically

[BToersche]

Adds support for uploading multiple artifacts using upload-artifact. Ofcourse, it is possible to upload multiple artifacts using multiple upload-artifact steps. However, this requires the amount of steps to be known beforehand. Sometimes this isn't the case however.

For instance, in Scala, SBT is usually used for building projects. This build tool allows for specifying which versions of the underlying language (Scala) should be used for a build (e.g. cross-build for Scala 2.12 and 2.13). The Github actions build doesn't know beforehand how many artifacts (cross-builds) are being created. Hence, given you don't want to change cross-build versions on multiple places (in build file and in Github actions), to support cross-building it is necessary to dynamically upload a compiled list of artifacts through a single upload-artifact step.

The way I've implemented this in this PR, is supporting JSON strings as an input:

- uses: actions/upload-artifact@v2
  with:
    name: '["my-artifact", "my-artifact-2"]'
    path: '["path/to/artifact/1/", "path/to/artifact/2/"]'

Ofcourse, it is backwards compatible and still allows for a normal string as input.
I've chosen JSON strings as a solution since Github actions already allows for this type in other fields when used in conjunction with ${{ fromJson(MY_JSON_STRING) }}.

Things like:

- uses: actions/upload-artifact@v2
  with:
    name: my-artifact
    path: '["path/to/artifact/1/", "path/to/artifact/2/"]'

work as well, in that case the artifacts will be named my-artifact and my-artifact_2 respectively.

When a JSON string is used for name, default artifact name is appended when not enough names are supplied:

- uses: actions/upload-artifact@v2
  with:
    name: '["an-artifact", "another-artifact"]'
    path: '["path/to/artifact/1/", "path/to/artifact/2/", "path/to/artifact/3/", "path/to/artifact/4/"]'

Artifacts will be named: an-artifact, another-artifact, artifact and artifact_2 respectively.

The way retentionDays is handled is done in the same way as name is handled (as described above).

Note that I did take notice of PR #138. However, the solution in that PR is limited to 1 file per artifact. This isn't going to work in the mentioned case above, since each artifact will require multiple files to be held (The build itself, the docs, sources and a pom file). I believe my solution will function for both, the case mentioned above and the log-files case mentioned in PR #138.

[konradpabjan]

At a high level, do you want multiple artifacts uploaded so it's easier to download each one. Something like this?

Or is the primary intention to reduce copy/paste YAML steps?

I'm brining this up because we have plans for adding something like a file browser for individual artifact files and allowing you to download individual files from an artifact.

It would be then possible to to do something like

- uses: actions/upload-artifact@v2
  with:
    name: my-artifact
    path: path/to/artifact

and in the UI you would be able to see

- my-artifact (button to download zip of everything in 1,2 and 3)
 - 1 (button to download zip of all files in 1)
   - ... to expand further
 - 2 (button to download zip of all files in 2)
   - ... to expand further
 - 3 (button to download zip of all files in 3)
   - ... to expand further

[BToersche]

To answer you question, yes it should be like the image you provided. Its primary intention is to allow for a dynamic amount of artifacts to be uploaded through your build action which are listed as separate artifacts in the artifacts list. So although it can reduce copy/pasting, this is not the primary intention.

The file browser feature seems to get pretty close to what I want, but is slightly off:

Common Ancestors

The feature you mentioned packages artifacts through their common ancestor, e.g.:

/my/path/to/artifact/1
/my/another/some/artifact/2

Will result in an artifact with top directory /my and subdirectories path/... and another/....
In the case of Maven (which is often used for Java/Scala projects) you would get:
/my-project/scala_2.12/build-version/artifact-files-here and /my-project/scala_2.13/build-version/artifact-files-here with common ancestor being /my-project.
This would include a directory in the scala_2.12 and scala_2.13 artifacts (zip files) with name /build-version. To be fair, this can be resolved by copying the files to a different directory structure in another build step, but it would be nicer if it were possible to use these paths directly through the upload step as multiple separate artifacts.

Listing artifacts separately

While using the file browser feature allows downloading artifacts separately, it sometimes might not make sense to allow for downloading all files at once. For instance, consider a build to have Linux/Windows/Mac runnable .exe artifact. It doesn't make much sense to combine these into a single artifact. For the user it makes more sense to list these separately.

The same applies for cross-building. It should be 2 separate artifacts which would normally be done through 2 upload-artifacts steps. (However, like I mentioned in my first comment, the amount of steps/uploads is dynamic depending on the amount of cross-build versions is supported.)

The artifacts themselves aren't related to each other (except them being build from the same source) and only one of them will be downloaded by a user. For instance, a user has software using Scala 2.12, in that case they will use the 2.12 build. In another case a user might have software using Scala 2.13, in that case they will use the 2.13 build. A single artifact containing both (Scala 2.12 and 2.13) builds wouldn't be preferred, its more user-friendly in this case to show both artifacts as separate items rather than a directory structure the user has to click through.

Immediate overview of versions

I'm expecting the directory structure you talk about to be collapsed by default (since it would likely be too unwieldy if it weren't). In that case, the user has to expand these directories first before being able to select the correct Scala version they need. So from first glance at looking at the artifacts listed it isn't clear which platforms are supported (be it Linux/Windows/Mac, or in this case, Scala_2.12 or Scala_2.13). A user has to first expand the artifact to see which versions there are, I suspect that users instead will download all three (Linux/Windows/Mac) versions, because it might not be clear at first glance they are packaged together, and then only use the one they need and drop the other two, which is a waste of time/bandwidth for the user and Github.

Final note

Note however I am definitely not against the directory structure feature you mentioned. In fact, it sounds like a pretty cool feature! I believe however that this PR and that feature could live side-by-side because their intent is different:

• This PR allowing you to upload a dynamic amount of artifacts to your artifact listing;
• The feature you mention allows for more control in downloading individual files/directories within a single artifact in the artifact listing.

[BToersche]

@konradpabjan Do you have any updates on this PR?

[elliot-nelson]

➕ I'd also like to see this feature (or something like it) merge.

The "file browser" addition that @konradpabjan mentioned would also be welcomed, but doesn't meet the same need: we really need to be able to target a list of discrete artifacts to upload, each one that contains a bunch of files, and each one can be targeted explicitly by downstream jobs and workflows. Having a "single big upload" that you can browse through wouldn't be enough in that scenario.

Right now we're forced to fork + use a copy of this branch internally which isn't ideal.

[initdc]

#354

[haadr]

Is there someone from github we can tag to get this reviewed and considered for merging? It's been two years and it would be good to know whether this is something that will happen or whether people should investigate alternatives.

[BToersche]

@haadr +1
I'd be willing to resolve the conflicts and make sure to bring this branch up-to-date.
But I'd first like to know whether this branch has any chance of getting merged or not before putting more time into it (which obviously would be a waste if this branch is going nowhere).

[t-lo]

We'd also find this feature very useful; we just ran into the one-artifact limitation when switching our CI over to Github Actions. Any chance of getting this merged, Github folks?

[filipsworks]

At a high level, do you want multiple artifacts uploaded so it's easier to download each one. Something like this?

Or is the primary intention to reduce copy/paste YAML steps?

I'm brining this up because we have plans for adding something like a file browser for individual artifact files and allowing you to download individual files from an artifact.

It would be then possible to to do something like

- uses: actions/upload-artifact@v2
  with:
    name: my-artifact
    path: path/to/artifact

and in the UI you would be able to see

- my-artifact (button to download zip of everything in 1,2 and 3)
 - 1 (button to download zip of all files in 1)
   - ... to expand further
 - 2 (button to download zip of all files in 2)
   - ... to expand further
 - 3 (button to download zip of all files in 3)
   - ... to expand further

Why create "sub artifact", and then maybe sub-sub artifact and all this kind of tree instead of letting people manage multiple artifacts with already existing actions?