Build vendor artifact

[catap]

Closes: #11760

[catap]

@ndeloof here it is

[ndeloof]

LGTM but should be declared in the merge workflow so this run for commits on main branch and tags

[ndeloof]

tidy is not relevant here, as we enforce go.sum to be up-to-date as part of the PR workflow

[catap]

Fixed

[catap]

LGTM but should be declared in the merge workflow so this run for commits on main branch and tags

ci workflow is triggered for by push into main brnach and tags as well. Also, this workflow makes all artifacts and it seems for me logical that it should be used to produce this one.

Have I missed something?

[ndeloof]

ci workflow is triggered by push to main but also by pull_requests, and this doesn't make sense to build this src archive for PRs
merge workflow is used for stuff to only happen on main branch and releases, like building bin image

[thaJeztah]

We should probably check if there's additional verification needed for these packages. The standard source tarballs on GitHub releases are produced from git, and therefore should always match the source code. This PR constructs a custom tar; while it's produced through GitHub actions, it's effectively still a "manual" upload, so we should check if there's ways to verify provenance and integrity.

Recent events with the xz backdoor showed that manually / custom uploaded sources were at the root of that backdoor (the xz project instructed users to not use the default tarballs produced by GitHub, but the ones that were manually uploaded).

cc @rachel-taylor-docker @gabriellavengeo

[catap]

@thaJeztah this is one of root cause why I'm doing this.

Right now OpenBSD port for docker-compose depends on .tar.gz which I build and host on my machine. Everyone should trust that I haven't been compromised and doesn't inject something very bad into this .tar.gz and has we seen such injection can be quite complicated.

So, the right way to do it is download from the same source where it sources is produced becuase this should be trusted source by defenition.

Regarding the way: as soon as I know go mod vendor respects checksums inside go.mod that should prevent from packing bad things. Or, let me say it that way: it should pack the same things which user should get if he run it locally.

[catap]

ci workflow is triggered by push to main but also by pull_requests, and this doesn't make sense to build this src archive for PRs merge workflow is used for stuff to only happen on main branch and releases, like building bin image

To keep things simpler I've added if inside this step.

Otherwise I need to move and duplicate code inside merge workflow which seems wrong from my point of view.

[thaJeztah]

Right now OpenBSD port for docker-compose depends on .tar.gz which I build and host on my machine. Everyone should trust that I haven't been compromised and doesn't inject something very bad into this .tar.gz and has we seen such injection can be quite complicated.

Yes, anyone producing binaries / artifacts for others to consume must have provenance. I think that goes beyond just these tars, and there would be many other ways for that build-environment / infra to be comprimised.

Regarding the way: as soon as I know go mod vendor respects checksums inside go.mod that should prevent from packing bad things. Or, let me say it that way: it should pack the same things which user should get if he run it locally.

Correct; it won't protect against a compromised tarball that has go.mod and go.sum tampered with though; so I'm looking if there's any way to have provenance for the uploaded tarball to verify if it's produced by the GitHub action, and not replaced in other ways.

The default tarballs produced by GitHub always match the git-commit of the release, but for other artifacts that's not the case, so some validation may be necessary for that.

[catap]

Regarding the way: as soon as I know go mod vendor respects checksums inside go.mod that should prevent from packing bad things. Or, let me say it that way: it should pack the same things which user should get if he run it locally.

Correct; it won't protect against a compromised tarball that has go.mod and go.sum tampered with though; so I'm looking if there's any way to have provenance for the uploaded tarball to verify if it's produced by the GitHub action, and not replaced in other ways.

The default tarballs produced by GitHub always match the git-commit of the release, but for other artifacts that's not the case, so some validation may be necessary for that.

I haven't see any differences with producing .tar.gz by this job with producing binary artifacts by the same github actions.

Second point: GitHub and other git hosting sometimes changes produced tar.gz from tag and they officialy confirm it, see: https://github.blog/2023-02-21-update-on-the-future-stability-of-source-code-archives-and-hashes/ and to avoid such mess they suggested to attach artifacts to releases.

[catap]

just a ping

[gabriellavengeo]

Is there a reason that we use actions/upload-artifact@v3 instead of v4? According to the action README, v3 will be deprecated in November.

[gabriellavengeo]

Speaking of v4, some of the breaking changed in it relate to my other comment bellow about the aritfact name.

2. Uploading to the same named Artifact multiple times.

Due to how Artifacts are created in this new version, it is no longer possible to upload to the same named Artifact multiple times. You must either split the uploads into multiple Artifacts with different names, or only upload once. Otherwise you will encounter an error.

[ndeloof]

aritfact name must be unique for a built commit, I don't think this has any impact for this workflow.

We still use upload-artifact@v3 on this repo as this breaking change require the whole workflow to be revisited to support the new behavior (see https://github.com/docker/compose/pull/11662/files)

[gabriellavengeo]

aritfact name must be unique for a built commit, I don't think this has any impact for this workflow.

Ah ok, then this should be fine.

[ndeloof]

Can you look into https://docs.github.com/en/actions/security-guides/using-artifact-attestations-to-establish-provenance-for-builds#generating-build-provenance-for-binaries so this additional artifact gets provenance attestation and you can safely rely on this for an offline build ?

[catap]

On Mon, 06 May 2024 13:54:21 +0100, Nicolas De loof ***@***.***> wrote: Can you look into https://docs.github.com/en/actions/security-guides/using-artifact-attestations-to-establish-provenance-for-builds#generating-build-provenance-for-binaries so this additional artifact gets provenance attestation and you can safely rely on this for an offline build ?

Thanks for the link. I've read it and I doesn't see how it can help me. OpenBSD ports, for example, ships its own checksum of artifacts which should be used to build port. It is checked by build servers which downloads artifacts from github.com or other plases and confirms that checksum holds.

…

-- wbr, Kirill