Generate SLSA provenance for binaries in GitHub releases

[laurentsimon]

Hi,

I'm reaching out on behalf of the Open Source Security Foundation (openssf.org). We work on improving the security of critical open source projects like yours.

Together with GitHub, we designed a free, easy-to-use method of code signing. It will help your users verify that your release binaries were built from your repository’s workflow and not altered by anyone. It’s just a few lines of code, but it will make your project more secure against third-party tampering and attacks like Codecov and CTX.

This PR shows how to add this seamless code signing to your workflow. You don’t have to be a cryptography expert or learn complicated tools and verification is simple for your users.

You can read more on the SLSA blog. Please reach out if you have any questions!

[laurentsimon]

@crazy-max sorry if I confused you. I've added the context from the corresponding issue #1242

Anything I can clarify?

[laurentsimon]

friendly ping. Any question I can help answer?

[crazy-max]

Sorry for the delay. I have some concerns and questions about this workflow.

First I would like to know what is the maturity/stability of this action/project? Looking at its usage across GitHub it seems only used by your OSSF scorecard project and 2 other small repos.

Another point quite relevant to me is "Why buildx is your first PR to bring this workflow up?". I would expect, as you're telling working with GitHub, that GitHub adopts it first or other major repos. Like https://github.com/cli/cli and also makes a blog post about it. Do you have more resources about it and your roadmap with GitHub?

Also it looks like the slsa-framework/slsa-github-generator action is not verified by GitHub and not available in their marketplace. I would expect an official action in actions org or github org like codeql.

Finally I have security concerns about artifacts that could be tampered. The composite action slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.2.0 being used invokes sub actions but does not build our project as it should and afaik is not SLSA 3 compliant. An SLSA releaser workflow needs to build, sign, generate provenance and push the artifacts with signed provenance:

[laurentsimon]

Sorry for the delay. I have some concerns and questions about this workflow.

First I would like to know what is the maturity/stability of this action/project? Looking at its usage across GitHub it seems only used by your OSSF scorecard project and 2 other small repos.

Great question. We are starting adoption with critical projects as of last week or so, which is why you don't see many repositories using it yet.

docker/compose#9702 and ko-build/ko#730 (comment) are considering using it too.

We have identified your repository as being critical, which is why we're reaching out early.

Another point quite relevant to me is "Why buildx is your first PR to bring this workflow up?". I would expect, as you're telling working with GitHub, that GitHub adopts it first or other major repos. Like https://github.com/cli/cli and also makes a blog post about it. Do you have more resources about it and your roadmap with GitHub?

We're working for wider adoption with GitHub, e.g., for Npm, see the mentions of slsa-framework/slsa-github-generatorhttps://github.com/npm/rfcs/blob/ab81e1c2be00002cb137dbb7ea2e3b2ce0d8c9c7/accepted/0000-link-packages-to-source-and-build.md.

The corresponding blog is https://github.blog/2022-08-08-new-request-for-comments-on-improving-npm-security-with-sigstore-is-now-open/

We are starting adoption as part of OpenSSF, which GitHub is a member of. I will add this repository to our list. Thanks for pointing this out.

Also it looks like the slsa-framework/slsa-github-generator action is not verified by GitHub and not available in their marketplace. I would expect an official action in actions org or github org like codeql.

Why do you mean by "verified"? Being on the market place? GitHub currently does not support verified "re-usable workflows", which is why it does not show up in the market place (they are working on it).

The Go builder is in the starter-workflow, meaning it is a "partnered action" and you can install via the GitHub "New workflow" tab.

The generic workflow is under review to be a partnered action as well, see actions/starter-workflows#1639 and will be merged soon. Thanks for pointing this out, I've asked for updates on the PR.

Finally I have security concerns about artifacts that could be tampered. The composite action slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.2.0 being used invokes sub actions but does not build our project as it should and afaik is not SLSA 3 compliant. An SLSA releaser workflow needs to build, sign, generate provenance and push the artifacts with signed provenance:

There are various requirements for SLSA3: build and provenance, see https://slsa.dev/spec/v0.1/requirements. This workflow will satisfy the provenance requirement, which provides a strong link between the source and the repository, as explained in GitHub's latest blog post https://github.blog/2022-08-08-new-request-for-comments-on-improving-npm-security-with-sigstore-is-now-open/.

Even though the workflow in this repository does not build, it satisfies the provenance level 3. In practice, this is enough for someone to determine the repository and hash commit at which the build was made, and manually inspect your workflow if they want to know more about how your build was made.

We are working on new builders that can take Dockerfile to define a build, in order to be flexible for different users (see tracking issue slsa-framework/slsa-github-generator#56)

Hope this clarifies the context. Happy to answer more questions if you have any.

Thanks again for the feedback.

[tonistiigi]

I don’t see added security value in this for the users. The build still happens in our workflow (and that creates the source connection to github that we already store in our release images and checksums). Afaics your “trusted workflow” just takes a digest that is already available/published, adds its custom signature file to it (without any way to verify that the binary digest had some authenticity compared to the actual source) and then creates a custom verifier binary for checking the file that the same project generated.

That’s not saying that I don’t see value in signed provenance, but that would need to happen within our actual build process so the provenance actually sees all the actual sources that make up the release(this is not just Git repo) and provides trusted verification path from these sources to the artifact. More like what your Go builder does, but we use containers to build and not run Go directly. Maybe moby/buildkit#2935 will get us closer to something similar in the future.

[laurentsimon]

The binding between the source repository and the final artifact is the value you get.
You could then verify from the GH Action https://github.com/docker/setup-buildx-action/blob/master/src/buildx.ts#L270, which would add transparency to users.

It's totally fine if you don't want to try it out. Appreciate your time and feedback.

[tonistiigi]

If this is just info for the user then they can already get it plainly from that URL. We also put this info into our build artifacts(binaries/images) during build.

If our CI would get compromised and this points to something malicious, then broken CI would just send the digest of the new malicious thing to you and your service would happily sign that as well. The signature (certificate of the signer) will contain the event that triggered the re-signing but there is no guarantee that the source code there was actually used to create the digest that was sent to you.

[laurentsimon]

If this is just info for the user then they can already get it plainly from that URL. We also put this info into our build artifacts(binaries/images) during build.

If our CI would get compromised and this points to something malicious, then broken CI would just send the digest of the new malicious thing to you and your service would happily sign that as well. The signature (certificate of the signer) will contain the event that triggered the re-signing but there is no guarantee that the source code there was actually used to create the digest that was sent to you.

This is actually not the threat SLSA protects against. SLSA is about provenance: a link to the original repo / workflow that triggered the build. The build steps are considered untrusted, even when you use a trusted builder. The trusted builder does use the final binary "generated" by the build steps, and the binary may just be wget http://malicious. SLSA still provides provenance about where the build came from. But it does not attest to whether your binary is malicious or not. A trusted builder who runs user-provided steps may also be pulling compromised binaries, or a compromised dependencies.

In the case of the Go builder, we're being restrictive and we only allow certain commands. In general, any slightly more complex build pipeline wont be able to record all the steps. For example: if we have a trusted builder that builds binaries using a Dockerfile provided by the user, we cannot record every steps in the Dockerfile. The trusted builder would record the top-level command like docker build or docker run, not RUN, etc. The provenance would provide a link to the original code, and a user can inspect the original workflow if they want.

You can think of it as traceability in the food industry. When there is a salmonella outbreak, they check the lot numbers and recall the items. The traceability number on the package does not attest to the quality of the product itself, it provides provenance information. You can add trust in the product if they have some labels (some SLSA level), if you have some amount of trust in certain companies and they have followed best practices

[laurentsimon]

Wanted to mention an early adopter, Java' container tool jib https://github.com/GoogleContainerTools/jib/blob/master/.github/workflows/jib-cli-release.yml

Thanks again all for the time you're taking to review this PR. I wanted to mention that none of this PR is for ever binding. If you were to try it out and decide later you don't want it, there is no key to revoke or key management to perform.

The feedback you may be able to provide would still be valuable.

[laurentsimon]

Something I forgot to mention:

The attestation PR moby/buildkit#2935 is interesting. We plan to allow building from dockerfile / container image (slsa-framework/slsa-github-generator#57). This is maybe something we could chat about offline using your work. I'm curious how you achieve isolation between the build and the collection of commands at run time. In our Go builder, the commands are generated / resolved without executing them, in a different VM. This ensure they cannot be altered by malicious build commands.

This PR, I believe, is compatible with later improvement that would build the binaries. What I mean is that all our builders are compatible for verification (same provenance format, same signing, etc). What this means is that you could start with the generic generator (this PR) for level 3 provenance, then later upgrade to a builder that build the binaries too (provenance + build level 3). The level of granularity of the commands reported could be the docker run commands (what we intend to do for our builders), or more granular (if we could re-use the work you're doing in buildkit). So there is a path to upgrades, without breaking compatibility for older binaries. Basically, all that would change in the provenance is the buildConfig field. With this PR, it's not generated. With our dockerfile / image builder (coming up in Q4), it would have the higher level docker run command (enough to try re-building); and later we could think of integrating your work for more granular commands.

The materials section of the provenance is also something that could be improved via the work you're doing in buildkit. There's still debate whether this should live in a separate SBOM or in SLSA. (the provenance can attest to an SBOM too)

I don't know exactly how much granularity is best. More commands is nice, but it's not obvious what is more actionable from a user's perspective. I don't know yet if users want to write policies for banning certain commands, etc.

Also note there is other work that could be interesting to brainstorm: policies at build time; for example, only build if the base image has provenance from a certain repo (e.g., distroless), etc

[laurentsimon]

If this is just info for the user then they can already get it plainly from that URL. We also put this info into our build artifacts(binaries/images) during build.

A project's maintainer can edit the binaries stored in a release assets, ie the assets are not immutable. There is no guarantees that the binaries are the result of running the GitHub workflow. SLSA provenance fills this gap, by "extending" the trust from source code to the binary. Users / consumers can verify that the binary they download was indeed the result of running your workflow. If you distribute binaries via other channels, the provenance still carries that proof. That's what provenance level 3 gives you (even if the workflow is the one that builds).

[laurentsimon]

Fyi, we onboarded another container project https://github.com/google/ko/blob/main/.github/workflows/release.yml#L38-L47

[laurentsimon]

Hi all, just wanted to ping to see if you have any updates. Happy to talk over video if you want, about either this PR or even collaboration

[tonistiigi]

@laurentsimon There is a dev branch on top of moby/buildkit#2935 in moby/buildkit@6b4e300...tonistiigi:buildkit:attestations-provenance to allow tracking the build from sources to the result. It's in early state with many missing things so not that ready for feedback yet. Example output https://gist.github.com/tonistiigi/003d3f49d646399d150a08ec30874398 . We should probably discuss it in some issue in Buildkit repo instead if you have comments.

We are in Docker community slack if you want to discuss something directly.

[laurentsimon]

Thanks. Sorry I'm OOO and will be back in a few weeks. I'll reach out when I'm back. Fyi, we're building a "bring your own builder" feature which would allow external builders (e.g., buildx) to be integrated with our SLSA generation easily.