ci: Generate SLSA signatures for GitHub released tarballs

[asraa]

Signed-off-by: Asra Ali asraa@google.com

Hi,

I'm reaching out on behalf of the Open Source Security Foundation (openssf.org). We work on improving the security of critical open source projects like yours.

Together with GitHub, we designed a free, easy-to-use method of code signing. It will help your users verify that your release tarballs were built from your repository’s workflow and not altered by anyone. It’s just a few lines of code, but it will make your project more secure against third-party tampering and attacks like attacks like Codecov and CTX.

I’ve created this PR to shows how to add this seamless code signing to your workflow. You don’t have to be a cryptography expert or learn complicated tools and verification is simple for your users.

Furthermore, I have tested the workflow on my own fork. You can see the release with the attached provenance here. I would love to add verification instructions as well, but don't see a place where this is mentioned yet. Verification looks like

$ go run ./cli/slsa-verifier -provenance ~/Downloads/attestation.intoto.jsonl -artifact-path ~/Downloads/go-containerregistry_Linux_arm64.tar.gz -source github.com/asraa/go-containerregistry
Verified signature against tlog entry index 3209559 at URL: https://rekor.sigstore.dev/api/v1/log/entries/40191448c1cf944ad66fe37e196e351775fc21b77215f30049ecf9101d05e97b
Verified build using builder https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.2.0 at commit 58229d2a58b3f3a118d2e48dc44662c02fe77959
PASSED: Verified SLSA provenance

You can read more on the SLSA blog. Please reach out if you have any questions!

[asraa]

Related: #1219
This PR does this for the GitHub release artifacts.

[codecov-commenter]

Codecov Report

Merging #1438 (58229d2) into main (7196cf3) will decrease coverage by 0.09%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1438      +/-   ##
==========================================
- Coverage   73.40%   73.31%   -0.10%     
==========================================
  Files         115      115              
  Lines        8757     8772      +15     
==========================================
+ Hits         6428     6431       +3     
- Misses       1688     1696       +8     
- Partials      641      645       +4     

Impacted Files	Coverage Δ
pkg/crane/append.go	55.17% <0.00%> (-15.67%)	⬇️
pkg/v1/stream/layer.go	88.05% <0.00%> (+0.38%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[imjasonh]

Hey Asra 👋

Thanks for this, and it looks good! When @laurentsimon proposed similar for ko we asked to have a simple verification step in the release process, to ensure what we produced was verifiable. This acts as both a working example for folks interested in verifying it themselves, and as an immediate check that it's verifiable, in case it breaks in the future.

That's here:

https://github.com/google/ko/blob/3baf14de6e43feda467200d1fb9c58304845f1da/.github/workflows/release.yml#L49

Could you add a similar workflow here as well? It might also be useful to add to other projects as you roll this out.

[abitrolly]

attacks like Codecov

Did I read that correctly, that they hosted public Docker image with committed keys?

"The attacker was able to extract an HMAC key for a Google Cloud Storage service account from an intermediate layer in our public Codecov Self-Hosted Docker image. "

[abitrolly]

@asraa what is the attack vector? From the command line it looks like I just need to forge slsa-verifier-linux-amd64.intoto.jsonl to correspond to modified slsa-verifier-linux-amd64. How SLSA makes it hard to do this?

go run ./cli/slsa-verifier -artifact-path ~/Downloads/slsa-verifier-linux-amd64 -provenance ~/Downloads/slsa-verifier-linux-amd64.intoto.jsonl -source github.com/slsa-framework/slsa-verifier -tag v1.3.0

[imjasonh]

attacks like Codecov

Did I read that correctly, that they hosted public Docker image with committed keys?

"The attacker was able to extract an HMAC key for a Google Cloud Storage service account from an intermediate layer in our public Codecov Self-Hosted Docker image. "

That's what it sounds like, yeah. This is actually pretty easy to do with a Dockerfile:

FROM debian
COPY secret.json .
RUN ./use-secret.sh secret.json && rm secret.json
...

This sounds like the secret.json isn't going to be present in the final image, and you can even run the image and see that it's not there since it was rmed, but of course the file is present in the layer tarball anyway.

Tools like Trivy and Grype will scan for secrets in these intermediate layers, and I wrote a tool to demonstrate it simply here: https://github.com/imjasonh/chaff

@asraa what is the attack vector? From the command line it looks like I just need to forge slsa-verifier-linux-amd64.intoto.jsonl to correspond to modified slsa-verifier-linux-amd64. How SLSA makes it hard to do this?

The attestation is also signed by the GitHub Actions OIDC identity (example here), and verification presumably includes checking that the signature was signed by who it says it was signed by:

$ cat attestation.intoto.jsonl | jq -r '.signatures[0].cert' | openssl x509 -noout -text
...
            X509v3 Subject Alternative Name: critical
                URI:https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.2.0
            1.3.6.1.4.1.57264.1.3:
                58229d2a58b3f3a118d2e48dc44662c02fe77959
            1.3.6.1.4.1.57264.1.5:
                asraa/go-containerregistry
            1.3.6.1.4.1.57264.1.2:
                push
            1.3.6.1.4.1.57264.1.4:
                goreleaser
            1.3.6.1.4.1.57264.1.1:
                https://token.actions.githubusercontent.com
            1.3.6.1.4.1.57264.1.6:
                refs/tags/v0.1
...

This block means that the GitHub OIDC identity got this cert from Sigstore's Fulcio root CA (also verified) after doing an OIDC exchange with Fulcio to verify its identity.

edit to add: The -source flag to the verifier command tells the verifier to only accept the attestation if it's signed by the specified repo's OIDC identity. I assume we could be even more specific, and require the specific commit and workflow reference, and maybe other annotations.

[abitrolly]

Does it really protect from Codecov attack? If goreleaser executes tamplered codecov bash script, the compiled executable will become tainted and signed, giving a false sense of security.

[imjasonh]

Does it really protect from Codecov attack? If goreleaser executes codecov bash script, the executable will become tainted a signed, giving a false sense of security.

Correct, we're putting some trust into goreleaser not to inject nasties into our binary prior to signing the attestation, and not to run scripts that inject nasties prior to signing the attestation.

This is why we run as little as possible before/during our goreleaser process, and do our best to pin to specific goreleaser versions.

The addition of a signed attestation doesn't obviate the need for diligent code reviews and release hygiene. It would help a consumer of the built artifacts have more confidence in how that artifact was built, when, and by what process.

[asraa]

Thanks! @imjasonh had it all correct:

The attestation is also signed by the GitHub Actions OIDC identity (example here), and verification presumably includes checking that the signature was signed by who it says it was signed by:

Yes, to this point, if you modify the attestation's Subject to a malicious binary, then it fails verification. Our verifier checks the signature and also compares the subject digest to the released binary you are verifying.

I'll send a follow-up with verification and an update to verification instructions!

[abitrolly]

Tools like Trivy and Grype will scan for secrets in these intermediate layers, and I wrote a tool to demonstrate it simply here: https://github.com/imjasonh/chaff

Maybe crane needs a prune command in addition to flatten, to keep layers for reuse. :D But that won't remove secrets sneaked in ENV. I am not even sure how people remove ENV entries from subsequent layers. Setting to an empty string perhaps.

The addition of a signed attestation doesn't obviate the need for diligent code reviews and release hygiene. It would help a consumer of the built artifacts have more confidence in how that artifact was built, when, and by what process.

That still doesn't guarantee there are no meanies in dependencies during like https://github.com/google/go-containerregistry/pull/1435/files unless people are motivated to stake their $$$ on reviews. The crypto junkie in me just can not leave this thing alone without thinking how the whole thing could be replaced with distributed storage for signed content addressable hashes and DHT. :D

[abitrolly]

Yes, to this point, if you modify the attestation's Subject to a malicious binary, then it fails verification. Our verifier checks the signature and also compares the subject digest to the released binary you are verifying.

This still depends on a root CA, which should give automated certificates. Which means I can fork the repo and generate this (spot the difference).

            X509v3 Subject Alternative Name: critical
                URI:https://github.com/sIsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.2.0
            1.3.6.1.4.1.57264.1.3:
                <<differenthash>>
            1.3.6.1.4.1.57264.1.5:
                asraa/go-containerregistry
            1.3.6.1.4.1.57264.1.2:
                push
            1.3.6.1.4.1.57264.1.4:
                goreleaser
            1.3.6.1.4.1.57264.1.1:
                https://token.actions.githubusercontent.com
            1.3.6.1.4.1.57264.1.6:
                refs/tags/v0.1

If need to download the signature the same way I download binary to check, then if there doubts that binary is valid, there are the same doubts that signature is not forged. To me signed hashes of binaries on the blockchain is an easier solution. But I don't know what specific attack vector SLSA is trying to block.

[laurentsimon]

If you fork the repo the origin in the cert will be different: abitrolly/ko instead of google/ko.
Your idea about DHT is one that has been floated around and is not a bad idea (see https://github.com/google/ent)! /cc @tiziano88

[laurentsimon]

there's also sget https://github.com/sigstore/cosign#sget which is a curl-like tool but can verify hash / signatures, and uses container registries to index artifacts.

[abitrolly]

If you fork the repo the origin in the cert will be different: abitrolly/ko instead of google/ko.

But if I fork to googie/ko then human won't be able to tell the difference. The only reliable way is to import the key from google/ko that has all the stars. Which means there is no need to have GitHub and this Fulcio intermediaries. Went to look up where this Fulcio is located, and it got even more complicated. :D Now I need to understand how to trust that Fulcio, as it doesn't seem like the certificate authority that is already present on my machine.

Maybe it is the tailless blockchain with hashes to check against.. but it is not immediately obvious, and I am not ready to study it yet.

[asraa]

But if I fork to googie/ko then human won't be able to tell the difference

The verification instructions prevent this, due to the flag --source google/gocontainer-registry. See here:
https://github.com/google/go-containerregistry/tree/main/cmd/crane#install-from-releases

If you were importing a key, then you as a client would require making sure that you've pulled the latest, most up-to-date key. Verifying against the identity prevents this.

Furthermore, not only does it absolve maintainers of needing to manage keys and issue revocations, but it also gives users much, much more information than just proof of ownership and tamper-resistiance. It allows a user to inspect the provenance manifest, and inspect what commit it was built from and which workflow it was generated from.

Now I need to understand how to trust that Fulcio, as it doesn't seem like the certificate authority that is already present on my machine.

This is done through TUF -- our utilities bundle a trusted root and verify a remote repository against that using a chain of signed roots. It fetches the latest cert to verify against.