Cosign 1.12.0 broke Flux build #899
Comments
|
Happy to help debug from the Sigstore side of things. If I had to guess, it's actually related to the fix to GHSA-8gw7-4j42-w388, which made verification more strict for If you can point me to the following, I'd be grateful:
|
|
@znewman01 to reproduce this locally do: curl -o ./checksums.txt -sfL https://github.com/fluxcd/golang-with-libgit2/releases/download/v0.2.0/checksums.txt
curl -o ./checksums.txt.pem -sfL https://github.com/fluxcd/golang-with-libgit2/releases/download/v0.2.0/checksums.txt.pem
curl -o ./checksums.txt.sig -sfL https://github.com/fluxcd/golang-with-libgit2/releases/download/v0.2.0/checksums.txt.sig
cosign verify-blob --cert ./checksums.txt.pem --signature ./checksums.txt.sig ./checksums.txt |
|
Will reproduce, 1 sec... In the meantime: does it still happen if you set |
|
With experimental enabled it doesn't fail but I see lots of duplicate warnings.... |
|
Okay, it's expected that With Sorry about the warnings....that's an unintended consequence of sigstore/cosign#2232. A fix is being tracked upstream: theupdateframework/go-tuf#376 . We're hoping to push a minor version bump soon to address that. |
|
@znewman01 shouldn't this be documented as a breaking change in Cosign release notes? |
|
Yup, filed sigstore/cosign#2253 |
|
Hi! Thank you for filing this issue! I want to call out two details too:
The signing command would add an output bundle flag. |
Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com> <!-- Hi, thanks for contributing! Please make sure you read our CONTRIBUTING guide. Also, add tests and the respective documentation changes as well. --> Cross-ref: fluxcd/source-controller#899 Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com>
|
After setting I have no clue why rerunning the job fixes it |
|
That's an intermittent failure, tracked here: sigstore/rekor#1055 |
|
Update on that flake: it was fixed and we rolled out the fix, but then had to rollback for an unrelated reason. As soon as we can roll forward again it should disappear. The previously-linked issue should have more details soon. |
|
@znewman01 is there any way to suppress the tuf logs? In #876 we can’t use the 1.12 packages because this will mess up Flux json logging. Given that Flux only deals with verification of OCI artifacts, is it safe to ship Flux with Cosign 1.11? |
|
Yeah, the TUF logging bit is very annoying, and there's no easy way to suppress it As you're alluding to, the most important thing about 1.12 is that it fixes [a vulnerability] when verifying blobs, not OCI artifacts. Looking at the fix, the only change to the |
|
Thank you @znewman01 We'll go ahead with |
|
Hey! I updated the sigstore code to remove the logging (sigstore/sigstore#701) yesterday, and am updating cosign right now. It should be in the v1.12.1 patch! |
|
v1.12.1 just went out; I think we missed it: https://github.com/sigstore/cosign/releases/tag/v1.12.1 |
|
Awesome! Thanks @asraa and @znewman01 We'll give this a try in #876. |
|
TUF logs are gone after updating the controller to 1.21.1 🎉 I'm going to close this issue now, thank a lot for all the help 🤗 |
|
Seems that blob verification is broken again (cosign v1.12.1), to reproduce do: curl -o ./checksums.txt -sfL https://github.com/fluxcd/golang-with-libgit2/releases/download/v0.2.0/checksums.txt
curl -o ./checksums.txt.pem -sfL https://github.com/fluxcd/golang-with-libgit2/releases/download/v0.2.0/checksums.txt.pem
curl -o ./checksums.txt.sig -sfL https://github.com/fluxcd/golang-with-libgit2/releases/download/v0.2.0/checksums.txt.sig
export COSIGN_EXPERIMENTAL=1
cosign verify-blob --cert ./checksums.txt.pem --signature ./checksums.txt.sig ./checksums.txtAnd the output is: |
|
FYI we can repro and are investigating. |
|
Do you know the entry that was originally uploaded? Maybe in the logs of a signin workflow? Got it: |
|
@asraa the full build log is here if you need it https://github.com/fluxcd/golang-with-libgit2/actions/runs/3060959453/jobs/4940247065 |
|
Thanks! We've identified the problem, working on fixes. |
|
Okay, I've only followed this secondhand, but I think this is what happened; correct me if I'm wrong, @asraa :
So:
EDIT: please ignore in favor of @asraa's comment below |
This was not quite the problem: Rekor only searched the active shard. It would fail if the entry was not on it. (It doesn't matter if the entry was on >1 shard.) With the fix, we'll now return all entries, even on previous shards. So if you were only on a previous shard, you'll get your entry. If in the case where someone duplicated the entry, then your client will reieve >1 entry and fail.
I would suggest not! If you do then you'll fail UNTIL the client updates. If you don't and stay on the inactive old shards with a single entry, old clients can continue verifying because they'll receive only one entry when querying, as was always the expectation. |
The libgit2 libraries are downloaded and verified before some of the make targets are executed. This assures the provenance of such files before using them and is very important specially for end users running such tests on their machines. Note that has been disabled specially due to recent issues we experienced at CI which can be seen in: fluxcd/source-controller#899 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
some of the make targets are executed. This assures the provenance of such files before using them and is very important specially for end users running such tests on their machines. Note that has been disabled specially due to recent issues we experienced at CI which can be seen in: fluxcd#899 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
|
We're working on fixing the client, but in the meantime, we'd recommend re-signing to unblock you. |
|
This sigstore/cosign#2294 issue tracks the fix! |
|
I've tested with latest Cosign and seems to work OK. Thanks! |
The libgit2 libraries are downloaded and verified before some of the make targets are executed. This assures the provenance of such files before using them and is very important specially for end users running such tests on their machines. Note that has been disabled specially due to recent issues we experienced at CI which can be seen in: fluxcd/source-controller#899 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
After Cosign v1.12.0 our builds are failing with:
We use keyless to sign the golang-with-libgit2 release assets and we use cosign verify-blob in all Flux controllers that make use of those assets. Since this basically broke Flux build all over, we have two options: remove cosign from our build system or revert to cosign 1.11.0 that is affected by CVE-2022-36056.
The text was updated successfully, but these errors were encountered: