cosign verify-attestation hangs indefinitely in GitHub Actions

[AliSajid]

Description

I have a GitHub Action that builds and signs an image and pushes it to GHCR and DockerHub. I verify the signatures in the same action. The verification for the image happens instantly but on the Verify-Attestataion for the SBOM, it hangs until it times out in six hours. I can verify that the attestation is pushed to the container registries and I can verify that locally on my Mac (M2) painlessly.

I'm using syft for SBOM generation and right now using a practically empty Dockerfile.

Version

cosign: v2.2.3
syft: v1.0.1

These are the logs from an example run.
logs_21813240831.zip

The workflow is here: https://github.com/AliSajid/aaprop/blob/next/.github/workflows/build_container.yaml

[haydentherapper]

This sounds like a one-off GHA failure, is it still occurring?

[AliSajid]

This has been consistently occurring over the past ~3 days. Sometimes it succeeds, but with an inordinately long time. An example of a very long run before success is here.

I have one action run happening right now which is going through the same process.

[ckotzbauer]

I can confirm the same behaviour in one of my actions.

[haydentherapper]

Without logs, I'm unable to reproduce this.

[ckotzbauer]

I create a repro build and share it here.

[ckotzbauer]

I created a simple reproduction repository and the workflow hung on the first execution: https://github.com/ckotzbauer/verify-attestation-repro/actions/runs/9044178111/job/24852568726

Between line 32 and 33/34 it took about 8 minutes.

[haydentherapper]

At a quick glance, the large payload stands out to me. Will have to dig in a bit more.

[AliSajid]

Hi. This is still an issue. Thank you, @ckotzbauer, for making the reproducible repo. @haydentherapper The large payload seems typical for what I've been doing. I may be doing it wrong but I've been generally following standard advice. I will appreciate any help.

[jku]

I have no idea yet why this would happen but I can confirm that PrintVerification() is fast locally but takes 8 minutes on GHA:

https://github.com/jku/test-cosign-verify-hang/actions/runs/9204640839/job/25318590422

EDIT:

As far as I can tell it's literally the fmt.Println() for payload printing that takes forever. It seems really weird: the printed data is not amazing (a single line of 300k characters) but I wouldn't expect that to take minutes

[jku]

We're likely hitting this: actions/runner#1031: the github runner is processing the line trying to find leaked secrets etc.

[jku]

As a workaround for folks affected: Here's a version of @ckotzbauer's workflow that runs in 4 seconds:
https://github.com/jku/test-cosign-verify-hang/blob/d6cddde9fa39d497e45ba86b3cbedfd3c07b0c55/.github/workflows/test.yml

• just redirecting STDOUT is enough to make it quick
• actions/upload-artifact is added to make sure the output is still available if needed

[ckotzbauer]

@jku Thanks for the workaround and some further analysis, much appreciated! 🎉

[AliSajid]

@jku Thank you so much. I did not even think of that being an issue. I hope GitHub fixes that. Meanwhile, I'll use the work around.