ECR security scanning fails on images built using BuildKit v0.11

[jedevc]

🐛 Originally reported by @farvour in docker/build-push-action#746 (comment), tracking here since the original issue is about adding attestations fields in docker/build-push-action.

cc @crazy-max @tonistiigi

With the release of Buildkit v0.11, by default, a minimal provenance attestation is created and pushed alongside the image, using the attestation storage. This apparently causes ECR image scans to fail.

docker/build-push-action#746 (comment):

I am using this action in our workflows to build. I am discovering that ECR does not like the attestation layers pushed when security scanning is enabled on a repository. It results in a Failed on the image UI state. Not a major issue, but sounds like this new feature to this action will let me turn these off with the newer docker buildx (0.10.0+) if my understanding is correct?

While the layers for the attestation storage cannot be unpacked, the OCI image-spec explicitly declares:

This descriptor property has additional restrictions for layers[]. Implementations MUST support at least the following media types:
...
Manifests concerned with portability SHOULD use one of the above media types. An encountered mediaType that is unknown to the implementation MUST be ignored.

Ideally, scanners should either:

• Skip over layers with unknown media types, as mentioned in the OCI spec
• Ignore manifests with unknown platform types

I don't think there's anything we can do buildkit side? As far as I'm aware, the attestation manifests generated by buildkit are compliant with OCI artifacts.

[tonistiigi]

With the release of Buildkit v0.11, by default, a minimal provenance attestation is created and pushed alongside the image,

Default is only in buildx level.

I don't think there's anything we can do buildkit side?

I think we are doing it correctly. The objects have proper mediatypes that show that blob is in-toto json and not a targz.

FYI @estesp

[farvour]

I did open a ticket with our AWS support to see what their response is. We are in US Gov Cloud in case that makes any difference.

It sounds like these layers just shouldn't be processed by Snyk or the failures bubble up to the UI. They are also un-deletable (as expected) since they are part of the list manifest.

We may have to wait for AWS ECR team to act for a resolution?

[farvour]

I did open a ticket with our AWS support to see what their response is. We are in US Gov Cloud in case that makes any difference.

It sounds like these layers just shouldn't be processed by Snyk or the failures bubble up to the UI. They are also un-deletable (as expected) since they are part of the list manifest.

We may have to wait for AWS ECR team to act for a resolution?

Spoke with our support and had a great call. Sounds like they will bring this back to their product and decide on how to present the layers to the user. I think it made the most sense to at least not try and scan the attestation layers, but still show them in the registry, even if unsupported. That makes sense.

Since none of this affects the actual functionality of the image, registry or tooling it's not blocking anything that I'm aware of (unless someone has alerting or monitoring set up on ECR layer failures, of course). Nonetheless, a resolution should happen.