c8d/list: Ignore attestation manifests #45124

vvoland · 2023-03-08T15:02:45Z

Related to:

Attestation manifests have an OCI image media type, which makes them being listed like they were a separate platform supported by the image:

$ echo 'FROM alpine' | docker buildx build - -t myimage
[+] Building 2.6s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                  0.0s
 => => transferring dockerfile: 49B                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                      2.0s
 => [1/1] FROM docker.io/library/alpine@sha256:69665d02cb32192e52e07644d76bc6f25abeb5410edc1c7a81a10ba3f0efb90a                                       0.4s
 => => resolve docker.io/library/alpine@sha256:69665d02cb32192e52e07644d76bc6f25abeb5410edc1c7a81a10ba3f0efb90a                                       0.0s
 => => sha256:af6eaf76a39c2d3e7e0b8a0420486e3df33c4027d696c076a99a3d0ac09026af 3.26MB / 3.26MB                                                        0.4s
 => exporting to image                                                                                                                                0.5s
 => => exporting layers                                                                                                                               0.0s
 => => exporting manifest sha256:e713f6240884609e976dd9d955de377ab272d2b5da3bb81cf3e0cc06fe92a2d5                                                     0.0s
 => => exporting config sha256:95f03905f1ed7ba6e36ab6c28bfd9c27ae83c6c9305fb852bfb3c26b4c8edf89                                                       0.0s
 => => exporting attestation manifest sha256:e7f3f0f7034c388bc64197f1086d1235e54106155f4a5ec7d2c5a55182f82bfd                                         0.0s
 => => exporting manifest list sha256:fd8f375da95ea8a50663e9499ae9abf56c2e0630ccd80575e35d36c83452b21d                                                0.0s
 => => naming to docker.io/library/myimage:latest                                                                                                     0.0s
 => => unpacking to docker.io/library/myimage:latest                                                                                                  0.5s


$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
myimage             latest              fd8f375da95e        1 second ago        3.26MB
myimage             latest              fd8f375da95e        1 second ago        2.62kB

$ ctr image ls
REF                              TYPE                                    DIGEST                                                                  SIZE    PLATFORMS                   LABELS
docker.io/library/myimage:latest application/vnd.oci.image.index.v1+json sha256:fd8f375da95ea8a50663e9499ae9abf56c2e0630ccd80575e35d36c83452b21d 3.1 MiB linux/arm64,unknown/unknown -

$ ctr content get sha256:fd8f375da95ea8a50663e9499ae9abf56c2e0630ccd80575e35d36c83452b21d | jq
{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:e713f6240884609e976dd9d955de377ab272d2b5da3bb81cf3e0cc06fe92a2d5",
      "size": 480,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:e7f3f0f7034c388bc64197f1086d1235e54106155f4a5ec7d2c5a55182f82bfd",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:e713f6240884609e976dd9d955de377ab272d2b5da3bb81cf3e0cc06fe92a2d5",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

- What I did
Fix docker images listing a separate image for attestation manifest.

- How I did it
Don't use images.Platforms and walk the manifest list ourselves looking for all manifests that are an actual image manifest.

- How to verify it

$ echo 'FROM alpine' | docker buildx build - -t myimage
$ docker images
myimage             latest              fd8f375da95e        1 second ago        3.26MB

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

vvoland · 2023-03-08T15:03:39Z

TBH, it feels wrong to actually work around this on our side, and I think attestation manifest should use a different media type.

daemon/containerd/image_list.go

vvoland · 2023-03-09T08:30:45Z

=== FAIL: github.com/docker/docker/integration/container TestHealthCheckProcessKilled (12.66s)
    health_test.go:111: polling check failed: expected "Health check exceeded timeout (50ms): logs logs logs\n", got "Health check exceeded timeout (50ms)"

Unrelated CI failure on Windows

daemon/containerd/image_list.go

thaJeztah · 2023-03-22T12:22:25Z

One thing worth mentioning is that we need to look at this / revisit this as part of the UX discussion in;

UX: improve / design UX for multi-arch images #44582

thaJeztah · 2023-03-22T12:33:57Z

daemon/containerd/image_list.go

+					}).Warn("checking availability of platform content failed")
+					return nil, nil
+				}
+				if !available || len(missing) > 0 {


Mostly a "note to self" (and others?) to check in what cases missing > 0, and to verify if there's cases in the containerd-integration;

are there cases where this would skip image(s) where we still want to show them?

cases where we must NOT ignore partials (thinking of delete image, prune image etc)

TBH, the containerd docs could use some improvement; https://github.com/containerd/containerd/blob/584d13d5cb350b48e64eb7c7b0e3e935b941e0d1/images/image.go#L294-L304

"missing will have the components that are part of required but not available in the provider."

what are components ?

what components are required ?

the whole signature is confusing (a bool, 3(!) slices, and an error)

^^ lots of ambiguity there

Looks like it was originally added in containerd/containerd@c555df5, and that commit message contains an example of some of this information;

images: support checking status of image content

The Check function returns information about an image's content components
over a content provider. From this information, one can tell which content is
required, present or missing to run an image.

The utility can be demonstrated with the check command:

$ ctr images check REF TYPE DIGEST STATUS SIZE docker.io/library/alpine:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:f006ecbb824d87947d0b51ab8488634bf69fe4094959d935c0c103f4820a417d incomplete (1/2) 1.5 KiB/1.9 MiB docker.io/library/postgres:latest application/vnd.docker.distribution.manifest.v2+json sha256:2f8080b9910a8b4f38ff5a55a82e77cb43d88bdbb16d723c71d18493590832e9 complete (13/13) 99.3 MiB/99.3 MiB docker.io/library/redis:alpine application/vnd.docker.distribution.manifest.v2+json sha256:e633cded055a94202e4ccccb8125b7f383cd6ee56527ab890db643383a2647dd incomplete (6/7) 8.1 MiB/10.0 MiB docker.io/library/ubuntu:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:60f835698ea19e8d9d3a59e68fb96fb35bc43e745941cb2ea9eaf4ba3029ed8a unavailable (0/?) 0.0 B/? docker.io/trollin/busybox:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:54a6424f7a2d5f4f27b3d69e5f9f2bc25fe9087f0449d3cb4215db349f77feae complete (2/2) 699.9 KiB/699.9 KiB

The above shows us that we have two incomplete images and one that is
unavailable. The incomplete images are those that we know the complete
size of all content but some are missing. "Unavailable" means that the
check could not get enough information about the image to get its full
size.

IIRC, I hit the missing > 0 case mostly after playing for some time with building images (building, removing, cancelling build) which possibly could leave some blobs in the content store (as a cache), and lead to a situation where the manifest for some platform is present, but its rootfs is not.

Yes, looks like for this case it's the right thing to do. The function on containerd is just quite confusing, and we likely do have to take some of these into account when pruning (not sure if that uses this function from the image-service though, but something we may have to check).

thaJeztah

LGTM, but;

left a suggestion for unmarshaling (let me know if you think that would make sense)
we should look at some test-cases for this (can be a follow-up); looks like it would be difficult to unit-test this 🤔

thaJeztah · 2023-03-22T12:57:21Z

daemon/containerd/image_list.go

@@ -303,3 +332,47 @@ func computeSharedSize(chainIDs []digest.Digest, layers map[digest.Digest]int, s
 	}
 	return sharedSize, nil
 }
+
+func getManifestPlatform(ctx context.Context, store content.Store, desc v1.Descriptor) (v1.Platform, error) {


Is this something we could (at some point?) use from

moby/vendor/github.com/containerd/containerd/image.go

Lines 432 to 444 in 1855a55

func (i *image) getManifestPlatform(ctx context.Context, manifest ocispec.Manifest) (ocispec.Platform, error) {

cs := i.ContentStore()

p, err := content.ReadBlob(ctx, cs, manifest.Config)

if err != nil {

return ocispec.Platform{}, err

}

var image ocispec.Image

if err := json.Unmarshal(p, &image); err != nil {

return ocispec.Platform{}, err

}

return platforms.Normalize(ocispec.Platform{OS: image.OS, Architecture: image.Architecture}), nil

}

Or are there improvements from readConfig() that we should contribute to containerd? (looks like our code has some more error-handling)

c8d version needs to be passed an already unmarshaled manifest object and our version takes care of it for us.
If we have more code using this kind of helper, we may think about contributing them directly to the c8d, but I wouldn't do that before we have some visible patterns in using it.

daemon/containerd/image_list.go

corhere · 2023-03-30T22:23:34Z

TBH, it feels wrong to actually work around this on our side, and I think attestation manifest should use a different media type.

It's unfortunate that the attestation manifest needs to use the image manifest mediaType for compatibility. The attestation doc says that the manifest's layers descriptors will have a specific mediaType, so could we decide which manifests to ignore based on that? Instead of playing whack-a-mole by enumerating all the manifest annotations which signify non-image manifests, could we filter out all pseudo-image manifests more generally by ignoring all manifests unless they have at least one layer with a known-good mediaType? (application/vnd.oci.image.layer.v1.tar(+COMPRESSION), application/vnd.oci.image.layer.nondistributable.v1.tar(+COMPRESSION), application/vnd.docker.image.rootfs.diff.tar.gzip)

thaJeztah · 2023-04-03T15:26:17Z

Do we need a tracking issue to "un-skip" these once we get to the tree-views as discussed in #44582 (and assuming these will still be included as "image")? Thinking if we want to show these in the tree (if they're there), or if we should continue hiding them from the user.

vvoland · 2023-04-03T15:26:20Z

I like the idea - added a layer check. Still left the explicit Annotation check though. It's cheap as it doesn't involve reading and deserializing the manifest json.

vvoland · 2023-04-03T15:30:19Z

Do we need a tracking issue to "un-skip" these once we get to the tree-views as discussed in #44582 (and assuming these will still be included as "image")? Thinking if we want to show these in the tree (if they're there), or if we should continue hiding them from the user.

I think they should be shown, but in a different way than an actual image.
So rather than creating a ticket for "un-skipping" these, I think we should create a ticket for actually designing their representation in the docker images output (although that could probably be a part of #44582 ?)

(Btw, in last force push I fixed a typo in comment, which made the linter sad)

corhere

LGTM!

daemon/containerd/image_list.go

Attestation manifests have an OCI image media type, which makes them being listed like they were a separate platform supported by the image. Don't use `images.Platforms` and walk the manifest list ourselves looking for all manifests that are an actual image manifest. Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>

vvoland · 2023-04-03T16:41:54Z

Unrelated buildkit test failure:

2023-04-03T16:07:55.8282819Z === CONT  TestIntegration/TestRunGlobalNetwork/worker=dockerd-containerd/frontend=client/network.host=granted
2023-04-03T16:07:56.5915961Z     dockerfile_runnetwork_test.go:193: 
2023-04-03T16:07:56.5917146Z         	Error Trace:	/src/frontend/dockerfile/dockerfile_runnetwork_test.go:193
2023-04-03T16:07:56.5922658Z         	            				/src/frontend/dockerfile/run.go:87
2023-04-03T16:07:56.5923221Z         	            				/src/frontend/dockerfile/run.go:196
2023-04-03T16:07:56.5924136Z         	Error:      	Received unexpected error:
2023-04-03T16:07:56.5925007Z         	            	process "/bin/sh -c ! nc -z 127.0.0.1 36915" did not complete successfully: exit code: 1
2023-04-03T16:07:56.5925422Z         	            	github.com/moby/buildkit/util/stack.Enable
2023-04-03T16:07:56.5925763Z         	            		/src/util/stack/stack.go:77
2023-04-03T16:07:56.5926513Z         	            	github.com/moby/buildkit/util/grpcerrors.FromGRPC
2023-04-03T16:07:56.5926900Z         	            		/src/util/grpcerrors/grpcerrors.go:197
2023-04-03T16:07:56.5928880Z         	            	github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
2023-04-03T16:07:56.5929399Z         	            		/src/util/grpcerrors/intercept.go:41
2023-04-03T16:07:56.5929764Z         	            	google.golang.org/grpc.(*ClientConn).Invoke
2023-04-03T16:07:56.5930136Z         	            		/src/vendor/google.golang.org/grpc/call.go:35
2023-04-03T16:07:56.5930554Z         	            	github.com/moby/buildkit/api/services/control.(*controlClient).Solve
2023-04-03T16:07:56.5930973Z         	            		/src/api/services/control/control.pb.go:2208
2023-04-03T16:07:56.5931343Z         	            	github.com/moby/buildkit/client.(*Client).solve.func2
2023-04-03T16:07:56.5931696Z         	            		/src/client/solve.go:258
2023-04-03T16:07:56.5932049Z         	            	golang.org/x/sync/errgroup.(*Group).Go.func1
2023-04-03T16:07:56.5932425Z         	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
2023-04-03T16:07:56.5932762Z         	            	runtime.goexit
2023-04-03T16:07:56.5933096Z         	            		/usr/local/go/src/runtime/asm_amd64.s:1594
2023-04-03T16:07:56.5933398Z         	            	failed to solve
2023-04-03T16:07:56.5933891Z         	            	github.com/moby/buildkit/client.(*Client).solve.func2
2023-04-03T16:07:56.5934243Z         	            		/src/client/solve.go:273
2023-04-03T16:07:56.5934582Z         	            	golang.org/x/sync/errgroup.(*Group).Go.func1
2023-04-03T16:07:56.5935999Z         	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
2023-04-03T16:07:56.5936438Z         	            	runtime.goexit
2023-04-03T16:07:56.5936753Z         	            		/usr/local/go/src/runtime/asm_amd64.s:1594
2023-04-03T16:07:56.5937561Z         	Test:       	TestIntegration/TestRunGlobalNetwork/worker=dockerd-containerd/frontend=client/network.host=granted
2023-04-03T16:07:56.5938018Z     sandbox.go:274: stdout: /usr/bin/dockerd
2023-04-03T16:07:56.5938307Z     sandbox.go:277: foo

thaJeztah

LGTM

vvoland added area/images kind/bugfix PR's that fix bugs area/builder/buildkit Issues affecting buildkit containerd-integration Issues and PRs related to containerd integration labels Mar 8, 2023

vvoland added this to the v-next milestone Mar 8, 2023

rumpl reviewed Mar 8, 2023

View reviewed changes

daemon/containerd/image_list.go Outdated Show resolved Hide resolved

vvoland force-pushed the c8d-list-ignore-attestations branch 3 times, most recently from 2e9adef to 4e97d90 Compare March 8, 2023 16:04

vvoland marked this pull request as ready for review March 9, 2023 09:58

thaJeztah reviewed Mar 10, 2023

View reviewed changes

daemon/containerd/image_list.go Outdated Show resolved Hide resolved

daemon/containerd/image_list.go Outdated Show resolved Hide resolved

daemon/containerd/image_list.go Outdated Show resolved Hide resolved

vvoland mentioned this pull request Mar 20, 2023

docker build -t <tag> ... creates 2 image references #45179

Closed

vvoland force-pushed the c8d-list-ignore-attestations branch from 4e97d90 to aae50bd Compare March 20, 2023 16:34

vvoland requested review from thaJeztah, neersighted, corhere and cpuguy83 March 22, 2023 10:57

thaJeztah reviewed Mar 22, 2023

View reviewed changes

thaJeztah approved these changes Mar 22, 2023

View reviewed changes

vvoland force-pushed the c8d-list-ignore-attestations branch from aae50bd to 0a468ab Compare March 22, 2023 14:35

vvoland force-pushed the c8d-list-ignore-attestations branch from 0a468ab to 2e011ad Compare March 30, 2023 16:38

thaJeztah added the status/2-code-review label Mar 30, 2023

vvoland force-pushed the c8d-list-ignore-attestations branch from 2e011ad to bb3dd25 Compare April 3, 2023 15:21

vvoland force-pushed the c8d-list-ignore-attestations branch from bb3dd25 to f3a3c03 Compare April 3, 2023 15:31

corhere approved these changes Apr 3, 2023

View reviewed changes

daemon/containerd/image_list.go Outdated Show resolved Hide resolved

vvoland force-pushed the c8d-list-ignore-attestations branch from f3a3c03 to 92e38b6 Compare April 3, 2023 15:51

thaJeztah approved these changes Apr 3, 2023

View reviewed changes

thaJeztah merged commit 781740c into moby:master Apr 3, 2023
91 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

c8d/list: Ignore attestation manifests #45124

c8d/list: Ignore attestation manifests #45124

vvoland commented Mar 8, 2023 •

edited by thaJeztah

vvoland commented Mar 8, 2023

vvoland commented Mar 9, 2023

thaJeztah commented Mar 22, 2023

thaJeztah Mar 22, 2023 •

edited

vvoland Mar 22, 2023

thaJeztah Mar 22, 2023

thaJeztah left a comment

thaJeztah Mar 22, 2023

vvoland Mar 22, 2023

corhere commented Mar 30, 2023

thaJeztah commented Apr 3, 2023

vvoland commented Apr 3, 2023

vvoland commented Apr 3, 2023 •

edited

corhere left a comment

vvoland commented Apr 3, 2023

thaJeztah left a comment

	func (i *image) getManifestPlatform(ctx context.Context, manifest ocispec.Manifest) (ocispec.Platform, error) {
	cs := i.ContentStore()
	p, err := content.ReadBlob(ctx, cs, manifest.Config)
	if err != nil {
	return ocispec.Platform{}, err
	}

	var image ocispec.Image
	if err := json.Unmarshal(p, &image); err != nil {
	return ocispec.Platform{}, err
	}
	return platforms.Normalize(ocispec.Platform{OS: image.OS, Architecture: image.Architecture}), nil
	}

c8d/list: Ignore attestation manifests #45124

c8d/list: Ignore attestation manifests #45124

Conversation

vvoland commented Mar 8, 2023 • edited by thaJeztah

vvoland commented Mar 8, 2023

vvoland commented Mar 9, 2023

thaJeztah commented Mar 22, 2023

thaJeztah Mar 22, 2023 • edited

Choose a reason for hiding this comment

vvoland Mar 22, 2023

Choose a reason for hiding this comment

thaJeztah Mar 22, 2023

Choose a reason for hiding this comment

thaJeztah left a comment

Choose a reason for hiding this comment

thaJeztah Mar 22, 2023

Choose a reason for hiding this comment

vvoland Mar 22, 2023

Choose a reason for hiding this comment

corhere commented Mar 30, 2023

thaJeztah commented Apr 3, 2023

vvoland commented Apr 3, 2023

vvoland commented Apr 3, 2023 • edited

corhere left a comment

Choose a reason for hiding this comment

vvoland commented Apr 3, 2023

thaJeztah left a comment

Choose a reason for hiding this comment

vvoland commented Mar 8, 2023 •

edited by thaJeztah

thaJeztah Mar 22, 2023 •

edited

vvoland commented Apr 3, 2023 •

edited