containerd-integration: listing multi-arch images is (partially) broken

[thaJeztah]

Description

Got something weird on my machine; it looks like it's returning duplicate entries for the same image;

docker image ls
REPOSITORY                  TAG           IMAGE ID       CREATED        SIZE
alpine                      3.16          b95359c25051   9 days ago     2.71MB
alpine                      3.16          b95359c25051   9 days ago     2.59MB
alpine                      latest        8914eb54f968   4 days ago     3.26MB
...

Hitting the API directly shows the same:

curl --unix-socket /var/run/docker.sock 'http://localhost/images/json' | jq

[
  {
    "Containers": -1,
    "Created": 1669291855,
    "Id": "sha256:b95359c2505145f16c6aa384f9cc74eeff78eb36d308ca4fd902eeeb0a0b161b",
    "Labels": null,
    "ParentId": "",
    "RepoDigests": [
      "alpine:3.16@sha256:b95359c2505145f16c6aa384f9cc74eeff78eb36d308ca4fd902eeeb0a0b161b"
    ],
    "RepoTags": [
      "alpine:3.16"
    ],
    "SharedSize": -1,
    "Size": 2711409,
    "VirtualSize": 5955584
  },
  {
    "Containers": -1,
    "Created": 1669291855,
    "Id": "sha256:b95359c2505145f16c6aa384f9cc74eeff78eb36d308ca4fd902eeeb0a0b161b",
    "Labels": null,
    "ParentId": "",
    "RepoDigests": [
      "alpine:3.16@sha256:b95359c2505145f16c6aa384f9cc74eeff78eb36d308ca4fd902eeeb0a0b161b"
    ],
    "RepoTags": [
      "alpine:3.16"
    ],
    "SharedSize": -1,
    "Size": 2594762,
    "VirtualSize": 6156288
  },
  {
    "Containers": -1,
    "Created": 1669712471,
    "Id": "sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4",
    "Labels": null,
    "ParentId": "",
    "RepoDigests": [
      "alpine:latest@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4"
    ],
    "RepoTags": [
      "alpine:latest"
    ],
    "SharedSize": -1,
    "Size": 3262843,
    "VirtualSize": 8138752
  }
]

I double-checked if it's not something in the Docker Desktop proxy, but using the .raw socket, I get the same;

docker run --rm -v /var/run/docker.sock.raw:/var/run/docker.sock docker:cli docker image ls
REPOSITORY                  TAG           IMAGE ID       CREATED        SIZE
alpine                      3.16          b95359c25051   9 days ago     2.71MB
alpine                      3.16          b95359c25051   9 days ago     2.59MB
alpine                      latest        8914eb54f968   4 days ago     3.26MB
busybox                     latest        fcd85228d7a2   43 hours ago   832kB

Within the Docker Desktop VM, I queried ctr, and also didn't see duplicates:

ctr -n moby -a /var/run/desktop-containerd/containerd.sock image ls -q
docker.io/library/alpine:3.16
docker.io/library/alpine:latest
docker.io/library/busybox:latest

So I'm suspecting it's somewhere in dockerd. Perhaps it's due to multi-arch; could it be I have the image pulled in 2 different architectures?

ctr -n moby -a /var/run/desktop-containerd/containerd.sock image ls
REF                                                                TYPE                                                      DIGEST                             SIZE      PLATFORMS   LABELS
docker.io/library/alpine:3.16                                      application/vnd.docker.distribution.manifest.list.v2+json sha256:b95359c2505145f16c6aa384f9cc74eeff78eb36d308ca4fd902eeeb0a0b161b 2.6 MiB   linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x   -
docker.io/library/alpine:latest                                    application/vnd.docker.distribution.manifest.list.v2+json sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 3.1 MiB   linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x   -
docker.io/library/busybox:latest                                   application/vnd.docker.distribution.manifest.list.v2+json sha256:fcd85228d7a25feb59f101ac3a955d27c80df4ad824d65f5757a954831450185 812.7 KiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/riscv64,linux/s390x  -

And with some testing; it looks like that's the case;

docker image rm alpine:3.16
Untagged: docker.io/library/alpine:3.16

docker image pull alpine:3.16
# ...
docker image ls
REPOSITORY       TAG          IMAGE ID       CREATED          SIZE
alpine           latest       8914eb54f968   4 days ago       3.26MB
busybox          latest       fcd85228d7a2   44 hours ago     832kB

docker image pull --platform=linux/amd64 alpine:3.16
# ...

docker image ls
REPOSITORY       TAG          IMAGE ID       CREATED          SIZE
alpine           3.16         b95359c25051   4 seconds ago    2.81MB
alpine           3.16         b95359c25051   4 seconds ago    2.71MB
alpine           latest       8914eb54f968   4 days ago       3.26MB
busybox          latest       fcd85228d7a2   44 hours ago     832kB

Reproduce

see above

Expected behavior

no duplicate images

docker version

Client:
 Cloud integration: v1.0.29
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:18 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.15.0 (93002)
 Engine:
  Version:          22.06.0-beta.0-902-g2708be0db4.m
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.18.4
  Git commit:       2708be0db4
  Built:            Wed Nov 23 09:32:50 2022
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

### docker info

```bash
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  base: Identify base image (Docker Inc., vn/a)
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.13.0)
  dev: Docker Dev Environments (Docker Inc., v0.0.5)
  extension: Manages Docker extensions (Docker Inc., v0.2.16)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.22.0)
  shell: Open a browser shell on the Docker Host. (thaJeztah, v0.0.1)

Server:
 Containers: 45
  Running: 13
  Paused: 0
  Stopped: 32
 Images: 14
 Server Version: 22.06.0-beta.0-902-g2708be0db4.m
 Storage Driver: stargz
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: bfpr7gj1w767qwq37qta7kqts
  Is Manager: true
  ClusterID: crv7mpxcck85em4ocnfci4esi
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.65.3
  Manager Addresses:
   192.168.65.3:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 5.8GiB
 Name: docker-desktop
 ID: c8ca0b32-cd5f-4ad2-9a97-bc30ebe413d4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Docker Desktop 4.15 with containerd integration, which gets confused when sorting the list of images, and starts duplicating them (as the images are no longer unique);

dashboard_duplicating.mov

[thaJeztah]

Oh! And I just realised this is introduced in;

• Handle multi-platform images when listing them rumpl/moby#113.

We need to work on the UX (as we were discussing some weeks ago in the containerd-integration sync call), so that multi-arch images are presented correctly.

The issue right now is that;

• images are shown multiple times (one time for each platform)
• But: the ID of the image is the digest of the manifest index (this is correct as this is the "source of truth" that ties the architectures together, and what's used to resolve the image)
• However, as all architectures share that same manifest index, entries in the list are no longer unique, which makes the list ambiguous, and this trips up docker dashboard.

We should have a single entry per multi-arch image (manifest index), but have a way to "unfold" / expand that list to allow users to interact with individual variants of the image.

I have a draft somewhere; will dust that off, and open a ticket.

/cc @rumpl @vvoland

[AkihiroSuda]

The current behavior seems correct, and it is consistent with containerd/nerdctl too

$ nerdctl images
REPOSITORY    TAG     IMAGE ID        CREATED           PLATFORM          SIZE       BLOB SIZE
alpine        3.16    b95359c25051    15 seconds ago    linux/amd64       5.9 MiB    2.7 MiB
alpine        3.16    b95359c25051    15 seconds ago    linux/arm64/v8    0.0 B      2.6 MiB

The docker images CLI will have to have the PLATFORM column though.

[thaJeztah]

Yes, that's the thing we need to look at for the UX; due to the missing PLATFORM column, rows are not uniquely identifiable. We need to work on the UX; I just (somewhat) finished up my notes for discussing the UX, and opened it as a ticket in #44582