Error inspecting local manifest-lists

[kingsleyzissou]

I am having difficulty inspecting a container from a local manifest list using both skopeo and the containers/image go library.
This could well be a user error and expected behaviour but I've gone through the open issues and docs to try find the answer and it's unclear. I have tried with both rootful and rootless podman.

Steps to reproduce:

podman manifest create multi:latest
podman build --platform linux/arm64,linux/amd64 --manifest localhost/multi:latest  .

The contents of the Containerfile:

FROM fedora-minimal
COPY README.md .

And here is the images list:

$ podman images
REPOSITORY                                 TAG         IMAGE ID      CREATED         SIZE
localhost/multi                            latest      603da7b1a137  25 minutes ago  1.05 kB
<none>                                     <none>      d5e5bd2654d3  25 minutes ago  138 MB
<none>                                     <none>      7eebab5c88a2  25 minutes ago  93.1 MB
registry.fedoraproject.org/fedora-minimal  latest      e92513728bce  2 hours ago     138 MB

When I try inspect using skopeo, I get the following error:

$ skopeo --debug inspect containers-storage:localhost/multi
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is being used
DEBU[0000] Cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]localhost/multi:latest"
FATA[0000] Error parsing manifest for image: fetching target platform image selected from image index: reading manifest for image instance "sha256:9dad0be14d515def44859dcf3e1cad962425340d449996bacba552bdfe6bd595": locating item named "manifest-sha256:9dad0be14d515def44859dcf3e1cad962425340d449996bacba552bdfe6bd595" for image with ID "603da7b1a13708d7474e51d92545c5aea1eb074ca4e785d212df7d68573e13aa" (consider removing the image to resolve the issue): file does not exist

I get a similar enough result running with the library in go, I have a small reproducer here:
https://github.com/kingsleyzissou/playground/blob/963a8f8a641842efef8144572945d50cdfa93c7a/main.go

panic: reading manifest for image instance "sha256:9dad0be14d515def44859dcf3e1cad962425340d449996bacba552bdfe6bd595": locating item named "manifest-sha256:9dad0be14d515def44859dcf3e1cad962425340d449996bacba552bdfe6bd595" for image with ID "603da7b1a13708d7474e51d92545c5aea1eb074ca4e785d212df7d68573e13aa" (consider removing the image to resolve the issue): file does not exist

And here is the output for the manifest id from storage/overlay/images.json:

{
  "id": "603da7b1a13708d7474e51d92545c5aea1eb074ca4e785d212df7d68573e13aa",
  "digest": "sha256:20b959ad5960230b65a77b746bdbf5d991ade4d7a129c2554e167acdcc990531",
  "names": [
    "localhost/multi:latest"
  ],
  "big-data-names": [
    "manifest",
    "instances.json",
    "artifacts.json"
  ],
  "big-data-sizes": {
    "artifacts.json": 2,
    "instances.json": 561,
    "manifest": 491
  },
  "big-data-digests": {
    "artifacts.json": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "instances.json": "sha256:04c2273093d3297a58bb8c9c936b2557fdc8f6f76d68fea9672d89181275f368",
    "manifest": "sha256:fc670131e7715c8b8b895e22df416b1cf696a5687a4007032d3dad9009acc74a"
  },
  "created": "2024-04-10T09:35:01.1314092Z"
}

Am I missing any steps? Or is this expected behaviour?

[kingsleyzissou]

$ skopeo --version
skopeo version 1.15.0 commit: 99e3c44865d4200bb93f38365622ef1e857c3a61

$ podman --version
podman version 5.0.1

[mtrmac]

I’m afraid this is expected. The podman manifest feature is not native to containers/storage, it is built on top in c/common/libimage/manifest.

That package might allow consuming the data in non-Podman processes, I’m not sure. Note that c/common explicitly doesn’t have a stable API promise.

[kingsleyzissou]

Thanks for the quick response.

Ultimately we're trying to do something similar to containers/skopeo#1554, is that issue not feasible for the same reason as this issue?

[mtrmac]

I think the two are orthogonal.

Also containers/skopeo#1554 is “only” about displaying information mostly known to the Go data structures; so if you are already writing Go, that data should be readily available if libimage/manifest can be called.

[kingsleyzissou]

Oh neat thanks for the information. Yeah we're mostly working with Go so I'll explore that

[kingsleyzissou]

We managed to do this by using querying the store and calling ImagesByDigest :)