keycloak version 20.0.5 skopeo copy failed

[rokkiter]

Description

when i use skopeo copy image to my repo

skopeo copy --all docker://quay.io/keycloak/keycloak:20.0.5 {myrepo}

it return error with

FATA[0018] Error initializing image from source docker://quay.io/keycloak/keycloak:20.0.5: unsupported OCIv1 media type: "application/vnd.in-toto+json" 

[huguesgr]

I'm having the same issue since 20.0.5.
I think this is related to a manifest issue. See the difference between 20.0.3 and 20.0.5:

$ docker manifest inspect quay.io/keycloak/keycloak:20.0.5
unsupported manifest media type and no default available: application/vnd.oci.image.manifest.v1+json

$ docker manifest inspect quay.io/keycloak/keycloak:20.0.3
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 903,
         "digest": "sha256:c167807890ff63fd10dacef2ab6fd2242487a940ce290a9417a373da66e862e9",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 903,
         "digest": "sha256:ead0c899efffce80b70f787dcd1f33f5332661191a546c2fd528703345828aae",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

It fails when trying to push to a private registry, using Kaniko:

error pushing image: failed to push to destination REDACTED_PRIVATE_REGISTRY/keycloak:21.0.1: PUT https://REDACTED_PRIVATE_REGISTRY-multi/keycloak/manifests/21.0.1: MANIFEST_INVALID: manifest invalid; map[]

Not sure this is related, but since 20.0.5 some unknown platforms have appeared in the repository: https://quay.io/repository/keycloak/keycloak?tab=tags

[frami]

The problem with the wrong mediaType attribute also occurs on OpenShift 3.11.

This manifest with v1 makes problem. It started with version 20.0.5:

	"mediaType": "application/vnd.oci.image.manifest.v1+json",
	"schemaVersion": 2,

This manifest with v2 works:

   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",

[shane-snyder]

The problem with the wrong mediaType attribute also occurs on OpenShift 3.11.

This manifest with v1 makes problem. It started with version 20.0.5:

	"mediaType": "application/vnd.oci.image.manifest.v1+json",
	"schemaVersion": 2,

This manifest with v2 works:

   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",

@frami For this issue in OCP 3.11, choose the specific sha256 for the architecture you'd like to run. OCP 3.11 does not have the ability to reach multi-arch OCI data. By choosing the specific image, the code does not have to understand the multi-arch data.

For amd64 arch, tag 21.1 run with the specific container image below.
quay.io/keycloak/keycloak@sha256:cee8a0b343993b699a3d3eeb90e0b1eca2fa19285a413200af9785db8b00dcd7

skopeo inspect --raw docker://quay.io/keycloak/keycloak:21.1
{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:cee8a0b343993b699a3d3eeb90e0b1eca2fa19285a413200af9785db8b00dcd7",
      "size": 1056,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    }

[frami]

In my case, it doesn't work to choose the specific image. Because the manifest also contains a "v1+json" media type.

docker manifest inspect quay.io/keycloak/keycloak@sha256:cee8a0b343993b699a3d3eeb90e0b1eca2fa19285a413200af9785db8b00dcd7
{
	"mediaType": "application/vnd.oci.image.manifest.v1+json",
	"schemaVersion": 2,
	"config": {
		"mediaType": "application/vnd.oci.image.config.v1+json",
		"digest": "sha256:d19174f3762d872c77756014d7313d262bb6bde0ba7f68d2c5649b0d86949220",
		"size": 6570
	},

My workaround is, to republish the image in our own registry. After that, it has a media type with "v2+json".

[shawkins]

This seems to be some limitation with the docker and skopeo tools. See moby/buildkit#3491 (comment)

I tried the suggestion "Use docker buildx imagetools inspect instead of docker manifest inspect --raw - it should work similarly, and supports all the different media types in the registry." And that correctly parsed the manifests.

skopeo copy with my fedora version of 1.4.1 fails with with containers/skopeo#1874 - as noted in the issue that should not happen with newer versions, but it looks like you may have to build / install those on your own.

@vmuzikar @stianst do you want any doc notes related to this? I'm assuming that we don't want to change anything about the image generation, correct?

[vmuzikar]

@shawkins Thanks for looking into it. Since it's essentially a confirmed skopeo issue, I think no action is needed on our end. More over AFAIK we don't document anywhere any skopeo uses or examples.

[shawkins]

Thanks @vmuzikar - closing the issue then.