--dest-compress doesn't change layers mediaType in manifest

[vrutkovs]

v1schema1 image cannot be properly converted to v2schema2 compatible for quay:

/home/vrutkovs/.local/bin/skopeo copy --format v2s2 docker://registry.ci.openshift.org/origin/4.6-2021-02-06-044755@sha256:fea04b4455ac8b0a69a12c712643662daf1692d6985cfef30712a3f69e4797de dir:4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator --dest-compress --dest-compress-format=gzip

produces gzipped files:

$ file 4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/*
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/0c983affa28412222c30175adad7784730fb1ab7358c541820624ab12eaa9d1e: gzip compressed data, original size modulo 2^32 19128320
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/1e64d61e08c723268a2febf65abceaf6d823c002ea5feb1bb64fc78bbf9f232e: JSON data
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/5e7394dd47ddf80c782ac475428241e4290d9a40576127ac6d5a95001ca59357: gzip compressed data, original size modulo 2^32 81669632
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/791696e2ff97c6fa7b01b9b161b381f5ebc8e04cb9ea729f704a2848e5bd0865: gzip compressed data, original size modulo 2^32 35350016
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/911e72d1d7d77f2c01d4939de037327e0caa10cf6743e65c5303057451deba45: gzip compressed data, original size modulo 2^32 1702912
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/f7db571f3a057b2e3fdad7c0f7cf7af581bb71f9d54acfe88b074a514abd5612: gzip compressed data, original size modulo 2^32 20480
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/fecf0868d3744dbd6e007e3dc099cdc3218df06c927a7c0c54c0181fabba2b94: gzip compressed data, original size modulo 2^32 210810880
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/manifest.json:                                                    JSON data
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/version:                                                          ASCII text

but manifest still refers to plain files:

/home/vrutkovs/.local/bin/skopeo inspect dir:4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator --raw | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 11272,
    "digest": "sha256:1e64d61e08c723268a2febf65abceaf6d823c002ea5feb1bb64fc78bbf9f232e"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 73956874,
      "digest": "sha256:fecf0868d3744dbd6e007e3dc099cdc3218df06c927a7c0c54c0181fabba2b94"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 1406,
      "digest": "sha256:f7db571f3a057b2e3fdad7c0f7cf7af581bb71f9d54acfe88b074a514abd5612"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 5182813,
      "digest": "sha256:0c983affa28412222c30175adad7784730fb1ab7358c541820624ab12eaa9d1e"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 465929,
      "digest": "sha256:911e72d1d7d77f2c01d4939de037327e0caa10cf6743e65c5303057451deba45"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 11473701,
      "digest": "sha256:791696e2ff97c6fa7b01b9b161b381f5ebc8e04cb9ea729f704a2848e5bd0865"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 33589962,
      "digest": "sha256:5e7394dd47ddf80c782ac475428241e4290d9a40576127ac6d5a95001ca59357"
    }
  ]
}

This breaks quay.io uploads, which requires application/vnd.docker.image.rootfs.diff.tar.gz mediaType

[vrothberg]

I tested with @nalind's PR containers/image#1138 and the good news is:

skopeo (master) $ file 4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/*
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/0c983affa28412222c30175adad7784730fb1ab7358c541820624ab12eaa9d1e: gzip compressed data, original size modulo 2^32 19128320
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/1e64d61e08c723268a2febf65abceaf6d823c002ea5feb1bb64fc78bbf9f232e: JSON data
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/5e7394dd47ddf80c782ac475428241e4290d9a40576127ac6d5a95001ca59357: gzip compressed data, original size modulo 2^32 81669632
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/791696e2ff97c6fa7b01b9b161b381f5ebc8e04cb9ea729f704a2848e5bd0865: gzip compressed data, original size modulo 2^32 35350016
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/911e72d1d7d77f2c01d4939de037327e0caa10cf6743e65c5303057451deba45: gzip compressed data, original size modulo 2^32 1702912
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/f7db571f3a057b2e3fdad7c0f7cf7af581bb71f9d54acfe88b074a514abd5612: gzip compressed data, original size modulo 2^32 20480
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/fecf0868d3744dbd6e007e3dc099cdc3218df06c927a7c0c54c0181fabba2b94: gzip compressed data, original size modulo 2^32 210810880
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/manifest.json:                                                    JSON data
4.6.0-0.okd-2021-02-06-044755-cluster-image-registry-operator/version:                                                          ASCII text

The layers are compressed but it seems that the manifest is not completely updated.

[mtrmac]

What version of Skopeo and c/image? This is most likely containers/image#733 , which should have been fixed by containers/image#1089 already. containers/image#1138 is a variant that only applies to builds with the blob cache enabled, IIRC.

Also, for reproducing, please make sure to establish the state of the blob info cache (~/.local/share/containers/cache/blob-info-cache-v1.boltdb or the like — whether it is deleted, first use, second use of the same repo) — and ideally collect --debug logs.

[mtrmac]

Actually, never mind; this has little to do with the original 733. That image on the registry is not v2s1; it is an already invalid v2s2 (gzip-compressed with non-gzip MIME types), probably due to 733.

So, the code correctly detects the files as compressed in the requested format = no change needed to layers, and therefore does not trigger updates to the MIME types.

The real fix is to change the original uploaded file, i.e. to update the build implementation (incl. possibly to a version that does not exist yet, for the blob cache case).

A possible workaround might be to instead discard the MIME types, by forcing a conversion to --format=v2s1, and then another conversion back to --format=v2s2 (which blindly assumes gzip).

[vrutkovs]

That image on the registry is not v2s1; it is an already invalid v2s2 (gzip-compressed with non-gzip MIME types), probably due to 733.

Right, iiuc things work correctly on valid alpine image

[vrutkovs]

What version of Skopeo and c/image?

I used lastest from master (1c4b0fc33dcd3e26aaf3e3568a5c0a4239593e0e) and then a version with c/images bumped to containers/image#1138

[vrutkovs]

A possible workaround might be to instead discard the MIME types, by forcing a conversion to --format=v2s1, and then another conversion back to --format=v2s2 (which blindly assumes gzip).

Yup, that did the trick, thanks!

[vrothberg]

Good catch, @mtrmac! Just rechecked and can confirm that initial image was already wrong.

[vrothberg]

I consider the issue fixed, please reopen if I am mistaken.

[vrutkovs]

sgtm, the image was broken in the first place, not sure if skopeo is intended to fix the mediaTypes - that would be an enhancement anyway