ctr pull images err archive/tar: invalid tar header: unknown

[system51]

Description

When employing ctr pull for mirroring, this error crops up, whereas no errors arise when using docker pull to fetch the image.

unpacking linux/amd64 sha256:4c7c85027d535bf93ca5e7580daaaa1d777d57b58ab6c2998c47f0cb6ce0713d...
INFO[0001] apply failure, attempting cleanup error="failed to extract layer sha256:178615070e3d6a5fcb2c6d23ebe1c245f05f6ba2f873a870cb4769ea1056dced: archive/tar: invalid tar header: unknown" key="extract-842707968-7z5- sha256:56970de1727c7ae3eaac7defed809523b0d646728183c90972fa96e0790eee5d"
ctr: failed to extract layer sha256:178615070e3d6a5fcb2c6d23ebe1c245f05f6ba2f873a870cb4769ea1056dced: archive/tar: invalid tar header: unknown

Steps to reproduce the issue

ctr -n k8s.io images pull -u 'xxxxxxxx:QkRjnyYO11iDWVm' harbor.test.com/xxxx/xxx-uc:test-4519e59e5a70b3b96d287f4fa1c866f2e4bde446-20240130164436

Describe the results you received and expected

Pull the image normally

What version of containerd are you using?

containerd containerd.io 1.6.28 ae07eda

Any other relevant information

[root@k8s-node-02 ~]# runc --version
runc version 1.1.12
commit: v1.1.12-0-g51d5e94
spec: 1.0.2-dev
go: go1.20.13
libseccomp: 2.5.2

[root@k8s-node-02 ~]# uname -a
Linux k8s-node-02 6.7.6-1.el8.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 17:04:38 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.

No response

[ckw017]

Seeing a similar problem, looks a lot like #3326, except we're using docker instead of buildah -- after upgrading docker we're seeing blobs that were previously marked as application/vnd.docker.image.rootfs.diff.tar.gzip (using docker 24.0.5) as application/vnd.oci.image.layer.v1.tar (using docker 25.0.3) -- feels like it should be application/vnd.oci.image.layer.v1.tar+gzip instead.

It looks like the docker pull is automatically detecting that the blob is gzipped, while containerd doesn't -- technically containerd's behavior is correct according to the spec ofc.

[ckw017]

Mostly grasping at straws, might be related to this string of commits, since seems like the main spots where application/vnd.oci.image.layer.v1.tar (i.e. ocispec.MediaTypeImageLayer) has been used thats new between 24.0 and 25.0

• moby/moby@2c45df2
• moby/moby@9f50e06
• moby/moby@c58acf0

[tianon]

Would either of you be able to make or share a public image that reproduces the problem? Or steps to build an image that reproduces the problem? 👀

[ckw017]

On my end it looks like this was an issue with bazel docker rules, unrelated to containerd/docker -- https://github.com/bazelbuild/rules_docker/blob/master/container/go/pkg/compat/image.go#L175-L180

It looks like in 25.0 docker started outputting OCI compliant tarballs, and it seems like by properly marking uncompressed layers as application/vnd.oci.image.layer.v1.tar, revealed a bug where the bazel rule always assumes that the media type is compressed.

[system51]

@tianon @ckw017 My solution to this problem was to rebuild the image, but it did not solve the underlying problem