support zstd as archive format

[grexe]

in addition to the currently supported archive formats zip, bz2 and tar.gz, it would be great to add support for the free and open ZSTD compression recently open sourced by Facebook.
The format provides excellent compression and speed, and is already supported by a variety of standard tools.

zstd on Github
related article

[pwaller]

I can't believe this hasn't had more love yet. This would surely make pushing and pulling quite a bit faster.

[pwaller]

I don't think separate dictionaries are useful unless the content you are compressing is very small.

The primary benefit of zstd is its very rapid compression and decompression (GB/sec), while also maintaining compression ratios similar to that of zlib for a variety of inputs.

[AkihiroSuda]

Link: OCI Image Spec added support for zstd recently: opencontainers/image-spec#788

This is already implemented by containerd, but still not implemented by Docker/Moby.

[cpuguy83]

Adding this to 20.03. It would be nice to be able to decompress zstd layers at least.

[cpuguy83]

#40820

[giuseppe]

is this issue closed by containerd/containerd#4809?

[thaJeztah]

No, because the containerd image store is not (yet) used; it's still separate

[giuseppe]

opened a PR to support extracting zstd layers: #41759

Is it enough to close the issue or do we need write support as well?

[hubutui]

We could not load a docker image by:

docker load image.tar.zstd

but we could use:

zstdcat image.tar.zstd | docker load

or

zstd -dcf image.tar.zstd | docker load

You need zstd installed on your host system, which is usually installed by default.

[edmorley]

We could not load a docker image by:

Do we need a separate issue open to track adding support to docker load --input, or is this issue sufficient?

[henry118]

It would be great to be able to build and push zstd compressed images via moby.

We do see the increasing demands of building and distributing images in zstd format due to the performance benefits. But the lack of toolchain supporting is the main blocker of the adoption among our customers.

Is it likely to be supported?

[cpuguy83]

I believe this should be supported in the 22.06 beta release.

[henry118]

I believe this should be supported in the 22.06 beta release.

Will 22.06 support building and pushing scenario as well? I thought it only supports decompressing zstd layers, rather than building/distributing.

[cpuguy83]

docker buildx build --output=type=image,compression=zstd

[henry118]

Thanks for the info.

I did experiment the buildx approach. With buildx the newly built zstd image must be pushed to remote registry with the docker-container driver.

Currently it seems that there is no native support in Moby to distribute (docker push) layers in any other format apart from gzip. This makes it hard for scenarios like pull and push.

Additionally a lot of our customers rely on native docker build instead of buildx.

My question is that will Moby add support of the scenarios described above in the near future?

Thanks!

[cpuguy83]

In 22.06 docker build is backed by buildx.

docker push has not changed, however and will push gzip layers only as before.
We are pushing (no pun intended 😆 ) to start the transition to using containerd for image storage and distribution in the next release after 22.06, and with that you should see more frequently releases as well.

[XciD]

Any update on this ? we use zstd + containerd on production system and would like to be able to debug some images locally

[gaby]

@hubutui Since #41759 the ommand you did is now supported. Your extension has to be .tar.zst not .tar.zstd

Save Image:

docker save example:latest | zstd -f - q -o example.tar.zst

Load image

docker load -i example.tar.zst

We need a way to save .zst images when doing docker save.

[gaby]

@thaJeztah Thoughts on this?

[thaJeztah]

Too complex. People want plain docker push to learn --compression zstd.

A --compression option on docker push will need a more in-depth discussion, and not sure if that's something we should implement (or at least, not without further discussion).

With the pre-containerd image store ("graphdrivers"), images are stored uncompressed, and image-layers as pulled from the registry are discarded after they've been pulled. When pushing an image, existing layers in the registry are skipped, and layers that don't exist in the registry are compressed during pull. Effectively, this means that the artifacts/layers that are "distributed" through the registry are created during push. The advantage of this approach is reduced storage (only the extracted layers are kept locally), but the downside is that images pushed are not reproducible (or at least, not guaranteed to be reproducible), due to compression not being 100% reproducible, and timestamps embedded in some parts of the image.

With the containerd image store integration ("snapshotters"), the situation is different; containerd stores BOTH the compressed ("distributed") and extracted content / layers. When pushing an image to a registry, no compression happens, as the compressed layers already exist. This means that push/pull is fully reproducible, at the cost of additional storage.

If a --compression option was added to docker push, doing so would mean that the existing content would have to be re-compressed. While (likely) technically possible, that would mean that the image you're pushing is created on the fly, and won't match the image you were pushing (i.e., the digest of the image you pushed won't match the digest of the image you're pushing; it's a new image that happens to have the same content).

[edmorley]

Do you have a sense of the timeline/transition plan for eventually making zstd compression the default? I presume due to backwards compatibility concerns this couldn't happen for some time (until support was dropped for Docker <23-24).

Or alternatively, are there plans to start dual-publishing popular images to Docker Hub using both the legacy compression formats and zstd (to the same repo; presuming this is supported via the same manifests approach used by multi-arch images?), so newer clients could at least benefit from zstd in the meantime?

[slonopotamus]

If a --compression option was added to docker push, doing so would mean that the existing content would have to be re-compressed

All these questions were already solved in other tools that operate on images, including the options to (not) recompress already existing things.

I'm not sure how docker build --compression zstd would work if docker build stores resulting image in uncompressed form?

are there plans to start dual-publishing popular images to Docker Hub using both the legacy compression formats and zstd

Please, no. People want to switch to zstd because gzip is slow. Pushing gzip and zstd would be even slower.

[thaJeztah]

Do you have a sense of the timeline/transition plan for eventually making zstd compression the default? I presume due to backwards compatibility concerns this couldn't happen for some time (until support was dropped for Docker <23-24).

No plans / nothing decided on yet.

Or alternatively, are there plans to start dual-publishing popular images to Docker Hub using both the legacy compression formats and zstd (to the same repo; presuming this is supported via the same manifests approach used by multi-arch images?), so newer clients could at least benefit from zstd in the meantime?

No, that's a big issue; there's no such option currently that allows "dual" formats, and probably should've been a reason to reject the changes in the OCI specs pending that. See a more in-depth discussion on opencontainers/image-spec#803

I'm not sure how docker build --compression zstd would work if docker build stores resulting image in uncompressed form?

This would be a feature when the containerd image-store (snapshotters) are used, not when using graphdrivers.

[edmorley]

Please, no. People want to switch to zstd because gzip is slow. Pushing gzip and zstd would be even slower.

I think you misunderstood. I wasn't proposing that the Docker CLI enable dual pushing by default, but instead that some of the popular "official" Docker Hub images (eg https://hub.docker.com/_/ubuntu/ and the rest) be dual published via their automated publishing pipelines, so downstream consumers of those images (eg myself, yourself) could at least benefit from zstd in the meantime when pulling from those repos.

No, that's a big issue; there's no such option currently that allows "dual" formats, and probably should've been a reason to reject the changes in the OCI specs pending that. See a more in-depth discussion on opencontainers/image-spec#803

Ah understood - than you for the discussion link.

[slonopotamus]

How I verify that docker buildx build . --output=type=image,oci-mediatypes=true,compression=zstd,push=true --tag=blabla does in fact use zstd? It looks like it does not. Because I even can specify compression=blablabla and it still silently uploads image with exactly the same size as gzip.

Will this work with graphdriver or only with containerd-snaphotter?

Will it still use zstd if I split this into two steps? Will this work with graphdriver?

1. docker buildx build . --output=type=image,oci-mediatypes=true,compression=zstd --tag=bla-bla
2. docker push bla-bla?

And what about Windows images?

Are you sure it is a good idea to tie zstd and containerd-snaphotter features together? As you said, containerd-snaphotter means there will be a noticeable storage overhead. Big images suffer from gzip the most. So they will have to trade suffering from gzip for suffering from increased disk usage. Why big images are not allowed to benefit from both things? In my case, the image is 14.5GB gzipped / 52GB unpacked. This means 28% of overhead. It's a lot.

[dsoprea]

@slonopotamus "docker push" will still use gzip. The --output line needs "push=true", but this preempts local storage, apparently. This was my question above. Image-tools inspections seems to still report gzip for all layers maybe from some previous build? It's not clear how to confirm that the output of the build actually used zstd because it's not clear where the output can even be found on the local system.

[dsoprea]

We're going to be leaving this issue closed, then?

[ReillyBrogan]

Will it still use zstd if I split this into two steps? Will this work with graphdriver?

1. `docker buildx build . --output=type=image,oci-mediatypes=true,compression=zstd --tag=bla-bla`

2. `docker push bla-bla`?

Unless I'm misunderstanding something you will need push=true in the docker buildx command in order to use zstd compression. Compression happens when the image is pushed to the repository, not at build time, so options that specify compression settings only work when the image is exported in some way. You can do this as push=true to do it as part of the buildx build itself, or you can use a different option to export the image (we export as an OCI image to a file since that's more convenient with our CI process, and then push it to a repository later with go-containerregistry tools).

I would recommend taking a look at buildx and buildkit (which buildx essentially wraps):

• https://github.com/docker/buildx
• https://github.com/moby/buildkit

Please, no. People want to switch to zstd because gzip is slow. Pushing gzip and zstd would be even slower.

I would note that the next version of containerd will support igzip which while still slower than zstd on decompression is at least not that much slower.

[slonopotamus]

The --output line needs "push=true", but this preempts local storage, apparently.

I fully wiped local storage and did docker buildx build . --output=type=image,oci-mediatypes=true,compression=zstd,push=true --tag=bla-bla. I also removed the tag from registry before pushing. Still, when pushing, I do not observe any change in image size at all.

$ docker info
Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: 4ffc61430bbe6d3d405bdf357b766bf303ff3cc5
 init version: de40ad007797e0dcd8b7126f27bb87401d224240
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.1.57-gentoo-dist
 Operating System: Gentoo Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.74GiB
 Name: noblesse
 ID: CA5U:PLGU:4PQJ:STD7:XUZU:WX2I:ESZ2:LJKX:UD4R:KOL4:GBBP:KZ5Z
 Docker Root Dir: /green/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additionally, oci-mediatypes=true is ignored. docker buildx imagetools inspect --raw says image has application/vnd.docker.distribution.manifest.v2+json media type.

[slonopotamus]

We're going to be leaving this issue closed, then?

Docker devs say Docker is already able to push zstd images, but nobody in current thread was able to reproduce that yet.

[ReillyBrogan]

You'll need force-compression=true in order to re-compress existing image layers. compression=zstd will only apply zstd compression to new image layers, assuming you are basing on an existing image FROM blahblahblah then the resulting image will be a mix of gzip layers and zstd layers without force-compression=trues.

I do not observe any change in image size at all

A better way to check is to see the mediatype of the image layers, and not the mediatype of the manifest.

but nobody in current thread was able to reproduce that yet

While we ultimately switched to using buildkit directly for image building I do know that I did have buildx working with building zstd images prior to the switch. I'm not in front of the scripts for that or I'd give you the exact options to use, but it does work if you use the right options.

[slonopotamus]

A better way to check is to see the mediatype of the image layers, and not the mediatype of the manifest.

I'm not sure how to do that.

Nope, even with force-compression=true it still says "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip"

[AkihiroSuda]

We're going to be leaving this issue closed, then?

Docker devs say Docker is already able to push zstd images, but nobody in current thread was able to reproduce that yet.

I can reproduce.

---

Dockerfile:

# The base image layers remain gzip by default.
# Specify force-compression=true to override this.
# ( See https://github.com/moby/buildkit/tree/v0.12.3#output )
FROM alpine

# compression=zstd applies to the layers below
RUN apk add neofetch

Build steps:

$ docker buildx create --use
nervous_clarke

$ docker buildx build --output type=image,name=akihirosuda/tmp-zstd-demo,oci-mediatypes=true,compression=zstd,push=true .
[+] Building 19.6s (9/9) FINISHED                                                               docker-container:nervous_clarke
 => [internal] booting buildkit                                                                                            8.8s
 => => pulling image moby/buildkit:buildx-stable-1                                                                         8.0s
 => => creating container buildx_buildkit_nervous_clarke0                                                                  0.7s
 => [internal] load build definition from Dockerfile                                                                       0.0s
 => => transferring dockerfile: 282B                                                                                       0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                           2.6s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                              0.0s
 => [internal] load .dockerignore                                                                                          0.0s
 => => transferring context: 2B                                                                                            0.0s
 => [1/2] FROM docker.io/library/alpine:latest@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978     0.6s
 => => resolve docker.io/library/alpine:latest@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978     0.0s
 => => sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa 3.40MB / 3.40MB                             0.4s
 => => extracting sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa                                  0.1s
 => [2/2] RUN apk add neofetch                                                                                             2.0s
 => exporting to image                                                                                                     5.4s
 => => exporting layers                                                                                                    0.1s
 => => exporting manifest sha256:a607b80f464e110b979794febc11f60a3033213e37a282f99d0500cc28dbe845                          0.0s 
 => => exporting config sha256:5c2effb34fa9ba83d127ef6f730ba0adb5610117e5f144e49c922dc7948c28a3                            0.0s 
 => => exporting attestation manifest sha256:baed245d68a2bcf1c81151836804b56289a77a0de7c4459901d7438cb3269761              0.0s 
 => => exporting manifest list sha256:c4a84a6dcc48aaf672db01432bd91442f2c0a996620440a32f9613c09f68f5de                     0.0s 
 => => pushing layers                                                                                                      3.5s
 => => pushing manifest for docker.io/akihirosuda/tmp-zstd-demo:latest@sha256:c4a84a6dcc48aaf672db01432bd91442f2c0a996620  1.7s
 => [auth] akihirosuda/tmp-zstd-demo:pull,push token for registry-1.docker.

Confirmation:

• sha256:a607b80f464e110b979794febc11f60a3033213e37a282f99d0500cc28dbe845 is the digest of the manifest (see the exporting manifest ... line above)
• the first layer is pushed as +gzip because force-compression=true is not specified
• the second layer is pushed as +zstd

$ docker buildx imagetools inspect --raw akihirosuda/tmp-zstd-demo@sha256:a607b80f464e110b979794febc11f60a3033213e37a282f99d0500cc28dbe845
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:5c2effb34fa9ba83d127ef6f730ba0adb5610117e5f144e49c922dc7948c28a3",
    "size": 812
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa",
      "size": 3401967
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:38e95ff4e4d377377e6391ce1bfa253e415b7623e12e90e76f6feb9f948169c0",
      "size": 2780349
    }
  ]
}

Docker version

$ docker version
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:07:41 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:41 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker info
Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-86-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.818GiB
 Name: lima-docker-rootful
 ID: 2ee359ad-2cf5-4c9b-ae65-f572cbc5628b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: akihirosuda
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[AkihiroSuda]

As an evidence, posting the result of force-compression=true too

$ docker buildx build --output type=image,name=akihirosuda/tmp-zstd-demo:force-compression,oci-mediatypes=true,compression=zstd,push=true,force-compression=true .
[+] Building 6.9s (8/8) FINISHED                                                                                                       docker-container:nervous_clarke
 => [internal] load build definition from Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 282B                                                                                                                              0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                  1.4s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                   0.0s
 => [1/2] FROM docker.io/library/alpine:latest@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978                                            0.0s
 => => resolve docker.io/library/alpine:latest@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978                                            0.0s
 => CACHED [2/2] RUN apk add neofetch                                                                                                                             0.0s
 => exporting to image                                                                                                                                            5.4s
 => => exporting layers                                                                                                                                           0.1s
 => => exporting manifest sha256:71bedc0a7b811473156f170e4de7f0dbbc3b2ab5433a3d671c93416448132833                                                                 0.0s
 => => exporting config sha256:5c2effb34fa9ba83d127ef6f730ba0adb5610117e5f144e49c922dc7948c28a3                                                                   0.0s
 => => exporting attestation manifest sha256:6e6fe413c2e9eb25f28b37c12a03c18c9aca517964565ae936e6aa13f80494dc                                                     0.0s
 => => exporting manifest list sha256:59baa7c0cbca27952dfdf0899115f8cef8ea53e33c238e6518c35223f949474a                                                            0.0s
 => => pushing layers                                                                                                                                             3.4s
 => => pushing manifest for docker.io/akihirosuda/tmp-zstd-demo:force-compression@sha256:59baa7c0cbca27952dfdf0899115f8cef8ea53e33c238e6518c35223f949474a         1.9s
 => [auth] akihirosuda/tmp-zstd-demo:pull,push token for registry-1.docker.io

$ docker buildx imagetools inspect --raw akihirosuda/tmp-zstd-demo:force-compression@sha256:71bedc0a7b811473156f170e4de7f0dbbc3b2ab5433a3d671c93416448132833
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:5c2effb34fa9ba83d127ef6f730ba0adb5610117e5f144e49c922dc7948c28a3",
    "size": 812
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:0bc1f746077d47db5c749b0e66ec2362d12462280b31bdace073d41be181e68e",
      "size": 3432705
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:38e95ff4e4d377377e6391ce1bfa253e415b7623e12e90e76f6feb9f948169c0",
      "size": 2780349
    }
  ]
}

[AkihiroSuda]

A better way to check is to see the mediatype of the image layers, and not the mediatype of the manifest.

I'm not sure how to do that.

Nope, even with force-compression=true it still says "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip"

You need to run docker buildx create --use first, to configure buildx to use the latest stable version of BuildKit (containerized)

[slonopotamus]

You need to run docker buildx create --use

I did that.

$ docker builder ls
NAME/NODE          DRIVER/ENDPOINT             STATUS   BUILDKIT             PLATFORMS
flamboyant_buck *  docker-container                                          
  flamboyant_buck0 unix:///var/run/docker.sock inactive                      
moby-buildkit      docker-container                                          
  moby-buildkit0   unix:///var/run/docker.sock inactive                      
default            docker                                                    
  default          default                     running  v0.11.6+0a15675913b7 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

Though I don't see how flamboyant_buck that was created differs from moby-buildkit builtin one.

[cpuguy83]

docker buildx build --output type=registry,compression=zstd -t cpuguy83/zstddemo -<<EOF
> FROM busybox
> RUN touch /foo
>
> EOF

busybox layer is gzip (because its unchanged) and the layer created by RUN is zstd:

https://explore.ggcr.dev/?image=cpuguy83/zstddemo@sha256:4f05074fea3e1718960e6a9ea5e59cae8066a33d210b1a59cdf7b8d3361b3079&mt=application%2Fvnd.oci.image.manifest.v1%2Bjson&size=668

[slonopotamus]

I did something and it started to work. Most likely buildx builder was not used for some reason. It it possible to prevent pushed image to be wrapped with application/vnd.oci.image.index.v1+json and only push application/vnd.oci.image.manifest.v1+json? GitLab Registry doesn't understand the former.

P.S. I still think it is too much invasive to force all these things on users instead of just docker push --compression zstd. Because I didn't want multi-image manifests, I didn't want buildx stuff, I just wanted zstd. And these are supposed to be absolutely independent features.

[ReillyBrogan]

It it possible to prevent pushed image to be wrapped with application/vnd.oci.image.index.v1+json and only push application/vnd.oci.image.manifest.v1+json?

Likely you are building and pushing a multi-arch image for some reason.

I didn't want buildx stuff

I think you misunderstand. docker-buildx IS the docker build engine now. docker build just uses buildx behind the scenes, which itself wraps buildkit. buildkit is the Docker build engine that was re-factored out of Docker and continued development and feature work, and then re-introduced as the buildx plugin (now the default build engine, which is why you likely noticed that the docker build messages changed format with Docker 23/24).

And these are supposed to be absolutely independent features.

That sounds like a personal opinion that doesn't have any basis in reality. Zstd compression support was implemented into Buildkit directly and support for it is only available through buildkit/buildx. You cannot have zstd-compressed images without buildx. You must specify OCI mediatypes because the Docker mediatype doesn't support zstd compression.

Like it or not you are trying to use advanced features that are still relatively new here, and so advanced configuration is expected at this stage. There is a lot of room for improved UX around this and that will come with time. As you are seeing zstd support is still fairly new in the ecosystem and there's still a lot of work to be done. This issue is specifically dealing with Docker supporting zstd-compressed images for pushing and pulling, which it does. Issue should remain closed.

GitLab Registry doesn't understand the former.

Sounds like a problem with Gitlab Registry. We use it too and these images do still work on pulling/pushing even if the webui doesn't pretty-print the metadata for tags with multi-image manifests like this.

[slonopotamus]

Likely you are building and pushing a multi-arch image for some reason.

Nope, it is single-arch, but I do have --platform linux/amd64.

[slonopotamus]

Just in case, this is how manifest looks like:

$ docker buildx imagetools inspect --raw  <image>
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:13814e78f4bc2038361763974c6bb0e96ac385f017fc1fffa1000fccc153cdb9",
      "size": 2578,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:ef6935c08ec0f6c633159a7581a6bd10fd56b261e99aa77924e09f02c4fdddaa",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:13814e78f4bc2038361763974c6bb0e96ac385f017fc1fffa1000fccc153cdb9",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]

[AkihiroSuda]

It it possible to prevent pushed image to be wrapped with application/vnd.oci.image.index.v1+json and only push application/vnd.oci.image.manifest.v1+json?

Yes, with attestation-inline=false

[slonopotamus]

Yes, with attestation-inline=false

Thanks, that worked!

[gaby]

You almost need a PhD to use Zstd with Docker 😂

[dsoprea]

Except most Docker users are completely oblivious to these levels. We need tee-shirts. Something wordy that ends with "and all I got was this stupid algorithm." :)

…

On Wed, Nov 22, 2023, 18:20 Juan Calderon-Perez ***@***.***> wrote: You almost need a PhD to use Zstd with Docker 😂 — Reply to this email directly, view it on GitHub <#28394 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFOW2TMVAOU2RZ7XD2PHG3YF2CCHAVCNFSM4CWGEH72U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGM3DCOBZGE2Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dsoprea]

Is there some way to retain the layers and apply the tag(s) when not pushing? Currently, in order to debug/validate the build process, "push" needs to be "true".

[vvoland]

You can't without the containerd integration enabled.

With graphdrivers the layers are stored in the uncompressed form so it doesn't matter what compression you set when building that image, it will have no compression when loaded into Docker.
The image manifest is also not stored and is reconstructed when pushing the image to the registry.

With containerd image store, the image is stored both in the original form (unmodified image content blobs) and uncompressed (snapshots). That means that pushing the image no longer recreates the image manifest, but transfers the original image form instead.

Currently Docker doesn't expose the raw image content directly, but you can use for example the ctr to read the manifest and layer content:

# force-compression=true is needed to force recompression of the busybox layer which is gzip compressed
# otherwise only new layers would get compressed with zstd (and in this example we don't create any new layers because we don't have any `ADD` or `COPY` instruction)
$ echo 'FROM busybox' | docker buildx build --output type=image,compression=zstd,force-compression=true -t my-image -
[+] Building 0.1s (5/5) FINISHED                                                                                                                                                                                                                         docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                     0.0s
 => => transferring dockerfile: 50B                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                          0.0s
 => CACHED [1/1] FROM docker.io/library/busybox:latest@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79                                                                                                                                           0.0s
 => => resolve docker.io/library/busybox:latest@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79                                                                                                                                                  0.0s
 => exporting to image                                                                                                                                                                                                                                                   0.0s
 => => exporting layers                                                                                                                                                                                                                                                  0.0s
 => => exporting manifest sha256:bdb6b6d656ed78eedf98d556213825dbc52fc5e1bc4e41499de3011964deb18a                                                                                                                                                                        0.0s
 => => exporting config sha256:037035b971fe4ba8cfd47ffb6000e02ab063ea90362e4811c9625df947270cd5                                                                                                                                                                          0.0s
 => => exporting attestation manifest sha256:c4a54e5dee008e8ae1bbb9ce79db12417bdd2347375f122327c724e071a0b2f2                                                                                                                                                            0.0s
 => => exporting manifest list sha256:466184b4ad471bbba3bfc4cbf226e92c1e4f07e480417836c9343994b4053e67                                                                                                                                                                   0.0s
 => => naming to docker.io/library/my-image:latest                                                                                                                                                                                                                       0.0s
 => => unpacking to docker.io/library/my-image:latest                                                                                                                                                                                                                    0.0s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/wo1qe8u25llwq2rxfhpxwzbgu

$ ctr -n moby content get sha256:bdb6b6d656ed78eedf98d556213825dbc52fc5e1bc4e41499de3011964deb18a
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:037035b971fe4ba8cfd47ffb6000e02ab063ea90362e4811c9625df947270cd5",
    "size": 605
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:45107fed83aca2fe20442edf446c0d63732f7ae0553232bd3aa39d314ddb7f0c",
      "size": 1951598
    }
  ]
}

$ ctr -n moby content get sha256:45107fed83aca2fe20442edf446c0d63732f7ae0553232bd3aa39d314ddb7f0c | file -
/dev/stdin: Zstandard compressed data (v0.8+), Dictionary ID: None

[dsoprea]

Sorry. Yes. I'm referring to a containerd-enabled context.

Thanks for the example, as well as bringing the ctr command to my general awareness.