Support for zstd compression layer

[utam0k]

Description

Support for zstd¹²³ compression.
zstd is said to be much faster than gzip. However, I believe zstd is still in the experimental stage.
opencontainers/image-spec#803

Moby has completed the corresponding implementation but it has not yet been released. buildkit has completed its support, as you can see from its implementation. nerdctl can be used to confirm that it works.

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:a2723dd59ae9025fd33b77ae7316a77406fcd08d0077bcf5819fec5e5e49f6e6",
    "size": 3930
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:fb0b3276a519f5e7085f51c75989b287b234b3508e1524cf2cdcbc397c06ec3d",
      "size": 28574451
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:f3bdd52e8a0a2c581486e84c029b6e45571656416d62d9f11561714a7e202ec1",
      "size": 139
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:21930dd1a8bd2c1d054c0c737ff3ff21fa0ae99fc830ab44df1136c72f94352b",
      "size": 6542185
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:6cf4a0e124673f39bd1ea8c7ff37f22e929e3db4217a5bf5e5d165a8efc88573",
      "size": 123049956
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:693583e208bc9a66aaf4096d4488d38768fcf4445602de8cda06e4dbcdc4c6e7",
      "size": 65642862
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:92d8ef9231fc6aa6a7b4776809c54257a2aa4924e11a276a8f00fd876dcadedc",
      "size": 33833622
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:e0ba2d4918a6886a1e4afd68840266200e7a5bc4433bce1d6029e1f8b1f1cebf",
      "size": 774
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:4d37bcae4df9957c8e7fc3dcbb8b1d0d206515b9fd088e4dd7e22ee02b9cc01a",
      "size": 336616
    }
  ],
  "annotations": {
    "dazzle.gitpod.io/base-ref": "localhost:5000/dazzle:base--bd2b6e028018842142b5de640d18f91d9ac8cb22eae851087e116b678ac95f36@sha256:0d19fc62337e316b122853b1a319d500fcc5857f1fe3f4b56ff14d47971e6881"
  }
}

Related Issue(s)

gitpod-io/workspace-images#951 (comment)

How to test

$ gp await-port 5000
$ go run main.go build --context example localhost:5000/dazzle --compression zstd
$ go run main.go combine --context example localhost:5000/dazzle --all --compression zstd
$ go install github.com/csweichel/oci-tool@latest
$ oci-tool fetch raw localhost:5000/dazzle:all | jq # you can see `tar+zstd`

You can confirm that nerdctl works from your terminal by opening port 5000.

Release Notes

dazzle: Support for zstd compression

Documentation

Footnotes

1. https://www.infoq.com/news/2022/09/amazon-gzip-zstd/ ↩

2. https://aws.amazon.com/blogs/containers/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/ ↩

3. https://www.slideshare.net/KoheiTokunaga/starting-up-containers-super-fast-with-lazy-pulling-of-images ↩

[codecov]

Codecov Report

Merging #58 (7da8f29) into main (d889756) will increase coverage by 0.08%.
The diff coverage is 68.18%.

@@            Coverage Diff             @@
##             main      #58      +/-   ##
==========================================
+ Coverage   51.54%   51.63%   +0.08%     
==========================================
  Files           4        5       +1     
  Lines         906      920      +14     
==========================================
+ Hits          467      475       +8     
- Misses        331      337       +6     
  Partials      108      108              

Impacted Files	Coverage Δ
pkg/dazzle/combiner.go	24.73% <0.00%> (ø)
pkg/dazzle/compression.go	50.00% <50.00%> (ø)
pkg/dazzle/build.go	59.32% <100.00%> (+0.21%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[utam0k]

@aledbf Shall we support estargz too?

[aledbf]

@aledbf Shall we support estargz too?

No.

Edit: please prioritize the scheduled work. This is only a nice thing to have.

[utam0k]

@aledbf Shall we support estargz too?

No.

Edit: please prioritize the scheduled work. This is only a nice thing to have.

Sure.

[kylos101]

Flipping this back to draft and removing from our project for now.

@aledbf @utam0k if you'd like to pick this up again, may I ask you to start with a RFC? This way we can visualize the involved components and overall idea.

[utam0k]

@aledbf Do you have the plan to support eStargz near future? I asked because I'm afraid that if you support it, we don't need to support zstd, right?
If you don't mind, I'd like to write an RFC about changing the compression algorithm of the image layers, eStargz or zstd.