zlib: add zstd support

[jkrems]

Adds ZstdCompress and ZsdDecompress to the zlib module which can be used to compress/decompress with the Zstandard ("zstd") algorithm.

Notable omissions:

• Providing dictionaries isn't implemented.
• The docs in zlib.md don't call out any params beyond the basic compression level.

The code follows similar patterns to the PR that added Brotli support. Just that instead of brotli, it adds the equivalent zstd APIs. Just like Brotli, this required separate compression/decompression context objects.

Zstd itself has been around and stable for multiple years but this PR is early in terms of web support: It only just starts shipping by default in Chrome 123. On the other hand, by shipping in Chrome it will soon be supported quite widely on the web. Firefox also signaled support (https://bugzilla.mozilla.org/show_bug.cgi?id=1301878#c65).

Official support in node.js would allow passing additional WPTs around fetch (nodejs/undici#2847).

[nodejs-github-bot]

Review requested:

• @nodejs/security-wg

[jkrems]

Alright, got up to making the basics (seemingly) work:

$ out/Debug/node -p 'zlib.zstdDecompressSync(zlib.zstdCompressSync("Hello World")).toString()'
Hello World

[github-actions]

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @richardlau.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

[jkrems]

/cc @terrelln @Cyan4973 from the zstd side in case this PR is doing a bad job representing how zstd should be used.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/57839/

[terrelln]

I've only looked at the usage of the Zstd API. It looks about right, I've left some minor comments. I'm not familiar with the code, and it isn't clear to me exactly how DoThreadPoolWork() is called, so I can't be completely confident. But broadly it looks right.

One thing to consider is that some advanced parameters directly can cause Zstd to allocate a lot of memory (e.g. setting ZSTD_c_hashLog == ZSTD_c_chainLog == ZSTD_c_windowLog == 31 will cause Zstd to allocate at least 6 GB during compression). Additionally, the parameter ZSTD_c_nbWorkers will cause Zstd to spawn worker threads during compression.

This normally isn't an issue, because the author of the code is trusted. I assume that this is also the threat model for NodeJS. But, because it is JS, I did want to raise the point.

[terrelln]

There's no .params() support even though zstd should support changing (some) params at runtime.

There must be at least compression level support ZSTD_c_compressionLevel. This is the most important parameter by far.

What do you mean at runtime? You can change any parameters you want before the (de)compression starts. After that, changing compression parameters generally isn't allowed.

No input size hints for compression because it looked like that API in zstd wasn't stable yet.

If the input size is known in advance, then you can use ZSTD_setPledgedSrcSize(), however it is an error to pass an incorrect value. Additionally, if the first call to ZSTD_compressStream2() uses ZSTD_e_end, Zstd will infer the input size.

If the input size is only approximately known, but may be wrong, then Zstd also supports setting ZSTD_c_srcSizeHint. Zstd will use this parameter for tuning its tradeoffs for that source size, but it is allowed to be wrong. But, as you say this is a newer feature that we haven't yet stabilized. It works and is fully supported, but all new features go through our experimental API before being stabilized, so we have the freedom to tweak or remove them in the future.

[jkrems]

What do you mean at runtime? You can change any parameters you want before the (de)compression starts. After that, changing compression parameters generally isn't allowed.

Thanks for confirming. The "at runtime" refers to the deflate-specific feature (deflateParams in zlib) that allows changing certain params within the same compression stream (at block boundaries IIUC). Node-level docs: https://nodejs.org/api/zlib.html#zlibparamslevel-strategy-callback

Updated the PR description to correct this part.

If the input size is known in advance, then you can use ZSTD_setPledgedSrcSize(), however it is an error to pass an incorrect value. Additionally, if the first call to ZSTD_compressStream2() uses ZSTD_e_end, Zstd will infer the input size.

Ah, thanks. Saw ZSTD_c_srcSizeHint but should've kept looking. Added pledgedSrcSize as an option.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/58350/

[jkrems]

Rebased after the fixes to ensure it still lands cleanly.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/58351/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/58412/

[jkrems]

Looking at the test failures, they look unrelated at first glance (different tests on different OS, not related to compression afaict). Resuming the build to see if the are flakes.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/58430/

[bricss]

Maybe it's possible to bump zstd to the latest version? 🤔

[jkrems]

Happy to rerun the import. I think right now I'm also looking for signal that this is something that other collaborators want in core. Otherwise rebasing (potentially) and reimporting wouldn't really be worth it - it would just bitrot again.

[bricss]

I cannot speak on behalf of collaborators, but imho W3C community would certainly like to see it in the core 🧐
Given the fact that Zstd Content-Encoding will soon be supported by majority of modern browsers 🧭