zstd failure for unsepcified buffer size

[martindurant]

Reported in kerchunk: fsspec/kerchunk#317 (comment) by @cgohlke

Minimal, reproducible code sample, a copy-pastable example if possible

>>> import numcodecs, cramjam
>>> numcodecs.Zstd().decode(cramjam.zstd.compress(bytes(1)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "numcodecs/zstd.pyx", line 219, in numcodecs.zstd.Zstd.decode
  File "numcodecs/zstd.pyx", line 153, in numcodecs.zstd.decompress
RuntimeError: Zstd decompression error: invalid input data

Problem description

According to zstd.h, note 2 "decompressed size is an optional field, it may not be present, typically in streaming mode."

This means, that we can always decompress chunks written by our own Zstd, but. not necessarily those written by others. This is another case in which we could simply use cramjam or imagecodecs (or even zstandard) rather than building out own cython extension. In most cases, we will know the size of the decompressed data, because it must fit the output (but other filters in the chain may spoil this).

Version and installation information

Please provide the following:

• version: main
• Version of Python interpreter: any