Question: snappy compression on large data is not compressed, and message size enlarged by 7 bytes.

[3AceShowHand]

Our service uses the snappy compression to encode large JSON data, the size is a few hundred kilobytes, even a few megabytes.

After the compression, we found that the message size is enlarged by around 7 bytes for all messages, and the original data content is not compressed.

The original data:
after-before.zip

Then we use the https://github.com/golang/snappy to compress the data, and it's compressed indeed, but the compression ratio is very small. For example, the original data size is around 550k, compressed data size is around 549k.

Question:

The original data is not compressed, is this by design or an unexpected behavior ?

[3AceShowHand]

It looks that the data is seemed as not compressible since n is 0 at the L243.

[klauspost]

Yes. It will check if it can do a "reasonable" compression. If the output gets bigger than this, it is stored as an uncompressed block, which is faster to decompress.

Limits are depending on the compression level:

• Default & Better: dstLimit := len(src) - len(src)>>5 - 5
• Best: dstLimit := len(src) - 5

So it must at least be able to remove 1 in 32 bytes in the input.

The 7 bytes are needed for it to be a valid block.

[3AceShowHand]

What do you mean by "reasonable", the github.com/golang/snappy compressed the data, even though the compression ratio is quite small, is this reasonable?

If the output gets bigger than this, it is stored as an uncompressed block, which is faster to decompress.

What is the difference between the Better and the Best?

It looks like the Better is the default, if the data does no need to be compressed, keep it the original.

Will the Best always compress the data, just like what the official implementation did ?

[klauspost]

Let's go through my answer. dstLimit := len(src) - len(src)>>5 - 5 describes the maximum output size before we reject the compression.

If you are unfamiliar with bit shifting, it can also be written as: dstLimit := len(src) - (len(src)/32) - 5.

This means that the limit is "the size if the input" minus "the size divided by 32", minus 5.

Take an input of 1000 bytes. We then replace and it becomes dstLimit = 1000 - (1000/32 = 31) - 5 = 964. If we get to or exceed this limit, we emit the block as uncompressed.

For best, the limit is 1000-5 = 995 bytes.

[3AceShowHand]

Thanks for your reply and detailed explanation, I got it.

It's by design that If the output size exceeds the limit, keep it the origin, to make decompress easy.

Best limit is larger than the Better limit, which means some input cannot be compressed by the Better, but might be compressed by the Best.