Feature request: Adjustable window size for deflate and gzip

[rojer]

My project targets embedded devices that cannot afford deflate's default window size of 32K.
The long and sad story of golang/go#3155 doesn't give me any hope of seeing it in the standard library so I'm looking for alternatives.
Yours seems like a high-quality, well used and actively maintained compression library that could be a good substitute for compress/flate, would you consider making logWindowSize an adjustable parameter? Even better if it could be pulled through all the way to the gzip writer.

I'm willing to help out with implementation if you agree in principle.

[klauspost]

Stateless compression should do this: https://github.com/klauspost/compress#stateless-compression

Basically the write sizes indicate the maximum backreference. So if you write no more than 4kb, that is max back ref.

I seem to remember it in reality has an 8kb window limit, but that is not a guarantee.

[rojer]

i'm looking at more like 512 or 1024 byte window size. and i need it guaranteed, or device won't be able to update.
(context: compressing firmware updates to save traffic, the device on the other side is very constrained)

[klauspost]

@rojer I think forking would be the easiest. If everything works as intended it is a one line patch, that makes merging back upstream changes trivial.

Since deflate (or gzip) doesn't communicate the window size, there is no real way to communicate a smaller window size, and it is a "trust" thing.

There will be a performance drop if it is made configurable, and since this is such a limited use case, I don't have a strong motivation to add it to be honest.

If you find issues where it doesn't respect the window size change, I would be happy to integrate fixes, though.

[rojer]

thanks for your response Klaus!

i try to minimize the number of forks we rely on in our projects, but if it comes to that, there's always the option, of course.
however, golang/go#3155 exists, so i know i'm not alone in wanting this. also, if implemented in a library such as yours, there's a better chance of getting movement in the standard library, so there's that motive :)

wrt trust - yes, of course, window size has to be agreed in advance between writer and reader. but i don't quite get why it is an argument not to make it an option for the writer, since even if the reader is not aware of the settings, it will still be able to read the bitstream. yes, a constrained reader has to trust the writer to use a particuar window size, but unconstrained reader can still read the stream, so... i guess i don't quite get the argument why not make it an option for the writer?

wrt performance - am i misunderstanding or are we talking const int vs var int access?

[klauspost]

wrt performance - am i misunderstanding or are we talking const int vs var int access?

Window size is used to allocate fixed size arrays. Take:

const(
	logWindowSize    = 10
	windowSize       = 1 << logWindowSize
	windowMask       = windowSize - 1
)

	var table[windowSize]uint32
... 
	table[x&windowMask] = y

The &windowmask makes it so the compiler can omit the bounds check, since there is no way it can over or underflow.

Secondary, yes, the const vs dynamic value in level 1-6 makes a difference.

I could make a single level (for example level 5) with a dynamic max offset. That should be fairly trivial compared to a generic case.

Let me make a PR with that.

[klauspost]

#843 - roundtrips work, but I don't have an easy way to test if max offsets are respected.