New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add dictionary compression support. #10
base: master
Are you sure you want to change the base?
Conversation
Merge 10XGenomics fork.
LZ4 supports using a dictionary when processing data, which can make a big difference on small messages/files. This change exposes the relevant functions in lz4-sys, and extends Encoder and Decoder to support using dictionaries.
@pmarks Please take a look, thanks. |
@thoren-d, thanks for the contribution! In one of my use cases I hold an How big are the dictionaries, typically? One option would be that the Decoder could hold a private copy of the dictionary so that there's no new lifetime. Another option would be to make a new type Any thoughts? |
The dictionaries can be up to 64 KiB so it would be inefficient to make copies of them, especially considering they're most useful when decoding/encoding many small messages. My thoughts were to allow Encoder and Decoder to have shared ownership of their respective dictionaries. We could do this with either
|
LZ4 supports using a dictionary when processing data, which can make a big difference on small messages/files. This change exposes the relevant functions in lz4-sys, and extends Encoder and Decoder to support using dictionaries.