Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Implement zran from zlib/examples #1444

Open
forrestfwilliams opened this issue Feb 20, 2023 · 2 comments
Open

Feature Request: Implement zran from zlib/examples #1444

forrestfwilliams opened this issue Feb 20, 2023 · 2 comments

Comments

@forrestfwilliams
Copy link

As discussed in the zlib FAQ, DEFATE-compressed data is not designed for random access:

  1. Can I access data randomly in a compressed stream?

No, not without some preparation. If when compressing you periodically use Z_FULL_FLUSH, carefully write all the pending data at those points, and keep an index of those locations, then you can start decompression at those points. You have to be careful to not use Z_FULL_FLUSH too often, since it can significantly degrade compression. Alternatively, you can scan a deflate stream once to generate an index, and then use that index for random access. See examples/zran.c.

However, as this quote mentions, Mark Adler created the zran utility, which allows for the creation of indexes that can enable reading of DEFLATE-compressed data from the beginning of DEFLATE blocks. This capability would be incredibly useful for accessing scientific data archives that are typically written once, but read many times.

Mark Adler recently updated this utility and it would be fantastic to see an adaptation of zran added to zlib-ng as well.

@nmoinvaz
Copy link
Member

Can't you just use his zran with zlib-ng?

@forrestfwilliams
Copy link
Author

If possible, it would be great to see zran become a part of zlib-ng, instead of an add-on utility that users would have to install separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants