Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider caching decrypted and decompressed data #158

Open
commial opened this issue Apr 26, 2023 · 0 comments
Open

Consider caching decrypted and decompressed data #158

commial opened this issue Apr 26, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@commial
Copy link
Contributor

commial commented Apr 26, 2023

When an archive is seeked, the reader has to:

  1. go back to the beginning of the corresponding encrypted block
  2. decrypt it
  3. decompress from the start of the compressed block

If another seek occurs, the uncompressed data and the decrypted data are lost, and the work must be done again.

This is not a problem when performing linear extraction, but some pattern suffers from this behavior, for instance:

  1. Create an archive with sparsed files, ie. [File 1 content 1][File 2 content 1][File 1 content 2][File 2 content 2] etc.
  2. Iterate over the file list (File 1, File 2) and read them

In the worst case of n tiny part for n files , every block could be decrypted and decompress n-times.

To avoid it, a cache could be use between layers. The implementation:

  • can be provided to each layer
  • could be a layer of its own
  • must have a strategy to avoid consuming the whole RAM in case of very large file (maybe with a LRU?)

Implementing #156 would be a way to check for the reality of performance increase

@commial commial added the enhancement New feature or request label Apr 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant