Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

db: dictionary compression #3453

Open
jbowens opened this issue Mar 25, 2024 · 0 comments
Open

db: dictionary compression #3453

jbowens opened this issue Mar 25, 2024 · 0 comments
Labels
A-storage T-storage w-amp potential to reduce write amplification
Projects

Comments

@jbowens
Copy link
Collaborator

jbowens commented Mar 25, 2024

Today Pebble's sstable compression is entirely per-block. If there's significant repetition across blocks within a sstable, the repetition is persisted. We could consider building dictionaries that are used across all the blocks of an sstable in the hopes of achieving better compression ratios. See RocksDB's feature:

See https://github.com/facebook/rocksdb/wiki/Dictionary-Compression

@jbowens jbowens added T-storage A-storage w-amp potential to reduce write amplification labels Mar 25, 2024
@jbowens jbowens added this to Incoming in Storage via automation Mar 25, 2024
@itsbilal itsbilal moved this from Incoming to Backlog in Storage Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-storage T-storage w-amp potential to reduce write amplification
Projects
Storage
  
Backlog
Development

No branches or pull requests

1 participant