Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Override DeltaBitPackEncoder Defaults #2282

Open
tustvold opened this issue Aug 2, 2022 · 0 comments
Open

Override DeltaBitPackEncoder Defaults #2282

tustvold opened this issue Aug 2, 2022 · 0 comments
Labels
enhancement Any new improvement worthy of a entry in the changelog help wanted

Comments

@tustvold
Copy link
Contributor

tustvold commented Aug 2, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The DeltaBitPackEncoder has a couple of parameters that users may wish to override, in particular:

  • Block size - the number of values per block (default: 128)
  • Miniblock count - the number of miniblocks per block (default: 4)

These defaults result in 32 elements per miniblock, the smallest possible. Small miniblocks may lead to smaller representation as the bit width is per miniblock, similarly small blocks may lead to smaller representation as the min delta is per block. However, for input data with a uniform distribution, larger blocks may yield better compression and faster decode.

See here for more information on what these parameters control

Describe the solution you'd like

ColumnProperties currently allows setting Encoding for a specific column, we should provide the ability to control the settings passed to the DeltaBitPackEncoder.

Describe alternatives you've considered

Additional context

#2276 adds vectorised unpacking of runs of 64x 64-bit numbers, however, this would need miniblocks containing 64 elements to benefit from this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog help wanted
Projects
None yet
Development

No branches or pull requests

1 participant