Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically Grow Parquet BitWriter Buffer #2226

Closed
tustvold opened this issue Jul 29, 2022 · 0 comments · Fixed by #2231 or #2259
Closed

Automatically Grow Parquet BitWriter Buffer #2226

tustvold opened this issue Jul 29, 2022 · 0 comments · Fixed by #2231 or #2259
Assignees
@tustvold
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The BitWriter within parquet requires knowing the exact size up front and will return an error if it runs out of space. This makes for a very confusing interface, results in a load of additional error handling code, and seems unnecessary

Describe the solution you'd like

BitWriter should grow its buffer as needed

Describe alternatives you've considered

Additional context
@tustvold tustvold added the enhancement Any new improvement worthy of a entry in the changelog label Jul 29, 2022
@tustvold tustvold self-assigned this Jul 29, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jul 29, 2022
@tustvold
Automatically grow parquet BitWriter (apache#2226)
f124959
@tustvold tustvold mentioned this issue Jul 29, 2022
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jul 29, 2022
@tustvold
Automatically grow parquet BitWriter (apache#2226)
2383ede
tustvold added a commit that referenced this issue Jul 31, 2022
@tustvold
Automatically grow parquet BitWriter (#2226) (~10% faster) (#2231)
99ad915 
* Automatically grow parquet BitWriter (#2226)

* Review feedback
tustvold added a commit to tustvold/arrow-rs that referenced this issue Aug 1, 2022
@tustvold
Remove fallibility from RLEEncoder (apache#2226)
27b7536
@tustvold tustvold mentioned this issue Aug 1, 2022
Merged
alamb pushed a commit that referenced this issue Aug 2, 2022
@tustvold
Remove fallibility from RLEEncoder (#2226) (#2259)
4222f5a
@alamb alamb added the parquet Changes to the parquet crate label Aug 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tustvold tustvold

Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
2 participants
@alamb @tustvold