Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazily materialize the null buffer builder for all array builders. #2125

Closed
HaoYang670 opened this issue Jul 22, 2022 · 1 comment · Fixed by #2127
Closed

Lazily materialize the null buffer builder for all array builders. #2125

HaoYang670 opened this issue Jul 22, 2022 · 1 comment · Fixed by #2127
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog

Comments

@HaoYang670
Copy link
Contributor

HaoYang670 commented Jul 22, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We have seen obvious performance improvement by lazily materializing the null buffer builder (please see the primitive builder or the boolean array builder). We want to use the optimization across all array builders.

Describe the solution you'd like
Instead of modifying each array builder one by one, I want to declare a new type: NullBufferBuilder

struct NullBufferBuilder {
    bitmap_builder: Option<BooleanBufferBuilder>,
    length: usize,
}

impl NullBufferBuilder {
    fn append {...}
    fn finish {...}
    ...
}

Then, let each array builder to use it.

Describe alternatives you've considered
We could not do this.

Additional context
Thank for the thought from @tustvold #2117 (comment)

@HaoYang670 HaoYang670 added the enhancement Any new improvement worthy of a entry in the changelog label Jul 22, 2022
@HaoYang670
Copy link
Contributor Author

I will work on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants