Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Omit validity buffer in PrimitiveArray::from_iter when all values are valid #1859

Conversation

jhorstmann
Copy link
Contributor

Which issue does this PR close?

Closes #1856.

Rationale for this change

This shouldn't add any overhead, but reduces memory usage slightly and maybe makes iteration a bit faster by not having to check a null buffer

What changes are included in this PR?

Are there any user-facing changes?

The behavior change can be observed by checking the memory consumption, like in the test_memory_size_primitive_nullable test that had to be adjusted, but semantically there is no change.

@github-actions github-actions bot added the arrow Changes to the arrow crate label Jun 13, 2022
arrow/src/array/array_primitive.rs Outdated Show resolved Hide resolved
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
@codecov-commenter
Copy link

Codecov Report

Merging #1859 (7bfc3f6) into master (fb697ce) will increase coverage by 0.00%.
The diff coverage is 95.00%.

@@           Coverage Diff           @@
##           master    #1859   +/-   ##
=======================================
  Coverage   83.49%   83.49%           
=======================================
  Files         201      201           
  Lines       56902    56915   +13     
=======================================
+ Hits        47511    47524   +13     
  Misses       9391     9391           
Impacted Files Coverage Δ
arrow/src/array/array_primitive.rs 94.97% <94.44%> (-0.09%) ⬇️
arrow/src/array/array.rs 89.69% <100.00%> (+0.02%) ⬆️
parquet_derive/src/parquet_field.rs 65.75% <0.00%> (-0.23%) ⬇️
arrow/src/array/array_struct.rs 89.20% <0.00%> (+0.79%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb697ce...7bfc3f6. Read the comment docs.

@nevi-me nevi-me merged commit 9bcd052 into apache:master Jun 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

PrimitiveArray::from_iter should omit validity buffer if all values are valid
5 participants