Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid unecessary copies in Arrow IPC reader #2437

Closed
alamb opened this issue Aug 13, 2022 · 1 comment · Fixed by #2510
Closed

Avoid unecessary copies in Arrow IPC reader #2437

alamb opened this issue Aug 13, 2022 · 1 comment · Fixed by #2510
Assignees
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog good first issue Good for newcomers performance

Comments

@alamb
Copy link
Contributor

alamb commented Aug 13, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The Arrow IPC format is designed to avoid memory copies when moving data from one implementation to another. However, as noted by @tustvold on #2369 (comment), the arrow-rs ipc reader implementation is actually copying data unnecessarily

Describe the solution you'd like
In the ipc code, create a Buffer initially and rewrite the ipc implementation in terms of Buffer rather than &[u8] (as the final output needs to be in a Buffer

Describe alternatives you've considered
N/A

Additional context

Came up in the context of #2369

Possibly also related to #189

@alamb alamb added good first issue Good for newcomers arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog performance labels Aug 13, 2022
@tustvold tustvold changed the title Avoid unecessary copes in Arrow IPC reader Avoid unecessary copies in Arrow IPC reader Aug 13, 2022
@HaoYang670
Copy link
Contributor

I'd like to have a try to get familiar with the IPC code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog good first issue Good for newcomers performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants