Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel fetching of column chunks when reading parquet files #2949

Closed
thinkharderdev opened this issue Jul 20, 2022 · 2 comments
Closed

Parallel fetching of column chunks when reading parquet files #2949

thinkharderdev opened this issue Jul 20, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@thinkharderdev
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and why for this feature, in addition to the what)

See apache/arrow-rs#2110 for the background.

Some object stores are able to efficiently parallelize reads so it would be great to take advantage of that in DataFusion.

apache/arrow-rs#2115 should add the plumbing in arrow-rs to allow DataFusion to decide how to best fetch multiple byte ranges (in this case, column chunks from a parquet file).

Describe the solution you'd like
A clear and concise description of what you want to happen.

I think there are two possible solutions:

  1. Make ParquetFileReader parallelize range requests by default, perhaps with some configurable options for max parallelism, etc.
  2. Push this back to the ObjectStore crate and just add a get_ranges(&self, location: &Path, ranges: Range<usize>) method and let the object store implementation decide what is the most efficient way to fetch multiple byte ranges.

I think 2 seems like a better solution but it involves making a change to a public trait in another crate (for the moment at least) which makes things logistically more complicated.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

We could not do anything.

Additional context
Add any other context or screenshots about the feature request here.

@alamb
Copy link
Contributor

alamb commented Aug 15, 2022

This would be awesome -- also related to #2504 / #2199 so we don't get uncontrolled parallelism

@thinkharderdev
Copy link
Contributor Author

@alamb I think we can close this one as we now have get_ranges in ObjectStore as well as #3051

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants