Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate Record Skipping into Column Reader Fuzz Tests #2198

Closed
tustvold opened this issue Jul 27, 2022 · 5 comments · Fixed by #2315
Closed

Integrate Record Skipping into Column Reader Fuzz Tests #2198

tustvold opened this issue Jul 27, 2022 · 5 comments · Fixed by #2315
Labels
development-process Related to development process of arrow-rs help wanted parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

parquet::arrow::arrow_reader contains a fairly extensive set of tests under run_single_column_reader_tests that are used to fuzz test the arrow readers. It would significantly boost the coverage of the record skipping logic if we could integrate the row skipping logic into this harness

Describe the solution you'd like

Add an optional row selection to TestOptions and add examples of using this to the all_options list in run_single_column_reader_tests.

Describe alternatives you've considered

We could not do this

Additional context

FYI @Ted-Jiang

@tustvold tustvold added enhancement Any new improvement worthy of a entry in the changelog help wanted labels Jul 27, 2022
@Ted-Jiang
Copy link
Member

Will cover it up 😄, hope no bug any more!

@tustvold
Copy link
Contributor Author

tustvold commented Aug 3, 2022

Where did we get with this, I think this is the major thing to be confident enough to make the APIs public?

@Ted-Jiang
Copy link
Member

Ted-Jiang commented Aug 3, 2022

I will try to finish this until this weekend.
I am free this evening try make a draft

@tustvold
Copy link
Contributor Author

tustvold commented Aug 3, 2022

Awesome, absolutely no rush on this, was purely curious if you wanted me to pick this up 😄

@Ted-Jiang
Copy link
Member

haha, i will try to figure this out ! I feel I can learn a lot from it😄

@alamb alamb added development-process Related to development process of arrow-rs parquet Changes to the parquet crate and removed enhancement Any new improvement worthy of a entry in the changelog labels Aug 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
development-process Related to development process of arrow-rs help wanted parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants