Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fallback for skip_page/peek_page without OffsetIndex in SerializedPag… #2490

Closed
wants to merge 3 commits into from

Conversation

thinkharderdev
Copy link
Contributor

@thinkharderdev thinkharderdev commented Aug 17, 2022

…eReader

Which issue does this PR close?

Closes #2459
Closes #2434

Rationale for this change

Need to be able to peek/skip pages even if we do not have an OffsetIndex

What changes are included in this PR?

Implement skip_next_page and peek_next_page in SerializedPageReader

Are there any user-facing changes?

@github-actions github-actions bot added the parquet Changes to the parquet crate label Aug 17, 2022
@@ -119,6 +119,7 @@ impl RowSelection {
}

/// Given an offset index, return the offset ranges for all data pages selected by `self`
#[allow(unused)]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clippy was unhappy because this method is only used with the async feature

@@ -484,13 +484,17 @@ pub(crate) fn decode_page(
Ok(result)
}

#[allow(clippy::large_enum_variant)]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Boxing buffered_header seemed counterproductive here so I just disabled the lint warning

@tustvold
Copy link
Contributor

I believe @Ted-Jiang is already working on this in #2460... Perhaps we could keep some of the tests from this?

@thinkharderdev
Copy link
Contributor Author

I believe @Ted-Jiang is already working on this in #2460... Perhaps we could keep some of the tests from this?

Ah, sorry about that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support SerializedPageReader::skip_page without OffsetIndex Error Reading Page Index When Not Available
2 participants