Row filtering #2271
Row filtering #2271
Conversation
| // TODO Why does this cause problems? | ||
| // if skipped != front.row_count { | ||
| // return Some(Err(general_err!( | ||
| // "failed to skip rows, expected {}, got {}", | ||
| // front.row_count, | ||
| // skipped | ||
| // ) | ||
| // .into())); | ||
| // } |
There was a problem hiding this comment.
I'm not sure what's up here. This is the correct assertion to make here but I think there is a bug somewhere in the skip logic that skips the right values but gets the accounting wrong. Leaving this check in causes issues and leaving it out seems to produce the correct results (if the selections are created correctly).
There was a problem hiding this comment.
@tustvold can you help out here? I haven't been following along with this PR / work closely enough to provide useful guidance here?
|
@thinkharderdev how would you like to proceed with this, in particular with regards to the suggestion on the ticket of pushing predicates into the ParquetRecordBatchReader? I would be happy to stub out an API for this if that would be helpful? |
I can take a crack at it and post a draft PR when the API is nailed down |
|
Closing in favor of #2310 |
Which issue does this PR close?
Closes #2270
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?