[feat] Support columns_sorted in row_filters with pageIndex
#3477
Conversation
Signed-off-by: yangjiang <yangjiang@ebay.com>
| @@ -79,6 +79,7 @@ object_store = "0.5.0" | |||
| ordered-float = "3.0" | |||
| parking_lot = "0.12" | |||
| parquet = { version = "22.0.0", features = ["arrow", "async"] } | |||
| parquet-format = "4.0.0" | |||
There was a problem hiding this comment.
Should we move this logic to arrow-rs to avoid import this parquet-format
There was a problem hiding this comment.
I think that makes sense
There was a problem hiding this comment.
I think parquet 23.0.0 will have properly exposed arrow definitions (added by @tustvold in apache/arrow-rs#2626)
Though now that I see this, I see the proposal is probably to move the functions columns_sorted and check_is_ordered into the parquet crate which makes sense to me (and will likely get reviewed by some others with more parquet knowledge)
@Ted-Jiang can you file a ticket / PR to do so?
| @@ -79,6 +79,7 @@ object_store = "0.5.0" | |||
| ordered-float = "3.0" | |||
| parking_lot = "0.12" | |||
| parquet = { version = "22.0.0", features = ["arrow", "async"] } | |||
| parquet-format = "4.0.0" | |||
There was a problem hiding this comment.
I think parquet 23.0.0 will have properly exposed arrow definitions (added by @tustvold in apache/arrow-rs#2626)
Though now that I see this, I see the proposal is probably to move the functions columns_sorted and check_is_ordered into the parquet crate which makes sense to me (and will likely get reviewed by some others with more parquet knowledge)
@Ted-Jiang can you file a ticket / PR to do so?
|
marking this PR as a draft pending resolution of @liukun4515 's review: #3477 (comment) |
|
Closing as this PR is over a year old. Please feel free to reopen it / rebase it if you plan to keep working on it |
Which issue does this PR close?
Closes #3476.
I think we can set true with only one col with pageIndex which is ordered. 馃
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?