You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I was wondering: does the datafusion_remote_tables filter push-down not support sorting? It seems that using filters and limits in the absence of a sort order could lead to un-expected results.
I'd be happy to help address this if this is indeed the case.
The text was updated successfully, but these errors were encountered:
Sorting itself is handled by DataFusion further down the data processing pipeline (i.e. once the data has been fetched) by a plan node above the scanning node in the plan AST.
While in principle filtering and sorting are commutative, the limit doesn't commute with sorting. DataFusion handles this by carefully deciding when to push-down the limit down into the scan (hence why it's an Option<usize>), though I forgot where exactly that occurs.
I was wondering: does the
datafusion_remote_tables
filter push-down not support sorting? It seems that using filters and limits in the absence of a sort order could lead to un-expected results.I'd be happy to help address this if this is indeed the case.
The text was updated successfully, but these errors were encountered: