You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What happened:
The filter expr didn't return expected rows. My table is relatively big so I tried to construct a minimal test to reproduce it, see below code.
Besides, from what I see in the log, my guess is:
delta scan is good, it successfully prune irrelated files.
One more suggestion:
Actually, we are able to return dynamic filter push down flag?
pub enum TableProviderFilterPushDown {
/// The expression cannot be used by the provider.
Unsupported,
/// The expression can be used to reduce the data retrieved,
/// but the provider cannot guarantee it will omit all tuples that
/// may be filtered. In this case, DataFusion will apply an additional
/// `Filter` operation after the scan to ensure all rows are filtered correctly.
Inexact,
/// The provider **guarantees** that it will omit **all** tuples that are
/// filtered by the filter expression. This is the fastest option, if available
/// as DataFusion will not apply additional filtering.
Exact,
}
when the expr only includes partition columns, we should return Exact.
Environment
Linux, Rust
Delta-rs version:
0.17.3
Binding:
Environment:
Bug
What happened:
The filter expr didn't return expected rows. My table is relatively big so I tried to construct a minimal test to reproduce it, see below code.
Besides, from what I see in the log, my guess is:
InExact
filter push down, so datafusion apply the same filter again, but however, the physical plan gets wrong column index.What you expected to happen:
How to reproduce it:
I wrote a unit test to check it, but it seems like I don't have permission to push it?
More details:
The text was updated successfully, but these errors were encountered: