Add SQL planner support for `Like`, `ILike` and `SimilarTo`, with optional escape character #3101

andygrove · 2022-08-10T22:46:00Z

Which issue does this PR close?

Rationale for this change

The Dask SQL project would like to support queries using LIKE, ILIKE,SIMILAR TO syntax in Postgres, with an optional escape character specified, so we want to add this support to the SQL query planner and the logical plan.

What changes are included in this PR?

We originally modeled LIKE and NOT LIKE as binary operations but this cannot support the optional escape character. For that reason, this PR adds new top-level expressions, but the old ones remain for backwards compatibility.

There is no support in the physical planner yet for supporting the optional escape character.

Are there any user-facing changes?

Yes. This is an API change.

codecov-commenter · 2022-08-24T20:12:23Z

Codecov Report

Merging #3101 (a18dec8) into master (e6378f4) will increase coverage by 0.06%.
The diff coverage is 60.56%.

@@            Coverage Diff             @@
##           master    #3101      +/-   ##
==========================================
+ Coverage   85.58%   85.64%   +0.06%     
==========================================
  Files         296      296              
  Lines       54252    54362     +110     
==========================================
+ Hits        46432    46561     +129     
+ Misses       7820     7801      -19

Impacted Files	Coverage Δ
datafusion/sql/src/planner.rs	`81.06% <18.75%> (-0.62%)`	⬇️
datafusion/physical-expr/src/planner.rs	`93.29% <77.77%> (-0.91%)`	⬇️
datafusion/proto/src/lib.rs	`93.85% <100.00%> (+0.33%)`	⬆️
benchmarks/src/bin/tpch.rs	`37.59% <0.00%> (-3.56%)`	⬇️
datafusion/optimizer/src/optimizer.rs	`90.90% <0.00%> (-1.40%)`	⬇️
datafusion/optimizer/src/reduce_outer_join.rs	`98.19% <0.00%> (-0.61%)`	⬇️
datafusion/proto/src/logical_plan.rs	`17.46% <0.00%> (-0.24%)`	⬇️
datafusion/common/src/scalar.rs	`85.06% <0.00%> (-0.07%)`	⬇️
datafusion/proto/src/bytes/mod.rs	`82.75% <0.00%> (ø)`
... and 12 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

andygrove · 2022-09-08T14:55:18Z

@ayushdg PTAL when you can

Dandandan · 2022-09-08T15:03:43Z

datafusion/optimizer/src/type_coercion.rs

+            Expr::Like { pattern, .. }
+            | Expr::ILike { pattern, .. }
+            | Expr::SimilarTo { pattern, .. } => match pattern.get_type(&self.schema)? {
+                DataType::Utf8 => Ok(expr.clone()),


What about LargeUtf8?

This is for the regular expression pattern, which is typically a literal string but could, in theory, be a column reference. I'm not sure if it would make sense to support LargeUtf8 here?

alamb · 2022-09-08T20:58:31Z

datafusion/sql/src/planner.rs

-                }
+                Ok(Expr::Like {
+                    negated,
+                    expr: Box::new(self.sql_expr_to_logical_expr(*expr, schema, ctes)?),


alamb · 2022-09-08T20:59:36Z

datafusion/optimizer/src/type_coercion.rs

                })
            }
+            Expr::Like { pattern, .. }
+            | Expr::ILike { pattern, .. }
+            | Expr::SimilarTo { pattern, .. } => match pattern.get_type(&self.schema)? {


This seems like the wrong place to be type checking the argument to Expr::Like, etc. as the argument types to other exprs are checked so why would we treat Expr::Like differently?

I would expect that to be done in the SQL planner perhaps? or in the physical_expr conversion?

We can't check the type of the pattern expression until we have a schema to resolve against (it could be a reference to a column or any other type of expression).

My goal was to do the validation in the logical plan for the benefit of other engines building on DataFusion that don't use the physical plan.

Never mind, we do have the schema in SQL planning ... I will make that change.

@alamb Ok, this PR just got a whole lot smaller. Maybe this really shouldn't have taken me 30 days to do 🤣

alamb · 2022-09-08T20:59:50Z

This is pretty neat, by the way 👍

alamb

Looks great to me 😍

@alamb Ok, this PR just got a whole lot smaller. Maybe this really shouldn't have taken me 30 days to do 🤣

Well, I think it had a bunch of dependencies that are all now sorted out, so the final PR "makes it look easy" even though a bunch of effort was needed

alamb · 2022-09-09T14:33:05Z

datafusion/sql/src/planner.rs

-                    }
+                let pattern = self.sql_expr_to_logical_expr(*pattern, schema, ctes)?;
+                let pattern_type = pattern.get_type(schema)?;
+                if pattern_type != DataType::Utf8 && pattern_type != DataType::Null {


ursabot · 2022-09-09T14:41:49Z

Benchmark runs are scheduled for baseline = 73447b5 and contender = c96f03e. c96f03e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

…ional escape character (apache#3101) * Make Like a top-level Expr * revert some changes * add type validation * Revert physical plan changes and reduce scope of the PR * Revert more changes * Revert more changes * clippy * address feedback * revert change to test * revert more changes

andygrove added the api change Changes the API exposed to users of the crate label Aug 10, 2022

github-actions bot added core Core datafusion crate logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Physical Expressions sql labels Aug 10, 2022

ayushdg mentioned this pull request Aug 16, 2022

[sqlparser-0.21] Update trimExpr members during planning #3181

Merged

This was referenced Aug 19, 2022

[DF] Remaining DataFusion / sqlparser work for the GA release dask-contrib/dask-sql#706

Closed

optimizer: add framework for the rule of pre-add cast to the literal in comparison binary #3185

Merged

andygrove marked this pull request as ready for review August 24, 2022 19:38

andygrove changed the title ~~[WIP] Make Like a top-level Expr and add SQL support for ILike and SimilarTo~~ Make Like a top-level Expr and add SQL support for ILike and SimilarTo Aug 24, 2022

andygrove marked this pull request as draft August 24, 2022 20:22

andygrove marked this pull request as ready for review August 24, 2022 20:52

andygrove marked this pull request as draft August 25, 2022 17:39

andygrove mentioned this pull request Aug 30, 2022

Add top-level Like, ILike, SimilarTo expressions in logical plan #3298

Merged

Make Like a top-level Expr

64e8559

andygrove force-pushed the like-expr branch from 7483e3e to 64e8559 Compare September 6, 2022 14:15

andygrove added 6 commits September 6, 2022 08:18

revert some changes

d1d6655

add type validation

bd14a93

Revert physical plan changes and reduce scope of the PR

41f7db1

Revert more changes

718c304

Revert more changes

64dbd31

upmerge

4882a98

andygrove marked this pull request as ready for review September 6, 2022 17:58

andygrove requested a review from alamb September 6, 2022 18:05

github-actions bot removed the logical-expr Logical plan and expressions label Sep 6, 2022

andygrove added 2 commits September 6, 2022 16:08

clippy

df73d16

upmerge

63bf702

andygrove requested a review from tustvold September 8, 2022 14:32

andygrove requested review from liukun4515, Dandandan, jimexist, yahoNanJing, viirya and yjshen September 8, 2022 14:32

andygrove changed the title ~~Make Like a top-level Expr and add SQL support for ILike and SimilarTo~~ Add SQL planner support for Like, ILike and SimilarTo, with optional escape character Sep 8, 2022

Dandandan reviewed Sep 8, 2022

View reviewed changes

ayushdg approved these changes Sep 8, 2022

View reviewed changes

alamb reviewed Sep 8, 2022

View reviewed changes

andygrove added 2 commits September 8, 2022 19:18

address feedback

3a86db8

revert change to test

8ea73b1

github-actions bot removed the core Core datafusion crate label Sep 9, 2022

revert more changes

a18dec8

github-actions bot removed the optimizer Optimizer rules label Sep 9, 2022

alamb approved these changes Sep 9, 2022

View reviewed changes

alamb merged commit c96f03e into apache:master Sep 9, 2022

yukkit mentioned this pull request Oct 26, 2022

Upgrade datafusion version to 13.0.0 cnosdb/cnosdb#719

Closed

andygrove deleted the like-expr branch January 27, 2023 18:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SQL planner support for `Like`, `ILike` and `SimilarTo`, with optional escape character #3101

Add SQL planner support for `Like`, `ILike` and `SimilarTo`, with optional escape character #3101

andygrove commented Aug 10, 2022 •

edited

codecov-commenter commented Aug 24, 2022 •

edited

andygrove commented Sep 8, 2022

Dandandan Sep 8, 2022

andygrove Sep 8, 2022

alamb Sep 8, 2022

alamb Sep 8, 2022

andygrove Sep 9, 2022

andygrove Sep 9, 2022

andygrove Sep 9, 2022

alamb commented Sep 8, 2022

alamb left a comment

alamb Sep 9, 2022

ursabot commented Sep 9, 2022

Add SQL planner support for Like, ILike and SimilarTo, with optional escape character #3101

Add SQL planner support for Like, ILike and SimilarTo, with optional escape character #3101

Conversation

andygrove commented Aug 10, 2022 • edited

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

codecov-commenter commented Aug 24, 2022 • edited

Codecov Report

andygrove commented Sep 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Sep 8, 2022

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ursabot commented Sep 9, 2022

Add SQL planner support for `Like`, `ILike` and `SimilarTo`, with optional escape character #3101

Add SQL planner support for `Like`, `ILike` and `SimilarTo`, with optional escape character #3101

andygrove commented Aug 10, 2022 •

edited

codecov-commenter commented Aug 24, 2022 •

edited