Upgrade to DataFusion 17.0.0 (#998)

* Upgrade to DataFusion 16.0.0 * fix * fix todo * Fix clippy style errors * Update getOffset and isUnbounded to better handle ScalarValue windowBounds * Window Plan: Fix field name order handling since the order now matches other aggregates (columns followed by agg columns) * Fix comment location * Predicate Filters: Df16 reverts changes in df15 and returns a simpler dnf based filter for some cases * testing * bump datafusion -> 16.1.0 * Fix 2 failing join pytests * cargo fmt --all * Ok really fix cargo fmt this time * Update join.rs to accept CAST as a JOIN condition type * Add explicit handling for RexAlias * Bump DataFusion -> 17.0.0 * clippy checks * Fix clippy warnings * Add protoc action to workflows for rust and style * use arrow re-exported from datafusion * Add protoc action tot conda.yml * Add protobuf install to import / conda build testing * Enable RUST_BACKTRACE=1 * Add RUST_BACKTRACE=1 to meta.yaml * Add Rust Toolchain action to conda.yml * Bump setuptools-rust version to 1.5.2 in hopes that helps * Enable RUST_BACKTRACE=full * Remove substrait package for now * Updates for BinaryExpr in join.rs and comment out FilterColumnsPostJoin temporarily * Remove protoc github action * Disable pytest that was using a disabled optimizer rule * Add libprotobuf=3 dependency * Remove libprotobuf dep * Remove Rust action from conda.yml * Address review --------- Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com> Co-authored-by: Jeremy Dyer <jdye64@gmail.com> Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com>
dask-contrib · Feb 13, 2023 · f895346 · f895346
1 parent 4f7e7f3
commit f895346
Show file tree

Hide file tree

Showing 37 changed files with 1,305 additions and 444 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -154,7 +154,7 @@ jobs:
           channel-priority: strict
       - name: Install dependencies and nothing else
         run: |
-          mamba install setuptools-rust
+          mamba install "setuptools-rust>=1.5.2"
           pip install -e . -vv
 
           which python

diff --git a/continuous_integration/environment-3.10-dev.yaml b/continuous_integration/environment-3.10-dev.yaml
@@ -30,7 +30,7 @@ dependencies:
 - pytest
 - python=3.10
 - scikit-learn>=1.0.0
-- setuptools-rust>=1.4.1
+- setuptools-rust>=1.5.2
 - sphinx
 - tpot
 - tzlocal>=2.1

diff --git a/continuous_integration/environment-3.8-dev.yaml b/continuous_integration/environment-3.8-dev.yaml
@@ -29,7 +29,7 @@ dependencies:
 - pytest
 - python=3.8
 - scikit-learn=1.0.0
-- setuptools-rust=1.4.1
+- setuptools-rust=1.5.2
 - sphinx
 - tpot
 - tzlocal=2.1

diff --git a/continuous_integration/environment-3.9-dev.yaml b/continuous_integration/environment-3.9-dev.yaml
@@ -30,7 +30,7 @@ dependencies:
 - pytest
 - python=3.9
 - scikit-learn>=1.0.0
-- setuptools-rust>=1.4.1
+- setuptools-rust>=1.5.2
 - sphinx
 - tpot
 - tzlocal>=2.1

diff --git a/continuous_integration/gpuci/environment.yaml b/continuous_integration/gpuci/environment.yaml
@@ -33,7 +33,7 @@ dependencies:
 - pytest
 - python=3.9
 - scikit-learn>=1.0.0
-- setuptools-rust>=1.4.1
+- setuptools-rust>=1.5.2
 - sphinx
 - tpot
 - tzlocal>=2.1

diff --git a/continuous_integration/recipe/meta.yaml b/continuous_integration/recipe/meta.yaml
@@ -18,16 +18,16 @@ build:
     - dask-sql-server = dask_sql.server.app:main
     - dask-sql = dask_sql.cmd:main
   string: py{{ python | replace(".", "") }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }}
-  script: {{ PYTHON }} -m pip install . --no-deps -vv
+  script: RUST_BACKTRACE=full {{ PYTHON }} -m pip install . --no-deps -vv
 
 requirements:
   build:
     - {{ compiler('rust') }}
-    - setuptools-rust >=1.4.1
+    - setuptools-rust >=1.5.2
   host:
     - pip
     - python
-    - setuptools-rust >=1.4.1
+    - setuptools-rust >=1.5.2
   run:
     - python
     - dask >=2022.3.0