Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cleanup CI #2933

Merged
merged 5 commits into from Oct 27, 2022
Merged

Cleanup CI #2933

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0dfcccf
Cleanup CI
tustvold Oct 26, 2022
db800c9
Fix object_store emulator tests
tustvold Oct 26, 2022
71e3261
Merge branch 'master' into cleanup-ci
tustvold Oct 27, 2022
3b43873
Update .github/actions/setup-builder/action.yaml
tustvold Oct 27, 2022
9aebb2e
Fix formatting
tustvold Oct 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 12 additions & 4 deletions .github/actions/setup-builder/action.yaml
View file
Open in desktop
Expand Up @@ -20,8 +20,12 @@ description: 'Prepare Rust Build Environment'
inputs:
rust-version:
description: 'version of rust to install (e.g. stable)'
required: true
required: false
default: 'stable'
target:
description: 'target architecture(s)'
required: false
default: 'x86_64-unknown-linux-gnu'
runs:
using: "composite"
steps:
Expand Down Expand Up @@ -51,7 +55,11 @@ runs:
shell: bash
run: |
echo "Installing ${{ inputs.rust-version }}"
rustup toolchain install ${{ inputs.rust-version }}
rustup toolchain install ${{ inputs.rust-version }} --target ${{ inputs.target }}
rustup default ${{ inputs.rust-version }}
rustup target add wasm32-unknown-unknown
echo "CARGO_TARGET_DIR=/github/home/target" >> $GITHUB_ENV
- name: Disable debuginfo generation
tustvold marked this conversation as resolved.
Show resolved Hide resolved
shell: bash
run: echo "RUSTFLAGS=-C debuginfo=1" >> $GITHUB_ENV
- name: Enable backtraces
shell: bash
run: echo "RUST_BACKTRACE=1" >> $GITHUB_ENV
100 changes: 29 additions & 71 deletions .github/workflows/arrow.yml
View file
Open in desktop
Expand Up @@ -42,18 +42,12 @@ jobs:
runs-on: ubuntu-latest
container:
image: amd64/rust
env:
# Disable full debug symbol generation to speed up CI build and keep memory down
# "1" means line tables only, which is useful for panic tracebacks.
RUSTFLAGS: "-C debuginfo=1"
steps:
- uses: actions/checkout@v3
with:
submodules: true
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: stable
- name: Test arrow-buffer with all features
run: cargo test -p arrow-buffer --all-features
- name: Test arrow-data with all features
Expand All @@ -66,11 +60,10 @@ jobs:
run: cargo test -p arrow-select --all-features
- name: Test arrow-integration-test with all features
run: cargo test -p arrow-integration-test --all-features
- name: Test arrow
- name: Test arrow with default features
run: cargo test -p arrow
- name: Test --features=force_validate,prettyprint,ipc_compression,ffi,dyn_cmp_dict,dyn_arith_dict,chrono-tz
run: |
cargo test -p arrow --features=force_validate,prettyprint,ipc_compression,ffi,dyn_cmp_dict,dyn_arith_dict,chrono-tz
- name: Test arrow with all features apart from simd
run: cargo test -p arrow --features=force_validate,prettyprint,ipc_compression,ffi,dyn_cmp_dict,dyn_arith_dict,chrono-tz
- name: Run examples
run: |
# Test arrow examples
Expand All @@ -87,50 +80,33 @@ jobs:
runs-on: ubuntu-latest
container:
image: amd64/rust
env:
# Disable full debug symbol generation to speed up CI build and keep memory down
# "1" means line tables only, which is useful for panic tracebacks.
RUSTFLAGS: "-C debuginfo=1"
steps:
- uses: actions/checkout@v3
with:
submodules: true
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: stable
- name: Check compilation
run: |
cargo check -p arrow
run: cargo check -p arrow
- name: Check compilation --no-default-features
run: |
cargo check -p arrow --no-default-features
run: cargo check -p arrow --no-default-features
- name: Check compilation --all-targets
run: |
cargo check -p arrow --all-targets
run: cargo check -p arrow --all-targets
- name: Check compilation --no-default-features --all-targets
run: |
cargo check -p arrow --no-default-features --all-targets
run: cargo check -p arrow --no-default-features --all-targets
- name: Check compilation --no-default-features --all-targets --features test_utils
run: |
cargo check -p arrow --no-default-features --all-targets --features test_utils
run: cargo check -p arrow --no-default-features --all-targets --features test_utils
- name: Check compilation --no-default-features --all-targets --features ffi
run: |
cargo check -p arrow --no-default-features --all-targets --features ffi
run: cargo check -p arrow --no-default-features --all-targets --features ffi
- name: Check compilation --no-default-features --all-targets --features chrono-tz
run: |
cargo check -p arrow --no-default-features --all-targets --features chrono-tz
run: cargo check -p arrow --no-default-features --all-targets --features chrono-tz

# test the --features "simd" of the arrow crate. This requires nightly Rust.
linux-test-simd:
name: Test SIMD on AMD64 Rust ${{ matrix.rust }}
runs-on: ubuntu-latest
container:
image: amd64/rust
env:
# Disable full debug symbol generation to speed up CI build and keep memory down
# "1" means line tables only, which is useful for panic tracebacks.
RUSTFLAGS: "-C debuginfo=1"
steps:
- uses: actions/checkout@v3
with:
Expand All @@ -140,46 +116,32 @@ jobs:
with:
rust-version: nightly
- name: Run tests --features "simd"
run: |
cargo test -p arrow --features "simd"
run: cargo test -p arrow --features "simd"
- name: Check compilation --features "simd"
run: |
cargo check -p arrow --features simd
run: cargo check -p arrow --features simd
- name: Check compilation --features simd --all-targets
run: |
cargo check -p arrow --features simd --all-targets
run: cargo check -p arrow --features simd --all-targets


# test the arrow crate builds against wasm32 in stable rust
# test the arrow crate builds against wasm32 in nightly rust
wasm32-build:
name: Build wasm32
runs-on: ubuntu-latest
container:
image: amd64/rust
env:
# Disable full debug symbol generation to speed up CI build and keep memory down
# "1" means line tables only, which is useful for panic tracebacks.
RUSTFLAGS: "-C debuginfo=1"
steps:
- uses: actions/checkout@v3
with:
submodules: true
- name: Cache Cargo
uses: actions/cache@v3
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
path: /github/home/.cargo
key: cargo-wasm32-cache3-
- name: Setup Rust toolchain for WASM
run: |
rustup toolchain install nightly
rustup override set nightly
rustup target add wasm32-unknown-unknown
rustup target add wasm32-wasi
- name: Build
run: |
cd arrow
cargo build --no-default-features --features=json,csv,ipc,simd,ffi --target wasm32-unknown-unknown
cargo build --no-default-features --features=json,csv,ipc,simd,ffi --target wasm32-wasi
rust-version: nightly
target: wasm32-unknown-unknown,wasm32-wasi
- name: Build wasm32-unknown-unknown
run: cargo build -p arrow --no-default-features --features=json,csv,ipc,simd,ffi --target wasm32-unknown-unknown
- name: Build wasm32-wasi
run: cargo build -p arrow --no-default-features --features=json,csv,ipc,simd,ffi --target wasm32-wasi

clippy:
name: Clippy
Expand All @@ -190,21 +152,17 @@ jobs:
- uses: actions/checkout@v3
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: stable
- name: Setup Clippy
run: |
rustup component add clippy
run: rustup component add clippy
- name: Clippy arrow-buffer with all features
run: cargo clippy -p arrow-buffer --all-targets --all-features
run: cargo clippy -p arrow-buffer --all-targets --all-features -- -D warnings
- name: Clippy arrow-data with all features
run: cargo clippy -p arrow-data --all-targets --all-features
run: cargo clippy -p arrow-data --all-targets --all-features -- -D warnings
- name: Clippy arrow-schema with all features
run: cargo clippy -p arrow-schema --all-targets --all-features
run: cargo clippy -p arrow-schema --all-targets --all-features -- -D warnings
- name: Clippy arrow-array with all features
run: cargo clippy -p arrow-array --all-targets --all-features
run: cargo clippy -p arrow-array --all-targets --all-features -- -D warnings
- name: Clippy arrow-select with all features
run: cargo clippy -p arrow-select --all-targets --all-features
run: cargo clippy -p arrow-select --all-targets --all-features -- -D warnings
- name: Clippy arrow
run: |
cargo clippy -p arrow --features=prettyprint,csv,ipc,test_utils,ffi,ipc_compression,dyn_cmp_dict,dyn_arith_dict,chrono-tz --all-targets -- -D warnings
run: cargo clippy -p arrow --features=prettyprint,csv,ipc,test_utils,ffi,ipc_compression,dyn_cmp_dict,dyn_arith_dict,chrono-tz --all-targets -- -D warnings
14 changes: 2 additions & 12 deletions .github/workflows/arrow_flight.yml
View file
Open in desktop
Expand Up @@ -43,18 +43,12 @@ jobs:
runs-on: ubuntu-latest
container:
image: amd64/rust
env:
# Disable full debug symbol generation to speed up CI build and keep memory down
# "1" means line tables only, which is useful for panic tracebacks.
RUSTFLAGS: "-C debuginfo=1"
steps:
- uses: actions/checkout@v3
with:
submodules: true
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: stable
- name: Test
run: |
cargo test -p arrow-flight
Expand All @@ -73,11 +67,7 @@ jobs:
- uses: actions/checkout@v3
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: stable
- name: Setup Clippy
run: |
rustup component add clippy
run: rustup component add clippy
- name: Run clippy
run: |
cargo clippy -p arrow-flight --all-features -- -D warnings
run: cargo clippy -p arrow-flight --all-features -- -D warnings
3 changes: 1 addition & 2 deletions .github/workflows/docs.yml
View file
Open in desktop
Expand Up @@ -51,5 +51,4 @@ jobs:
with:
rust-version: ${{ matrix.rust }}
- name: Run cargo doc
run: |
cargo doc --document-private-items --no-deps --workspace --all-features
run: cargo doc --document-private-items --no-deps --workspace --all-features
3 changes: 1 addition & 2 deletions .github/workflows/miri.yaml
View file
Open in desktop
Expand Up @@ -49,5 +49,4 @@ jobs:
env:
RUST_BACKTRACE: full
RUST_LOG: "trace"
run: |
bash .github/workflows/miri.sh
run: bash .github/workflows/miri.sh
34 changes: 24 additions & 10 deletions .github/workflows/object_store.yml
View file
Open in desktop
Expand Up @@ -37,11 +37,10 @@ jobs:
image: amd64/rust
steps:
- uses: actions/checkout@v3
- name: Setup Rust toolchain with clippy
run: |
rustup toolchain install stable
rustup default stable
rustup component add clippy
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
- name: Setup Clippy
run: rustup component add clippy
# Run different tests for the library on its own as well as
# all targets to ensure that it still works in the absence of
# features that might be enabled by dev-dependencies of other
Expand Down Expand Up @@ -71,8 +70,6 @@ jobs:
# Disable full debug symbol generation to speed up CI build and keep memory down
# "1" means line tables only, which is useful for panic tracebacks.
RUSTFLAGS: "-C debuginfo=1"
# https://github.com/rust-lang/cargo/issues/10280
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a workaround for CircleCI that survived the import from to arrow-rs and can be removed

RUST_BACKTRACE: "1"
# Run integration tests
TEST_INTEGRATION: 1
Expand Down Expand Up @@ -121,6 +118,23 @@ jobs:
OBJECT_STORE_AWS_ACCESS_KEY_ID: test
OBJECT_STORE_AWS_SECRET_ACCESS_KEY: test
OBJECT_STORE_AWS_ENDPOINT: http://localhost:4566
run: |
# run tests
cargo test -p object_store --features=aws,azure,gcp
run: cargo test -p object_store --features=aws,azure,gcp

# test the object_store crate builds against wasm32 in stable rust
wasm32-build:
name: Build wasm32
runs-on: ubuntu-latest
container:
image: amd64/rust
steps:
- uses: actions/checkout@v3
with:
submodules: true
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
target: wasm32-unknown-unknown,wasm32-wasi
- name: Build wasm32-unknown-unknown
run: cargo build -p object_store --target wasm32-unknown-unknown
- name: Build wasm32-wasi
run: cargo build -p object_store --target wasm32-wasi