Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to arrow 16.0.0 #2718

Merged
merged 4 commits into from Jun 14, 2022
Merged

Upgrade to arrow 16.0.0 #2718

merged 4 commits into from Jun 14, 2022

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Jun 10, 2022

Draft until arrow 16.0.0 is released

re: apache/arrow-rs#1809

This PR validates that DataFusion can be upgraded to the latest arrow

Closes #2732
Closes #2731
Closes #2733

@github-actions github-actions bot added the core Core datafusion crate label Jun 10, 2022
expected_error
),
}
let e = execution_plan.unwrap_err().to_string();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great -- arrow 16.0.0 contains support for casting Utf8 to bool (apache/arrow-rs#1738 - Thanks @MazterQyou) -- so I updated the test to try and cast Utf8 to some struct.

I also took the opportunity to clean up the test in general

Expr::Literal(ScalarValue::Utf8(Some("a".to_string()))),
Expr::Literal(ScalarValue::Int64(Some(1))),
];
let list = vec![lit("a"), lit(1i64)];
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used more concise construction of ScalarValues

@github-actions github-actions bot added logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Physical Expressions sql labels Jun 13, 2022
@alamb alamb changed the title WIP: Upgrade to arrow 16.0.0 Upgrade to arrow 16.0.0 Jun 13, 2022

Ok(())
}

/// Return a `null` literal representing a struct type like: `{ a: bool }`
fn struct_literal() -> Expr {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Casting from Utf8 to Struct is still not supported

@alamb alamb marked this pull request as ready for review June 14, 2022 12:32
@andygrove andygrove merged commit 5096ff4 into apache:master Jun 14, 2022
@alamb alamb deleted the alamb/df_upgarde branch August 8, 2023 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core datafusion crate logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Physical Expressions sql
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants