New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
optimize: no validation for decimal array for cast/take #2551
Conversation
The performance of casting decimal has been improve significantly for the decimal128 to decimal128 about 30%, when the situation meet the no validation requirement. |
The performance changes of take operation |
.collect::<Result<Decimal128Array>>()?; | ||
|
||
// specify the precision/scale without the validation | ||
unsafe { | ||
array.with_precision_and_scale_without_validation( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is something extremely bizarre about this API, we are collecting into a Decimal128Array
which to be consistent should be performing validation based on the default decimal precision/scale, and we are then converting it into a different array. I can't help feeling we should be consistently either validating or not validating, at the moment we are basically doing a random combination
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
collecting the i128 data to construct the DecimalArray128, we use the default precision which is MAX precision(38), so all the element will not be overflow.
The take operation will don't change the data and precision, so we can skip the validation, If we use the strict validation mode.
As we don't consistently validate the contents of Decimal arrays, we can't meaningfully reason about if a given operation will overflow, and so I don't really know how to review this or what this is even really doing... I'll try to drive some consensus forward on #2387 |
I believe this will be superseded by #2637 so marking as draft |
#2857 removes decimal validation |
Which issue does this PR close?
part of #2313
From the closed pr #2357
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?