New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Validate ArrayData type when converting to Array (#2834) #2835
Changes from all commits
21e6dec
102f86f
2fa08ee
7d73899
e900fad
a6d9618
a2d9865
c600114
d4266e6
4f1069b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -26,7 +26,8 @@ use crate::errors::{ParquetError, Result}; | |
use crate::schema::types::ColumnDescPtr; | ||
use arrow::array::{ | ||
ArrayDataBuilder, ArrayRef, BooleanArray, BooleanBufferBuilder, Decimal128Array, | ||
Float32Array, Float64Array, Int32Array, Int64Array,TimestampNanosecondArray, TimestampNanosecondBufferBuilder, | ||
Float32Array, Float64Array, Int32Array, Int64Array, TimestampNanosecondArray, | ||
TimestampNanosecondBufferBuilder, UInt32Array, UInt64Array, | ||
}; | ||
use arrow::buffer::Buffer; | ||
use arrow::datatypes::{DataType as ArrowType, TimeUnit}; | ||
|
@@ -169,15 +170,21 @@ where | |
.null_bit_buffer(self.record_reader.consume_bitmap_buffer()); | ||
|
||
let array_data = unsafe { array_data.build_unchecked() }; | ||
let array = match T::get_physical_type() { | ||
PhysicalType::BOOLEAN => Arc::new(BooleanArray::from(array_data)) as ArrayRef, | ||
PhysicalType::INT32 => Arc::new(Int32Array::from(array_data)) as ArrayRef, | ||
PhysicalType::INT64 => Arc::new(Int64Array::from(array_data)) as ArrayRef, | ||
PhysicalType::FLOAT => Arc::new(Float32Array::from(array_data)) as ArrayRef, | ||
PhysicalType::DOUBLE => Arc::new(Float64Array::from(array_data)) as ArrayRef, | ||
PhysicalType::INT96 => { | ||
Arc::new(TimestampNanosecondArray::from(array_data)) as ArrayRef | ||
} | ||
let array: ArrayRef = match T::get_physical_type() { | ||
PhysicalType::BOOLEAN => Arc::new(BooleanArray::from(array_data)), | ||
PhysicalType::INT32 => match array_data.data_type() { | ||
ArrowType::UInt32 => Arc::new(UInt32Array::from(array_data)), | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is this a bug fix? Seems like a good improvement to me There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sort of, it doesn't actually matter as the PrimitiveArray is discarded by the parent StructArrayReader, which is only concerned with the ArrayData portion There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess I was wondering "what is the rationale for this change" There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, because now it would panic complaining that it was creating an Int32Array from DataType::UInt32 ArrayData. |
||
ArrowType::Int32 => Arc::new(Int32Array::from(array_data)), | ||
_ => unreachable!(), | ||
}, | ||
PhysicalType::INT64 => match array_data.data_type() { | ||
ArrowType::UInt64 => Arc::new(UInt64Array::from(array_data)), | ||
ArrowType::Int64 => Arc::new(Int64Array::from(array_data)), | ||
_ => unreachable!(), | ||
}, | ||
PhysicalType::FLOAT => Arc::new(Float32Array::from(array_data)), | ||
PhysicalType::DOUBLE => Arc::new(Float64Array::from(array_data)), | ||
PhysicalType::INT96 => Arc::new(TimestampNanosecondArray::from(array_data)), | ||
PhysicalType::BYTE_ARRAY | PhysicalType::FIXED_LEN_BYTE_ARRAY => { | ||
unreachable!( | ||
"PrimitiveArrayReaders don't support complex physical types" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍