Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optionally require alignment when reading IPC, respect alignment when writing #5554

Merged
merged 14 commits into from
Apr 4, 2024
8 changes: 7 additions & 1 deletion arrow-buffer/src/buffer/scalar.rs
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,15 @@ impl<T: ArrowNativeType> ScalarBuffer<T> {
/// This method will panic if
///
/// * `offset` or `len` would result in overflow
/// * `buffer` is not aligned to a multiple of `std::mem::size_of::<T>`
/// * `buffer` is not aligned to a multiple of `std::mem::align_of::<T>`
/// * `bytes` is not large enough for the requested slice
pub fn new(buffer: Buffer, offset: usize, len: usize) -> Self {
assert_eq!(
buffer.as_ptr().align_offset(std::mem::align_of::<T>()),
0,
"buffer is not aligned"
);
hzuo marked this conversation as resolved.
Show resolved Hide resolved

let size = std::mem::size_of::<T>();
let byte_offset = offset.checked_mul(size).expect("offset overflow");
let byte_len = len.checked_mul(size).expect("length overflow");
Expand Down
1 change: 1 addition & 0 deletions arrow-flight/src/decode.rs
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@ impl FlightDataDecoder {
&state.schema,
&mut state.dictionaries_by_field,
&message.version(),
false,
)
.map_err(|e| {
FlightError::DecodeError(format!("Error decoding ipc dictionary: {e}"))
Expand Down
1 change: 1 addition & 0 deletions arrow-flight/src/sql/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -590,6 +590,7 @@ pub fn arrow_data_from_flight_data(
&dictionaries_by_field,
None,
&ipc_message.version(),
false,
)?;
Ok(ArrowFlightData::RecordBatch(record_batch))
}
Expand Down
1 change: 1 addition & 0 deletions arrow-flight/src/utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ pub fn flight_data_to_arrow_batch(
dictionaries_by_id,
None,
&message.version(),
false,
)
})?
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,7 @@ async fn receive_batch_flight_data(
&schema,
dictionaries_by_id,
&message.version(),
false,
)
.expect("Error reading dictionary");

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@ async fn record_batch_from_message(
dictionaries_by_id,
None,
&message.version(),
false,
hzuo marked this conversation as resolved.
Show resolved Hide resolved
);

arrow_batch_result
Expand All @@ -330,6 +331,7 @@ async fn dictionary_from_message(
&schema_ref,
dictionaries_by_id,
&message.version(),
false,
);
dictionary_batch_result
.map_err(|e| Status::internal(format!("Could not convert to Dictionary: {e:?}")))
Expand Down