Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always validate the array data when creating array in IPC reader #2541

Closed
HaoYang670 opened this issue Aug 21, 2022 · 1 comment · Fixed by #2547
Closed

Always validate the array data when creating array in IPC reader #2541

HaoYang670 opened this issue Aug 21, 2022 · 1 comment · Fixed by #2547
Labels
arrow Changes to the arrow crate bug

Comments

@HaoYang670
Copy link
Contributor

IMO we should probably be validating all of them, I'm not sure why we aren't validating some of them... IPC is inherently not trusted

Originally posted by @tustvold in #2525 (comment)

@HaoYang670
Copy link
Contributor Author

HaoYang670 commented Aug 22, 2022

After replacing build_unchecked with build().unwrap() in create_primitive_array in the ipc reader, 5 tests fail:

failures:

---- ipc::reader::tests::read_generated_streams_014 stdout ----
thread 'ipc::reader::tests::read_generated_streams_014' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("-11697 is too small to store in a Decimal128 of precision 3. Min is -999")', arrow/src/ipc/reader.rs:497:18

---- ipc::reader::tests::read_generated_files_014 stdout ----
thread 'ipc::reader::tests::read_generated_files_014' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("-11697 is too small to store in a Decimal128 of precision 3. Min is -999")', arrow/src/ipc/reader.rs:497:18

---- json::reader::tests::test_row_type_validation stdout ----
thread 'json::reader::tests::test_row_type_validation' panicked at 'assertion failed: `(left == right)`
  left: `"Json error: Expected JSON record to be an object, found Array([Number(1), String(\"hello\")])"`,
 right: `"Json error: Expected JSON record to be an object, found Array [Number(1), String(\"hello\")]"`', arrow/src/json/reader.rs:2624:9

---- ipc::writer::tests::read_and_rewrite_generated_streams_014 stdout ----
thread 'ipc::writer::tests::read_and_rewrite_generated_streams_014' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("-11697 is too small to store in a Decimal128 of precision 3. Min is -999")', arrow/src/ipc/reader.rs:497:18

---- ipc::writer::tests::read_and_rewrite_generated_files_014 stdout ----
thread 'ipc::writer::tests::read_and_rewrite_generated_files_014' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("-11697 is too small to store in a Decimal128 of precision 3. Min is -999")', arrow/src/ipc/reader.rs:497:18

@tustvold @viirya Any idea about these?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate bug
Projects
None yet
2 participants