Add full data validation for ArrayData::try_new()

[alamb]

Which issue does this PR close?

Closes #817

Rationale for this change

Improve the security arrow-rs by validating untrusted user input, and resolves several outstanding potential security issues by doing deep checks of the data passed to ArrayData::try_new()

What changes are included in this PR?

1. Add full validation to ArrayData::try_new() based on the checks in the C++ implementation, kindly pointed out (and contributed) by @pitrou

Are there any user-facing changes?

• ArrayData::try_new() may return an Err now on some (invalid) inputs rather than succeeding and will take more time.
• There is a new pub ArrayData::validate_full() that performs full ArrayData validation

Testing

• All the arrow-rs tests pass (other than some different error messages) when I temporarily changed the code to call validate_full() always as in (DEMO) tests pass when validate_full is forced #987 (review)
• All the datafusion tests pass with these changes (using https://github.com/alamb/arrow-rs/tree/alamb/full_array_data_construction_validation_df which has the changes in this PR cherry-picked)

Follow On:

• Add full validation logic for Union in Update Union Array to add UnionMode, match latest Arrow Spec, and rename new -> unsafe new_unchecked() #885 (not backwards compatible)

[codecov-commenter]

Codecov Report

Merging #921 (2e6f3af) into master (9fb2a5f) will increase coverage by 0.01%.
The diff coverage is 86.78%.

❗ Current head 2e6f3af differs from pull request most recent head 4436472. Consider uploading reports for the commit 4436472 to get more accurate results

@@            Coverage Diff             @@
##           master     #921      +/-   ##
==========================================
+ Coverage   82.29%   82.31%   +0.01%     
==========================================
  Files         168      168              
  Lines       48761    49031     +270     
==========================================
+ Hits        40129    40358     +229     
- Misses       8632     8673      +41     

Impacted Files	Coverage Δ
arrow/src/array/data.rs	81.16% <85.60%> (+1.37%)	⬆️
arrow/src/array/array_binary.rs	93.55% <100.00%> (+0.13%)	⬆️
arrow/src/array/array_dictionary.rs	88.75% <100.00%> (+0.28%)	⬆️
arrow/src/array/array_string.rs	97.08% <100.00%> (-0.83%)	⬇️
parquet_derive/src/parquet_field.rs	65.98% <0.00%> (-0.46%)	⬇️
arrow/src/array/transform/mod.rs	85.10% <0.00%> (-0.14%)	⬇️
parquet/src/encodings/encoding.rs	93.71% <0.00%> (+0.19%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fb2a5f...4436472. Read the comment docs.

[alamb]

Update: I have a bunch of tests written, so that is good 🎉

I think there is some more testing needed here as when I tried to apply validate_full to every ArrayData construction I found some corner cases (like what if an offset array is marked as null? the spec I think says this is ok, but I think the C++ errors, which seems sensible to me)

[alamb]

This PR (that has full ArrayData validate) is now ready for review . I hope it allows us to close the currently open RUSTSEC entries for arrow-rs: https://github.com/rustsec/advisory-db/tree/main/crates/arrow

cc @jhorstmann @bjchambers @Jimexist @jorgecarleitao

I know it is a large ask to review this PR, so thank you if you have time to do so.

As for testing, if you have a project that uses arrow 6.x.0 perhaps you would be willing to test it using the method in apache/datafusion#1386 which would be very helpful.

[alamb]

Thanks again everyone for the help. I'll backport this PR and hopefully we can get it shipped in 6.4.0