Added support for JSON ser/de records layout

[AnIrishDuck]

Prior discussion in #1178

Serialization was easy: individual streaming iterators already produce a record at a time, so we just need to salami slice them to transpose the results.

Deserialization was more complex:

• Define Preallocate for arrays that have a backing that can be opportunistically resized.
  • Add expand to MutableListArray. This is the workhorse that makes later arbitrarily-nested deserialization possible.
  • Declare an impl MutableArray for Box<dyn MutableArray>. This is mostly trivial delegation, so it may be worth investigating the delegate crate for boilerplate reduction. We need this to support arbitrary recursion at the type level. This allows us to define a MutableListArray<O, Box<dyn MutableArray>>.
  • Finally, implement schema inference and deserialization. Here be dragons:
    • Schema inference basically recurses down to the first element for each list, and assumes that element and array shape for all future records. There are obvious pitfalls here.
    • For deserialization, we want to avoid an expensive transpose. So we instead allocate mutable arrays for each record in the schema, and expand any nested list arrays recursively after filling the recursive sub-level. This requires converting a bunch of deserialize_xxx methods into deserialize_xxx_into methods that extend an existing array instead of returning a new array. We can generally use adapters like fill_array_from to avoid code duplication in the original case where we just need a plain list returned.

It's worth noting that this approach may be useful in general to avoid some of the allocations that the json deserializer currently performs (creating Vec of row references).

[codecov]

Codecov Report

Base: 83.04% // Head: 83.13% // Increases project coverage by +0.08% 🎉

Coverage data is based on head (06757db) compared to base (27e109d).
Patch coverage: 77.59% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1275      +/-   ##
==========================================
+ Coverage   83.04%   83.13%   +0.08%     
==========================================
  Files         363      363              
  Lines       38442    38884     +442     
==========================================
+ Hits        31926    32326     +400     
- Misses       6516     6558      +42     

Impacted Files	Coverage Δ
src/array/mod.rs	64.93% <50.00%> (-3.62%)	⬇️
src/io/json/read/infer_schema.rs	93.15% <74.28%> (-2.33%)	⬇️
src/io/json/read/deserialize.rs	74.52% <74.35%> (+1.96%)	⬆️
src/array/list/mutable.rs	76.14% <86.04%> (+3.98%)	⬆️
src/io/json/write/serialize.rs	92.33% <88.88%> (-0.40%)	⬇️
src/array/fixed_size_list/mutable.rs	67.93% <100.00%> (+18.72%)	⬆️
src/io/json/write/mod.rs	98.59% <100.00%> (+5.73%)	⬆️
src/compute/cast/mod.rs	93.54% <0.00%> (-0.51%)	⬇️
src/io/ipc/read/schema.rs	95.29% <0.00%> (-0.30%)	⬇️
src/io/ipc/read/file.rs	96.87% <0.00%> (+0.44%)	⬆️
... and 10 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[AnIrishDuck]

Thoughts on using match-downcast to get rid of the if-else ugliness? Clippy just reminded me that it is indeed, quite gross.

[ritchie46]

Thoughts on using match-downcast to get rid of the if-else ugliness? Clippy just reminded me that it is indeed, quite gross.

I don't think that's worth another crate pulled in.

[jorgecarleitao]

This looks ready great

My only major comment is to keep things private - i.e. keep all changes needed for the json private to this crate, since they are only used in the context of the crate.

Great PR!!

[jorgecarleitao]

I suspect that this is because line 144 uses false on the nullability of the inner field, but the result is a nullability true

[AnIrishDuck]

Yup, this was it. Kinda an interesting philosophical question on whether they are actually equal... value-wise, they are identical, even though one could hold nullable values. While I'm not 100% sure I agree with the conclusion, I understand and respect the reasoning.

[AnIrishDuck]

The last thought I'll leave here is that the error when they are not equal is pretty confusing:

thread 'io::json::read::read_json_records' panicked at 'assertion failed: `(left == right)`
  left: `ListArray[[[1.1, 2, 3], [2, 3], [4, 5, 6]], [[3, 2, 1], [3, 2], [6, 5, 4]]]`,
 right: `ListArray[[[1.1, 2, 3], [2, 3], [4, 5, 6]], [[3, 2, 1], [3, 2], [6, 5, 4]]]`', tests/it/io/json/read.rs:114:9

Not sure if the solution is to somehow mark that one is nullable and one is not.

[jorgecarleitao]

I agree - I was also bitten by this many times. The problem challenge is that the ListArray can become pretty complex if we expose the inner field in the debug :(

[AnIrishDuck]

Yup, that makes sense. Especially with deeply nested types.

[jorgecarleitao]

Thank you so much! Awesome feature and PR! 🙇