Move JSON Test Format To integration-testing #2724
Conversation
|
Benchmark runs are scheduled for baseline = 4f52a25 and contender = 5146663. 5146663 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
There was a problem hiding this comment.
Looks good. The json format should be for integration test purpose. Although I think it may be possible like @alamb said that it is used for other purpose as it is public now, it is out of the design of the format and we can do extraction later if it is really needed.
|
I was using the JSON format for schemas in https://github.com/domoritz/csv2parquet. Could you provide a crate for reading/writing schemas from JSON? I don't need the same format. |
|
@domoritz The schema types implement serde::Serialize and Deserialize, and can therefore be used directly with serde_json. Would that be workable? |
|
That should probably work. Do you have a code snippet I could look at? |
|
Perfect. That works! |
|
@alamb was right, someone (my company) was using this code, and now we're broken when trying to move beyond arrow 22; specifically we are dependent on `Schema::from::<serde_json::Value> |
The actual 13.0.0 DF release uses Arrow 24.0.0, but we need to pick up 25.0.0, since it brings back the Arrow Schema/Field-to-JSON serialization code (albeit in a different crate for integration tests). apache/arrow-rs#2868 apache/arrow-rs#2724
The actual 13.0.0 DF release uses Arrow 24.0.0, but we need to pick up 25.0.0, since it brings back the Arrow Schema/Field-to-JSON serialization code (albeit in a different crate for integration tests). apache/arrow-rs#2868 apache/arrow-rs#2724
Which issue does this PR close?
Part of #2594
Part of #2300
Related to #2723
Rationale for this change
Collocates the code related to the integration tests, reducing coupling to serde_json, potentially improves compilation times.
What changes are included in this PR?
Follow on to #2598, this moves the JSON schema parsing logic to integration-testing. I had incorrectly assumed this was a canonical representation, however, it is a custom format specific to these tests - https://github.com/apache/arrow/blob/master/docs/source/format/Integration.rst#json-test-data-format. I therefore think it can be safely moved to live alongside the other logic.
Are there any user-facing changes?
Yes, these APIs were public, but I'm fairly confident they weren't in use, especially as the logic to actually parse the files themselves along with the array data, is not public.