ARROW-11417: Add integration files for buffer compression

[pitrou]

No description provided.

[pitrou]

I'm not sure what the naming scheme should be. I chose "1.0.0" for the format version, but is it the Arrow library version that generated the files instead? (in which case it should be "2.0.0")
@wesm

[pitrou]

The JSON files are refused by the Java Arrow integration test:

Incompatible files
Expected END_OBJECT but got FIELD_NAME
16:30:12.557 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalStateException: Expected END_OBJECT but got FIELD_NAME
	at org.apache.arrow.vector.ipc.JsonFileReader.readToken(JsonFileReader.java:779)
	at org.apache.arrow.vector.ipc.JsonFileReader.readFromJsonIntoVector(JsonFileReader.java:740)
	at org.apache.arrow.vector.ipc.JsonFileReader.read(JsonFileReader.java:219)
	at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:203)
	at org.apache.arrow.tools.Integration.run(Integration.java:118)
	at org.apache.arrow.tools.Integration.main(Integration.java:69)
	Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (16648)
Allocator(ROOT) 0/16648/49416/2147483647 (res/actual/peak/limit)

		at org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:437)
		at org.apache.arrow.memory.RootAllocator.close(RootAllocator.java:29)
		at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:185)
		... 2 common frames omitted

It looks like the problem is that it contains an empty "children" field for primitive batches, and that displeases the Java reader (which for some unknown reason uses a low-level token-by-token parsing technique).

[pitrou]

Ok, I edited the JSON files by hand to remove the offending field and now Java manages to read them (cringe), though it fails to instantiate the compression:

java.lang.IllegalArgumentException: Compression type not supported: 0
	at org.apache.arrow.vector.compression.CompressionUtil.createCodec(CompressionUtil.java:57)
	at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:59)
	at org.apache.arrow.vector.ipc.ArrowReader.loadRecordBatch(ArrowReader.java:205)
	at org.apache.arrow.vector.ipc.ArrowFileReader.loadNextBatch(ArrowFileReader.java:153)
	at org.apache.arrow.vector.ipc.ArrowFileReader.loadRecordBatch(ArrowFileReader.java:184)
	at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:205)
	at org.apache.arrow.tools.Integration.run(Integration.java:118)
	at org.apache.arrow.tools.Integration.main(Integration.java:69)

[emkornfield]

Strange, I wonder why main integration tests aren't broken for the JSON issue. Regarding versioning, I used the library version that generated them. It seems like a bug that Java can't even read uncompressed buffers? Or maybe the java error message is bad. Either way something for Java contributors to figure out. Thanks for adding these.

[emkornfield]

LGTM assuming rename to use library version that generated them.

[pitrou]

Strange, I wonder why main integration tests aren't broken for the JSON issue.

Because the JSON files that are used in the integration tests are generated by the Python datagen harness (in archery.integration).
The ARROW_TO_JSON mode of operation of native integration runners isn't used in the integration tests (as opposed to the JSON_TO_ARROW and VALIDATE mdoes).

[pitrou]

I renamed to "2.0.0-compression", will merge.

[liukun4515]

@pitrou Hi, how do you generated the compression file?
I'm doing compression work in rust version apache/arrow-rs#1855
But the test failed when reading the 2.0.0-compression` test file

[pitrou]

@liukun4515 The compression file was generated using Arrow C++ IIRC.