Support DurationType in cudf parquet reader via arrow:schema

[mhaseeb123]

Description

This PR adds the support for reading and using the arrow:schema struct from the serialized arrow:ipc message written at the key-value metadata section of the Parquet file with ARROW:schema key. This allows cudf to read and interop with arrow for non-standard parquet types (DurationType in this PR).

Arrow uses Google flatbuffers (inside Schema.fbs) to serialize the arrow:Schema structure (containing column descriptors) and puts it (padded for 8 byte alignment) into the header of an empty ipc:Message (also a flatbuffer-serialized structure inside Message.fbs). The ipc:Message is prepended with two integers containing a validity message and the size of the header (the arrow:Schema + padding). The final message is endoded as a base64 string and written to Parquet file footer key-value metadata using "ARROW:schema" key.

In this PR, we base64-decode the ipc:Message, then we decode the validity message and the header size, and offset pointers to the arrow:Schema flatbuffer. We then use Flatbuffer structs to walk the arrow:Schema and collect information on columns of interest as an unordered_map (using column name as key). This unordered_map is used inside select_columns function to build cudf Table columns and get the correct dtype.

Closes #13410

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[mhaseeb123]

CC: @GregoryKimball @vuule

[etseidl]

Some early comments. That's a lot of code for one data type 😄. Do we want to include all the google code, or add a library dependency? Or follow the thrift/protobuf model and roll our own parser?

Have you an idea of how long it takes to parse the schema? I'm wondering if it's better to make it optional (like add a use_arrow_schema reader option).

[etseidl]

Github ate some of my comments ☹️ A note about the column naming.

[mhaseeb123]

Some early comments. That's a lot of code for one data type 😄. Do we want to include all the google code, or add a library dependency? Or follow the thrift/protobuf model and roll our own parser?

Have you an idea of how long it takes to parse the schema? I'm wondering if it's better to make it optional (like add a use_arrow_schema reader option).

Thank you for looking at this @etseidl. I am also thinking of making it optional as well in the updates I am working on. I am also thinking of removing all the pushed flatbuffer code and adding it as dependency instead.

[mhaseeb123]

Thank you for the helpful input @etseidl and @galipremsagar. I am going to remove the use_arrow_schema from the public Python APIs as I also agree keeping our args consistent with other readers. I will keep a default=True use_arrow_schema option in the Cython code in case we need to expose it at a later point.

[galipremsagar]

Thank you for the helpful input @etseidl and @galipremsagar. I am going to remove the use_arrow_schema from the public Python APIs as I also agree keeping our args consistent with other readers. I will keep a default=True use_arrow_schema option in the Cython code in case we need to expose it at a later point.

Sounds good to me 👍 Thanks for helping me understand @etseidl !

[mhaseeb123]

@mroeschke

Thank you for the helpful input @etseidl and @galipremsagar. I am going to remove the use_arrow_schema from the public Python APIs as I also agree keeping our args consistent with other readers. I will keep a default=True use_arrow_schema option in the Cython code in case we need to expose it at a later point.

@mroeschke Apologies for the back and forth but since we have decided to remove the option from Python side altogether (above discussion) , I am going to remove the 2nd part of the tests (which was using the assert_neq) altogether.

[galipremsagar]

We can just do this.

[galipremsagar]

/okay to test

[mhaseeb123]

/ok to test

[galipremsagar]

/okay to test

[mhaseeb123]

/ok to test

[mhaseeb123]

/merge