Writing records row by row

[ishitatsuyuki]

I'm currently working on a profiler that uses Parquet for the on-disk profile format. We don't do much of columnar processing but would like to take advantage of the compression and efficiency benefits.

As such, the records come in a row-by-row manner, but the arrow2 crate's writer only takes an entire column in a row-group. It should be technically possible to serialize things on a tighter manner, e.g. for delta bitpacked, you can serialize as soon as you have buffered a minibatch-worth of data. Doing so allows the in-memory buffer to have a smaller footprint.

As such, it might be a worthy addition to the API to be able to write things one-by-one.

[jorgecarleitao]

Hey @ishitatsuyuki and thanks for the issue. I converted it to a discussion for now since it seems a bit broad to be actionable (yet).

Maybe the primary question I have here: do you need to go through Arrow? It seems to me that you can skip arrow entirely and work directly with parquet.

I am asking because it seems to me that you would like to write to parquet on a streaming fashion and thus having "arrow arrays" does not seem beneficial.

Thinking through this angle,

• is there a subset of parquet types that you would use or is it everything?
• do you need to support nested types (struct, or array of lists)?

[ishitatsuyuki]

Maybe the primary question I have here: do you need to go through Arrow? It seems to me that you can skip arrow entirely and work directly with parquet.

I think conceptually I should be dealing with the parquet library directly, however in the case of parquet2 it's so "raw" (in being untyped and others) to be used directly, and therefore I'm feeling I have to use the arrow2 crate instead.

is there a subset of parquet types that you would use or is it everything? do you need to support nested types (struct, or array of lists)?

The plan right now is to use only the basic types, mostly integers, strings (but with a custom dictionary impl described later), and some convenience types like timestamps. I'm rolling a custom string dictionary because for exporting purposes I want the strings to be in a single large table, instead of being in the header of each row group.

EDIT: The dictionary would just be another parquet file with a single string column.

So for the second question, it doesn't involve struct or lists.

[jorgecarleitao]

Got it. how would you think the API look like from arrow2's point of view?

[deleted]

@ishitatsuyuki I'm curious if you ever solved this problem.

I have an application where I need to create arrow chunks row-by-row because I'm streaming data from a row-major source (a database table) and want to output an arrow IPC HTTP response. Records come out of the database in a RowStream, and need to be transposed into arrow chunks.

It seems like to accomplish this, I'm going to need to make some kind of mutable chunk-builder struct that holds a schema and a vec of vecs, and can emit an arrow2 Chunk when the program decides it's accumulated enough data rows.

I think the arrow2 crate could benefit from having something like this, though I'm curious how many other users are doing this kind of transposition operations. If I come up with a solution that works, I'll try to remember to post it here.