Optionally require alignment when reading IPC, respect alignment when writing

[hzuo]

Which issue does this PR close?

Closes #5553

Rationale for this change

FileWriter and StreamWriter should ensure that the data is written with appropriate alignment such that arrays can be used without copying to a more-aligned buffer.

In particular, as of Rust 1.77.0 and LLVM 18, i128 now has a 16-byte alignment requirement even on x86 (ARM always had this requirement), i.e. std::mem::align_of::<i128> == 16. So Decimal128Arrays must be aligned to a 16-byte boundary when serialized into an IPC buffer. The pad_to_8 used everywhere in the IPC code causes it to pad insufficiently.

This prevents readers of the IPC data generated by this crate from doing true zero-copy reads (e.g. mmapping) since the data may be insufficiently aligned. See #5165, #5153.

On some platforms, SIMD may also be significantly slower if the beginning of the IPC block isn't aligned to a 16-, 32-, or 64- byte boundary (as discussed in the Arrow spec document).

(copied from issue)

What changes are included in this PR?

• Make all the IPC code actually respect the alignment specified in IpcWriteOptions
• Add new require_alignment option to FileDecoder, read_record_batch, and read_dictionary, which enforces that the read is a zero-copy read. Previously if the buffer was unaligned then the code automatically allocates an aligned buffer and performs a memcpy (build_aligned => align_buffers), which users of Arrow as a zero-copy format may not expect / may want to enforce does not happen.

Are there any user-facing changes?

Yes:

• alignment in IpcWriteOptions::try_with cannot be larger than 64 bytes - previously unbounded (breaking but minor)
• New require_alignment argument on read_record_batch and read_dictionary (breaking but minor)
• New option with_require_alignment to FileDecoder (non-breaking)

[hzuo]

This test is equivalent to the current behavior without the changes in this PR, it's equivalent to using pad_to_8 everywhere.

Because the Decimal128Array is insufficiently aligned, this prevents readers from doing a true zero-copy read (e.g. mmapping), because they need to copy the block into a more-aligned buffer.

[hzuo]

previously a Vec was being allocated every time padding was needed

[tustvold]

I'm travelling for the next few days, but will try to get to this on my return if nobody else gets there first

[alamb]

Thank you very much for this contribution @Huzo 🙏

I think it looks great overall and makes sense. I had a few small comments but I also think this PR could be merged as is.

It would probably be good to get someone else like @tustvold t give it a review prior to merge, but all in all very nice

[alamb]

What do you think about calling this concept / field require_alignment which describes what is required (rather than how it is enforced)?

It might also help make the intent clearer to users (that the IPC decoder will error if the buffers are not aligned properly)

I don't feel super strongly about this, but I was a little confused about what enforce_zero_copy meant until I found this comment

[hzuo]

Yeah I agree, will make this change.

My original thinking was that there might be a variety of reasons why IPC decoders may need to create a copy of the input data, and this would be a catch-all so that the users don't have to understand the intricacies of why or why not copies need to be made.

But agree that it ends up being kinda vague, and in the future if there are other reasons copies need to be made, probably better to expose each of the "reasons" as a granular "requirement" - if there ends up being a ton, we can expose a catch-all, but no reason to anticipate that today at the expense of clarity.

[alamb]

❤️
Thanks again @hzuo

[alamb]

so close!

[tustvold]

This looks good, my only concern is the breaking API change. We have had some community feedback requesting we reduce the frequency of these - #5368