Iterators on non-optional elements

[ncpenke]

Hey @jorgecarleitao, for arrow2_convert, we're considering an API simplification which would be possible if we can have iterators on non-optional elements. If an arrow2::array has no optional elements and doesn't need a valid bitmask, then the array would map straightforwardly to the rust equivalent of Vec<T> (as opposed to Vec<Option<T>>).

I was wondering if you think there are any downsides to non-optional iterators in general, and if not whether that abstraction would belong in arrow2 or arrow2_convert. Thanks!

[aldanor]

Correct me if I'm wrong, but (especially within Rust!), I would assume that the majority of use cases would consist of dealing with non-nullable data.

This way, would it perhaps make sense to have two APIs for iteration? (just thinking out loud)

• Iterate over T. Fallible operation - but fallible only at the start of the operation and not on each element. If there's a bitmask, fail right away when initializing the operation.
• Iterate over Option<T>. Always works, non-fallible, but messier to work with (and in some cases, slower - when dealing with nested containers and such) and in some edge cases may require unwraps.

Not sure how to wrap this nicely, but (ignoring potential API breakage for a second etc, just hypothesizing), it could even be something like

.iter()?
.iter_nullable()

What's not so nice is the fallability of instantiating non-optional iteration because you don't know in advance if there's a bitmask or not? But don't think there's ways around it.

[jorgecarleitao]

Most arrays have non-optional iterators, iter_values. They ignore the bitmap altogether so they need to be used when the bitmap is guaranteed to be empty or their usage is semantically incorrect.

[ncpenke]

Thanks @jorgecarleitao that's perfect, had missed that. Great addition!

[ncpenke]

A follow up question - if we explicitly check that the validity() is None, then that should be sufficient to use iter_values, correct?

[jorgecarleitao]

correct :)

[ncpenke]

@jorgecarleitao I'm using values_iter() and ran into an issue for FixedSizeBinaryArray. Looks like the iterator type for FixedSizeBinaryArray which is FixedSizeBinaryValuesIter is intentionally private.

I can think of two possible solutions, but both need changes to arrow2:

1. Make FixedSizeBinaryValuesIter public
2. Introduce a IntoValueIter (or similarly named trait), that's the analog of IntoIterator but on the values. This will allow the underlying iterator type to be accessed and referenced, but without making the type public.

Do either of these sound reasonable? Please lmk if you have any other ideas.

[jorgecarleitao]

I agree. For now a solution is to use array.values().chunks_exact(array.size()), FixedSizeBinaryValuesIter is just that.

Sorry for the late reply, had limited time for arrow2 last 2 months.

[aldanor]

@jorgecarleitao If you think any of the above suggestions would be reasonable additions, would be always glad to help with a PR.

[jorgecarleitao]

Thanks! I think that this is addressed in #1270

[ncpenke]

Thanks @jorgecarleitao