MapArrayReader Does Not Understand Nesting

[tustvold]

Describe the bug

The logic within MapArrayReader will not work if the MapArray is itself nested, in particular:

• It does not expose the def_level / rep_levels of its children
• It assumes that a repetition level of 0 corresponds to a new map entry

To Reproduce

A nullable StructArray with a child MapArray will likely result in an error, similarly a ListArray with a child MapArray will likely error

Expected behavior

This should be properly supported

[frolovdev]

@tustvold maybe I get your idea wrong but I can't reproduce it in any ways, trying with such incoming data

message table {
            repeated group my_group_map {
                REQUIRED BYTE_ARRAY kek;
                optional group map (MAP) {
                    repeated group key_value {
                        REQUIRED BYTE_ARRAY key;
                        OPTIONAL group nested_map (MAP) {
                            repeated group key_value {
                                REQUIRED BYTE_ARRAY key;
                                REQUIRED BYTE_ARRAY value;
                            }
                        }
                    }
                }
            }
            
        }

[tustvold]

Hi @frolovdev. I'm afraid I'm away from a computer for the next few days, but taking a cursory look, lines like

        let entry_len = rep_levels.iter().filter(|level| **level == 0).count();

Are not able to correctly handle the case of a MapArrayReader inside another list, as it starts a new run when the repetition level is 0, as opposed to the repetition level of the map array (which will only be 0 if it is at the root).

If you compare the logic with the logic in ListArrayReader, that may help, as a MapArray is just a special case of a ListArray and the levels handling logic should be identical.

One way to fix this, might be to remove MapArrayReader and just add a teeny bit of additional logic to ListArrayReader to handle MapArray. We do something similar for the level computation on the writer side.