DynComparator/LexicographicalComparator Incorrectly Handles Dictionary Nulls

[tustvold]

Describe the bug

DynComparator does not inspect null masks. This is by design as it isn't provided a SortOptions specifying how to order them.

Unfortunately the values array of a dictionary is permitted to contain nulls, which causes the comparator to break.

To Reproduce

The builders will never put a null in a dictionary, but if you manually construct the DictionaryArray from ArrayData it is possible.

let values = Int32Array::from_iter([None, Some(1)]).into_data();
let keys =
    Int32Array::from_iter([Some(0), None])
        .into_data();

let data = keys
    .into_builder()
    .data_type(DataType::Dictionary(
        Box::new(DataType::Int32),
        Box::new(DataType::Int32),
    ))
    .child_data(vec![values])
    .build()
    .unwrap();
let dictionary = DictionaryArray::<Int32Type>::from(data);

If you then construct a comparator on this dictionary it will say that the two rows are not equal, when they are technically both null. If you remove the Some(1) it will panic.

Expected behavior

This should be handled correctly

Additional context

Longer term I intend for these to be deprecated and replaced with the row format #2677, and so perhaps we just defer on fixing this.

[alamb]

Maybe we could simply start by generating an error

[tustvold]

This same issue also impacts the ability to add support for nested types, such as structs #3187