RFC: Encode Scalars as dyn Any in Scalar dyn Kernels

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

As expanded upon in #2837 currently the encoding of scalars in dyn kernels is inconsistent. The arithmetic kernels must be explicitly type hinted with the ArrowPrimitiveType, whilst the comparison kernels coerce the scalar type to the desired type.

Taking a step back, users just want to give a kernel an array and a scalar of the correct type and have it work, with all the dispatch logic handled for it.

Describe the solution you'd like

I would like to suggest encoding scalars as & dyn std::any::Any, for example,

pub fn eq_scalar_dyn(
    left: &dyn Array,
    right: &dyn std::any::Any,
) -> Result<BooleanArray> {
    downcast_primitive_array! {
        left => match right.downcast_ref() {
            Some(r) => eq_scalar(left, *r),
            None => Err(ArrowError::ComputeError(format!("Failed to downcast scalar for {}", left.data_type())))
        },
        DataType::Boolean => match right.downcast_ref() {
            Some(r) => eq_bool_scalar(as_boolean_array(left), *r),
            None => Err(ArrowError::ComputeError(format!("Failed to downcast scalar for {}", left.data_type())))
        }
        DataType::Utf8 => match right.downcast_ref() {
            Some(r) => eq_utf8_scalar(as_string_array(left), *r),
            None => Err(ArrowError::ComputeError(format!("Failed to downcast scalar for {}", left.data_type())))
        }
        DataType::Dictionary(_, _) => downcast_dictionary_array! {
            left => {
                let dict_compare = eq_scalar_dyn(left.values().as_ref(), right)?;
                unpack_dict_comparison(left, dict_compare)
            }
            _ => unreachable!()
        }
        t => Err(ArrowError::ComputeError(format!("Unsupported datatype for eq_dyn_scalar_any {}", t)))
    }
}

Users can then write

eq_scalar_dyn(array, &false)?;
eq_scalar_dyn(array, &3_i32)?;
eq_scalar_dyn(array, &5.2_f32)?;
eq_scalar_dyn(array, &"hello")?;

This has the following advantages:

• The dyn kernels don't require any type hinting
• The kernels don't perform any implicit type coercion

The only major downside is that users will need to be careful to correctly hint scalar types, i.e. &1_i64.

Describe alternatives you've considered

We could not do this

Additional context

[alamb]

I think this sounds like a great idea. I didn't realize Any was implemented for the basic types

[viirya]

Hm, will this affect performance?

[tustvold]

I wouldn't expect it to have a material performance impact. It will perform a downcast of the scalar value once when it downcasts the array, and then proceed as normal

[tustvold]

One wrinkle the comes up here is that we need a way to encode the data type of the scalar, e.g. to allow adding both interval and duration scalars of varying types to temporal types such as dates and timestamps. We may need to introduce a Scalar trait 🤔