Source generated formatter support of sparse/patch objects via BitArray

[AndriySvyryd]

Is your feature request related to a problem? Please describe.

In my use case it's common to send updated data to the client. Currently, there is no built-in way to do so without also sending redundant data.

This is related to #265, #678 and could be leveraged for #107. It would also allow versioning with int keys without a significant perf penalty.

Describe the solution you'd like

My proposal is to leverage a BitArray property that would contain bits corresponding to the keyed properties that should be (de)serialized. null array is equivalent to an array with all bits set. One of the bits would correspond to the BitArray itself, it can be used to indicate whether to set the array value in the deserialized object, but it could be just ignored.

The feature would be activated for any type with int keys that have a BitArray property assigned to 0.

I'm only interested in the source generated approach, but this could also be implemented dynamically.

If there's enough interest I can contribute the implementation.

Additional context

Here's an example of a generated formatter using this feature:

MapItem

[MessagePackObject]
public class MapItem
{
    [Key(0)]
    public BitArray? ChangedProperties { get; set; } = new BitArray(8);

    [Key(1)]
    public int Id = -1;

    [Key(2)]
    public ItemType Type;

    [Key(3)]
    public string BaseName = "";

    [Key(4)]
    public string Name = "";

    [Key(5)]
    public int LevelX = -1;

    [Key(6)]
    public int LevelY = -1;

    [Key(7)]
    public bool IsCurrentlyPerceived;
}

MapItemFormatter

public sealed class MapItemFormatter : global::MessagePack.Formatters.IMessagePackFormatter<global::MapItem>
{
    public void Serialize(ref global::MessagePack.MessagePackWriter writer, global::MapItem value, global::MessagePack.MessagePackSerializerOptions options)
    {
        if (value == null)
        {
            writer.WriteNil();
            return;
        }

        global::MessagePack.IFormatterResolver formatterResolver = options.Resolver;
        var values = value.ChangedProperties;
        var length = 8;
        if (values != null)
        {
            length = values.GetCardinality();
            if (length > 0 && !values[0])
            {
                length++;
            }
            if (length > 8)
            {
                length = 8;
            }
        }
        writer.WriteArrayHeader(length);
        global::MessagePack.FormatterResolverExtensions.GetFormatterWithVerify<global::System.Collections.BitArray>(formatterResolver).Serialize(ref writer, values, options);
        if (values == null || values[1])
        {
            writer.Write(value.Id);
        }
        if (values == null || values[2])
        {
            global::MessagePack.FormatterResolverExtensions.GetFormatterWithVerify<global::ItemType>(formatterResolver).Serialize(ref writer, value.Type, options);
        }
        if (values == null || values[3])
        {
            global::MessagePack.FormatterResolverExtensions.GetFormatterWithVerify<string>(formatterResolver).Serialize(ref writer, value.BaseName, options);
        }
        if (values == null || values[4])
        {
            global::MessagePack.FormatterResolverExtensions.GetFormatterWithVerify<string>(formatterResolver).Serialize(ref writer, value.Name, options);
        }
        if (values == null || values[5])
        {
            writer.Write(value.LevelX);
        }
        if (values == null || values[6])
        {
            writer.Write(value.LevelY);
        }
        if (values == null || values[7])
        {
            writer.Write(value.IsCurrentlyPerceived);
        }
    }

    public global::MapItem Deserialize(ref global::MessagePack.MessagePackReader reader, global::MessagePack.MessagePackSerializerOptions options)
    {
        if (reader.TryReadNil())
        {
            return null;
        }

        options.Security.DepthStep(ref reader);
        global::MessagePack.IFormatterResolver formatterResolver = options.Resolver;
        var length = reader.ReadArrayHeader();
        var ____result = new global::MapItem();

        global::System.Collections.BitArray values = null;

        for (int i = 0; i < length; i++)
        {
            if (values != null && !values[i])
            {
                continue;
            }

            switch (i)
            {
                case 0:
                    values = global::MessagePack.FormatterResolverExtensions.GetFormatterWithVerify<global::System.Collections.BitArray>(formatterResolver).Deserialize(ref reader, options);
                    if (values != null)
                    {
                        length = values.Length;
                        if (length == 0 || values[0])
                        {
                            ____result.ChangedProperties = values;
                        }
                    }
                    break;
                case 1:
                    ____result.Id = reader.ReadInt32();
                    break;
                case 2:
                    ____result.Type = global::MessagePack.FormatterResolverExtensions.GetFormatterWithVerify<global::ItemType>(formatterResolver).Deserialize(ref reader, options);
                    break;
                case 3:
                    ____result.BaseName = global::MessagePack.FormatterResolverExtensions.GetFormatterWithVerify<string>(formatterResolver).Deserialize(ref reader, options);
                    break;
                case 4:
                    ____result.Name = global::MessagePack.FormatterResolverExtensions.GetFormatterWithVerify<string>(formatterResolver).Deserialize(ref reader, options);
                    break;
                case 5:
                    ____result.LevelX = reader.ReadInt32();
                    break;
                case 6:
                    ____result.LevelY = reader.ReadInt32();
                    break;
                case 7:
                    ____result.IsCurrentlyPerceived = reader.ReadBoolean();
                    break;
                default:
                    reader.Skip();
                    break;
            }
        }

        reader.Depth--;
        return ____result;
    }
}

[AArnott]

I'm only interested in the source generated approach

In that case, let's hold onto this till source generation for v3 stabilizes and see whether adding this makes sense at that point.

[AArnott]

How important is it that the BitArray be an actual field on the class? What if the formatter did the BitArray magic without the field being declared?
Instead of the app having to explicitly set this BitArray, could we use a [Default] attribute on the members (or assume default(T)) as a hint that a member can be skipped? Or we might use the ShouldXBeSerialized property pattern.

It might be interesting to see an example where the [Key] attributed members left a 'hole' in the array, simulating a removed property.

[AndriySvyryd]

How important is it that the BitArray be an actual field on the class? What if the formatter did the BitArray magic without the field being declared?

It doesn't need to be exposed as BitArray, but there should be a way to indicate that a given BitArray property should be used for this purpose. It allows the app logic to determine which properties were deserialized in a fast way. It also works well with the INotifyPropertyChanged pattern.

Instead of the app having to explicitly set this BitArray, could we use a [Default] attribute on the members (or assume default(T)) as a hint that a member can be skipped? Or we might use the ShouldXBeSerialized property pattern.

I'm ok supporting the ShouldXBeSerialized/ShouldSerializeX/XSpecified pattern. But I wonder whether there should be an attribute or something else that would be an opt-out for this feature, either at class-level or property-level.

To avoid proliferation of methods we could also support bool ShouldSerialize(int) signature that takes a key value.

It might be interesting to see an example where the [Key] attributed members left a 'hole' in the array, simulating a removed property.

Essentially, the holes in keys translate into holes in the BitArray

[AArnott]

It also works well with the INotifyPropertyChanged pattern.

It sounds like you have a scenario in mind where you use messagepack to serialize only changes to an object. That's a notable scope change in the library. I've never heard of a general purpose serializer that was designed to do that.
It's an interesting scenario and I wouldn't mind supporting it, but we'll want to be thoughtful about what responsibilities are in the library vs. in patterns that we ask users to follow. For example, if the library offered a feature to allow an object to indicate it wanted array-compressed serialization of only select properties, it would enable your scenario if we accept that the developer had to follow the required pattern, including implementing the INotifyPropertyChanged and updating whatever state drove the library's insight into which properties to serialize.

[AndriySvyryd]

It sounds like you have a scenario in mind where you use messagepack to serialize only changes to an object.

Yes, and I wanted the library logic to be minimal and only handle the actual (de)serialization so that the responsibility for doing comparisons or otherwise detecting changes would still be in the developer's hands.