Support more attribute value types

[adriangb]

The allowed values in attributes seem to contain some notable omissions:

• objects
• null/none values
• integer support is capped at i64 (I think)

I understand that the recommendation for objects is to "flatten" them to foo.bar instead of something like {"foo": {"bar": ...}} but that doesn't mesh well with arrays, i.e. I don't think {"foo": [{"bar": ...}]} or just [{"foo": ...}] can be represented cleanly.

As far as I know there is no way to put a null/none value in attributes since the only types are integers, strings, bytes, floats and lists of these.

I ran into the limited size integers problem when I tried to stick a python integer into attributes.

I understand that most of these limitations come from protobuf encoding itself, but it seems to me like otel should be able to work around this and not let the most common (but not only!) encoding leak into the specification. For example, we could automatically take large integers and encode them as bytes then have SDK functions to encode/decode from protobuf into language structs that take this into account and automatically unpack into large integers / variable sized integers for each language. Null values could similarly be handled by having a null field on the AnyValue model that is a boolean or something like that.

Most of this should be backwards compatible to implement (given that these things are totally unsupported / error right now).

[jack-berg]

On the objects / arrays of objects front, that is supported at the protocol level but not in the APIs / SDKs (with the exception of logs). #2888 tried to extend API / SDK support generally, but wasn't merged. I would comment on that PR or the #376 issue.

I'm not sure about the use case for null or none. My gut reaction is that null / none is trivially different from using boolean with false. Maybe you can open an issue and expand on the use case?

I haven't come across anyone ask for integers larger than 64 bits, but I definitely understand the use case. I recommend opening an issue for that as well. As a workaround, and assuming you can coordinate with the OTLP server, you could encode the large integer as a string? Obviously not great.

[adriangb]

They can be made to mean the same thing. The amount of information is the same. So theoretically there is no difference.

But then everyone has to go around writing recursive functions to pre process all of their data before putting it in attributes to replace nulls with absence of the key or a separate boolean. Queries into traces also have to handle this. So practically speaking yes, they are very different.

And again even if one can come up with a way to handle this the fact that no implementations will agree on what the workaround is means it's a lot less useful than handling it at the protocol level.

Do you feel it would be hard to handle? It seems pretty straightforward to handle to me.

[adriangb]

To clarify my use case: as a Python user I would like to be able to take any object I could pass to json.dumps and pass it to attributes. That includes all of the three cases I pointed out above.

[jack-berg]

But then everyone has to go around writing recursive functions to pre process all of their data before putting it in attributes to replace nulls with absence of the key or a separate boolean.

The spec actually does take a stance on this here:

Attribute values of null are not valid and attempting to set a null value is undefined behavior.

In the java implementation, we simply ignore null attribute values anywhere attributes are used. Since the behavior is undefined, this behavior is acceptable but not required.

Do you feel it would be hard to handle? It seems pretty straightforward to handle to me.

I think its hard to change stable spec documents.

To clarify my use case: as a Python user I would like to be able to take any object I could pass to json.dumps and pass it to attributes. That includes all of the three cases I pointed out above.

The key thing blocking you is the failed attempt to expand attributes to support maps and heterogenous arrays. That was a hotly debated issue we weren't able to resolve. Restarting that conversation would require someone to champion the issue, and probably present new evidence / arguments that break the stalemate.

One idea that comes to mind for you is to use the Event API (or maybe the Log API) instead. Logs are a bit of an anomaly in that the log data model has a AnyValue body field, and an map<string, AnyValue> attributes field. These looser restrictions were meant to accommodate bridging from other log frameworks, which may not have the same restrictions as OpenTelemetry attributes have elsewhere. Could you emit a log record / event with your json.dumps data?

[adriangb]

I think its hard to change stable spec documents.
Isn't undefined behavior -> working well-defined behavior, not a breaking change?

One idea that comes to mind for you is to use the Event API ...
There are already several workaround options, but unfortunately, they all have pretty large drawbacks, a more permanent solution is needed.

[adriangb]

#2888 was very helpful, thank you for linking @jack-berg. I'll have to read and explore some more and may reply in that PR (since that's where most of the discussion seems to be).