[Java] Questions on Attributes, key semantics and formatting.

[hagbard]

I asked this question on SO, before I found this discussion forum, so I thought I'd repost here with more context:

https://stackoverflow.com/questions/77297754/how-should-open-telemetry-users-prevent-accidentally-creating-attributes-with-ma

I'm looking at putting together an OTel friendly backend for Google's Flogger logging API (rather than going via a simplistic bridging backend), but am wondering how to manage attributes and attribute keys.

Flogger already has a robust context metadata mechanism, and it mostly maps well to Attributes ... mostly.

Some issues arise however because while Flogger doesn't impose any restrictions on key labels, OTel seems to (esp. in the sense that values are stored in maps and duplicates are removed). So I'm faced with the undesirable prospect of having to ignore user provided metadata with distinct key instances which (possible accidentally) share duplicate labels. This is something Flogger itself doesn't do currently so I am keen to avoid it if possible.

I can use Floggers metadata handling to avoid directly using "Attributes", "AttributesBuilder" etc. during processing, but I still need to make calls to logRecordBuilder.setAttribute() at some point. However, I can find no documentation on the semantics of this call. Specifically:

1. what if multiple calls are made with the same key instance?
2. what if multiple calls are made with equal keys (I assume it's the same as 1)?
3. what if multiple calls are made with keys which share the same label string but are of a different type?

None of the above happen when using the default Attributes implementation due to it's deduplication, but nothing in the docs says that this behaviour must be reflected in the calls to setAttribute() itself.

I can address some of the issues if I can make my own keys with automatic prefixes (e.g. "otel:user_defined_label" and "flogger:user_defined_label"), but I also can't find documentation on what the label syntax should be and whether it's promised that the key-string used to create an AttributeKey must be what is passed to the log record.

Finally, I'm interested in what the semantics of printing attribute values should be. I saw some code which just used "Attributes.toString()" but it's not clear if that's required, or just an option. I say this because Flogger goes to some length to make the metadata representation it uses correct JSON (including escaping string contents of values). The Attributes class does not do this.

Specifically it:

1. Only quotes strings without escaping them (allowing for [value="foo" bar="baz"] which is not reliably parseable).
2. It doesn't quote strings in string-array attributes (it just calls toString() on the list value) and does not quote the elements or the result.

Would it be acceptable to create string representations of attributes without these drawbacks, or is this part of a serialization specification somewhere?

[hagbard]

Oh yes, I'm also interested in whether calling setAttribute in "sorted key-string order" or some other stable order is required behaviour.

[hagbard]

Would it be acceptable to then merge the String and Long values into a STRING_ARRAY attribute (same label) with appropriate formatting?

[jkwatson]

This would require a change to the otel attributes specification, since this is not the defined behavior currently.

[hagbard]

My goal is essentially to avoid ever needing to discard an attribute value because of a key clash. Unless key labels have some notion of a global namespace (a'la Java packages or Maven artifacts for example) you will inevitably risk having accidental duplicate keys, possibly with distinct types, possibly in shared library code that an application cannot modify/fix.

When I designed Flogger's metadata system, I made sure it never needed to discard information, but I only achieved this by treating attribute key/value pairs as a list, not a map, and allowing duplicate keys. I figured that when debugging, it's actually unlikely you need to care about key uniqueness, since you often only want to do things like "find all logs with the key/value pair 'x=y' " which isn't affected if there's also an 'x=z' pair. In Flogger the context metadata can contain all of { foo="bar", foo=123, foo=true }.

I don't suggest people try and make clashing key labels, but I figured it will happen accidentally in any sufficiently large codebase, so it's best to preserve everything. I figure this problem also exists for Attributes unless there's strong guidance on making keys unique.

Do you have a link to the "otel attributes specification" I can read?

[jkwatson]

https://opentelemetry.io/docs/specs/otel/common/#attribute-collections

[hagbard]

Thanks, that's useful.
I think I can work around the duplication issue for Flogger metadata if I can ensure that the OTel specific metadata keys users can generate for use with Flogger can never clash with Flogger metadata labels. To do this, I can prefix Flogger metadata labels when they are emitted as attributes. However, for documentation purposes, it would be nice to know what the set of supported characters for key strings is, and any existing best practice in defining them. At least knowing some existing commonly keys would help (but a spec would be better).

Nevermind, I found it: https://opentelemetry.io/docs/specs/otel/common/attribute-naming/

[jkwatson]

Any formatting in the toString implementations in any OTel classes is for visual debuging purposes only, and should never be relied upon for parsing. [I know this only addresses one thing in your post, but it's the one that I know for sure.]