opentelemetry: add semconv exception fields #2135
Conversation
There was a problem hiding this comment.
So, this seems correct overall!
I am not sure if this is really going to be all that useful in practice yet, though: this branch will emit the semconv exception fields if a Span records a value of type Error. However, in typical tracing use, it's much more common for errors to be recorded by an event inside of a span. I think we may want to consider also attaching the exception fields to the current span whenever an event is recorded with an Error value. This should probably be configurable behavior, since some users may not want it.
If we do that, we'll also have to figure out what happens if multiple events with Errors occur in the same span. How does OpenTelemetry handle multiple fields with the same name?
|
@lilymara-onesignal any updates on this? |
|
Whoops, sorry I missed the comments on this. I'll see about adding configurability for these otel fields and adding them to the event's current span. On duplicate fields, it seems that the otel library will drop the older field and report only the newer field. |
lilymara-onesignal
force-pushed
the
exception
branch
from
ebae7af
to
543ebb1
Compare
|
@hawkw I think this may be ready for another pass, the behavior is configurable and I've added docs/tests |
|
Ok I think I've addressed everything but the defaults - on the fence there also. It seems to me like walking the "cause" chain is the most expensive part of all of this, and that's enabled by default and can't currently be turned off. I don't feel extremely strongly either way. |
|
There's also the possibility that we could set the status code on the event/span whenever we record an error event |
All otel attribute collections have unique key requirements, generally last write wins.
It's not always clear that the status of the span has changed when an error event occurs (e.g. operations with retries or concurrent alternatives) |
|
@hawkw could I get another review on this? I think the only outstanding item is the default behavior, and I'm not sure how to come to an answer on that. |
There was a problem hiding this comment.
@lilymara-onesignal sorry I didn't get back to you on this sooner. I think this PR seems good overall, but I think we should have the exception fields be off by default, making them opt-in rather than opt-out, due to the additional overhead they incur. If we can find a way to reduce the overhead of recording these fields (perhaps if the changes to opentelemetry i proposed in open-telemetry/opentelementry-rust#809 are made?) we could change them from opt-in to opt-out.
Does that seem reasonable to everyone (cc @jtescher)?
|
No problem, I think at this point I've resolved all of the points of contention |
|
Hmm, not really sure what to do about that MSRV check |
That's fixed on |
When an error value is recorded, it should add the `exception.message` and `exception.stacktrace` fields from the opentelemetry semantic conventions to the span/event. Ideally the `exception.type` field would be provided also, but there is currently not a mechanism to determine the type name of a trait object.
lilymara-onesignal
force-pushed
the
exception
branch
from
d578dd8
to
6ab5511
Compare
jtescher
added
the
crate/opentelemetry
Currently, the `tracing-opentelemetry` docs indicate that the support for OpenTelemetry's exception semantic conventions added in #2135 is enabled by default. However, this is not the case --- this feature was changed to opt-in rather than opt-out prior to merging PR #2135. This branch updates the docs to state that these features are disabled by default, rather than enabled by default.
## Motivation Currently, the `tracing-opentelemetry` docs indicate that the support for OpenTelemetry's exception semantic conventions added in #2135 is enabled by default. However, this is not the case --- this feature was changed to opt-in rather than opt-out prior to merging PR #2135. ## Solution This branch updates the docs to state that these features are disabled by default, rather than enabled by default.
# 0.17.4 (July 1, 2022) This release adds optional support for recording `std::error::Error`s using OpenTelemetry's [semantic conventions for exceptions][exn-semconv]. ### Added - `Layer::with_exception_fields` to enable emitting `exception.message` and `exception.backtrace` semantic-convention fields when an `Error` is recorded as a span or event field ([#2135]) - `Layer::with_exception_field_propagation` to enable setting `exception.message` and `exception.backtrace` semantic-convention fields on the current span when an event with an `Error` field is recorded ([#2135]) Thanks to @lilymara-onesignal for contributing to this release! [thread-semconv]: https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/exceptions/ [#2135]: #2135
# 0.17.4 (July 1, 2022) This release adds optional support for recording `std::error::Error`s using OpenTelemetry's [semantic conventions for exceptions][exn-semconv]. ### Added - `Layer::with_exception_fields` to enable emitting `exception.message` and `exception.backtrace` semantic-convention fields when an `Error` is recorded as a span or event field ([#2135]) - `Layer::with_exception_field_propagation` to enable setting `exception.message` and `exception.backtrace` semantic-convention fields on the current span when an event with an `Error` field is recorded ([#2135]) Thanks to @lilymara-onesignal for contributing to this release! [thread-semconv]: https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/exceptions/ [#2135]: #2135
# 0.17.4 (July 1, 2022) This release adds optional support for recording `std::error::Error`s using OpenTelemetry's [semantic conventions for exceptions][exn-semconv]. ### Added - `Layer::with_exception_fields` to enable emitting `exception.message` and `exception.backtrace` semantic-convention fields when an `Error` is recorded as a span or event field ([#2135]) - `Layer::with_exception_field_propagation` to enable setting `exception.message` and `exception.backtrace` semantic-convention fields on the current span when an event with an `Error` field is recorded ([#2135]) Thanks to @lilymara-onesignal for contributing to this release! [thread-semconv]: https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/exceptions/ [#2135]: tokio-rs/tracing#2135
When an error value is recorded, it should add the
exception.messageandexception.stacktracefields from the opentelemetry semantic conventions to thespan/event. Ideally the
exception.typefield would be provided also, but thereis currently not a mechanism to determine the type name of a trait object.
Motivation
Motivated by comment on another PR - #2122 (comment)