Why is there an indirection from span::Id to registry::SpanRef via Context/LookupSpan?

[Swatinem]

Can someone explain to me why the Layer API needs an indirection via Context and LookupSpan to a SpanRef?

There are a few shortcomings I see with the current API:

• It is a bit cumbersome to implement a Layer, since I have to copy-paste all kinds of trait bounds which are entirely not obvious to me.
• The span lookup seems to be fallible, though even tracing_subscriber::fmt treats it as not, and has a bunch of unwraps in there.

See https://docs.rs/tracing-subscriber/0.3.1/src/tracing_subscriber/fmt/fmt_layer.rs.html#561 for examples.
Having panicking code adds a bit of overhead (code size, etc).
But in general I feel like these are runtime checks for something that should rather be solved at compiletime by making these mistakes impossible to express in the type system.

I think maybe the reasoning for all this comes from the fact that the core tracing API only outputs a flat stream of span events, and has no notion of hierarchy and attached data at all?
Registry would be the thing that offers this functionality, making sure that spans are put into a hierarchy, and that they can have data attached to them, right?

[hawkw]

I think maybe the reasoning for all this comes from the fact that the core tracing API only outputs a flat stream of span events, and has no notion of hierarchy and attached data at all?
Registry would be the thing that offers this functionality, making sure that spans are put into a hierarchy, and that they can have data attached to them, right?

This is correct. Depending on the use-case, it may be desirable to not store span data in the program at all, and simply write that data out to a file or network IO immediately. Therefore, we chose not to make in-memory storage of any span data a core part of tracing, and instead, made it opt-in when required.

For example, I've seen tracing used in low-memory embedded code, where it's not feasible to store any span-related data in memory. Instead, when spans are created, the data is immediately written out over a serial port if debugging is enabled. Then, each span's fields and other data can be hydrated from the ID by the dev machine on the other side of the serial connection, without having to store it in memory on the embedded device.

Similar designs are possible in use-cases other than embedded systems, any time some external component is responsible for processing or viewing trace data. In fmt, where we perform all formatting of log lines in-process, it's necessary to store span data persistently in memory in order to format those log lines. But, not all uses of tracing will require this.

Another motivation for decoupling these components is to allow span data storage to be modular, so that the Registry type can be swapped out for another component that also implements the LookupSpan trait, although this isn't really done in practice yet.

So, yes, there is some additional overhead for implementing span data storage at a higher level than in tracing-core. However, the goal for this is to avoid making the overhead of storing span data at all be a required part of using tracing, so that it can be avoided when it's unnecessary (or impossible --- for example, I've written a #![no_std]tracingSubscriber` that performs no allocations at runtime).

Hope that answers your main question!

Regarding this point:

• The span lookup seems to be fallible, though even tracing_subscriber::fmt treats it as not, and has a bunch of unwraps in there.

I've considered the possibility of changing the Layer trait so that rather than accepting an Id and a Context, and looking up the span by Id, the SpanRef is simply passed in to methods like on_enter and on_new_span. However, this requires changing the trait so that it requires the wrapped subscriber to implement LookupSpan. It might be possible to add a new trait which provides a version of these methods where SpanRefs are automatically looked up and passed in, and implement Layer<S: LookupSpan> for all implementations of that trait...

[Swatinem]

Yes, the first part is a very good explanation!

And the tracing-core API does make sense, considering its just a stream of events.

Also, changing Layer in a way that all of its methods get SpanRefs directly, without the indirection via Id/Context is the suggestion that I kind of implied by my question.

IMO, a way to compose/chain different low-level Subscriber implementations together should be different from the high-level Registry concept which offers hierarchy and span data. Also in a sense that the high-level API can be really high level and ergonomic to use, vs the ctx.span(id).unwrap() that has to be done right now.

---

So from my point of view as maintainer of sentry-rust, the question that was in my mind was if Layer is actually the right trait to implement.
Ideally, we would want to have a low-level API independent of the tracing crate, which essentially allows users to create a flat stream of events. That means it makes more sense to implement a Subscriber, as we would probably have to find our own solution for hierarchy and span-data that fits with the sentry data model anyway for a more low-level API.

[davidbarsky]

So from my point of view as maintainer of sentry-rust, the question that was in my mind was if Layer is actually the right trait to implement.
Ideally, we would want to have a low-level API independent of the tracing crate, which essentially allows users to create a flat stream of events. That means it makes more sense to implement a Subscriber, as we would probably have to find our own solution for hierarchy and span-data that fits with the sentry data model anyway for a more low-level API.

I say the following at the risk of misunderstanding your concerns: while portions of the Layer/Registry interface can be improved, I'd argue that implementing the Layer trait is the correct choice, in that it allows other users of tracing to compose with sentry-rust (which Subscriber does not do). I'd consider using the registry as a starting point before molding the data that tracing records into something that Sentry expects. This is a pretty common approach and pattern taken by crates such as tracing-opentelemetry or tracing-error.