Trace exemplars

[kolesnikovae]

We already have an integration with distributed tracing solutions that enables traces to profiles navigation. We should extend the scenario to enable the other way around: profiles to traces.

From the user perspective this could look like trace exemplars in metrics:

Given a query expression (series labels + call site filter), we could retrieve top K samples with tracing attribution (by count/sum) and place them in the timeline. The links would include trace_id:span_id pairs, which allows to navigate users straight to the trace span view. In turn, in the trace span view, there would show up a profile for this specific span.

This will require a few adjustments of the storage schema and SDKs (we only collect root span_id and span_name as of now).

It's possible that the trace referred by a profile was not sampled by the trace collector (tail sampling). This should be addressed in the collector

[knylander-grafana]

I like the idea you have in this issue. If we do this, we'll want to add doc to cover concepts and any required configurations.