metrics tag missing from datadog exporter v3

[rex-remind101]

Looking at dd-trace we see that "sampling.priority" is added as a tag implicitly https://github.com/DataDog/dd-trace-js/blob/3419a4c09584a3109952b7c7da7d84924c36b866/packages/dd-trace/src/format.js#L98

Given that "sampling.priority" is represented by a number it should be exported as part of the metrics part of the span data as shown here https://github.com/DataDog/dd-trace-js/blob/3419a4c09584a3109952b7c7da7d84924c36b866/packages/dd-trace/src/format.js#L146

However, looking at the datadog exporter v3, there does not appear to be any mention of forwarding "sampling.priority" nor mention of metrics https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-datadog/src/exporter/model/v03.rs

Is this inserted elsewhere? If not what would be required to add it?

[rex-remind101]

fyi, it appears that when the user tags "sampling.priority" it gets converted to x
https://github.com/DataDog/dd-trace-js/blob/c89a35f7d27beb4a60165409376e170eacb194c5/packages/dd-trace/src/priority_sampler.js#L106
and
https://github.com/DataDog/dd-trace-js/blob/7b27059fe3b9de6788c0517a2b499ff6b78da5ff/ext/tags.d.ts#L6
and
https://github.com/DataDog/dd-trace-js/blob/c89a35f7d27beb4a60165409376e170eacb194c5/packages/dd-trace/src/constants.js#L4

[rex-remind101]

pr #792

[TommyCpp]

Thanks for bringing it up. I think maybe we can just drop the span when exporting instead of encode it and send out.

Have you tested it against a datadog endpoint? I lost my access to datadog a while ago so I can't test it myself.

[rex-remind101]

We need to export even in cases where spans are not sampled because the agent needs to be aware of metrics to properly calculate request/error statistics on datadog's end. The agent will then perform sampling on its end anyway afaiu.

[rex-remind101]

Actually not sure if i'm understanding you, what do you mean by "drop the span when exporting"?

[TommyCpp]

For unsampled spans, we don't have to pass it to the span exporter as per the spec.

Sampled flag in TraceFlags on SpanContext. This flag is propagated via the SpanContext to child Spans. For more details see the W3C Trace Context specification. This flag indicates that the Span has been sampled and will be exported. Span Exporters MUST receive those spans which have Sampled flag set to true and they SHOULD NOT receive the ones that do not.

[rex-remind101]

Yet judging from everything I've looked at, datadog's agent requires receiving metrics for every span or it will fail to correctly calculate statistics. So it seems like datadog does not precisely follow the spec. Also I believe datadog allows the user to setup their own ingestion controls through their website for how we wish to sample.

I think this means we'll need to force every span to be sent to datadog and make sure that metrics appropriately sets the sampling priority as I did in that pr for datadog's metrics to work correctly.

Does this make sense?

[TommyCpp]

Yeah that make sense. Let me take a deep look on the datadog document but generally I think your argument is valid

[rex-remind101]

Have you had time to take a deep look yet? We are hoping this will resolve the described issue with our telemetry and would like to get it in.

[TommyCpp]

Sorry I have been traveling this week so cannot spend much time. I will try to get a review for the PR this weekend

[rex-remind101]

looks like we can close this out now, how often do new versions come out so we may upgrade?

[rex-remind101]

@TommyCpp ?

[TommyCpp]

looks like we can close this out now, how often do new versions come out so we may upgrade?

We are planning to release 0.18 on #779. I'd say probably some time next month?