Overriding Decisions of the Built-in Trace Samplers via a "sample.priority" Like Span Attribute

[stoty]

I propose adding a feature to the built in Sampler classes (AlwaysOn, AlwaysOff, TraceIdRatioBased) which would allow the sampling decision can be overriden by setting a Span attribute.

This would let users force tracing on or off when starting a Span, whitout writing and registering a custom Sampler.

span = tracer.spanBuilder("configurable").setAtttribute("sample.priority", 1).startSpan().

This is somehwat similar to #530 , but the objective here is not
pausing/terminating traces inside the chain, but overriding the sampling flag from code when creating the root span.

Background:

I am trying to migrate Apache Phoenix from HTrace to OpenTelemetry tracing.
Apache Phoenix is an SQL Layer over HBase. HBase has built-in support for OpenTelemetry.
It depends on opentelemetry-api, and by default expects the implementation to be provided by the OpenTelemetry java agent, and configured via its environment variables.

One of my goals is making the tracing configurable from the JDBC API and the statement text.

Either on the connection level by setting a propery, or adding a JDBC URL parameter to
https://docs.oracle.com/javase/8/docs/api/java/sql/DriverManager.html#getConnection-java.lang.String-java.util.Properties-

Or on a query by query basis with a query hint like
SELECT /*+ TRACE=ON */ * FROM TABLE
Doing this is harder than I expected.

The Trace API doesn't seem to allow setting sampling explicitly, only via the Sampler object, which is not available in opentelemetry-api, only in opentelemetry-sdk.

I have looked at using the "parentbased_always_on" built-in sampler, and simply not starting a Span from Phoenix if don't want to enable tracing. However, this wouldn't work, because the underlaying HBase client library would start sampled spans for each call into it anyway, which is not what I need.

So I need a way to start either sampling on non-sampling Spans based on information in Phoenix.
The way to do it seems to be writing a custom sampler, setting some attribute on the Span, and then checking this in the Sampler to make the decision.

I believe that this a viable soultion, but I'd need to add a compile-time dependency on opentelemetry-sdk for this, and I am afraid that adding my custom Sampler to an agent based setup may cause unforeseen complications.

Searching for solutions, I came across the sample.priority semantic attribute for Opentracing, which seems to do what I want, but I could not find a similar feature in OpenTelemetry

Looking at TraceIdRatioBasedSampler, it would be super easy to add an override based on a Span attribute, all it takes is a hash lookup and a comparison.

The only hard part is that ideally this should go into the specification, and be added for each language SDK, which is a lot of changes in a lot of languages.

[adriangb]

Both this and #530 seem like good ideas. There are going to be varied use cases that require varied APIs:"

• globally configured sampling (exists)
• per processor/exporter sampling (exists)
• per span sampling (this proposal)
• changing sampling once a span is already started based on things that happened between it starting and ending (Use Context to stop tracing #530)

[stoty]

Thank you.
In the meantime I dug more into what Phoenix implements currently:

Turn tracing on/off or do no nothing : could be "sample.priority" 1/0/not specified
Override tracing probability: could "sample.probability" or similar. 0..1/not specified

Really those are the only two things that make sense for the default tracers.
(Overriding the probability would only have an effect for TraceIdRatioBasedSampler, of course.)

[jmacd]

The Sampling SIG meets weekly on Thursdays at 8am PST. There is already a precedent for the thing you're asking for and it would be fairly easy to see adding this to the spec.

[adriangb]

Does someone need to join the sig and advocate for this, or are you saying the sig will discuss this just based on this discussion?

[PeterF778]

I do not think that this proposal, as written, is a good idea. The issue with it is that is would allow the application to force tracing on or off when starting a Span, overriding any decisions made by the (default) samplers.
One of the goals of Open Telemetry is to provide a foundation for observability. I believe observability will be compromised if the application can control what data is collected and what is not.
Your use case is unique in that you seem to be both the application owner and the OpenTelemetry user.

However, sampling priority (specified by the application) makes sense as a hint. For example, TraceIdRatioBased sampler can use different thresholds depending on sampling priority.

[stoty]

I see your point about allowing to negatively influence head sampling decisions being a problem.

These features could be gated behind another agent system property/environment variable, like "allow sampling rate/decision override" or added in new Samplers.

Alternatively/Additionally, would it be acceptable if these hints were only used to turn ON sampling ?
i.e. If the sample hint is set, it is always sampled, if it is not, then the default policies are used, and the frequency hint is only observed if it increases the default ?

For my (and I imagine most) use case, it's perfectly to fine to use the default policies for the negative case, what I really need is selectively enabling sampling.

I can imagine a lot of cases where an application/service would want to make sure to get traces of some operations, but handling that in tail samplers would cause too high a load on the tracing infrastructure (as this would require the AlwaysOn head sampler).

[PeterF778]

Even if we limit the hints to encourage/force sampling, such decisions can conflict with the sampling policy used by the actually configured sampler. Would you expect that AllwaysOffSampler makes exception and samples spans with the sampling.priority attribute set to 1? What about any rate limiting sampler whose mandate is to sample no more than N spans per time unit?

I think it is better to develop a special sampler which looks for the sampling.priority attribute and makes final sampling decision if it is present, and delegates the decision to a fallback sampler if the attribute is missing.

[stoty]

Would you expect that AllwaysOffSampler makes exception and samples spans with the sampling.priority attribute set to 1

Basically yes. That behaviour could be could be gated behind a sampler config parameter.

What about any rate limiting sampler whose mandate is to sample no more than N spans per time unit?

Custom samplers would be free to ignore either hint, or re-interpret it as needed.
In my mind, the feature is about avoiding having to use a custom sampler, and making the feature available from otel-agent.

I think it is better to develop a special sampler which looks for the sampling.priority attribute and makes final sampling decision if it is
present, and delegates the decision to a fallback sampler if the attribute is missing.

That is totally fine by me.
It would let the application control sampling. It wouldn't allow for direct overriding the sampling frequency, but my use case doesn't require that, and the application could still use an internal frequency based sampler.

This modified proposal would be (just the logic, it would not necessarily be implemented in separate classes):
Add a HintedSampler similar to ParentBasedSampler to the opentelemetry SDK.

This could be chained similarly to the parentBased logic, sou you could have these new combinations:

Hinted > parentBased > Ratio/alwaysOn/AlwaysOn:
Check the hinted flag first, if it is set, honor it, otherwise behave like then current parentBased...

ParentBased > Hinted > Ratio/alwaysOn/AlwaysOn:
Check the parent first. If there is a parent, use its decision. Otherwise check hint, and honor if it's set. Otherwise use ratio/on/off
(This is the one I'd used for my use case)

Hinted > Ratio/alwaysOn/AlwaysOn
Ignore the parent. Check the hint, and honor it if set. Otherwise use ration/on/off

A slight problem is configuring it.
At the moment there are 2x4=8 possible values for OTEL_TRACES_SAMPLER (ignoring jaeger)
With the above setup there would be 5x4 = 20 permutations.

[PeterF778]

Yes, I think putting the new functionality in a single sampler (HintedSampler) is the cleanest solution.

[svrnm]

@stoty can you take a look at #3205, this sounds like a very similar discussion (but I might miss the nuances between your proposal and what has been discussed there)

[stoty]

#3205 is a far more abitious proposal than this one.

This is a much smaller subset of that functionality:

• provide a property to force enable (disable is up for debate) HEAD sampling for span from code
• provide a way to configure the default samplers in the OTEL SDK to honor it.
• optionally: provide a property to influence the sampling frequence, and a way to enable it

This propsal is not about pruning the traces, and how to handle pruned sections, it assumes no changes in the inner workings beyond influencing the sampling decision.

The intended use case is letting the application turn on sampling selectively, and the propagating this decision via standard parent based samplers in every component.

[stoty]

For reference:
I was able to write a custom hintable sampler with the corresponding SPI Provider, and specify this jar as an agent extension.

This works fine, though having to build, ship and specify a separate OTEL agent extension jar with Phoenix is not ideal.

[stoty]

Based on this dicussion, I feel that most people agree that something like this would be useful, but a desire for a very generic solution to solve all span pruning and customization needs gets in the way of simple fix for this specific feature gap.

Would it help if I made a PR for the java SDK that adds the hinting feature based on the discussion above ?

[trask]

@stoty would open-telemetry/oteps#240 (if support for it was implemented in the Java agent) address your need?

[stoty]

Not in itself.

open-telemetry/oteps#240 is about configuring and composing existing samplers, so it would help with defining how the hintable sampler is used for the sampling decision (first, last, only for root spans, only if matches somehting, etc), but but it does not seem to include the hintable sampler itself, neither does it define a standard hint/attribute for applications to configure it.

AFAICT the hintable sampler itself would still have to be provided by the application separately, and the hint attribute would still be application specific.

Going through the links in that document, now I think that RuleBasedRoutingSampler could ALMOST work for my use case, and could be configured to use for example "Parent Based Always" for hinted spans, and "Parent based TraceIdRatioBasedSampler" for non-hinted ones.

The problem is that the only way to configure that is from Java code.
So if RuleBasedRoutingSampler's rules could be configured via open-telemetry/oteps#240, that would work for me.

However, I still feel that defining a (some) standard hint(s), and including a mechanism to enable it (them) in the main (non-contrib) agent would be a useful addition.

[stoty]

Short term, I could add a few SPI provider classes like https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/LinksParentAlwaysOnSamplerProvider.java to implement some of the hinted sampler combinations I have mentioned above.