How to get a ReadableSpan in a SpanProcessor?

[jebeaudet]

Hi!

I'm trying to implement a small SpanProcessor to implement a propagating attribute logic : I would like to have this to have nested spans inherit some of the properties set by their parents. I'm using the javaagent in my case, alongside the spring-boot-starter. Here is the simple implementation of my processor :

public class AttributePropagatingSpanProcessor implements SpanProcessor
{
    @Override
    public void onStart(Context parentContext, ReadWriteSpan span)
    {
        Span parentSpanFromContext = Span.fromContextOrNull(parentContext);
        if (!(parentSpanFromContext instanceof ReadableSpan)) {
            return;
        }

        ReadableSpan readableParentSpan = (ReadableSpan) parentSpanFromContext;
        readableParentSpan.toSpanData().getAttributes().forEach((key, value) -> {
            if (key instanceof PropagatingAttributeKey) {
                span.setAttribute((AttributeKey<String>) key, (String) value);
            }
        });
    }

    @Override
    public boolean isStartRequired()
    {
        return true;
    }

    @Override
    public void onEnd(ReadableSpan span)
    {
    }

    @Override
    public boolean isEndRequired()
    {
        return false;
    }
}

PropagatingAttributeKey is defined next to this class and available from within the application. For now, it's only String attributes but you get the idea.

Now, my problem with this is that I never get a ReadableSpan from within the extension context. Debugging and checking the agent source code, I realized that I'm getting an ApplicationSpan that is created by the Bridging and AgentContextWrapper classes to account for the shaded classes between the application and the agent. Unfortunately, that ApplicationSpan only implements Span, even though the wrapped span might be a ReadableSpan or even a ReadWriteSpan. In my debugging setup, the wrapped implementation was a SdkSpan instance.

Is there any way around this for now? Or would it make sense to wrap the Span instance into a class that implements the right extra interfaces (ReadWriteSpan and ReadableSpan), something like :

  public static Span toApplication(io.opentelemetry.api.trace.Span agentSpan) {
    if (!agentSpan.getSpanContext().isValid()) {
      // no need to wrap
      return Span.getInvalid();
    } else if (agentSpan instanceof ReadWriteSpan) {
      return new ReadWriteApplicationSpan(agentSpan);
    } else if (agentSpan instanceof ReadWriteSpan) {
      return new ReadableApplicationSpan(agentSpan);
    } else {
      return new ApplicationSpan(agentSpan);
    }
  }

Thanks!

[jebeaudet]

Adding some details here, it seems that the code works for span instances created by the java agent but it does not for spans created programatically in the code, in my case it's something like :

public class Foo {
    @Autowired
    private OpenTelemetry openTelemetry;


    public void execute()
    {
            var outerSpan = openTelemetry.getTracer("someTracer")
                                         .spanBuilder("outer")
                                         .setParent(Context.current())
                                         .startSpan();
            try (Scope scope = outerSpan.makeCurrent()) {
                var innerSpan = openTelemetry.getTracer("someTracer")
                                             .spanBuilder("inner")
                                             .startSpan();
                try (Scope innerScope = innerSpan.makeCurrent()) {
                    System.out.println("hello");
                }
                innerSpan.end();
            }
            outerSpan.end();
   }

I guess it makes sense because spans created by the agent do not need to be wrapped.

[laurit]

I'm using the javaagent in my case, alongside the spring-boot-starter.

My guess is that is what is causing the issue. Java agent and spring starter both create their own OpenTelemetry instances. My guess is that your span processor is added to the one created by the spring starter. To allow allow agent and manual instrumentation to coexist agent instruments the opentelemetry api that is used in the application. I guess you end up with a weird situation where span is create by the opentelemetry from the spring starter but is stored in the context that is managed by the agent. Imo the best way around this would be choose whether you want to use the spring starter or the agent, it doesn't really make sense to use both of them at the same time as they contain duplicate instrumentations.

[jebeaudet]

Hi @laurit
Thanks for your answer. For precision, my span processor is added on "both sides" since I use the otel.javaagent.extensions with a shaded built JAR and I also link on my otel-extension maven reactor module and processor gets correctly registered through SPI.

I agree that there's an overlap here, and I've been quite vocal where I work about the discrepencies of using both :D. This being said, thanks and I'll look shortly (and report back here) into ditching the spring-boot stuff and just use the agent and the OpenTelemetry bean configured as documented here.

In all cases though, I still think handling the different sub-interfaces of Span could be an improvement to the java agent code and would provide a more accurate portrait to the consumer.

[jebeaudet]

@laurit
Thanks again for your feedback, I now get a proper SdkSpan instance in my processor! I've improved our posture with the whole spring-boot integration on top of the java agent, I now rely solely on the agent configuration using this simple configuration :
Replace opentelemetry-spring-boot-starter with :

        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
        </dependency>

And simply add the global OpenTelemetry instance as a bean for easy wiring :

    @Bean
    public OpenTelemetry openTelemetry()
    {
        return GlobalOpenTelemetry.get();
    }

Beware not to use AutoConfiguredOpenTelemetrySdk.initialize().getOpenTelemetrySdk() as mentioned in the documentation as this would create another instance (you'll get a warning printed out to the syslog when doing that).

This being said, unfortunately, my processor still not work for a similar reason, the Bridging class will convert my the Attributes to barebones AttributeKey(here). This restrict my processor to using the attribute key instead of using another implementation of AttributeKey like I was doing. It is a bit limiting since I now need an allowlist which would hardcode which attribute I want to propagate, or use a naming scheme (ie. propagated.attributeKey) to decide whether the attribute is propagated or not.

Am i SOL here or there are options I'm missing? I'm assuming the former because of classpath issues between my extension, the application and the agent which are all different.

TIA!

[laurit]

Replace opentelemetry-spring-boot-starter with :

    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
    </dependency>

you shouldn't really need these. Add opentelemetry-api if you need to do manual instrumentation.

This being said, unfortunately, my processor still not work for a similar reason, the Bridging class will convert my the Attributes to barebones AttributeKey(here). This restrict my processor to using the attribute key instead of using another implementation of AttributeKey like I was doing.

Agent and application are isolated from each other. The PropagatingAttributeKey inside you application is not visible for the extension that runs inside the agent and vice versa. To share code you'd need to place this class in boot loader or apply some kind of bridging that would allow converting the key between application and the agent code. Introducing api that would let extensions customize how attribute keys are converted might help.

[jebeaudet]

you shouldn't really need these. Add opentelemetry-api if you need to do manual instrumentation.

I'm following the example here, my understanding is that only including the api is for librairies, a bit like slf4j is for logging. Unless I misunderstood something?

Agent and application are isolated from each other. The PropagatingAttributeKey inside you application is not visible for the extension that runs inside the agent and vice versa. To share code you'd need to place this class in boot loader or apply some kind of bridging that would allow converting the key between application and the agent code. Introducing api that would let extensions customize how attribute keys are converted might help.

Yes that's what I figured out and you now confirm that there's no easy way to obtain this, with the current setup of the java agent.

[laurit]

I'm following the example here, my understanding is that only including the api is for librairies, a bit like slf4j is for logging. Unless I misunderstood something?

You have misunderstood. That example talks about configuring an OpenTelemetry instance yourself, since you are using the one form the agent you won't need to include opentelemetry-sdk-extension-autoconfigure.

[jebeaudet]

You're right! It makes sense, my app is using the API and the implementation is provided by the java agent, therefore I don't need the autoconfigure nor the exporter since it's all taken care of by the agent itself.

Thanks again for your help, with the recent addition of querying parent/any/root span attributes in honeycomb, propagating attributes is less required, though the parent attribute is only 1 level so I would still like to have it! But I can manage, or do a allow list implementation based on the attribute key.