New link creates second ghost link

[SgtSchultz02]

I'm using open-telemetry for Java version 1.36.0 and creating a new Span (using SpanBuilder) and adding a single link to a related span. That just one link was added via the API is verified by some logging. The link creation works and we get the link to the related traceId and spanId. We also get a link to the current traceId with a spanId of all 0's. (Viewed in Grafana)

Expected:

trace (id=1)

span (id=15)
link (related - traceId=2, spanId = 20)
Actual:

trace (id=1)

span (id=15)
link (related - traceId=1, spanId = 00...) - not a valid spanId
link (related - traceId=2, spanId = 20)

Is this a unique instance of a problem or are other people seeing this?

Java version - 17

    SpanBuilder spanBuilder =tracer.spanBuilder(spanName)
                     .setSpanKind(kind)
                     .setNoParent();
                     
    if (linkedSpans != null && !linkedSpans.isEmpty()) {
        for (Span linked:  linkedSpans) {
            log.debug("Adding link to span {} - {}", spanName, linked.getSpanContext().getSpanId());
            spanBuilder.addLink(linked.getSpanContext());
        }
    }

    io.opentelemetry.api.trace.Span oSpan = spanBuilder.startSpan();

Expectation: A single link within the new span

Tried: updating to the latest version of open-telemetry, logging to ensure a link was only added once. Also logged the toString of the new span after spanbuilder.startSpan() and the totalRecordedLinks=1.

[jkwatson]

Probably we need a bit more information here. Where are you exporting spans to? How are they getting exported? What versions of the OTel libraries are you using?

[SgtSchultz02]

1. Exporting to : collector version 0.94.0
2. exporting spans using io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter / io.opentelemetry.sdk.trace.export.BatchSpanProcessor
3. open-telemetry for Java version 1.36.0

[SgtSchultz02]

Also ... thanks for the quick response.

[jack-berg]

That just one link was added via the API is verified by some logging. The link creation works and we get the link to the related traceId and spanId. We also get a link to the current traceId with a spanId of all 0's

Hmm.. we have an integration test that creates a span with a link, and verifies that after a round trip (java app w/ sdk -> otel collector -> java app) the link is as expected.

I wonder if grafana or your backend is doing something wrong with the link. Could you run the same scenario, but use the OtlpJsonLoggingSpanExporter (i.e. logging-otlp if using autoconfigure)? The logging otlp exporter will log the full OTLP JSON encoding of the spans, so we'll be able to know if its a problem with the SDK, or elsewhere.

[jack-berg]

The span in question is one of three scoped spans in the log generated by OltpJsonLoggingSpanExporter.

The OtlpJsonLoggingSpanExporter prints out the full OTLP representation of the spans, including all span links. Do you see span links as expected in the log output?

[SgtSchultz02]

Yes. The log is an accurate representation of what we want.

[jack-berg]

Then I have to conclude that the SDK is behaving correctly, and there is some sort of bug in either the back end you are sending data to or in the visualization layer.

The OTLP exporters use the same serialization logic as the OtlpJsonLoggingSpanExporter.

[SgtSchultz02]

What are our next steps?

[jack-berg]

Investigate how your backend / visualization layer's supports links, and consider opening an issue with the appropriate group. It appears to be outside of the bounds of what this repository controls.