Migrating Jaeger tracing from OpenTracing to OpenTelemetry; Add OTLP exporter #5411

metonymic-smokey · 2022-06-09T11:40:30Z

part of fixing #1972

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

This PR provides two changes that brings us closer to closing the gap between migrating from OpenTracing to OpenTelemetry, in particular:
a) This moves Jaeger client from OpenTracing to OpenTelemetry, without breaking changes. This means users can keep using their existing tracing configuration (some options will now not have any effect, but these are the ones that are obsolete after move to OTEL; read more in the updated documentation in the PR)
b) This adds the OTLP exporter, meaning now users will be able to leverage this protocol as well, with support of both gRPC and HTTP client.

Verification

Added E2E tests to validate Jaeger tracing
Manual tests

pkg/tracing/jaeger/jaeger.go

metonymic-smokey · 2022-06-12T13:55:05Z

@matej-g could you please review my current progress, when you're free.

I've created a function which returns an OTLP exporter but I'm not sure how to use the exporter.
Working on E2E testing and pushed a draft commit for it, but I'll probably take some time for it.

TODO:

E2E:
todo - Fix E2E draft.
Jaeger:
Add more options from config.
Unit tests(similar to google_cloud.go)
OTLP Package:
Test locally.
Update docs with example config for OTLP package.

Thanks!

pkg/tracing/jaeger/jaeger.go

pkg/tracing/client/factory.go

matej-g · 2022-06-15T14:14:19Z

pkg/tracing/client/factory.go

 )

 type TracingConfig struct {
 	Type   TracingProvider `yaml:"type"`
 	Config interface{}     `yaml:"config"`
 }

+// NewOTELTracer returns an OTLP exporter based tracer.
+func NewOTELTracer(ctx context.Context, logger log.Logger) trace.Tracer {


So my thought was to simply add a new TracingProvider const to the list we have above, simply named OTLP. It would be yet another package under tracing directory, just as we have jaeger, google_cloud etc.

Then, an OTLP exporter should be simply one of the switch cases in the factory function below. We should also probably take some config from YAML, same we do for other tracing provider.

I have created a new package otlp in a recent commit with some options for configuration. I will try to add some more config options if possible.
Please let me know what you think.

matej-g · 2022-06-15T14:16:59Z

Thank you @metonymic-smokey! I think this is going in the right direction, I provided some feedback which will hopefully unblock you (if not feel free to ask for more details).

As for checking traces in E2E test, we could simply do a HTTP request to Jaeger and parse the reply for the trace data we expect. You can take a look at an example from our other project here: https://github.com/observatorium/api/blob/main/test/e2e/traces_test.go#L195

pkg/tracing/client/factory.go

pkg/tracing/jaeger/jaeger.go

matej-g

Nice progress! 💪 I added a few more hints for OTLP exporter.

pkg/tracing/otlp/otlp.go

test/e2e/tracing_test.go

pkg/tracing/jaeger/jaeger.go

pkg/tracing/otlp/otlp.go

test/e2e/e2ethanos/services.go

test/e2e/tracing_test.go

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

Signed-off-by: Matej Gera <matejgera@gmail.com>

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

metonymic-smokey · 2022-10-01T12:28:29Z

@GiedriusS @matej-g i've updated the E2E tests and looks like they all pass :)
Thank you so much for working with me on this one!

GiedriusS

…exporter (thanos-io#5411) * draft: standalone OTLP tracer Signed-off-by: Aditi Ahuja <aditi.ahuja@couchbase.com> * added jaeger exporter Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: tracing test Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * minor refactor and cleanup Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * created package tracing/otlp Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * minor refactor Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: added some more config options for jaeger Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: extended e2e test Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: added OTLP GRPC options Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * minor fixes; temp prints Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * refactor: removed OT code, changed tag parsing to OTel tags Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * jaeger refactor; minor updates Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * jaeger: added some more config options Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * otlp: added retry options Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * organizing and pruning Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * added TLS config for OTLP Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: temp test fix for CI Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * added unit tests for jaeger Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: added sampler types to jaeger Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * replaced objstore with exthttp Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: test fixes Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * added jaeger remote sampler Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: reverted E2E test fix Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * lint and doc test fixes Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * minor refactor for OTLP Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * jaeger: modified sampler selection Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * review fixes Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: jaeger rate limiting sampler added Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * added OTLP unit tests; minor refactor Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * minor review fixes Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: rate limiting sampler implementation Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * review fixes Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * removed OT tracer struct Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * Use service name in resource Signed-off-by: Matej Gera <matejgera@gmail.com> * Fix tracing E2E test Signed-off-by: Matej Gera <matejgera@gmail.com> * Adjust trace ID extraction for exemplars Signed-off-by: Matej Gera <matejgera@gmail.com> * Add OTLP docs and adjust Signed-off-by: Matej Gera <matejgera@gmail.com> * Add CHANGELOG Signed-off-by: Matej Gera <matejgera@gmail.com> * review fixes Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> * draft: updated E2E tests Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> Signed-off-by: Aditi Ahuja <aditi.ahuja@couchbase.com> Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com> Signed-off-by: Matej Gera <matejgera@gmail.com> Co-authored-by: Matej Gera <matejgera@gmail.com> Signed-off-by: utukj <utukphd@gmail.com>

pull-request-size bot added the size/M label Jun 9, 2022

metonymic-smokey force-pushed the otlp branch from 684b52a to 6691f1e Compare June 9, 2022 11:56

sonatype-lift bot reviewed Jun 12, 2022

View reviewed changes

pkg/tracing/jaeger/jaeger.go Outdated Show resolved Hide resolved

metonymic-smokey force-pushed the otlp branch from b73bddd to ac38ca2 Compare June 12, 2022 05:47

pull-request-size bot added size/L and removed size/M labels Jun 12, 2022

metonymic-smokey force-pushed the otlp branch 3 times, most recently from 7a436e5 to 16d3134 Compare June 12, 2022 10:17

metonymic-smokey marked this pull request as ready for review June 12, 2022 11:15

metonymic-smokey marked this pull request as draft June 12, 2022 11:24

metonymic-smokey force-pushed the otlp branch from efd413d to 3e225b5 Compare June 12, 2022 14:26

matej-g reviewed Jun 15, 2022

View reviewed changes

metonymic-smokey marked this pull request as ready for review June 16, 2022 14:57

metonymic-smokey force-pushed the otlp branch 2 times, most recently from 73f9ab2 to 1a8134b Compare June 16, 2022 15:04

sonatype-lift bot reviewed Jun 16, 2022

View reviewed changes

pkg/tracing/client/factory.go Outdated Show resolved Hide resolved

metonymic-smokey force-pushed the otlp branch 2 times, most recently from e9e42d2 to 96d2ccb Compare June 17, 2022 15:40

sonatype-lift bot reviewed Jun 19, 2022

View reviewed changes

pkg/tracing/jaeger/jaeger.go Outdated Show resolved Hide resolved

metonymic-smokey force-pushed the otlp branch from 14bbd28 to 5d0cb36 Compare June 20, 2022 03:00

matej-g reviewed Jun 20, 2022

View reviewed changes

pkg/tracing/otlp/otlp.go Outdated Show resolved Hide resolved

pkg/tracing/otlp/otlp.go Outdated Show resolved Hide resolved

metonymic-smokey force-pushed the otlp branch 4 times, most recently from 1292434 to d9572bd Compare June 22, 2022 16:01

metonymic-smokey commented Jun 23, 2022

View reviewed changes

test/e2e/tracing_test.go Show resolved Hide resolved

matej-g reviewed Jun 24, 2022

View reviewed changes

metonymic-smokey and others added 20 commits October 1, 2022 17:00

replaced objstore with exthttp

c11294f

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

draft: test fixes

57eb264

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

added jaeger remote sampler

ab1ad19

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

draft: reverted E2E test fix

565fc67

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

lint and doc test fixes

7b57c0e

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

minor refactor for OTLP

8e15037

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

jaeger: modified sampler selection

97cd0f5

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

review fixes

02367ea

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

draft: jaeger rate limiting sampler added

d29cbec

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

added OTLP unit tests; minor refactor

0913eb3

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

minor review fixes

200845f

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

draft: rate limiting sampler implementation

bab3fbb

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

review fixes

b9ab731

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

removed OT tracer struct

83d005a

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

Use service name in resource

5aa3ea6

Signed-off-by: Matej Gera <matejgera@gmail.com>

Fix tracing E2E test

8ccb9e6

Signed-off-by: Matej Gera <matejgera@gmail.com>

Adjust trace ID extraction for exemplars

ed724e0

Signed-off-by: Matej Gera <matejgera@gmail.com>

Add OTLP docs and adjust

1984fd6

Signed-off-by: Matej Gera <matejgera@gmail.com>

Add CHANGELOG

bfac872

Signed-off-by: Matej Gera <matejgera@gmail.com>

review fixes

3d04e76

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

metonymic-smokey force-pushed the otlp branch 2 times, most recently from d0b558c to bf311ad Compare October 1, 2022 11:53

draft: updated E2E tests

ed85390

Signed-off-by: Aditi Ahuja <ahuja.aditi@gmail.com>

metonymic-smokey force-pushed the otlp branch from bf311ad to ed85390 Compare October 1, 2022 12:03

GiedriusS approved these changes Oct 2, 2022

View reviewed changes

GiedriusS merged commit 64e7e3b into thanos-io:main Oct 2, 2022

alanprot mentioned this pull request Oct 12, 2022

Update Prometheus to v0.39.1 and Thanos to main (a4e334152549) cortexproject/cortex#4903

Merged

ahurtaud mentioned this pull request Nov 7, 2022

query: v0.29.0. Opentracing config create gopanic SIGSEGV #5872

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrating Jaeger tracing from OpenTracing to OpenTelemetry; Add OTLP exporter #5411

Migrating Jaeger tracing from OpenTracing to OpenTelemetry; Add OTLP exporter #5411

metonymic-smokey commented Jun 9, 2022 •

edited by matej-g

metonymic-smokey commented Jun 12, 2022 •

edited

matej-g Jun 15, 2022

metonymic-smokey Jun 20, 2022

matej-g commented Jun 15, 2022

matej-g left a comment

metonymic-smokey commented Oct 1, 2022

GiedriusS left a comment

Migrating Jaeger tracing from OpenTracing to OpenTelemetry; Add OTLP exporter #5411

Migrating Jaeger tracing from OpenTracing to OpenTelemetry; Add OTLP exporter #5411

Conversation

metonymic-smokey commented Jun 9, 2022 • edited by matej-g

Changes

Verification

metonymic-smokey commented Jun 12, 2022 • edited

matej-g Jun 15, 2022

Choose a reason for hiding this comment

metonymic-smokey Jun 20, 2022

Choose a reason for hiding this comment

matej-g commented Jun 15, 2022

matej-g left a comment

Choose a reason for hiding this comment

metonymic-smokey commented Oct 1, 2022

GiedriusS left a comment

Choose a reason for hiding this comment

metonymic-smokey commented Jun 9, 2022 •

edited by matej-g

metonymic-smokey commented Jun 12, 2022 •

edited