Replace Prometheus with OpenTelemetry

[sagikazarmark]

Is your feature request related to a problem?

OpenTelemetry will be ready soon (it's already in RC).

Describe the solution you'd like to see

We should replace the Prometheus lib with OpenTelemetry.

We should probably keep the OpenMetrics export, but we should add an (configurable) OpenTelemetry agent target as well.

[nabokihms]

There are a few words about OpenTelemetry Prometheus compatible exporter.

I managed to replace our current implementation of metrics endpoint with the combination of an otel-prometheus-exporter and a handler wrapper. It seems like both metrics exporter and traces exporter works fine. During my test, I looked through traces via stdout exporter.

@sagikazarmark If it is ok for you, I would like to be in charge of this issue.

The plan is:

• Replace prometheus with otel without losing observability
• Write a (design?) doc about traces exporting
• Add ability to export traces to jaeger
• Add ability to export traces to otl-collector

[sagikazarmark]

Sure, feel free to take it. A couple thoughts:

• I think the otel collector protocol for traces should be enough. AFAIK Jaeger will support it natively.
• I would also implement metrics for the otel collector, meaning that metrics should be collected by both a pull and a push collector. See Combine the Push and Pull metric controllers open-telemetry/opentelemetry-go#1378 for details.

Honestly, I'd start with a minimum replacement of the prometheus lib with open telemetry and go from there.

[nabokihms]

@sagikazarmark played with it a little last weekend.

Some concerns/problems I have faced:

• GRPC version in dex is v1.26.0, but for otel instrumentation, it is required to be around v1.35.0
• It looks impossible to count errors with otel net/http instrumentation. This method's code shows that we can propagate a "labeler" to add custom labels. We can extract info from the request (such as method name) and add it to labels, but unfortunately, we can't add status code or other things from the response.

[sagikazarmark]

* GRPC version in dex is v1.26.0, but for otel instrumentation, it is required to be around [v1.35.0](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/v0.18.0/instrumentation/google.golang.org/grpc/otelgrpc/go.mod#L15)

Once we can upgrade etcd to 3.5, we can upgrade gRPC as well.

* It looks impossible to count errors with otel net/http instrumentation. [This method's code](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/v0.18.0/instrumentation/net/http/otelhttp/handler.go#L164) shows that we can propagate a "labeler" to add custom labels. We can extract info from the request (such as method name) and add it to labels, but unfortunately, we can't add status code or other things from the response.

Metrics is probably not yet fully ready.

[nabokihms]

I opened the draft, but there are still some problems.

• No metrics for grpc
• New metric names (breaking change)
• I opened a PR to add total request counters with error codes, but there is no feedback yethttps://github.com/net/http: add requests count metric with status code label open-telemetry/opentelemetry-go-contrib#771

[sagikazarmark]

One possible workaround is to use OpenCensus with the available bridge.

[sagikazarmark]

Now that the status code label thing is resolved, we should probably revive this topic as well.