Jaeger: DEV ES v8.x Compatibility

[Souheil-Yazji]

After correctly configuring the ES service thanks to Will, we ran into an issue where the Jaeger Collector, a component created by the operator, was running into the following error:
"msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): unknown key [template] in the template

After doing some research, it seems that the latest release of Jaeger (1.41) does not yet support ES v8.x+. There is a long-run issue open in the Jaeger repository for this.
Also supporting this discovery https://www.jaegertracing.io/docs/1.41/deployment/#elasticsearch, does not include 8.

For reference, our ElasticSearch instance is currently at v8.5.0

According to ES, it should be possible to integrate https://www.elastic.co/guide/en/apm/guide/current/jaeger-integration.html

[Jose-Matsuda]

Leaving a comment here with info I have synthesized from talking with Pat and Souheil. (if the two of you want to have comments leaving what you have learned go ahead but this will be my property to edit.

Attempting to continue to leverage Jaeger

• We tried to use cassandra's k8ssandra to use as the span-storage-backend but found that when trying to do this on dev we ran into various errors (one was permission) that we were unsure how it happened as Souheil was able to start it up in minikube in 10 minutes. The helm install via terraform had exceeded 40 minutes in trying to get something going but nothing.
• As said above, leveraging APM is a way forward; albeit a bit weird I feel as APM has it's own offerings that are similar to Jaeger. Note that one of the devs of APM has stated that the only benefit of using Jaeger in this case is if you're already using Jaeger -- that means you don't have to change your instrumentation when talking to a user who wanted to integrate an existing APM setup with a possible new Jaeger setup

Other options to Jaeger

Leveraging APM

• We could use APM which seems to have similar features to Jaeger, and really depends on what we need out of this (do we just want those fancy span graphs?)

Caveats

• Currently it does not seem like APM supports getting traces directly from istio. If we wanted to use those istio traces we would need the jaeger-agent in some form to consume said traces.
• If we don't want jaeger at all, we would need to inject an agent into the pods to allow APM to get data (since cant get it from istio).

Leveraging OTel and APM

OpenTelemetry is not an observability back-end like Jaeger or Prometheus. Instead, it supports exporting data to a variety of open source and commercial back-ends. which we can combine with APM. Using the opentelemetry operator we can set the mode to be in sidecar to inject into our pods.

Leveraging OTel and Jaeger

• From above, OTel seems like what would generate the data, so in our current setup it would be like the istio sidecar generating and sending traces. Once applications have the OTel library in them (idk how feasible that is)
  

• In the video "if you have a jaeger instrumentation library... you can just point it to the collector, and the collector would take advantage of that and send it to your destination"

• In the demo (also in the video screenshotted above) they present OpenTel with Jaeger + Elastic

• I don't think this solves the existing problem of needing a "span storage" area though...

Zipkin

Maybe I shouldn't be surprised but the storage component for zipkin is also elasticsearch (tested against es 6-7.x no action on this issue for ES 8.3.0) and cassandra

[Jose-Matsuda]

The OpenTelemetry and ES APM pivot (details to be edited in)

Rationale: Jaeger was proving difficult to work with in our environment, and the workarounds needed introduced more components than thought necessary. We are moving in favour of using Elastic APM which seems to provide similar functionality to Jager
Because APM does not support using istio traces to get spans, we need span data somehow by either injecting APM Agents or using OpenTelemetry agents
With talks of looking at OpenSearch as a path forward we decided on leveraging OpenTelemetry instead of digging too deep into Elastic which does support Otel. Do note however that envoy is looking to send Otel compatible data so possibly in the future we might be able to do away with whatever agent we add at current time of implementation.

Plan Moving forward (Souheil or Pat feel free to edit this section)

Following bullets are to be split into tasks

• Set up Elastic APM in dev #1512 - Set up for OTel
• Install the open-telemetry-operator in sidecar mode?
  • "How can I instrument applications without touching their code?". Further reading on OTel without manual instrumentation
  • Configuration for the collector
  • As it turns out OpenTelemetry's automatic instrumentation is only available for a few languages atm. May want to move to intstrumenting our Go apps with the Elastic APM agents while we possibly wait for envoy to send otel compatible data

[Jose-Matsuda]

Closing as completed