Custom http headers for tracing

[krmcbride]

What do you want to improve?

Similar to #4323 but for the tracing config -- I'm using a multitenant Grafana Tempo deployment which needs a X-Scope-OrgID header set with a tenant ID. I can connect to Grafana Mimir with the prometheus.custom_headers config:

spec:
  # ...
  external_services:
    prometheus:
      custom_headers:
        "X-Scope-OrgID": "<the tenant id>"

But there doesn't appear to be an equivalent for tracing/tempo.

What is the current behavior?

Requests to Tempo without the X-Scope-OrgID set result in a

HTTP/1.1 401 Unauthorized
Content-Type: text/plain; charset=utf-8

no org id

response, which causes a JSON unmarshalling error.

As a temp workaround I'm proxying requests from Kiali to Tempo through an nginx pod with

location ^~ /api {
  proxy_pass        https://my-grafana-tempo-url$request_uri;
  proxy_set_header  X-Scope-OrgID "<the tenant id>";
}

to set the correct tenant header, and trace info appears in Kiali as expected.

What is the new behavior?

Logically I'd expect there to be an optional spec.external_services.tracing.custom_headers map, same as for the prometheus client.

[jshaughn]

I think this makes sense. We support query_scope, but that adds tags to queries in order to narrow the queried data, when trace data is unified. This is different as it is adding headers.

I will add this to the backlog, @krmcbride, would you be interested in contributing this feature?

[jmazzitelli]

I did the original impl for the prom headers, so I can try to do this since its very similar

[jmazzitelli]

@krmcbride Do you want to test this locally to see if it addresses your issue? I uploaded the dev images on my quay repo:

• Server: quay.io/jmazzitelli/kiali:issue7266 -> https://quay.io/repository/jmazzitelli/kiali?tab=tags
• Operator (if you are using the operator to install): quay.io/jmazzitelli/kiali-operator:issue7266 -> https://quay.io/repository/jmazzitelli/kiali-operator?tab=tags

The new configuration should look like:

external_services:
  tracing:
    custom_headers:
      HeaderName1: HeaderValue1

[jmazzitelli]

The two PRs are ready for review. In my testing, I can see the custom headers get passed using kubeshark to examine the requests.

[krmcbride]

Thanks @jmazzitelli, tested your issue7266 images in a sandbox cluster with the expected tempo custom_headers and it works great 👍