Allow setting custom HTTP headers in otelhttp trace exporter.

[lukasbindreiter]

Hi,

we are using SpiceDB which in turn uses this library for configuration.
In all our other services we are exporting our open telemetry traces to https://axiom.co/. We would also like to do the same for SpiceDB.
Axiom offers an otel compatible http endpoint for traces, which we theoretically could use directly to accept traces configured using the parameters and environment configuration options provided by cobrautil/cobraotel.
https://axiom.co/docs/send-data/opentelemetry

The only issue is right now that each request to that endpoint requires some custom headers:

X-Axiom-Dataset=<your axiom dataset to ingest the traces to>
X-Axiom-Org-Id=<your axiom org>
Authorization=Bearer <api key>

I know that we could also go the route of adding an opentelemetry collector to our deployment, and sending traces to that and then forwarding it to axiom. But currently our deployment is still rather small in scale, so this step is not yet really warranted.

Plus just for getting started quickly that anyways would always be a nice feature to have I assume.

Implementation wise this could look something like this:

headers := make(map[string]string)
// these could probably be parsed then from a newly created 'otel-endpoint-headers' parameter?
headers["CUSTOM-HEADER-1"] = "custom-value"
opts = append(opts, otlptracehttp.WithHeaders(headers))

// already like this in the code
exporter, err = otlptrace.New(context.Background(), otlptracehttp.NewClient(opts...))

(here in the code: https://github.com/jzelinskie/cobrautil/blob/main/cobraotel/cobraotel.go#L127)

What do you think, does adding support to custom HTTP headers make sense in your opinion?
If you think so, I could probably also contribute the implementation as a PR 😄

[jzelinskie]

I'm not opposed to adding headers to OTEL, but I am opposed to adding it in a bespoke way.
Let's find some examples of how other programs handle this and/or any documentation from OpenTelemetry folks as to a standard environment variable etc...

[lukasbindreiter]

I'm not opposed to adding headers to OTEL, but I am opposed to adding it in a bespoke way. Let's find some examples of how other programs handle this and/or any documentation from OpenTelemetry folks as to a standard environment variable etc...

Opentelemetry Collector Exporters are configured using this config struct and in this case the Headers map.

An example configuration for that in YAML can be found here:
https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp#client-configuration

I've done some digging on how this headers map can be configured in various OTEL SDKs, and for example in the official Javascript SDK this is possible using the OTEL_EXPORTER_OTLP_HEADERS environment variable.

https://github.com/open-telemetry/opentelemetry-js/blob/5608bba35957f59a66ac224e5bf12267b2cddc86/packages/opentelemetry-core/src/utils/environment.ts#L102

I'm not aware of any CLI applications to use as an example directly from OTEL, but I think this does show enough precedence that a feature like this is for sure warranted.