Why can't Prometheus scrape metrics with a custom HTTP header?

[aSemy]

I would like to scrape metrics from an HTTP endpoint that requires an API key. However, Prometheus does not allow this.

This feature was requested in 2016, but it was closed. I'm really confused by the reason, so I would like to find out why.

From what I can tell, it was closed because users might... use it? That's the point of a feature request though.

And also because it might be complicated? But then the alternative (set up a reverse proxy) is even more complicated! It's just an HTTP header, I shouldn't need to spend time learning about reverse proxies and nginx config and request forwarding.

It's fine if the issue was closed, but it's frustrating that the issue was locked, which shuts down the discussion and prevents sharing workarounds. One user replied with some ngix config, but this is not very thoroughly explained. There are a couple of answers on StackOverflow, but they don't provide a clear way of adding headers. It's bizarre how complicated such a simple request has been made so complicated, and has really killed my progress.

[bboreham]

Reading that discussion, I think the key statement was:

I think we'd need a stronger use case to justify adding this complexity.

Please supply details of your use-case.

such a simple request

Feel free to illustrate how simple it is with a PR.

[aSemy]

The details of my use-case are in the original post.

I would like to scrape metrics from an HTTP endpoint that requires an API key.

Feel free to illustrate how simple it is with a PR.

From what I can tell, a PR would be against the contributors guide, which states an comment on the issue is required. But since the Prometheus issue was closed, this is not possible.

Besides - the request is simple, but the implementation is not. Although I don't see how it could be difficult - Prometheus already allows defining HTTP headers, but the types are restricted. All that's needed is allowing the header key to be configurable.

[roidelapluie]

Prometheus has to be very careful about security and reliability. Every secret put in custom http header is a challenge. We do support basic auth and full customisation of the Authorization header (the credential type and the secret).

On the reliability side, custom HTTP headers could change the behaviour of the target in an unwanted way, and always having the same http headers sent makes it easier to debug them.

The last change we made was #8512 which was the flexibility around http tokens.

Without knowing more about your use case and why your targets can't support standard authentication mechanisms, it will be difficult to change the consensus.

[aSemy]

On the reliability side, custom HTTP headers could change the behaviour of the target in an unwanted way, and always having the same http headers sent makes it easier to debug them.

I'm really struggling to imagine how adding a header could affect the metrics that a Prometheus endpoint provides. Perhaps an example would help, if you're able to give one.

And even if it did, so what? Someone makes a mistake, and defines some invalid config? But a mistake is made more likely, because now I have to hack around and add additional systems which I'm unfamiliar with, and from my point of view leads to a more precarious situation.

Without knowing more about your use case and why your targets can't support standard authentication mechanisms, it will be difficult to change the consensus.

Is there something unclear about adding a header, with a custom key value? I'm not sure how else to explain it. In all HTTP clients I've used it's adding a header is a basic operation. The client may come with some utilities to add common headers, but a header is just a string key/value pair.

Here's an example of how to add a header with Curl.

curl https://my-service.com/actuator/prometheus   -H "The-Api-Key: password123"

I'm not able to do this in Prometheus because there is no option to add headers. Instead, in the locked issue #1724 the alternative suggested is run a reverse proxy, just to add a header. The example that a user suggested doesn't work for me, and so I have to dig around and to learn and debug ngnix and Docker configuration.

[roidelapluie]

HTTP header can tell the other side to close the connection instead of leaving it open, or could specify body length. When we manage Prometheus upstream, it's the kind of annoyances that makes it hard to support our users.

As I said, the current solution of fully customizing the authorization header seems to satisfy the vast majority of users.

I'd note that we are flexible where we can, enabling users to chose it they want http 2, which TLS version they need, etc. The positions from 5 years ago have moved in many cases, but so far you have not proven that your application could not switch to the standard http header Authorization.

[aSemy]

HTTP header can tell the other side to close the connection instead of leaving it open, or could specify body length. When we manage Prometheus upstream, it's the kind of annoyances that makes it hard to support our users.

These problems and others are made more likely, and harder to diagnose, by requiring workarounds like a reverse proxy just to add a header.

so far you have not proven that your application could not switch to the standard http header Authorization.

That is not possible because of organizational requirements and technological limitations. The application itself has no knowledge of the API key, this is defined somewhere in the network config. As the company might say, the API key header seems to satisfy the majority of users. I would consider switching to use a standard Authorization header if I could, but that's just not feasible.

Again, it's fine if Prometheus can't handle a custom HTTP header. But it's frustrating that the original issue was locked, because this prevents sharing of work-arounds.

[roidelapluie]

yeah, we lock old issues once they are closed. I am open to discuss custom http headers anyway, as after 2 year they did not cause issues in remote write/read. In that case, we also have a limitation on certain headers that can't be set. However, they are not considered secrets. Maybe we would need http_headers and secret_http_headers if we implement that, so users can decide if they want the header value to be printed on the configuration page or not.

[aSemy]

That sounds like a good plan 👍

[toxyl]

Any update on this? Just like the OP I find it very frustrating that I can't just set an "x-api-key" header.

[toxyl]

In my email inbox I saw a reply to this from @roidelapluie (thanks!) but I don't see it in this discussion, so if someone else struggles with this, this might help:

You can do it.

Use:

authorization:
  type: x-api-key
  credentials: thekey  

I have not yet tested if this works as expected, tho.

[roidelapluie]

I deleted it because it does not do what you want. It still deals with the standard header

[toxyl]

Ah, that explains it, hehe. This is the Authorization: xapikey thekey version then, right?

[roidelapluie]

yes

[roidelapluie]

Yes Le sam. 17 juin 2023, 18:04, Tox ***@***.***> a écrit :

…

Ah, that explains it, hehe. This is the Authorization: xapikey thekey version then, right? — Reply to this email directly, view it on GitHub <#11506 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACHHJXV7YYSIHK56PRLQBTXLXIP7ANCNFSM6AAAAAARTG62JA> . You are receiving this because you were mentioned.Message ID: ***@***.***>