Support basic_auth_users in web configuration for Prometheus and Alertmanager

[jack1902]

What did you do?

Looked at previous issues but basic-auth flags a few but they all appear to be prometheus -> endpoint rather than say grafana -> prometheus.

I'm looking to setup prometheus per cluster i have and then have a single grafana instance hit the prometheuses as seperate data-sources. In order for me to feel comfortable about exposing prometheus apis externally (even behind a whitelist) i would ideally configure basic-auth on prometheus itself. See https://prometheus.io/docs/guides/basic-auth/

Did you expect to see some different?

I was expecting to see something around setting a user/password combo or list of usernames passwords for an instance of prometheus.

Anything else we need to know?:

Documentation that states its possible to enable basic auth on prometheus itself (again not endpoints prometheus hits). https://prometheus.io/docs/guides/basic-auth/

I can also see in Grafana that it is possible to configure basic auth for the prometheus data source:

[fpetkovski]

Currently the prometheus operator cannot configure basic auth for the prometheus HTTP endpoints. There are two options I see

• Configure mutual TLS: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#webtlsconfig
• Configure basic auth (or any other auth) at the ingress level. An nginx ingress is pretty flexible in this regard.

[jack1902]

Configuring at the ingress level will work for now but ideally a solution directly on prometheus would be ideal. NGINX does unblock me for this but if its possible to extend the prometheus object to include some form of basic auth (pulling the password from a secret) that would be great.

The TLS side of things would again ideally be handled by an ingress as this is a common setup for most applications.

Thanks for the help though, let me know if it would be possible to extend the specs to include basicAuth for prometheus and i might try and contribute it back (feels like something might be needed at the docker command level unless it just sits inside the larger config file)

[fpetkovski]

We are always looking for contributors and if you are interested in implementing the feature, I'd be more than happy to help with reviewing PR and answering any questions you might have.

As you indicated, we need to extend the web spec to accept basic auth configuration. There is already a BasicAuth type which can be reused: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#basicauth

[simonpasquier]

Also related to #4274

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[heylongdacoder]

I am working on it.

[a0s]

I am using kube-prometheus-stack. I created a new secret custom-prometheus-web-config as described in https://prometheus.io/docs/guides/basic-auth/#creating-web-yml. Now i need a way how to mount this secret instead of default prometheus-kube-prometheus-stack-prometheus-web-config

[heylongdacoder]

@a0s If I am not mistaken, currently prometheus-operator don't support the mounting of a custom web-config.

[nbjohnson]

Any movement on this? I would like to be able to set basic_auth directly in prometheus. If there is no immediate plan to support basic_auth_users in the web config, can it be made possible to override the web.config.file option either with additionalArgs or with specifying an existing secret? Right now even with prometheus supporting basic_auth (experimentally) there is no way to enable it when using the operator

[simonpasquier]

@nbjohnson it should be possible to provide your own web config file via .spec.secrets and leverage .spec.additionalArgs to inject the --web.config.file argument.

[nicolastakashi]

If I'm not wrong me and @ArthurSens last week went over an issue about basic auth and prometheus.
Seems when it's enabled the probes will fail because the healthy and ready path are requiring auth as well and there's no way to configure basic auth for probes in k8s.

[heylongdacoder]

prometheus/exporter-toolkit#151, someone is working hard on this 😄

[nbjohnson]

I guess it makes sense you are waiting on the ability to exclude a health endpoint from the auth as that is the cleaner method, but I thought I saw it was possible to set httpHeaders for those probes to be able to pass an auth header. I haven't tested it but I thought I saw that was possible. Now it is definitely not ideal having the auth in your deployment and the ability to exclude a health endpoint is better but might be a possible workaround in the meantime

[simonpasquier]

I wouldn't set HTTP headers in the probe config because it would expose sensitive information in clear-text. Instead I agree that Prometheus should allow /-/ready and /-/healthy endpoints without authentication.

[mingmingshiliyu]

prometheus/exporter-toolkit#151, someone is working hard on this 😄

dude,is there any progress on prometheus operator basic_auth? i saw you committed some code,was that workable for this issue?

[tomikonio]

@nbjohnson it should be possible to provide your own web config file via .spec.secrets and leverage .spec.additionalArgs to inject the --web.config.file argument.

Hi @simonpasquier I tried to specify the above, but the operator fails to create a prometheus statefulset with the following error:

level=error ts=2024-04-24T10:41:33.678620916Z caller=klog.go:126 component=k8s_client_runtime func=ErrorDepth msg="sync \"monitoring/prometheus-kube-prometheus-prometheus\" failed: making statefulset failed: make StatefulSet spec: can't set arguments which are already managed by the operator: web.config.file"

can't set arguments which are already managed by the operator: web.config.file
Is there a way to remove the operator managed argument?

[simonpasquier]

Is there a way to remove the operator managed argument?

unfortunately not.