Alertmanager secrets

[OfficerKoo]

How are we supposed to deploy Alertmanager secrets(OpsGenie API keys for example), if they are stored in plaintext in config file.
Vanilla Alertmanager allows you to store secrets in a file, but this is disabled in Mimir Alertmanager API, and there is no other way to upload tenant configurations.

Is there some best practise to do this?
I really don't want to add some templating engine to a pipeline, or inject header with reverse proxy.

DEBU[0001] checking response                             status="400 Bad Request"
ERRO[0001] response                                      body="error validating Alertmanager config: setting smtp_auth_password_file, password_file, bearer_token_file, auth_password_file or credentials_file is not allowed\n" status="400 Bad Request"
mimirtool: error: POST request to https://mimir.internal.infra.loveos.io/api/v1/alerts failed: server returned HTTP status: 400 Bad Request, body: "error validating Alertmanager config: setting smtp_auth_password_file, password_file, bearer_token_file, auth_password_file or credentials_file is not allowed\n", try --help

[dimitarvdimitrov]

You can use environment variables inside the configuration.

"Injecting credentials" shows how to do it for the Helm chart.

And "Use environment variables in the configuration" is a general paragraph about how environment variables in the config work.

[dimitarvdimitrov]

I see. I think I misunderstood the original issue.

We can possibly expand environment variables in alertmanager tenant configuration as well (i.e. replace $SECRET_VARIABLE in the user config with password123). One possible complication with that is that notification templates can reference the labels from alerts via $labels (or have other variables referenced via $).

In those cases the alertmanager shouldn't attempt to replace variables.

[hobbsh]

Is there any update to this workflow? It seems like configs uploaded via mimirtool do not expand variables in the config, despite their presence in the environment of the container.

[lblazewski]

Hey, I would like to bump this since I have stumbled upon this issue recently. I think this is a serious limitation and secrets support should be added to the alertmanager config parsing.

[lukasmrtvy]

@lblazewski there's no issue created afaik, maybe this would be a good start

[taobojlen]

I created a feature request here: #8259