Add secret provider interface under feature flag

[TheSpiritXIII]

Fixes: #8551
Partially fixes: prometheus/alertmanager#3108
Group Discussion: https://groups.google.com/g/prometheus-developers/c/WKOej_pnhXg
Initial Design Doc: https://docs.google.com/document/d/1EqHd2EwQxf9SYD8-gl3sgkwaU6A10GhiN7aw-2kx7NU/edit

Description

Interface

This PR adds a Provider interface, which is registered similarly to service discovery interfaces. There is a respective RegisterProvider which can let us register a provider within an init method.

Implementers implement Apply(configs []Config[yaml.Node]) ([]Secret, error). A sample config looks like this:

secrets:
- name: secret_a
  kubernetes_sp_config:
    api_server: "localhost:8080"
    secret:
      namespace: ns1
      name: secret1
      key: password

Your scrape config would look as follows:

scrape_config:
  job_name: job1
  basic_auth:
    username: admin
    password_ref: secret_a # use secret_a which loads from a Kubernetes secret.

Providers output a Secret which is just a func(ctx context.Context, node *yaml.Node) -> (string, error). This allows secrets to be fetched on-demand, or cached. Up to the implementer.

Usage

Before using this functionality, users must enable the "secret-providers" feature flag.

Default Providers

This PR only adds two default providers out of the box: inline and file, which allows users to configure their existing secrets with the new configuration, e.g. for BasicAuth replace password and password_file respectively.

secrets:
- name: secret_a
  inline: hunter2

Or:

secrets:
- name: secret_a
  file: password.txt

Maintainability

Test coverage here is ≥90 to ensure no breaking changes.

One note is that I've provided both yaml.v2 and yaml.v3 versions of the Configs object.

Pull Requests

1. Use common interface to fetch secrets in HTTP client config common#538
2. Add support for secret refs via a secret manager common#572
3. This one!
4. Kubernetes Secret Provider: [WIP] Add a native Kubernetes secret provider #13956

[bwplotka]

This is great, but I would add that once there is a need for yaml.v3 parsing of Prometheus config 🤔

[beorn7]

bwplotka requested a review from beorn7 last week

Generally, this feature looks good to me. For a serious review, I needed to get deeper into the subject, for which I won't have time anytime soon. I have a huge review backlog already, and we want to get native histograms stable for the Prom 3 release in September, so I'm afraid I cannot add more things to my place right now.