Further generalizing http_sd configuration for truly generic discovery

[mrnetops]

Looking at the projects that are jumping into http_sd, I'm seeing some divergence in how folks are using it that seems to be handled in rewrite rules, when I think we can converge for truly generic service discovery

fastly-exporter

• service ids for targets
• needing to reuse the http_sd_configs -> url because all of the services discovered are coming back to the same endpoint

relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: service
      - target_label: __address__
        replacement: 127.0.0.1:8080

prometheus-vcd-sd

• __meta_vcd_network_internal -> address relabel
• __meta_vcd_name -> instance

  relabel_configs:
  - source_labels: [__meta_vcd_network_internal]
    replacement: "${1}:9100"
    target_label: __address__
  - source_labels: [__meta_vcd_name]
    target_label: instance
    target_label: instance

suggestions

• param field in http_sd output, similar to labels with key:value parameters that are used for the url.
• either make target optional, where if not present a single element array consisting of the http_sd_configs -> url hostname:port combination is used, or have a similar effect with a dedicated keyword (url? http_sd_configs_url?)

Ideally we should have an interface that could be used generically without having to understand and map each individual http_sd implementation.

This would also hopefully be a step along the road to auto discovering service discovery endpoints, ala automatic service discovery on things like auto discovered kubernetes endpoints.

@roidelapluie ^

[roidelapluie]

Service discovery is mean to be generic, it can be used for intrumented monitoring and/or for external (prober) monitoring, but also for loki, parca and other downstreams. If there is business logic, relabel_configs are the way to go.

[mrnetops]

I think the point is that with some tweaks we can make it a generic entry point, wheras right now, it has pretty much immediately diverged where the http_sd endpoint needs to be individually understood per project (in some cases per project/per endpoint), and we can nip a lot of that in the bud early with some additional work on the interface provided.

To be clear, this is less about business logic relabeling, and more about most of the projects currently implementing it needing custom url construction, vs genericizing the url construction a bit more where they work out of the box without needing arbitrary relabel_configs.

Without being overly project specific, take fastly-exporter for example (where I started playing with this).
The suggestions I proposed would let its service discovery endpoint work out of the box without needing to implicitly understand the endpoint and doing custom relabel_configs

[
  {
    "params": {
      "target": "<service>"
    }
  },
  ...
]

Where

• a missing targets (top level field, I know, confusing with two targets) -> come back to the same endpoint
• params values -> add ?target=<service> to the url

[mrnetops]

I think perhaps the core of the issue is that http service discovery specifies targets as host when the http_sd endpoint needs the ability to specify (at least optionally) all of the url components for targets

• scheme
• host[:port]
• path
• query string

probably with a couple of meta components

• host[:port] supporting __meta_host to come back to the same host as http_sd_config -> url, to address natting or the like
• path supporting __meta_path & __meta_path_basename , to address proxy rewriting or the like

This can be done by adding additional string/array fields, or perhaps ideally by making the targets specification more flexible.

as an example for fastly-exporter

[
  {
    "targets": [ "__meta_host/metrics?target=<service>", ... ],
  }
]

[roidelapluie]

Fastly exporter and vcd sd are completely different. One is a "traditional SD" the other one is "blackbox exporter" type. They could expose __params_targets, instance, __metrics_path__, __address__ as labels, to make ready to use but I would not recommend it because that would be business logic.