HTTP 404 returned on Consul metrics endpoint

[obourdon]

What did you do?

Using the great Grafana Observability demo and more specifically
the version I derived from it, I came across an issue that I was unable
to fix in another way than this commit in my prometheus fork of tag v2.28.1

In fact, the issue comes from the fact that using a Consul registry for 'discovering' services targets there may be some issue when having to access metrics on a non standard path.

What I get in the prometheus targets screen is:

http://172.17.0.1:8500/v1/agent/metrics%3Fformat=prometheus => DOWN => server returned HTTP status 404 Not Found

If you look into this part of the Consul documentation, you see that it is stated that the Prometheus configuration should be adapted to such cases by adding:

metrics_path: '/v1/agent/metrics'
params:
  format: ['prometheus']

however, considering my current knowledge and proficiency in both Consul and Prometheus, I was not able to make this work with my own dynamic config below.

What confused me in the first place and therefore led to lots of time spent trying to understand why it worked in Netdata case and not in Consul one can be summarized as follows:

# From within the prometheus docker container:
/prometheus $ wget -q -O - 172.17.0.1:19999/api/v1/allmetrics?format=prometheus | wc -l
1831
/prometheus $ wget -q -O - 172.17.0.1:19999/api/v1/allmetrics%3Fformat=prometheus | wc -l
1831
/prometheus $ wget -q -O - 172.17.0.1:8500//v1/agent/metrics?format=prometheus | wc -l
1225
/prometheus $ wget -q -O - 172.17.0.1:8500//v1/agent/metrics%3Fformat=prometheus | wc -l
wget: server returned error: HTTP/1.1 404 Not Found
0

meaning that Netdata is more clever than Consul in the sense that it supports both standard URL with the query part but also the encoded one with %3F

What did you expect to see?

What did you see instead? Under which circumstances?

Environment
see details above in the various repositories

VAGRANT_VAGRANTFILE=Vagrantfile.flatcar vagrant up

• System information:

I am pretty much sure that any linux based system would trigger the same behaviour

• Prometheus version:

v2.28.1 (but surely later versions also exhibit this behaviour)

• Prometheus configuration file:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'self'
    consul_sd_configs:
      - server: '172.17.0.1:8500'
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: '.*,prometheus,.*'
        action: keep
      - source_labels: [__meta_consul_service]
        regex: '.*-sidecar-proxy'
        action: drop
      - source_labels: [__meta_consul_service]
        target_label: 'job'
      - source_labels: [__meta_consul_node]
        target_label:  'instance'
      - source_labels: [__meta_consul_service_metadata_external_source]
        target_label: 'source'
      - source_labels: [__meta_consul_service_id]
        regex: '_nomad-task-([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})-.*'
        target_label:  'task_id'
        replacement: '$1'
      - source_labels: [__meta_consul_tags]
        regex: '.*,(app|monitoring|demo),.*'
        target_label:  'group'
        replacement:   '$1'
      - source_labels: [__meta_consul_tags]
        regex:         '.*,metrics_path=([^,]*),.*'
        target_label:  '__metrics_path__'
        replacement:   '$1'
      - source_labels: [__meta_consul_tags]
        regex:         '.*,source=([^,]*),.*'
        target_label:  'source'
        replacement:   '$1'

• Logs:

level=debug ts=2021-09-30T08:37:41.268Z caller=scrape.go:1236 component="scrape manager" scrape_pool=self target="http://172.17.0.1:8500/v1/agent/metrics%3Fformat=prometheus" msg="Scrape failed" err="server returned HTTP status 404 Not Found"
...

• Other interesting configuration files/excerpts:

# Consul self registration as a service for Prometheus scraping
# /etc/consul.d/consul-service.json
{
  "service": {
    "name": "consul-service",
    "address": "172.17.0.1",
    "port": 8500,
    "tags": ["monitoring","prometheus","source=system","metrics_path=/v1/agent/metrics?format=prometheus"],
    "checks": [
      {
        "http":     "http://172.17.0.1:8500/",
        "interval": "10s"
      }
    ]
  }
}

• Other interesting thoughts:
  I also fully understand that my fix could also lead to some security issues (URL decoding and reinjection without any checks) and was also considering implementing just a single & simple string replacement of first %3F occurence by ? which would
  do the trick without exposing too much to vulnerabilities

[roidelapluie]

You can use:

    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: '.*,metrics_path=([^,?]*)[?,].*'
        target_label:  '__metrics_path__'
        replacement: '$1'
      - source_labels: [__meta_consul_tags]
        regex: '.*,metrics_path=([^,?]*)\?format=([^,]*),.*'
        target_label:  '__param_format'
        replacement: '$2'

[obourdon]

@roidelapluie I was hoping for such kind of answer but was not able to derive if from either official docs & others issues or alike
will try this early week after next for verification. Many thanks