Prometheus special labels

[danielerepici]

Hello,

I have been working with Prometheus configuration for some time, which involved file-based service discovery, special labels such as __metrics_path__, __scheme__, __scrape_interval__ and so on, and I think I built up a fair amount of understanding about how all of these things wire up altogether. The problem is that the only evidence I have of my understanding is the fact that my system works as intended, but I could not find any validation in the official Prometheus documentation, so I came here hoping that some "insider" can confirm (or reject) what I have understood.

For context, I am configuring my monitoring system to only work upon file-based service discovery. So in my Prometheus configuration I only have one single Scrape Configuration. Any job will be passed through file. Unfortunately, all you can put into a job configuration is a targets and a labels element. Therefore, I was looking for a way to be able to customize stuff like metrics path, scheme, scrape interval in my job configuration files to be auto-discovered. That's when I bumped into the special labels.

Now that the context is clear, here's my findings. Thanks to this specific Prometheus documentation bit, I was able to reconstruct the timeline of the relevant steps that Prometheus goes through just before scraping.

1. Before doing anything, it adds the following labels:
   1. It sets the job label to the value of the scrape config's job_name element
   2. It sets the __address__ label to <host>:<port> address of the target.
2. relabel_configs is evaluated, so some labels are turned into other new labels
3. The next following labels are added now:
   1. instance set to __address__
   2. __metrics_path__ set to scrape config's metrics_path element
   3. __scheme__ set to scrape config's scheme element
   4. __param_<name> set to the value of the URL parameter
   5. __scrape_interval__ set to scrape config's scrape_interval element
   6. __scrape_timeout__ set to scrape config's scrape_timeout element
4. Scraping now happens
5. After scraping, metric_relabel_configs and write_relabel_configs are evaluated but this is not relevant in my context so you can ignore that.
6. Eventually, all labels starting with __ are dropped and not shown downstream

Ok, all of the above can be easily found in the documentation. But that wouldn't actually help me solving my problem, if I didn't find out the following "insight knowledge", which I could not find anywhere in the official documentation.

Finding 1: Prometheus behaviour is not driven by the configuration elements. It is driven by the special labels!

I'm not sure if this should be obvious by the documentation, but I wasn't able to find any mentioning of this anywhere. But once I started suspecting this, all that behaviour I described above of Prometheus setting these special labels just before scraping started making a lot of sense finally. Why would otherwise Prometheus bother setting these special labels if its behaviour was then just driven by the original elements (from which those special labels were populated)? So I came to the conclusion that Prometheus behaviour is actually driven by these special labels, and not by the original elements themselves (by original elements I mean the job_name, metrics_path, scrape_interval etc elements defined in the Scrape Configuration). So it's something like this:

[original elements] --determine--> [special labels] --drive--> [Prometheus scraping behaviour]

Q&A Is my understanding correct? Is it just me or is this not documented anywhere?

Finding 2: At any stage, if Prometheus is about to set a special label, and finds that that special label already happens to have a value, then it doesn't touch it (and leaves the value as-is)!

This was very important to solve my problem! Suppose I have defined my file-based discoverable configuration as the following:

[
  {
    "targets": [ "myhost:1234" ],
    "labels": {
      "job": "my-specific-job",
      "__metrics_path__": "/api/actuator/prometheus",
      "__scheme__": "https",
      "__scrape_interval__": "30s"
    }
  }
]

Then, referring to my timeline at the top, during step 1, Prometheus would set the job label to whatever the scrape config's job_name element is (e.g. "file-sd"), but it won't! It won't because it finds that a job label has already been set. In fact, among my Prometheus targets, I can see the job label has value "my-specific-job" and not "file-sd".

During step 3, Prometheus would set the __metrics_path__ label to whatever the scrape config's metrics_path element is (or the default), but it won't! It won't because it finds that a __metrics_path__ label already has a value. In fact, in my case this is correctly scraping from /api/actuator/prometheus and not from /metrics.

All these tests I did are also evidence of my "Finding 1", that is Prometheus behaviour is driven by the special labels.

Q&A (same questions) Is my understanding correct? Is it just me or is this not documented anywhere?

[danielerepici]

Turns out both my findings are correct (sort of).

• Finding 1: correct. The resulting labels after relabel_configs turn into a per-target scrape loop configuration item.
• Finding 2: (sort of) correct. Special labels are overwritten by discovery's data.

Also yes, the documentation about this can definitely be improved.

Source: I had a chat with @SuperQ in the IRC channel who kindly answered my questions. (Feel free to chip in if I've missed anything).