Prometheus continuously re-discovers all existing targets.

[darkl0rd]

What did you do?

Setup Prometheus with Docker Swarm Service Discovery using both Swarm basic filters and the workaround/solution provided by Brian Brazil in issue #8073 to ensure that the targets aren't constantly re-discovered

  - job_name: 'alertmanager'
    dockerswarm_sd_configs:
      - host: http://proxy_swarm-socket:9090
        role: tasks
        port: 80
        filters:
          - name: desired-state
            values: ["running"]
          - name: label
            values: ["prometheus-job=alertmanager"]
    # Use static port above, to avoid re-discovery - see #8073 
    relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):80'
        target_label: __address__
        replacement: "${1}:9093"

What did you expect to see?

Services not being continuously rediscovered.

What did you see instead? Under which circumstances?

Services being continuously rediscovered, leading to an extremely high network load; snippet from the Caddy container (swarm api proxy):

{"level":"info","ts":1638891728.1137774,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"10.0.139.13:42320","proto":"HTTP/1.1","method":"GET","host":"prometheus_swarm-api:9090","uri":"/v1.40/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22label%22%3A%7B%22prometheus-job%3Dalertmanager%22%3Atrue%7D%7D","headers":{"User-Agent":["Prometheus/2.31.0"]}},"common_log":"10.0.139.13 - - [07/Dec/2021:15:42:08 +0000] \"GET /v1.40/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22label%22%3A%7B%22prometheus-job%3Dalertmanager%22%3Atrue%7D%7D HTTP/1.1\" 200 6293","user_id":"","duration":0.523735961,"size":6293,"status":200,"resp_headers":{"Api-Version":["1.40"],"Content-Type":["application/json"],"Docker-Experimental":["true"],"Ostype":["linux"],"Date":["Tue, 07 Dec 2021 15:42:08 GMT"],"Server":["Caddy","Docker/19.03.15 (linux)"]}}
{"level":"info","ts":1638891753.5051885,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"10.0.139.12:55914","proto":"HTTP/1.1","method":"GET","host":"prometheus_swarm-api:9090","uri":"/v1.40/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22label%22%3A%7B%22prometheus-job%3Dalertmanager%22%3Atrue%7D%7D","headers":{"User-Agent":["Prometheus/2.31.0"]}},"common_log":"10.0.139.12 - - [07/Dec/2021:15:42:33 +0000] \"GET /v1.40/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22label%22%3A%7B%22prometheus-job%3Dalertmanager%22%3Atrue%7D%7D HTTP/1.1\" 200 6293","user_id":"","duration":0.341778595,"size":6293,"status":200,"resp_headers":{"Api-Version":["1.40"],"Server":["Caddy","Docker/19.03.15 (linux)"],"Content-Type":["application/json"],"Docker-Experimental":["true"],"Ostype":["linux"],"Date":["Tue, 07 Dec 2021 15:42:33 GMT"]}}
{"level":"info","ts":1638891787.7132037,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"10.0.139.13:42320","proto":"HTTP/1.1","method":"GET","host":"prometheus_swarm-api:9090","uri":"/v1.40/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22label%22%3A%7B%22prometheus-job%3Dalertmanager%22%3Atrue%7D%7D","headers":{"User-Agent":["Prometheus/2.31.0"]}},"common_log":"10.0.139.13 - - [07/Dec/2021:15:43:07 +0000] \"GET /v1.40/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22label%22%3A%7B%22prometheus-job%3Dalertmanager%22%3Atrue%7D%7D HTTP/1.1\" 200 6293","user_id":"","duration":0.122917304,"size":6293,"status":200,"resp_headers":{"Server":["Caddy","Docker/19.03.15 (linux)"],"Content-Type":["application/json"],"Docker-Experimental":["true"],"Ostype":["linux"],"Date":["Tue, 07 Dec 2021 15:43:07 GMT"],"Api-Version":["1.40"]}}

The example here has a 'small' response, in reality we're dealing with responses as big as "size":2995782" on the same interval as above causing unnecessary network pressure.

Environment

• Prometheus version:

  2.32.0-rc0, same problem exist in previous versions.

[roidelapluie]

Service are discovered on the basis of the "refresh_interval", which you seem to use the default value of 30s.

[darkl0rd]

@roidelapluie I understand - please note Brian's comment in referenced ticket in which I was advised to use a relabel config to avoid the same service from being rediscovered all the time.

[roidelapluie]

can you paste your full configuration?

[darkl0rd]

The above is the configuration as used for my alertmanager, the same blocks exist for other services.
Static port, relabel config to set the correct port.

[roidelapluie]

Can I see the debug logs? I am looking for msg="Starting provider" lines.

[darkl0rd]

ts=2021-12-07T16:27:26.504Z caller=main.go:996 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dns/0 subs=[consul]
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=static/1 subs=[vmware]
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/2 subs=[docker]
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/3 subs=[censored]
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/4 subs=[censored]
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/5 subs="[barman cadvisor]"
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=static/6 subs=[cloudwatch]
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/7 subs=[promtail]
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/8 subs=[node-exporter]
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/9 subs=[postgres]
ts=2021-12-07T16:27:26.515Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dns/10 subs=[pushgateway]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/11 subs=[blackbox-swarm]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dns/12 subs=[minio]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/13 subs=[alertmanager]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/14 subs=[censored]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dns/15 subs=[traefik]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/16 subs=[censored]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dns/17 subs=[node-exporter-other]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=static/18 subs=[blackbox-static]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dockerswarm/19 subs=[prometheus]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager scrape" msg="Starting provider" provider=dns/20 subs=[censored]
ts=2021-12-07T16:27:26.516Z caller=manager.go:196 level=debug component="discovery manager notify" msg="Starting provider" provider=dns/0 subs=[config-0]
ts=2021-12-07T16:27:26.520Z caller=manager.go:214 level=debug component="discovery manager scrape" msg="Discoverer channel closed" provider=static/18
ts=2021-12-07T16:27:26.520Z caller=manager.go:214 level=debug component="discovery manager scrape" msg="Discoverer channel closed" provider=static/1
ts=2021-12-07T16:27:26.520Z caller=manager.go:214 level=debug component="discovery manager scrape" msg="Discoverer channel closed" provider=static/6
ts=2021-12-07T16:27:26.571Z caller=main.go:1033 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=66.770637ms db_storage=2.492µs remote_storage=3.513µs web_handler=1.837µs query_engine=32.672µs scrape=644.739µs scrape_sd=2.246113ms notify=39.113µs notify_sd=55.673µs rules=54.896861ms

[roidelapluie]

So, you do not seem to be using relabeling, but you are also having different filters for all the jobs. Prometheus indeed does not reuse the same servie discovery in this case.

You could try:

  - job_name: 'alertmanager'
    dockerswarm_sd_configs:
      - host: http://proxy_swarm-socket:9090
        role: tasks
        port: 80
        filters:
          - name: desired-state
            values: ["running"]
          - name: label
            values: ["prometheus-job=alertmanager", "prometheus-job=promtail",]
    # Use static port above, to avoid re-discovery - see #8073 
    relabel_configs:
      - source_labels: [__meta_dockerswarm_service_label_prometheus_job_alertmanager]
        action: keep
      - source_labels: [__address__]
        regex: '(.*):80'
        target_label: __address__
        replacement: "${1}:9093"

[roidelapluie]

#9187 might block us :(

[darkl0rd]

Each service has its own discovery block as they have different relabel actions further (dropping specific metrics, renaming specific labels etc.) - I can't combine them into one block. That being said, having one block or having a hundred shouldn't make a difference - my problem is that each block (I checked), keeps being re-processed over and over; I don't mind the initial "overhead" of processing all the distinct blocks once.

The re-labeling I have in place, is as as per above/referenced ticket, that is I relabel the statically defined port (80) to "${1}:port-number". How should I change that to prevent service discovery from continuously (on refresh_interval interval) pulling out all services?

As for the issue you just referenced, I'm aware of that issue and actually use container labels - so I can perform re-labeling actions on the prometheus level. That however, doesn't solve my problem - I try to filter as much as I can on the docker swarm level to avoid pulling in unnecessary services, but if the swarm socket is going to be polled every 30s and pull out all services each time, that's a very costly operation. Filtering (re-labeling) after the fact on the prometheus level, does not reduce the amount of data pulled in the first place, or am I misunderstanding this?

[roidelapluie]

It would if you use the same filter everywhere. Put a larger filter and use relabel configs for keeping targets relevant for the current job. In the log lines, you should have multiple subs per discovery.

[darkl0rd]

I am not quite sure what you mean? I currently have a dockerswarm_sd_configs block per job. Each block filters (on the discovery level) on both the desired-state and on the label. On the prometheus relabeling side, I rewrite the static port for each of these blocks to the actual port of the exporter.
Alone for that reason, I cannot use one single block, since each exporter listens on a different port; besides, different (custom) relabel actions need to be taken depending on the exporter, so I can't provide further grouping.

Or, are you saying that if I apply the same filtering which I currently already apply on the swarm side (to minimise the data set being pulled) in the relabel block as well, that prometheus will stop re-evaluating the targets all the time?