Docker Target Always Showing State as 'DOWN'

[conradwt]

What did you do?

1. updated the Docker Engine to the following:

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": true,
  "metrics-addr": "127.0.0.1:9323"
}

2. Restarted Docker For Mac (Command + R)

3. Restarted Grafana and Prometheus services

docker compose down
docker compose up --build -d

4. Refreshed the Prometheus site (http://localhost:9090/targets)

What did you expect to see?

I expected that both targets would have the State as UP.

What did you see instead? Under which circumstances?

System information

Darwin Conrads-MacBook-Pro-2.local 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64

Prometheus version

/prometheus $ /bin/prometheus --version
prometheus, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:37:33
  go version:       go1.20.6
  platform:         linux/arm64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A example scrape configuration for running Prometheus with Docker.

scrape_configs:
  # Make Prometheus scrape itself for metrics.
  - job_name: 'prometheus'
    static_configs:
      - targets: ['host.docker.internal:9090']

  # Create a job for Docker daemon.
  #
  # This example requires Docker daemon to be configured to expose
  # Prometheus metrics, as documented here:
  # https://docs.docker.com/config/daemon/prometheus/
  - job_name: 'docker'
    static_configs:
      - targets: ['127.0.0.1:9323']

  # Create a job for Docker Swarm containers.
  #
  # This example works with cadvisor running using:
  # docker run --detach --name cadvisor -l prometheus-job=cadvisor
  #     --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock,ro
  #     --mount type=bind,src=/,dst=/rootfs,ro
  #     --mount type=bind,src=/var/run,dst=/var/run
  #     --mount type=bind,src=/sys,dst=/sys,ro
  #     --mount type=bind,src=/var/lib/docker,dst=/var/lib/docker,ro
  #     google/cadvisor -docker_only
  - job_name: 'docker-containers'
    docker_sd_configs:
      - host: unix:///var/run/docker.sock # You can also use http/https to connect to the Docker daemon.
    relabel_configs:
      # Only keep containers that have a `prometheus-job` label.
      - source_labels: [__meta_docker_container_label_prometheus_job]
        regex: .+
        action: keep
      # Use the task labels that are prefixed by `prometheus-`.
      - regex: __meta_docker_container_label_prometheus_(.+)
        action: labelmap
        replacement: $1
# remote_write:
#   - url: '<Your Prometheus remote_write endpoint>'
#     basic_auth:
#       username: '<Your Grafana Username>'
#       password: '<Your Grafana API key>'

Alertmanager version

N/A

Alertmanager configuration file

N/A

Logs

prometheus  | ts=2023-09-05T06:01:08.086Z caller=main.go:541 level=info msg="No time or size retention was set so using the default time retention" duration=15d
prometheus  | ts=2023-09-05T06:01:08.086Z caller=main.go:585 level=info msg="Starting Prometheus Server" mode=server version="(version=2.46.0, branch=HEAD, revision=cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)"
prometheus  | ts=2023-09-05T06:01:08.086Z caller=main.go:590 level=info build_context="(go=go1.20.6, platform=linux/arm64, user=root@42454fc0f41e, date=20230725-12:37:33, tags=netgo,builtinassets,stringlabels)"
prometheus  | ts=2023-09-05T06:01:08.086Z caller=main.go:591 level=info host_details="(Linux 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 d9c42cb32862 (none))"
prometheus  | ts=2023-09-05T06:01:08.086Z caller=main.go:592 level=info fd_limits="(soft=1048576, hard=1048576)"
prometheus  | ts=2023-09-05T06:01:08.086Z caller=main.go:593 level=info vm_limits="(soft=unlimited, hard=unlimited)"
prometheus  | ts=2023-09-05T06:01:08.087Z caller=web.go:563 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
prometheus  | ts=2023-09-05T06:01:08.088Z caller=main.go:1026 level=info msg="Starting TSDB ..."
prometheus  | ts=2023-09-05T06:01:08.088Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693809731249 maxt=1693814400000 ulid=01H9FS4DHWS56XRZN702Z1WJN1
prometheus  | ts=2023-09-05T06:01:08.088Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693814400000 maxt=1693821600000 ulid=01H9HKYP44FNNG1DYY638Y2EFK
prometheus  | ts=2023-09-05T06:01:08.088Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693821612293 maxt=1693828800000 ulid=01H9HKYP5NG3MD0Z464QC98VG4
prometheus  | ts=2023-09-05T06:01:08.088Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693882210428 maxt=1693886400000 ulid=01H9HY8FBN1F0B21GNPP0HK1Y2
prometheus  | ts=2023-09-05T06:01:08.088Z caller=tls_config.go:274 level=info component=web msg="Listening on" address=[::]:9090
prometheus  | ts=2023-09-05T06:01:08.088Z caller=tls_config.go:277 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
prometheus  | ts=2023-09-05T06:01:08.091Z caller=head.go:595 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
prometheus  | ts=2023-09-05T06:01:08.092Z caller=head.go:676 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=434.375µs
prometheus  | ts=2023-09-05T06:01:08.092Z caller=head.go:684 level=info component=tsdb msg="Replaying WAL, this may take a while"
prometheus  | ts=2023-09-05T06:01:08.104Z caller=head_wal.go:425 level=warn component=tsdb msg="Unknown series references" samples=32999 exemplars=0 histograms=0 metadata=0
prometheus  | ts=2023-09-05T06:01:08.104Z caller=head.go:720 level=info component=tsdb msg="WAL checkpoint loaded"
prometheus  | ts=2023-09-05T06:01:08.105Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=19 maxSegment=25
prometheus  | ts=2023-09-05T06:01:08.105Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=20 maxSegment=25
prometheus  | ts=2023-09-05T06:01:08.105Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=21 maxSegment=25
prometheus  | ts=2023-09-05T06:01:08.106Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=22 maxSegment=25
prometheus  | ts=2023-09-05T06:01:08.106Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=23 maxSegment=25
prometheus  | ts=2023-09-05T06:01:08.107Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=24 maxSegment=25
prometheus  | ts=2023-09-05T06:01:08.109Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=25 maxSegment=25
prometheus  | ts=2023-09-05T06:01:08.109Z caller=head.go:792 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=12.674458ms wal_replay_duration=4.583167ms wbl_replay_duration=167ns total_replay_duration=17.711833ms
prometheus  | ts=2023-09-05T06:01:08.110Z caller=main.go:1047 level=info fs_type=EXT4_SUPER_MAGIC
prometheus  | ts=2023-09-05T06:01:08.110Z caller=main.go:1050 level=info msg="TSDB started"
prometheus  | ts=2023-09-05T06:01:08.110Z caller=main.go:1231 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
prometheus  | ts=2023-09-05T06:01:08.111Z caller=main.go:1268 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=827.167µs db_storage=792ns remote_storage=917ns web_handler=334ns query_engine=750ns scrape=161.208µs scrape_sd=78.166µs notify=25.583µs notify_sd=7.875µs rules=1µs tracing=2.125µs
prometheus  | ts=2023-09-05T06:01:08.111Z caller=main.go:1011 level=info msg="Server is ready to receive web requests."
prometheus  | ts=2023-09-05T06:01:08.111Z caller=manager.go:1009 level=info component="rule manager" msg="Starting rule manager..."
prometheus  | ts=2023-09-05T06:01:08.111Z caller=refresh.go:80 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:02:08.115Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:03:08.114Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:04:08.114Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:05:08.112Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:06:08.112Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:07:08.141Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:08:08.144Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:09:08.142Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:10:08.148Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:11:08.145Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:12:08.148Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:13:08.147Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:14:08.146Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:15:08.148Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:16:08.152Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:17:08.149Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:18:08.150Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:19:08.150Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:20:08.154Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
prometheus  | ts=2023-09-05T06:21:08.152Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=docker config=docker-containers msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"