Prometheus can't access to docker.sock

[coolapso]

Sorry if this is not the right place, been spinning around, reading docs and how other people done it, nothing seems to work, IRC seems pretty dead and unresponsive. =/ hope someone can point me in the right direction.

What did you do?

Follow documentation about prometheus and docker swarm using:
https://prometheus.io/blog/2015/06/01/advanced-service-discovery/
and
https://prometheus.io/docs/guides/dockerswarm/
and
https://github.com/prometheus/prometheus/blob/release-2.22/documentation/examples/prometheus-dockerswarm.yml

What did you expect to see?

Prometheus should be able to access docker.sock

What did you see instead? Under which circumstances?

level=error ts=2020-11-15T19:36:07.282Z caller=refresh.go:98 component="discovery manager scrape" discovery=dockerswarm msg="Unable to refresh target groups" err="error while listing swarm services: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/tasks\": dial unix /var/run/docker.sock: connect: permission denied"

Environment

CentOS, Docker swarm, using prom/prometheus:latest

• System information:

# docker --version
Docker version 19.03.13, build 4484c46d9d

# uname -srm
Linux 4.18.0-193.28.1.el8_2.x86_64 x86_64

• Prometheus version:

# docker exec -ti 636014888e15 /bin/sh
/prometheus $ prometheus --version
prometheus, version 2.22.1 (branch: HEAD, revision: 00f16d1ac3a4c94561e5133b821d8e4d9ef78ec2)
  build user:       root@516b109b1732
  build date:       20201105-14:02:25
  go version:       go1.15.3
  platform:         linux/amd64

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'dockmaster'

    static_configs:
    - targets: ['redacted:9100']

  - job_name: 'vms'

    static_configs:
      - targets: ['redacted:9100', 'redacted:9100']

  - job_name: 'home-servers'

    static_configs:
      - targets: ['redacted:9100', 'redacted:9100']

  - job_name: 'pis'

    static_configs:
      - targets: ['redacted']

  - job_name: 'docker'
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: nodes
    relabel_configs:
      # Fetch metrics on port 9323.
      - source_labels: [__meta_dockerswarm_node_address]
        target_label: __address__
        replacement: $1:9323

  - job_name: 'dockerswarm'
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: tasks
    relabel_configs:
      # Only keep containers that should be running.
      - source_labels: [__meta_dockerswarm_task_desired_state]
        regex: running
        action: keep

• Prometheus Docker compose file:

version: '3.3'

  private:
    image: prom/prometheus
    networks:
      - dockadmin_rp
      - private
    volumes:
      - /srv/data/prometheus/config:/etc/prometheus
      - /srv/data/prometheus/data:/prometheus
      - /var/run/docker.sock:/var/run/docker.sock:0444
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.labels.type == private 
      labels:
        traefik.enable: "true"
        traefik.docker.network: "dockadmin_rp"
        traefik.http.routers.pm02.service: "pm02"
        traefik.http.routers.pm02.tls: "true"
        traefik.http.routers.pm02.tls.certResolver: "default"
        traefik.http.routers.pm02.middlewares: "defaultsecheaders@file, pmauth"
        traefik.http.routers.pm02.rule: "Host(`pm02.mydomain.xyz`)"
        traefik.http.middlewares.pmauth.basicauth.usersfile: "/auth/prometheus"
        traefik.http.services.pm02.loadbalancer.server.port: 9090 

  cadvisor:
    image:  gcr.io/google-containers/cadvisor
    command: -docker_only
    networks:
      - private
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    deploy:
      mode: global
      restart_policy:
        condition: on-failure


networks:
  private:
  dockadmin_rp:
    external: true

• Logs:

level=error ts=2020-11-15T19:45:07.282Z caller=refresh.go:98 component="discovery manager scrape" discovery=dockerswarm msg="Unable to refresh target groups" err="error while listing swarm nodes: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/nodes\": dial unix /var/run/docker.sock: connect: permission denied"

[roidelapluie]

Can you please read and react there: https://groups.google.com/g/prometheus-users/c/EuEW0qRzXvg/m/0aqKh_ZABQAJ

thanks!

[coolapso]

Thanks @roidelapluie .. but i can't do anything on that topic tough ...

Edit: Just disregard.

[coolapso]

@roidelapluie sorry to insist, and to be honest... reading that topic ... this should be considered a bug .. or at least is a documentation issue .. is it possible to reopen this and get some more traction / attention on this?

[marc-x-andre]

I'm facing the same issue on v2.22.0.

[Ruppsn]

Following the documenation stated above and there is still not one line in the documenation that gives a hint...Why? You just have to add one line like: Mounting docker sock is not possible at the moment becouse prometheus is not running as root in standard configuration....

[roidelapluie]

It depends what is a standard configuration. I works for me with the socket and I do not run Prometheus as root. Any improvement to the documentation is welcome.

[Ruppsn]

It depends what is a standard configuration means (for me) that i followed your documentation without any extra.

I am doing exactly what is described at https://prometheus.io/docs/guides/dockerswarm/

Setting up Prometheus

For this guide, you need to setup Prometheus. We will assume that Prometheus runs on a Docker Swarm manager node and has access to the Docker socket at /var/run/docker.sock.

I can mount the socket. But i cant read it
level=error ts=2021-04-21T10:00:35.696Z caller=refresh.go:79 component="discovery manager scrape" discovery=dockerswarm msg="Unable to refresh target groups" err="error while listing swarm nodes: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/nodes\": dial unix /var/run/docker.sock: connect: permission denied"

---

How do you give prometheus access without running it as root? Are you using something like socketproxy? The docker.sock is only readable by root! Prometheus Container runs as user nobody. I am using image: prom/prometheus:v2.26.0

Thank you for replying!

[roidelapluie]

Here is how this happens on my machine:

$ ls -l /var/run/docker.sock 
srw-rw---- 1 root docker 0 Apr 21 09:23 /var/run/docker.sock

and my prometheus user belongs to the docker group.

[Ruppsn]

So your prometheus user (nobody) is part of the docker group. The docker group which grants them the ability to run containers which can be used to obtain root privileges on the Docker host. This is not possible in our production environment. But thank your for clearing things up for me.
Have a nice day/night wherever your are.

[roidelapluie]

I don't run prometheus as nobody or in docker.

[coolapso]

"works on my machine therefore its fine" .. sounds about right! i am with @Ruppsn some warning or info in the documentation page would be of really good help. but it seems that this keeps being ignored by the maintainers which is sad.

a simple change could save a lot of time and trouble either to maintainers and users.

[roidelapluie]

As I said we accept PR to improve the documentation.

[coolapso]

As I said we accept PR to improve the documentation.

will see if i have some time to go around it during weekend and get that PR up :)

[coolapso]

@roidelapluie where would be the best page to put a PR up to?

This? https://github.com/prometheus/docs/blob/master/content/docs/guides/dockerswarm.md

@Ruppsn and others who may stumble here, as discussed here, https://groups.google.com/g/prometheus-users/c/EuEW0qRzXvg/m/0aqKh_ZABQAJ one possible solution is to run the container as root by adding user: root

example:

services:
  prometheus:
    image: prom/prometheus
    user: root
    networks: 
    ...
    volumes:
      ...

[roidelapluie]

Yes

[roidelapluie]

Changing the docker.sock permissions will not only reset but will leave it exposed for anything else on the docker node which is IMO is even worse to recommend.

This is not true. The recommendation from docker ( https://docs.docker.com/engine/install/linux-postinstall/ ) is to create a docker group and add the user to the group. So only users in this group have access to the socket.

[coolapso]

Here is what the documentation of docker has to say:

https://docs.docker.com/engine/install/linux-postinstall/

We could link to that page, telling people to add the prometheus user to the docker group?

This would force users to create a Prometheus user with the same uuid that Prometheus is using inside the container and add it to the docker group which might not be possible in many cases ... not even sure how well this would work.

I don't see why running Prometheus inside the container as root is an issue since its such a common practice, cadvisor itself is running as root inside the container, however i can understand that Prometheus devs and maintainers wanna stay away from doing and recommending so.

I think we are mainly dealing with a sad limitation of docker swarm by not allowing the usage of privileged: true therefore having to work around it.

Changing the docker.sock permissions will not only reset but will leave it exposed for anything else on the docker node which is IMO is even worse to recommend.

This is not true. The recommendation from docker ( https://docs.docker.com/engine/install/linux-postinstall/ ) is to create a docker group and add the user to the group. So only users in this group have access to the socket.

Maybe i wasn't clear ... i was referring to changing permissions directly on the socket as it can be found in so many suggestions over the internet ... like chmod 777 /var/run/docker.sock things like that!

[Ruppsn]

Maybe i wasn't clear ... i was referring to changing permissions directly on the socket as it can be found in so many suggestions over the internet ... like chmod 777 /var/run/docker.sock things like that!

Before you do that..just leave the documentation as it is...this would be a stupid suggestion.

This would basically gives everyone on your host access to the docker.sock and this is NOT what you want to recommend. Furthermore it will be reseted everytime you will restart your host or relog.

You really need to understand that this opens up your docker instance to everyone. For a thorough explanation this is a must read (https://docs.docker.com/engine/security/). chmod 777 is never the correct solution (well... unless the sticky bit is also set and your REALLY know what you are doing).

[coolapso]

@Ruppsn that is exactly what i was saying above :)

[Ruppsn]

Sorry than i missunderstood you :)

[coolapso]

Sorry than i missunderstood you :)

Its okay, i think we are all in the same page ... and lets see if we can find a solution for the documentation itself ...

Reading the tests i did back then here: https://groups.google.com/g/prometheus-users/c/EuEW0qRzXvg/m/0aqKh_ZABQAJ?pli=1

What should not be done:

Adding nobody to the docker group its not really a good production practice either
changing permissions with chmod on docker.sock

Possible but not tested

1 - Create custom prometheus docker image and have prometheus running inside the container with different user (i.e prometheus), create the same user with the same uuid on the Docker swam manager(s) and add it to the docker group ( I Personally would stay AWAY from doing this)

Viable Options

1 - use docker socket proxy
2 - add user: root to the docker-compose file
3 - build custom image and have Prometheus running as root inside the container

I personally think the usage of docker socket proxy is probably the safest approach.

[coolapso]

This being said .. What to add to the documentation?

NOTE: When using docker swarm bind mounting the docker socket into container won't provide Prometheus with access to it, please consider using a docker socket proxy, or read This thread for other alternatives.

Any suggestions?

Guess at this point this thread and this https://groups.google.com/g/prometheus-users/c/EuEW0qRzXvg/m/0aqKh_ZABQAJ?pli=1 has enough data, and someone has to make a decision which is not either up to me or @Ruppsn, so leaving up to Prometheus maintainers to make a decision about it.

I will be more than glad to put up that PR once a decision has been taken.

[Ruppsn]

NOTE: When using docker swarm bind mounting the docker socket into container won't provide Prometheus with access to it, please consider using a docker socket proxy, or read This thread for other alternatives.

IMO: This alone would be enough for people considering security to do their own research. I do not think there is a golden solution to this but with this note everyone can use his/her/its brain. In prod environments everyone has their own consideration to make.

[roidelapluie]

I think we should cover the setup of prometheus in swarm. Currently the recommended way is the tarball.

I would not recommend linking to threads or discussions, but rather providing actual options.

[coolapso]

@roidelapluie not really making this easier ... and even tough i really want to help my time is limited and its needed elsewhere.

To Cover the swarm setup,
Someone has to make the call on what is the recommended method for Prometheus.
Test it and provide a working example of docker-compose file compatible with docker-swarm.

I personally have it running with user: root and don't really have the time to test it with docker socket proxy.

[ktpktr0]

I also encountered this problem, adding 65534 users to the docker group through the host, which could not be solved.

[brunocascio]

I just leave this example for future references using prometheus with docker socat proxy:

version: "3.9"

networks:
  socat:
  traefik:
    name: "traefik-gateway-test_default"
    external: true

volumes:
  prometheus_data:

configs:
  prometheus-config:
    name: $PROM_CONFIG_NAME
    file: ./prometheus.yml

services:

  prometheus:
    image: prom/prometheus:v2.30.1
    networks:
      - traefik
      - socat
    configs:
      - source: prometheus-config
        target: /configs/prometheus.yml
        mode: 0444
    command:
      - "--config.file=/configs/prometheus.yml"
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    volumes:
      - prometheus_data:/prometheus
    deploy:
      placement:
        constraints:
          - node.labels.prometheus==yes
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.prometheus.entrypoints=http"
        - "traefik.http.routers.prometheus.rule=Host(`example.com`)"
        - "traefik.http.services.prometheus.loadbalancer.server.port=9090"

  docker-api-socat:
    image: tecnativa/docker-socket-proxy:0.1
    networks:
      - socat
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      NODES: 1
      NETWORKS: 1
      SERVICES: 1
      TASKS: 1
    logging:
      # Socat logs send to black hole (we don't need them)
      driver: none
    deploy:
      mode: global
      resources:
        reservations:
          memory: 5M
          cpus: '0.05'
        limits:
          memory: 10M
          cpus: '0.1'
      update_config:
        parallelism: 1
        order: start-first
        failure_action: rollback
      rollback_config:
        parallelism: 1
        order: start-first
      placement:
        constraints:
          - node.role == manager

prometheus.yml

scrape_configs:
  # Make Prometheus scrape itself for metrics.
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

  # Create a job for Docker daemons.
  - job_name: 'docker'
    dockerswarm_sd_configs:
      - host: tcp://tasks.docker-api-socat:2375
        role: nodes
    relabel_configs:
      # Fetch metrics on port 9323.
      - source_labels: [__meta_dockerswarm_node_address]
        target_label: __address__
        replacement: $1:9323
      # Set hostname as instance label
      - source_labels: [__meta_dockerswarm_node_hostname]
        target_label: instance

[till]

In case someone needs a simple override and doesn't want socat or another proxy, here's a Dockerfile:

FROM quay.io/prometheus/prometheus:v2.37.2
ADD config.yml /etc/my-config
USER root
CMD ["--config.file=/etc/my-config"]

You can build this, run it and even add your configuration, and if you need to override all options/flags with a custom CMD stanza.

version: "3.7"

services:
  prometheus:
    image: my-image
    volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro

[sunbear87]

FYI, I got this working without needing a docker proxy by using the below arguments with my docker run command.

--user 0:$(stat -c '%g' /var/run/docker.sock)

Starts the container with the docker group.

Not sure if security issues or not but it works for me.

[Minikloon]

What worked for me is to add

volumes:
      - /var/run/docker.sock:/var/run/docker.sock

in the prometheus service config.