etc/prometheus/prometheus.yml: operation not permitted

[SarpongAbasimi]

What did you do?

When starting up the prometheus container locally, I get the following error.

ts=2023-06-23T19:18:52.158Z caller=main.go:
450 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" 
file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: operation not permitted"

This issues seems to have been going on for a while as seen here
#5976

Has a solution been for for the error?

This is the image being used

  prometheus:
    image: prom/prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'

I have also tried this

  prometheus:
    image: prom/prometheus
    user: root
    ports:
      - 9090:9090
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'

But I get the same error.

Expected Behaviour
Prometheus should start up without any hickups

Current Behaviour
Prometheus never starts up.

Possible Solution
Update the Dockerfile and take necessary steps to eliminate the permission denied issue.

What did you expect to see?

A working prometheus container.

What did you see instead? Under which circumstances?

ts=2023-06-23T19:18:52.158Z caller=main.go:
450 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" 
file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: operation not permitted"

Current Behaviour
Prometheus never starts up.

System information

No response

Prometheus version

latest

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

ts=2023-06-23T19:18:52.158Z caller=main.go:
450 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" 
file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: operation not permitted"

[roidelapluie]

From the error message, it seems like the Prometheus process in the container is not allowed to read the configuration file at /etc/prometheus/prometheus.yml. This could be a Docker-related issue.

First, let's ensure that the prometheus.yml file exists on your local system and it is in the correct location i.e ./prometheus/prometheus.yml.

Ensure the file permissions are correct on the host system, that is, the file can be read by the user in Docker. You could try changing the permissions using chown, for example, chown 65534 ./prometheus/prometheus.yml on your local machine and see if that helps.

[SarpongAbasimi]

@roidelapluie Thanks for responding. This is the structure of the application

as you can already see, I have the prometheus.yml file in the correct directory and even after doing chown 65534 ./prometheus/prometheus.yml I still get the same error.

This is what I have in my docker compose file obviously I have prometheus_data: {} added to my volume.

  prometheus:
    image: prom/prometheus
    user: root
    ports:
      - 9090:9090
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'

[SarpongAbasimi]

@roidelapluie And just to show , I have changed the permissions on the file as suggested.

[SarpongAbasimi]

I have also tried mounting the who directory like so

      - ./prometheus/:/etc/prometheus/

but that too does not help because I get the same error log.

[SarpongAbasimi]

@roidelapluie are you able to help with this?
I still have not found the solution to the issue but I am going to close the issue since it is taking a while.
If you do know what is going on please leave the answer here else thank you for your time.

[PrinceP]

sudo docker run --mount type=bind,source="$(pwd)"/prom.yaml,target=/etc/prometheus/prometheus.yml,readonly --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus