Behavior if read-only file system is detected #13127
Unanswered
sthellmuth
asked this question in
Q&A
Replies: 1 comment
-
I don't think this can be configured inside Prometheus, however you could write a Prometheus init container to do the check and exit with an error. Also you could file a feature request (and even a PR!), that read-only filesystem should cause Prometheus to exit with an error. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello,
we are running many promehteus instances inside of a kubernetes cluster. Very sporadically we are running in a problem. The prometheus is not able to store new data, but pod,container etc. looks fine from outside. (Healthcheck)
I see these msg in the log, so prometheus is aware of it.
msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/wal/00000029: read-only file system"
I would like to restart the pod/container automatically since a manual restart solves the problem.
So for example if the health check would fail Kubernetes it self could do that.
Is there a way to configure the behavior if a read-only filesystem is detected? ...It would also be ok for me if the whole prometheus instance would crash.
Beta Was this translation helpful? Give feedback.
All reactions