lock DB directory: resource temporarily unavailable

[mousimin]

Hi,
We were running a single Prometheus with agent mode on our server, and we observed bellow error:

caller=dir_locker.go:77 level=warn msg="A lockfile from a previous execution already existed. It was replaced" file=/var/lib/agent/wal/lock
Sep 04 14:11:39 agent[3208744]: ts=2023-09-04T12:11:39.858Z caller=agent.go:1113 level=error err="opening storage failed: lock DB directory: resource temporarily unavailable"

After we deleted the lock file and restart prometheus, we observed warning:

level=warn msg="found sample referencing non-existing series" skipped_series=5105866
level=warn msg="found sample referencing non-existing series" skipped_series=5297834
level=warn msg="found sample referencing non-existing series" skipped_series=4438412

Later, some other errors appeared:

Sep 06 12:00:40 agent[12580]: ts=2023-09-06T10:00:40.442Z caller=dedupe.go:112 component=remote level=error remote_name=3c74ba url=https://collector-cortex/api/prom/push msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 400 Bad Request: user=Storage: err: out of bounds. timestamp=2023-09-06T10:00:27.973Z, series={...}"

Could you please help to check the issue? any suggestion will be appreciate.

[hezhizhen]

I manually deleted the lock file and then recreated the prometheus deployment and it worked for me.