inotify inode leak in file discovery

[GoneLikeAir]

What did you do?

We use file discovery for target discovery, and save the files in the same directory.

What did you expect to see?

each job only needs to watch to the files they care about, do not watch all the files in the directory.

What did you see instead? Under which circumstances?

As more and more jobs are created, the inotify inode may run out and encounter an error of "too many open files".
We save the files used for target discovery from multiple jobs in the same directory. By reviewing the source code (kqueue. go), we found that when discovering targets via a specified file, it will watch all files in the same directory (in fsnotify, watching to a directory will actually traverse all files). This results in each job watching to all the files in the directory, but in reality, each job only needs to watch to the files they care about.

System information

Linux 3.10.0-1160.90.1.el7.x86_64 x86_64

Prometheus version

prometheus 2.39.1

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[machine424]

Watching individual files is not recommended by the used library: fsnotify as it’s not resilient.
Additionally, because we support globs, we want to listen for directory changes to detect new files (changing that may break some use cases).

Maybe the optimal intermediate solution would be to watch the directory itself (excluding the files) in addition to the necessary files. However, it appears that fsnotify generalizes inotify’s behavior, which involves watching all of a directory’s files as soon as the directory is watched, to other implementations.

I believe the best solution for you would be to isolate each job’s files in a separate folder. Note that recent kernel versions seem to adjust max_user_watches based on the available RAM.

[machine424]

Maybe we should mention this in the docs somewhere.

[GoneLikeAir]

Watching individual files is not recommended by the used library: fsnotify as it’s not resilient. Additionally, because we support globs, we want to listen for directory changes to detect new files (changing that may break some use cases).

Maybe the optimal intermediate solution would be to watch the directory itself (excluding the files) in addition to the necessary files. However, it appears that fsnotify generalizes inotify’s behavior, which involves watching all of a directory’s files as soon as the directory is watched, to other implementations.

I believe the best solution for you would be to isolate each job’s files in a separate folder. Note that recent kernel versions seem to adjust max_user_watches based on the available RAM.

We have deployed Prometheus in k8s and are using ConfigMaps to mount its configuration files. If we store these files in different directories, this would necessitate creating a separate ConfigMap for each job and mounting them into the Pod, which would result in Pod restarts. This approach is clearly not elegant.
A more compromising alternative is to employ a singleton pattern in the usage of fsnotify without altering the current watch mode. This involves adopting an approach similar to "reference counting" for watching file changes. Specifically, for each File SD instance, maintain a channel; when fsnotify get an event, it sends the event to the channel of every relevant File SD instance.