User-space recursive watcher #18
Comments
|
|
The question becomes then if we want to fallback to whatever is the most efficient way of doing subtree wachtes on an OS that succeeds, and if we want to tell the user that the fallback happened. Or if the user should be able to configure whether a fallback is allowed. The reason this might be handy is that a user might prefer to not have directory watching at all instead of polling, which might put too much stress on the system. Plainly telling the user that the feature is disabled or something. I, for one, would absolutely love subtree watches, but understand the difficulties on getting it right for cross-platform. |
|
Is the implementation in the https://github.com/xyproto/recwatch package correct, or are there issues? I ask because it looks as if this is a more subtle, tricky problem. cc @nathany @xyproto Cheers |
|
It should work. Most of the tricky issues are with too many resources being consumed. Also, we would ideally take advantage of native recursive watching on the operating systems that support it. |
This comment was marked as spam.
This comment was marked as spam.
|
No update for fsnotify, but take a look at https://github.com/rjeczalik/notify as an alternative. |
|
Rafal's notify library uses FSEvents on OS X instead of kqueue, so it doesn't have that problem on OS X. |
|
FSEvents sounds like a good approach. The notify package has a few unresolved issues, though. |
|
As does fsnotify. |
|
Neither github.com/rjeczalik/notify nor recwatch place a watch on newly-added directories. This is fine if the directory structure is static, so for things like anomaly detection in a server environment or corporate asset. I have a use case (github.com/unixist/cryptostalker) that aims to detect crypto ransom malware on end-user computers. In such an environment, directories are expected to come and go all the time. Is there a potential for resource exhaustion that I'm unaware of, leading to this feature not being implemented more widely (and recursive at all for fsnotify)? |
|
OK that might be a fluke in my testing since I created directories with mkdir -p. Still any comments welcome. |
|
@landaire, I tore down my test system, but I'm thinking it was a premature post. Though I'm unsure what it was about the multiple directory creation that threw fsnotify for a loop. I'd be interested to know what you find doing the mkdir -p test on your code. |
Could you provide test case that reproduces this? @rjeczalik's notify package should add new directories when watching recursively. It is a bug if it doesn't. |
|
I tore down my test environment, but you can test by placing a watch on On Wed, Apr 13, 2016 at 12:07 AM, Pawel Knap notifications@github.com
|
Yes, it does. |
|
Hi. The fswatch may be a good place for fsnotify, especially for this issue. They state it has "no known limitations" on Solaris kernels (with FEN). I.e. also with recursive monitoring. |
This comment was marked as spam.
This comment was marked as spam.
|
Could someone provides an example about how to work around this problem? I want to watch the generated folders and files underneath by some tools. |
|
I have made a little change ,it work orderly ,how about that |
Recursive watches can be added by using a "/..." parameter, similar to
the Go command:
w.Add("dir") // Behaves as before.
w.Add("dir/...") // Watch dir and all paths underneath it.
w.Remove("dir") // Remove the watch for dir and, if
// recursive, all watches underneath it too
w.Remove("dir/...") // Behaves like just "dir" if the path was
// recursive, error otherwise (probably
// want to add recursive remove too at some
// point).
The advantage of using "/..." vs. an option is that it can be easily
specified in configuration files and the like.
This should be expanded to other backends too; I started with Windows
because the implementation is the both the easiest and has the least
amount of control (just setting a boolean parameter), and we can focus
mostly on writing tests for it, and we can then match the inotify and
kqueue behaviour to the Windows one.
Updates #18
Recursive watches can be added by using a "/..." parameter, similar to
the Go command:
w.Add("dir") // Behaves as before.
w.Add("dir/...") // Watch dir and all paths underneath it.
w.Remove("dir") // Remove the watch for dir and, if
// recursive, all watches underneath it too
w.Remove("dir/...") // Behaves like just "dir" if the path was
// recursive, error otherwise (probably
// want to add recursive remove too at some
// point).
The advantage of using "/..." vs. an option is that it can be easily
specified in configuration files and the like.
This should be expanded to other backends too; I started with Windows
because the implementation is the both the easiest and has the least
amount of control (just setting a boolean parameter), and we can focus
mostly on writing tests for it, and we can then match the inotify and
kqueue behaviour to the Windows one.
Updates #18
Recursive watches can be added by using a "/..." parameter, similar to
the Go command:
w.Add("dir") // Behaves as before.
w.Add("dir/...") // Watch dir and all paths underneath it.
w.Remove("dir") // Remove the watch for dir and, if
// recursive, all paths underneath it too
w.Remove("dir/...") // Behaves like just "dir" if the path was
// recursive, error otherwise (probably
// want to add recursive remove too at some
// point).
The advantage of using "/..." vs. an option is that it can be easily
specified in configuration files and the like; for example from a TOML
file:
[watches]
dirs = ["/tmp/one", "/tmp/two/..."]
Options for this were previously discussed at:
#339 (comment)
This should be expanded to other backends too; I started with Windows
because the implementation is the both the easiest and has the least
amount of control (just setting a boolean parameter), and we can focus
mostly on writing tests and documentation and the for it, and we can
then match the inotify and kqueue behaviour to the Windows one.
Updates #18
|
In case it's of interest to folks following here, we just created https://pkg.go.dev/github.com/cue-lang/cuelang.org@v0.0.0-20230505131944-cc11e9153697/internal/fsnotify. As the package documentation notes:
Note this is only availalable in the alpha branch for now: https://github.com/cue-lang/cuelang.org/tree/alpha/internal/fsnotify I have not had great experience with recursive watcher implementations in the past, so one goal here was to establish good tests to cover edge cases. Feedback welcome on the API. |
|
Are there any other libraries that do the recursive watcher thing? Its sadly bit of a non starter for me without it (The tree being watched is kind of labrynth) |
If running on linux, try https://github.com/pablodz/inotifywaitgo was a workaround that I developed for this edge case |
|
There is also https://github.com/rjeczalik/notify - still a potential project on my back-backlog to merge the two projects. Basically use fsnotify within notify for the underlying non-recursive watching. (Also for the avoidance of doubt: Back-backlog means unlikely to ever happen :) ). |
|
It might be worth putting up a bit of notice on the front page as unfortunately its not really obvious that its not supported until it doesnt work, and I suspect 90% of use-cases would require that recursive watch (As in I cant imagine any usecases where it wouldnt be a requirement?), edit: My bad, it is actually in the FAQ. |
and others
@jbowtie requested this feature at howeyc/fsnotify#56.
Windows and FSEvents (OS X) can watch a directory tree, in fact FSEvents is built entirely for this purpose:
Ideally fsnotify would expose a way to use this functionality on Windows and Macs.
Unfortunately inotify (Linux, Android), kqueue (BSD, iOS), and FEN (Solaris) don't have this capability.
There is some debate as to whether or not fsnotify should include a user-space recursive watcher to make up the difference. If not, perhaps it can be implemented a separate wrapper or utility library. In any case, it's something often requested, so we should definitely be thinking about it.
Implementation
In two parts:
The text was updated successfully, but these errors were encountered: