Inconsistent event paths from Docker on macOS

[mokiat]

Describe the bug

Hi,

I have been developing a tool for automatic rebuilding and restart of Go applications. It uses fsnotify to track for file changes in a recursive way. The problem is that in some situations the events produced by fsnotify are not as per documentation.

For example, the following scenario breaks it (Linux):

1. Create a folder foo

   CREATE        "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   CHMOD         "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   CHMOD         "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   

2. Create file bar inside foo

   CREATE        "/home/momchil/Workspace/Tools/gocrane/example/internal/foo/bar"
   WRITE         "/home/momchil/Workspace/Tools/gocrane/example/internal/foo/bar"
   CHMOD         "/home/momchil/Workspace/Tools/gocrane/example/internal/foo/bar"
   

3. Delete folder foo

   RENAME        "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   RENAME        "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   

4. Create folder foo again

   CREATE        "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   CHMOD         "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   CHMOD         ""
   

5. Create file bar inside foo again

   CREATE        "/bar"
   CHMOD         "/bar"
   

6. Empty Trash

   RENAME        "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   CHMOD         "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   REMOVE        "/home/momchil/Workspace/Tools/gocrane/example/internal/foo/bar"
   REMOVE        "/home/momchil/Workspace/Tools/gocrane/example/internal/foo"
   

At step 4 and 5 things start to break. It reports events with empty names or not relative to anything.

I observe similar when using my tool inside Docker container on a MacOS host. Except that there the events are REMOVE instead of RENAME.

In the past I have used my own internal tracking of files and folders that I have passed to fsnotify watcher. That way, when a file or folder is removed, I manually call Watcher.Remove for it and all its children. This helps when doing the simple scenario from above but under hight stress (go mod vendor and the likes) it fails again.

To Reproduce

Not a one-liner. I can try and provide a branch of my tool that can reproduce it consistently.

Which operating system and version are you using?

Observed both on Linux (where RENAMEs are reported), as well as in Docker container on MacOS (where REMOVEs are reported).

Which fsnotify version are you using?

1.6.0

Did you try the latest main branch?

No

[arp242]

Can you try the latest main branch? There's been quite a few changes, so it may already be fixed/improved. I should probably tag a new version (I wanted to get the recursive watcher finished first, but that's been taking a longer than I expected).

Also, if it still fails, a (short) program to reproduce the issue would be helpful (though not required); there's a ./cmd/fsnotify program you can use to print out events, and a small shell script in addition to that is enough.

[mokiat]

I tried using the latest in main. It works on Linux but still has issues in Docker on MacOS, where similar root-level REMOVE events are emitted.

[arp242]

I only have a QEMU machine for macOS, which just barely functions as-is. Trying to run Docker inside that would probably melt the laptop, desk, chair, and quite likely the rest of the house (if it even works at all).

I could try Docker on Linux host, but if I can't reproduce it there then it's highly unlikely I'll work on it any time soon.

[mokiat]

I understand. For the time being I will try to use a heuristic to solve the problem, knowing that some paths will drop from watching under bad conditions.

Maybe one day someone with a MacOS could pick this up or maybe I can find the time to dive into the depths of kqueue and try and tackle it myself 🤷 .

Anyway, thanks for the quick replies and trying to help out.

[arp242]

Actually looking at this again, you shouldn't get any events for step 2, since you're watching ".../internal", and not ".../internal/foo", so I guess you're setting up a watcher for that directory?

I think this may be related to a specific usage pattern rather than "docker on macOS".

This should have a program to reproduce the issue, or a test script. In the main branch you can just add those in testdata, like so:

diff --git c/testdata/watch-dir/bug-584 w/testdata/watch-dir/bug-584
new file mode 100644
index 0000000..fff55ce
--- /dev/null
+++ w/testdata/watch-dir/bug-584
@@ -0,0 +1,15 @@
+# https://github.com/fsnotify/fsnotify/issues/584
+
+watch /
+
+mkdir /dir
+#watch /dir
+touch /dir/file
+rm -r /dir
+
+mkdir /dir
+#watch /dir
+touch /dir/file
+
+Output:
+       create   /dir
+       remove   /dir
+       create   /dir

And then run it with something like:

TMPDIR=/home/martin/xxx go test -run TestScript/watch-dir/bug-584

Don't need to set TMPDIR, but may be useful in case it can only be reproduced on some filesystems/directories.

For more docs on that, see: https://github.com/fsnotify/fsnotify/blob/main/CONTRIBUTING.md#writing-new-tests

[mokiat]

Yes, you are right. I traverse directories and add watches to them manually.

It's been a while since I last played around with the tool (since the older version works ok-ish and does the job). Once I have a bit more free time, I will give your test script a try. It looks like a good way to narrow it down.

[arp242]

Cheers, or any way to reproduce it. I don't care about the exact form, some Go code combined with a shell script is also fine.