Fanotify Support

[opcoder0]

Initial work on adding fanotify support. Much of the code is from https://github.com/opcoder0/fanotify.

These are some of the things that are on my to do list -

• Adding features for these fanotify flags:
• unix.FAN_UNLIMITED_QUEUE
• unix.FAN_UNLIMITED_MARKS
• unix.FAN_REPORT_TID
• unix.FAN_ENABLE_AUDIT
• I have not yet been able to test on certain kernel versions to test the need for getFileHandle and getFileHandleWithName.
• Figure out how to wire fanotify tests into the current structure (requires CAP_SYS_ADMIN).
• Figure out how to get the docs show up correctly.
• Add examples

In addition to any other comments, I would appreciate feedback on the APIs as I am not sure how to maintain the API compatability with inotify and fanotify.

Closes #114

[arp242]

I don't think that any fanotify support should add new symbols or features, other than a setting to select that you want to use it. Beyond that, it should just provide the same feature set the existing backends do.

Need to see what the best way would be to select the fanotify backend; for now I'd just make a WatcherFanotify or something, so we can focus only on fanotify. I did a bit of work for options in #521.

Things like additional event types and such are not cross-platform. New event types like fsnotify.Read, fsnotify.Execute, etc. should be implemented on all platforms (and that should be a different PR too). There's #67 to add support for non-portable events.

[arp242]

I don't think we should have any references to fanotify in the backend_inotify.go file.

What I'd do for now is adjust the build tags on backend_inotify.go so it never compiles and then implement a new Watcher in backend_fanotify.go. This way all the tests should run against the fanotify backend.

I need to think a bit on how to best architecture things on this so you can tell you prefer to use fanotify; my idea was that you can select the backend per-watch.

[opcoder0]

Sounds good. A step closer. Will update back soon.

[opcoder0]

I don't think that any fanotify support should add new symbols or features, other than a setting to select that you want to use it. Beyond that, it should just provide the same feature set the existing backends do.

Need to see what the best way would be to select the fanotify backend; for now I'd just make a WatcherFanotify or something, so we can focus only on fanotify. I did a bit of work for options in #521.

Things like additional event types and such are not cross-platform. New event types like fsnotify.Read, fsnotify.Execute, etc. should be implemented on all platforms (and that should be a different PR too). There's #67 to add support for non-portable events.

Thanks for the comments and information regarding the issues.

Regarding #521 AddWith is going to be useful for specifying custom event types like it's been shown in an example on the issue or to specify customizations after the watcher is initialized. With Fanotify I see the watcher initialization parameters vary depending on what needs to be watched and the settings are mutually exclusive. For instance to watch the entire mount point the (fanotify_init) flags are different when compared to monitoring files under a set of paths. And flags are different for permissions. Once the NewWatcher is initialized for monitoring a set of paths AddWith with mount point functions should either return errors or ignore the settings. I'll try to think a bit more about this after addressing the other two comments.

[arp242]

My idea was that you can set the backend per-watch. This will be useful especially with the poll watcher: for example if you want to watch /home/foo and /mnt/nfs where /mnt/nfs doesn't support anything other than a poll watcher on account of being an NFS drive. You'd have to create 2 watchers otherwise, and it will all be very awkward to use. You can also fairly easily do things like "start watching with inotify, and if that doesn't work remove the watch and re-add it as poll" or "start watching with fanotify and if that returns an error because it's not supported on my kernel remove it and re-add using inotify".

For instance to watch the entire mount point the (fanotify_init) flags are different when compared to monitoring files under a set of paths.

This isn't really something we can easily support in fsnotify in the first place, since the goal is to make all features work well on all supported platforms; it's a bit of a "lowest common denominator" thing. The downside is that you can't use these kind of (often useful) features, but the upside is you can write your code on your Linux laptop and be reasonably confident it will also work well for people on macOS, OpenBSD, Windows, etc.

I'm not against adding support for this kind of thing, somehow, because in a number of use cases you just don't care about cross-platform support and obviously these kind of things are useful. But need to think about how to do this in a way that's 1) not ugly, 2) doesn't lead to people accidentally writing applications that will only work correct on fanotify, and 3) is compatible (so we don't need to make a v2). I'm not yet 100% sure how to best do that.

But in general, fsnotify won't be as "fully featured" like the fanotify Go library you wrote because the cross-platform support makes everything a lot harder.

[arp242]

The path forward would be something like:

1. Figure out a way to set the backend, merge that (I can work on that).
2. Implement fanotify_backend.go, supporting all the features inotify does (and nothing more), and merge that.
3. Figure out a way to deal with platform-dependent options you want to pass to a watcher and/or platform-dependent events, and merge that. The hard part here is just figuring out a good API.
4. Implement some platform-dependent things for fanotify.

Also, worth pointing out I'm working on adding recursive watch support for inotify (and the other backends): using w.Add("/tmp/path/...") will recursively watch all files and directories in /tmp/path`. Would probably be a good idea to also add support for that from the get-go here. Related PRs: #540, #472.

Come to think of it, we can perhaps use "watch entire mount point" as an optimisation if you do w.Add("/mnt/...") 🤔

[opcoder0]

The path forward would be something like:

1. Figure out a way to set the backend, merge that (I can work on that).
2. Implement fanotify_backend.go, supporting all the features inotify does (and nothing more), and merge that.
3. Figure out a way to deal with platform-dependent options you want to pass to a watcher and/or platform-dependent events, and merge that. The hard part here is just figuring out a good API.
4. Implement some platform-dependent things for fanotify.

Also, worth pointing out I'm working on adding recursive watch support for inotify (and the other backends): using w.Add("/tmp/path/...") will recursively watch all files and directories in /tmp/path`. Would probably be a good idea to also add support for that from the get-go here. Related PRs: #540, #472.

Come to think of it, we can perhaps use "watch entire mount point" as an optimisation if you do w.Add("/mnt/...") thinking

Sounds great. I'll then proceed with the (2) one for now. I think I'll be able to disable inotify on my feature branch and make fanotify have all the features inotify provides. And then run the existing tests. If Fanotify supports all features of Inotify then the NewWatcher would be for watching paths only for now.

Also, worth pointing out I'm working on adding recursive watch support for inotify (and the other backends): using w.Add("/tmp/path/...") will recursively watch all files and directories in /tmp/path`. Would probably be a good idea to also add support for that from the get-go here. Related PRs: #540, #472.

I can try and explore on an idea to make the Fanotify Watcher a composition of multiple fanotify watcher types one for each (paths, permissions, mount point) that way the composite watcher is the same while the user can still call w.Add("/mnt/...") and w.Add("/home/foo"). I will have to explore when if I can properly determine when w.Add("/path/...") is called if the path is a directory or the entire mountpoint. May be I can query the /proc/mounts to match the path. Not too sure yet. But there might be a way.

[arp242]

I can try and explore on an idea to make the Fanotify Watcher a composition of multiple fanotify watcher types one for each (paths, permissions, mount point) that way the composite watcher is the same while the user can still call w.Add("/mnt/...") and w.Add("/home/foo"). I will have to explore when if I can properly determine when w.Add("/path/...") is called if the path is a directory or the entire mountpoint. May be I can query the /proc/mounts to match the path. Not too sure yet. But there might be a way.

Personally I'd just keep things as simple/minimal as possible for the initial version, and add additional things later on. It doesn't all have to be in one PR, and just getting the initial implementation right will already be plenty of work. I'd be fine just not doing recursion at all for the initial version and adding that later on.

But it's your PR so you can decide.

[opcoder0]

I can try and explore on an idea to make the Fanotify Watcher a composition of multiple fanotify watcher types one for each (paths, permissions, mount point) that way the composite watcher is the same while the user can still call w.Add("/mnt/...") and w.Add("/home/foo"). I will have to explore when if I can properly determine when w.Add("/path/...") is called if the path is a directory or the entire mountpoint. May be I can query the /proc/mounts to match the path. Not too sure yet. But there might be a way.

Personally I'd just keep things as simple/minimal as possible for the initial version, and add additional things later on. It doesn't all have to be in one PR, and just getting the initial implementation right will already be plenty of work. I'd be fine just not doing recursion at all for the initial version and adding that later on.

But it's your PR so you can decide.

Yep I agree. I wasn't planning on making those changes here :). As a first step making Fanotify and Inotify functionally equivalent is good.

[opcoder0]

@arp242 with the recent commits I've addressed -

• Removing dependency on gocapability
• The Watcher + API is now simple (no permission events). Similar to Inotify provides.
• Disabled inotify build on Linux
• Fixed tests; I had to add special case for fanotify output as they don't always align with inotify / linux case
• Fixed some race issues around Close method

NOTE - I had to introduce a temporary flag isFanotify in the Watcher to skip certain tests.

I am seeing a couple of problems -

• When I run all the tests; Some tests like TestWatchCreate (and a few others) fail. However when those tests are run by themselves they pass.
• TestWatchStress always fails.

[arp242]

How do you run the tests? I tried:

% go test -short -failfast
--- FAIL: TestWatch (0.00s)
    --- FAIL: TestWatch/multiple_creates (0.00s)
        helpers_test.go:333: newWatcher: require CAP_SYS_ADMIN capability

% go test -c
% doas setcap cap_sys_admin=+ep ./fsnotify.test

% ./fsnotify.test -test.failfast
--- FAIL: TestWatch (0.00s)
    --- FAIL: TestWatch/watch_same_dir_twice (0.75s)
        helpers_test.go:386: operation not permitted

I added a bit more context with the below patch, and it seems to fail on the unix.OpenByHandleAt call:

--- FAIL: TestWatch (0.00s)
    --- FAIL: TestWatch/watch_same_dir_twice (0.76s)
        helpers_test.go:386: unix.OpenByHandleAt: operation not permitted

---

diff --git backend_fanotify_event.go backend_fanotify_event.go
index ec2d77b..1432db6 100644
--- backend_fanotify_event.go
+++ backend_fanotify_event.go
@@ -405,14 +405,12 @@ func (w *Watcher) start() {
 			continue
 		}
 		if err != nil {
-			if err == unix.EINTR {
-				continue
-			} else {
-				if !w.sendError(err) {
+			if err != unix.EINTR {
+				if !w.sendError(fmt.Errorf("Watcher.start: %w", err)) {
 					return
 				}
-				continue
 			}
+			continue
 		}
 		if fds[1].Revents != 0 {
 			if fds[1].Revents&unix.POLLIN == unix.POLLIN {
@@ -460,9 +458,9 @@ func (w *Watcher) checkPathUnderMountPoint(path string) (bool, error) {
 // for subdirectories. Calling AddWatch to mark the entire mountpoint results in
 // [os.ErrInvalid]. To watch the entire mount point use [WatchMount] method.
 // Certain flag combinations are known to cause issues.
-//  - [FileCreated] cannot be or-ed / combined with [FileClosed]. The fanotify system does not generate any event for this combination.
-//  - [FileOpened] with any of the event types containing OrDirectory causes an event flood for the directory and then stopping raising any events at all.
-//  - [FileOrDirectoryOpened] with any of the other event types causes an event flood for the directory and then stopping raising any events at all.
+//   - [FileCreated] cannot be or-ed / combined with [FileClosed]. The fanotify system does not generate any event for this combination.
+//   - [FileOpened] with any of the event types containing OrDirectory causes an event flood for the directory and then stopping raising any events at all.
+//   - [FileOrDirectoryOpened] with any of the other event types causes an event flood for the directory and then stopping raising any events at all.
 func (w *Watcher) fanotifyAddPath(path string) error {
 	if w.isClosed {
 		return ErrClosed
@@ -566,7 +564,7 @@ func (w *Watcher) readFanotifyEvents() error {
 			continue
 		}
 		if err != nil {
-			if !w.sendError(err) {
+			if !w.sendError(fmt.Errorf("readFanotifyEvents: read from buf: %w", err)) {
 				return err
 			}
 		}
@@ -587,7 +585,7 @@ func (w *Watcher) readFanotifyEvents() error {
 				procFdPath := fmt.Sprintf("/proc/self/fd/%d", metadata.Fd)
 				n1, err := unix.Readlink(procFdPath, name[:])
 				if err != nil {
-					if !w.sendError(err) {
+					if !w.sendError(fmt.Errorf("readFanotifyEvents: converting data: %w", err)) {
 						return err
 					}
 					i += int(metadata.Event_len)
@@ -648,7 +646,7 @@ func (w *Watcher) readFanotifyEvents() error {
 				}
 				fd, errno := unix.OpenByHandleAt(int(w.mountPointFile.Fd()), *fileHandle, unix.O_RDONLY)
 				if errno != nil {
-					if !w.sendError(errno) {
+					if !w.sendError(fmt.Errorf("unix.OpenByHandleAt: %w", errno)) {
 						// fmt.Println("oops something wrong. returning:", errno)
 						return errno
 					}

[opcoder0]

How do you run the tests? I tried:

% go test -short -failfast
--- FAIL: TestWatch (0.00s)
    --- FAIL: TestWatch/multiple_creates (0.00s)
        helpers_test.go:333: newWatcher: require CAP_SYS_ADMIN capability

% go test -c
% doas setcap cap_sys_admin=+ep ./fsnotify.test

% ./fsnotify.test -test.failfast
--- FAIL: TestWatch (0.00s)
    --- FAIL: TestWatch/watch_same_dir_twice (0.75s)
        helpers_test.go:386: operation not permitted

I added a bit more context with the below patch, and it seems to fail on the unix.OpenByHandleAt call:

--- FAIL: TestWatch (0.00s)
    --- FAIL: TestWatch/watch_same_dir_twice (0.76s)
        helpers_test.go:386: unix.OpenByHandleAt: operation not permitted

diff --git backend_fanotify_event.go backend_fanotify_event.go
index ec2d77b..1432db6 100644
--- backend_fanotify_event.go
+++ backend_fanotify_event.go
@@ -405,14 +405,12 @@ func (w *Watcher) start() {
 			continue
 		}
 		if err != nil {
-			if err == unix.EINTR {
-				continue
-			} else {
-				if !w.sendError(err) {
+			if err != unix.EINTR {
+				if !w.sendError(fmt.Errorf("Watcher.start: %w", err)) {
 					return
 				}
-				continue
 			}
+			continue
 		}
 		if fds[1].Revents != 0 {
 			if fds[1].Revents&unix.POLLIN == unix.POLLIN {
@@ -460,9 +458,9 @@ func (w *Watcher) checkPathUnderMountPoint(path string) (bool, error) {
 // for subdirectories. Calling AddWatch to mark the entire mountpoint results in
 // [os.ErrInvalid]. To watch the entire mount point use [WatchMount] method.
 // Certain flag combinations are known to cause issues.
-//  - [FileCreated] cannot be or-ed / combined with [FileClosed]. The fanotify system does not generate any event for this combination.
-//  - [FileOpened] with any of the event types containing OrDirectory causes an event flood for the directory and then stopping raising any events at all.
-//  - [FileOrDirectoryOpened] with any of the other event types causes an event flood for the directory and then stopping raising any events at all.
+//   - [FileCreated] cannot be or-ed / combined with [FileClosed]. The fanotify system does not generate any event for this combination.
+//   - [FileOpened] with any of the event types containing OrDirectory causes an event flood for the directory and then stopping raising any events at all.
+//   - [FileOrDirectoryOpened] with any of the other event types causes an event flood for the directory and then stopping raising any events at all.
 func (w *Watcher) fanotifyAddPath(path string) error {
 	if w.isClosed {
 		return ErrClosed
@@ -566,7 +564,7 @@ func (w *Watcher) readFanotifyEvents() error {
 			continue
 		}
 		if err != nil {
-			if !w.sendError(err) {
+			if !w.sendError(fmt.Errorf("readFanotifyEvents: read from buf: %w", err)) {
 				return err
 			}
 		}
@@ -587,7 +585,7 @@ func (w *Watcher) readFanotifyEvents() error {
 				procFdPath := fmt.Sprintf("/proc/self/fd/%d", metadata.Fd)
 				n1, err := unix.Readlink(procFdPath, name[:])
 				if err != nil {
-					if !w.sendError(err) {
+					if !w.sendError(fmt.Errorf("readFanotifyEvents: converting data: %w", err)) {
 						return err
 					}
 					i += int(metadata.Event_len)
@@ -648,7 +646,7 @@ func (w *Watcher) readFanotifyEvents() error {
 				}
 				fd, errno := unix.OpenByHandleAt(int(w.mountPointFile.Fd()), *fileHandle, unix.O_RDONLY)
 				if errno != nil {
-					if !w.sendError(errno) {
+					if !w.sendError(fmt.Errorf("unix.OpenByHandleAt: %w", errno)) {
 						// fmt.Println("oops something wrong. returning:", errno)
 						return errno
 					}

I usually ran with sudo (as root). But this command below one gives permissions to the system calls fanotify_init (cap_sys_admin), open_by_handler_at (cap_owner_dac) and unlink (cap_fowner). and seems to run the tests -

go test -c
sudo setcap cap_dac_read_search,cap_sys_admin,cap_fowner+ep ./fsnotify.test

I am seeing some issue with determining the mount ID when I run with the above command. That fails a few tests. I'll try to fix that in my next commit.

[opcoder0]

The cap_dac_override was required for unlinking a file that was in the unreadable directory. This works for all tests -

go test -c
sudo setcap cap_dac_read_search,cap_sys_admin,cap_fowner,cap_dac_override+epi ./fsnotify.test

[arp242]

Are you still interested in working on this?

[arp242]

BTW what this needs is to be reworked to be much simpler, not adding any new features and things like that. Basically something like the below patch, except, well, working, passing all the tests, etc. – this is just something I cooked up quickly.

Adding new event types, mountpoint watching, and things like that are all out of scope here. Also just get it running on the latest Linux; don't worry about supporting older Linux systems (there's quite a lot of code for this).

commit d019cd7
Author: Martin Tournoij <martin@arp242.net>
Date:   Sat Apr 27 21:47:58 2024 +0100

    wip

diff --git a/backend_fanotify.go b/backend_fanotify.go
new file mode 100644
index 0000000..937c322
--- /dev/null
+++ b/backend_fanotify.go
@@ -0,0 +1,192 @@
+//go:build linux && !appengine
+
+package fsnotify
+
+import (
+	"fmt"
+	"os"
+	"sync"
+	"time"
+	"unsafe"
+
+	"golang.org/x/sys/unix"
+)
+
+type Watcher struct {
+	Events chan Event
+	Errors chan error
+
+	fd     int
+	done   chan struct{}
+	doneMu sync.Mutex
+}
+
+func NewWatcher() (*Watcher, error) {
+	return NewBufferedWatcher(0)
+}
+
+func NewBufferedWatcher(sz uint) (*Watcher, error) {
+	fd, err := unix.FanotifyInit(
+		unix.FAN_CLASS_NOTIF|unix.FAN_REPORT_FID|unix.FAN_REPORT_DIR_FID|unix.FAN_REPORT_NAME|
+			unix.FAN_NONBLOCK|
+			unix.FAN_CLOEXEC,
+		unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NONBLOCK,
+	)
+	if err != nil {
+		return nil, fmt.Errorf("fanotify_init: %w", err)
+	}
+
+	w := &Watcher{
+		fd:     fd,
+		Events: make(chan Event, sz),
+		Errors: make(chan error),
+		done:   make(chan struct{}),
+	}
+	go w.readEvents()
+	return w, nil
+}
+
+func (w *Watcher) sendEvent(e Event) bool {
+	select {
+	case <-w.done:
+		return false
+	case w.Events <- e:
+		return true
+	}
+}
+
+func (w *Watcher) sendError(err error) bool {
+	if err == nil {
+		return true
+	}
+	select {
+	case <-w.done:
+		return false
+	case w.Errors <- err:
+		return true
+	}
+}
+
+func (w *Watcher) isClosed() bool {
+	select {
+	case <-w.done:
+		return true
+	default:
+		return false
+	}
+}
+
+func (w *Watcher) Close() error {
+	w.doneMu.Lock()
+	if w.isClosed() {
+		w.doneMu.Unlock()
+		return nil
+	}
+	close(w.done)
+	w.doneMu.Unlock()
+
+	return nil
+}
+
+func (w *Watcher) Add(path string) error { return w.AddWith(path) }
+
+var faMask uint64 = unix.FAN_ONDIR |
+	unix.FAN_CREATE |
+	unix.FAN_MODIFY |
+	unix.FAN_DELETE | unix.FAN_DELETE_SELF |
+	unix.FAN_ATTRIB |
+	unix.FAN_MOVED_FROM | unix.FAN_MOVED_TO | unix.FAN_MOVE_SELF
+
+func (w *Watcher) AddWith(path string, opts ...addOpt) error {
+	if w.isClosed() {
+		return ErrClosed
+	}
+	if debug {
+		fmt.Fprintf(os.Stderr, "FSNOTIFY_DEBUG: %s  AddWith(%q)\n",
+			time.Now().Format("15:04:05.000000000"), path)
+	}
+
+	return unix.FanotifyMark(w.fd, unix.FAN_MARK_ADD, faMask, -1, path)
+}
+
+func (w *Watcher) Remove(path string) error {
+	if w.isClosed() {
+		return nil
+	}
+	if debug {
+		fmt.Fprintf(os.Stderr, "FSNOTIFY_DEBUG: %s  Remove(%q)\n",
+			time.Now().Format("15:04:05.000000000"), path)
+	}
+	return unix.FanotifyMark(w.fd, unix.FAN_MARK_REMOVE, faMask, -1, path)
+}
+
+func (w *Watcher) WatchList() []string {
+	if w.isClosed() {
+		return nil
+	}
+	return nil
+}
+
+const sizeOfFanotifyEventMetadata = uint32(unsafe.Sizeof(unix.FanotifyEventMetadata{}))
+
+func (w *Watcher) readEvents() {
+	defer func() {
+		close(w.Errors)
+		close(w.Events)
+	}()
+
+	var (
+		buf  [4096 * sizeOfFanotifyEventMetadata]byte
+		path [unix.PathMax]byte
+	)
+	for {
+		if w.isClosed() {
+			return
+		}
+		_, err := unix.Read(w.fd, buf[:])
+		if !w.sendError(err) {
+			return
+		}
+
+		type fanotifyEventInfoFID struct {
+			Header struct {
+				InfoType uint8
+				pad      uint8
+				Len      uint16
+			}
+			fsid       [2]int32
+			fileHandle byte
+		}
+		i := 0
+		metadata := (*unix.FanotifyEventMetadata)(unsafe.Pointer(&buf[i]))
+
+		mask := metadata.Mask
+		if mask&unix.FAN_ONDIR == unix.FAN_ONDIR {
+			mask = mask ^ unix.FAN_ONDIR
+		}
+		p := 0 // TODO
+		if !w.sendEvent(w.newEvent(string(path[:p]), mask)) {
+			return
+		}
+	}
+}
+
+func (w *Watcher) newEvent(path string, mask uint64) Event {
+	e := Event{Name: path}
+	if mask&unix.FAN_CREATE == unix.FAN_CREATE || mask&unix.FAN_MOVED_TO == unix.FAN_MOVED_TO {
+		e.Op |= Create
+	}
+	if mask&unix.FAN_DELETE == unix.FAN_DELETE || mask&unix.FAN_DELETE_SELF == unix.FAN_DELETE_SELF {
+		e.Op |= Remove
+	}
+	if mask&unix.FAN_MODIFY == unix.FAN_MODIFY || mask&unix.FAN_CLOSE_WRITE == unix.FAN_CLOSE_WRITE {
+		e.Op |= Write
+	}
+	if mask&unix.FAN_MOVE_SELF == unix.FAN_MOVE_SELF || mask&unix.FAN_MOVED_FROM == unix.FAN_MOVED_FROM {
+		e.Op |= Rename
+	}
+	if mask&unix.FAN_ATTRIB == unix.FAN_ATTRIB {
+		e.Op |= Chmod
+	}
+	return e
+}
diff --git a/backend_inotify.go b/backend_inotify.go
index 2f0d0a0..178f9fd 100644
--- a/backend_inotify.go
+++ b/backend_inotify.go
@@ -1,4 +1,4 @@
-//go:build linux && !appengine
+//go:build ignore
 
 // Note: the documentation on the Watcher type and methods is generated from
 // mkdoc.zsh
diff --git a/backend_inotify_test.go b/backend_inotify_test.go
index 40b9ef0..92761f2 100644
--- a/backend_inotify_test.go
+++ b/backend_inotify_test.go
@@ -1,4 +1,4 @@
-//go:build linux
+//go:build ignore
 
 package fsnotify