Polling fallback

[nathany]

Whether or not fsnotify implements polling support itself, some thought should be given into how polling could work as an alternative to native OS events.

GoConvey uses polling exclusively to avoid the "too many files" error #8.

"we walk the file system every quarter second and use the sum of the last mod time stamp and size of each go file as a quick comparison. Along the way we make note of new and deleted packages, all the while skipping 'ignored' packages. It's actually quite speedy." - @mdwhatcott, Gopher Slack

If you use an approach like what I've done in GoConvey, make sure to count empty directories as well... - @mdwhatcott

https://github.com/smartystreets/goconvey/blob/master/web/server/watcher/scanner.go

@pifantastic reports that node's gaze (https://github.com/shama/gaze#errors) library detects EMFILE errors and falls back to polling.

Polling could be opt-in for network file systems (NFS), Vagrant or Plan 9 -- where OS events don't work or are unavailable.

[mdwhatcott]

FYI, the link (above) to the goconvey scanner is no longer valid (I've recently rewritten that package. My approach, however, is unchanged:

https://github.com/smartystreets/goconvey/tree/master/web/server/watch

[omeid]

👍 for polling.

It is not ideal, but it is better than none and instead of everyone developing it on their own, it makes sense to have one that is made better by many people.

In terms of the opt-in option, fsnotify could have an option to "register" different "watch providers", in the spirit of sql package. This would also allow people to develop their own providers for other services, say Dropbox, s3, and so forth, but maybe that is jumping ahead of ourselves.

[nathany]

I've never even thought of extending it to webhook notifications like Dropbox and Google Drive provide. Interesting idea. Not sure if it belongs in this package, but having a common interface in top could be cool.

The sql driver model is one I've been thinking about, particularly if there are multiple options for a given OS.

[omeid]

Yeah, it really depends how you look at it, I wouldn't consider them as general webhooks as you can think of dropbox or s3 just another type of filesystem.

Regardless of that, having a watch-driver interface and different driver is a better design IMHO.

[nathany]

Indeed. It also fits with another desire I have, which is to make it easier to contribute to without being knowledgable of every single platform. Separate drivers for inotify, kqueue, etc. would be one way to achieve that.

[szechyjs]

👍 polling would be great. I'm trying to migrate a python application from watchdog to fsnotify. Unfortunately I think this is the roadblock, as the filesystem that needs monitored is on NFS and polling is the only option.

[nathany]

Yah, polling is the only option for NFS as far as I know. You can also check https://github.com/rjeczalik/notify but I don't think it has polling yet either.

[calavera]

You might be interested in checking how docker support fsnotify and polling at the same time. We have a compatible interface and a fallback initialization for when fsnotify is not supported:

https://github.com/docker/docker/blob/master/pkg/filenotify/filenotify.go

maybe @cpuguy83 would be interested in moving the polling here so we can maintain only one package.

[cpuguy83]

That would be great.
The reason I did not submit here is it seemed rather unknown if polling support was desired and we really needed to implement it.

[nathany]

Please do submit a pull request.

[omeid]

@nathany Any plans for the v2 with driver interfaces? I will be happy to help out.

[nathany]

@omeid I don't have any plans for what a driver interface would look like yet (and to be honest, I haven't done much work on fsnotify lately).

Personally, I'd prefer to see the current code base cleaned up before doing a big API change. The Windows internals are particularity crufty.
https://github.com/go-fsnotify/fsnotify/milestones/v2%20Internals

Maybe we could start a new issue to figure out the details of transitioning to a driver model?

[cpuguy83]

@nathany Can you clarify what you'd like to see?

[nathany]

@cpuguy83 Do you think you would be able to add polling without changing the API? Maybe just for operating systems that fsnotify doesn't currently support?

There are other situations where polling would be desirable, but I'm not sure how to detect them, or if it should be done more manually (which is why a driver-style API is relevant to this discussion).

[nathany]

In terms of API, here are related issues #104 and #75.

Polling should also help with #45 for Windows users.

[nathany]

@mdwhatcott @cpuguy83 Would you be willing to build a stand-alone fsnotify/polling package that could be incorporated into fsnotify as a secondary step?

If so, I'll create a repo called polling or poller or whatever you prefer.

For me the key considerations are:

• providing a speedy implementation as outlined in the original post
• a simple and idiomatic Go API specifically for polling
• solid unit and/or integration tests (preferably without any external testing dependencies to keep it simple)
• well documented both in the README, examples, and API docs

[mdwhatcott]

@nathany - Sounds like a fun project but I can't commit to it at this time.

[nathany]

ok.

Here is a polling watcher by @radovskyb https://github.com/radovskyb/watcher

[nathany]

@radovskyb Hey Benjamin, Would you be interested in transferring watcher into the fsnotify organization and working on it here? Still as a stand-alone repository for the time being.

The API already looks pretty close to fsnotify.

Once some of the low-level bits are extracted from fsnotify (e.g. #173) I'd like to incorporate polling into fsnotify as a fallback, while still allowing people to use the poller directly if that's all they want.

[dtaniwaki]

I really wanted to use this feature, but since I can't find any stable one in public, I implemented an extensible polling logic for HDFS which implements Walk method in the HDFS gateway of argo-events by myself.

[liu-cong]

Is someone working on this now or is there any plan? If not, I am interested in contributing to this.

[Helflym]

I need this package in order to port auditbeat software (https://github.com/elastic/beats) on AIX.
However, AIX doesn't provide an easier way to watch files than polling them.
I'm planning to take some part of @radovskyb's work. But we don't have the end of your slack conversation here, @nathany and @radovskyb. I would like to know if any of you is against such thing ?

[serdarkalayci]

Just wanted to add my two cents about polling mechanism. In Kubernetes, when a ConfigMap which is attached as a file on a pod changes, the file inside the pod changes as well but its last modified timestamp does not change. Size may not change depending on the change. So the GoConvey's polling approach will not work on this scenario.
Francisco Beltrao from Microsoft offers a different approach which checks the hash of files instead of timesptamp. Here's a .NET Core implementation:
https://github.com/fbeltrao/ConfigMapFileProvider
It'll be slower than the aforementioned approach but it covers more ground.

[nathany]

@bep Recently added a polling fallback in Hugo that may be worth taking a look at.
gohugoio/hugo#8723