add Solaris FEN support

[isaacdavis]

This pull request fixes #12.

I'm an employee of Joyent, Inc., and have completed the Google corporate CLA.

What does this pull request do?

This pull request adds fsnotify support for Solaris and illumos via Solaris FEN. It is heavily based on #196, authored by @gfrey. As such, and with his approval, I have added @gfrey in the AUTHORS file in addition to myself.

One major revision to @gfrey's work concerns liveness when the watcher is closed when there is an unconsumed event in the channel - the old pull request would sometimes hang on the TestWatcherClose test. The new concurrency model is borrowed from fsnotify's inotify implementation.

Another major revision concerns proper translation of FEN events to fsnotify events, in particular the FILE_RENAME_TO FEN event.

Where should the reviewer start?

These pages are useful as an introduction to FEN:

• https://xnet.fi/solaris/
• https://solarisrants.wordpress.com/2013/07/24/solaris-file-event-notification/

The illumos man pages for port_create and related functions are also helpful for understanding how to interface with FEN.

The discussion on #196 is also worth reading. Some points I'll address here:

• The use of syscall appears necessary here because golang.org/x/sys/unix does not (to my knowledge) provide a way to get atime and ctime.
• cgo is used because, for illumos, the officially supported interface to the OS is libc rather than syscalls, and thus user-level code should refrain from making syscalls directly.

How should this be manually tested?

I ran all tests on a SmartOS host. On SmartOS, Go can be installed from Joyent's pkgsrc repository. fsnotify can then be built and tested as normal.

[isaacdavis]

The Travis CI build is failing only on MacOS on go 1.8. I'm not sure why this is, given that the code I changed affects only Solaris. I'm additionally unable to reproduce the failure on my own Mac. If anyone can offer insight, I would much appreciate it...

[gfrey]

I created issue #264. Seems to be a flake (it was locally and last PR builds seem to have worked).

[nathany]

#264 is now fixed on master. Thanks @gfrey. Please rebase.

Is there a CI solution for Solaris?

Is this sufficiently reviewed by people familiar with Solaris? I'll also do a review before it is merged.

[gdey]

kick integrations.

[isaacdavis]

Looks like macOS builds are still failing even with the fix.

@nathany Joyent would be happy to offer a Triton container on the Joyent Public Cloud in which to build and test fsnotify. Under-the-hood this is a SmartOS zone that we'd administer and give you access to. I could assist with installing Go and setting up the build. You could then integrate this with the CI system of your choice. Please let me know if you're interested or have questions - feel free to email isaac.davis@joyent.com.

[nathany]

@isaacdavis A CI solution would be great, though I can't say I have any idea how to integrate a Triton container with a hosted solution like Travis CI.

Also, before adding Solaris support, I want to be sure there is a point person (or people would be better) for Solaris support that are handling any issues. /cc @gfrey

We also want to be sure that Solaris support is documented, especially if there are behaviour differences between operating systems that users need to know about.

One other note: at GopherCon we have been discussing standalone packages for each OS that fsnotify relies on. This would likely look like github.com/fsnotify/fsnotify/fen. @gdey is starting on a shared test helper package to reuse across packages.

[isaacdavis]

@nathany It's a little hacky, but I suppose in Travis you could write a script that ssh'ed in to the container and ran the tests there, sending back the results. This would also work with any other hosted CI solution, I think.

[isaacdavis]

I've discovered a bug in the current implementation under review:

The port_associate function accepts a pointer to an object describing the entity being watched. port_get, when returning from an event, returns the same pointer as the portev_object member of the port_event_t struct returned from port_get. Thus, if the event-handling code uses this pointer to retrieve information about the file affected (as we currently do), it must still point to valid memory.

In associateFile, we construct a file_obj and pass a pointer to it to port_associate. The Go runtime can relocate or garbage-collect this file_obj, rendering the pointer invalid. We thus may get garbage data back when we retrieve the object upon returning from port_get.

This bug does not manifest in the test suite, as there is not enough memory churn to trigger an overwrite of the file_obj memory. I discovered it when running Prometheus on SmartOS using fsnotify to watch config files.

I'll have a fix committed by tomorrow (2018-09-06) at noon PST.

[isaacdavis]

I'd like to check up on the status of this - what work needs to be done to get this merged? Please let me know what I can do to help.

[nathany]

Thanks for your work on this Isaac.

For this review, please keep in mind that I'm not familiar with FEN, nor have I exercised this code or even ensured that it compiles (I don't have a Solaris environment, nor am I personally planning to set one up).

Structurally, I would prefer if the cgo code were more isolated as we have done for fsevents, but I'm not sure if that can even be done to the same extent.

It appears that FEN requires that a malloced file_obj_t remain allocated for the lifetime of the watch, is that correct?

[nathany]

This list of of files includes subdirectories.

Is this the behaviour you want? To call associateFile/disassociateFile on a subdirectory?

(keeping it non-recursive for now is probably best, as the other platforms are not recursive yet)

[isaacdavis]

My understanding is that, if we add a watch to a directory, we want to watch all of its immediate children for changes - this would include children that are directories. This isn't recursive in that we don't descend into those subdirectories and watch the files/directories therein, walking down the entire tree - we just watch the subdirectory vnode itself and stop there.

Is this incorrect? Should children that are directories be explicitly omitted?

[nathany]

That's correct.

[isaacdavis]

fixed.

[nathany]

@isaacdavis Apologies for the delay in reviewing your work. Here are some considerations before merging this.

I'm not personally willing to take on the responsibility of supporting Solaris and FEN myself, setting up CI, or any of that. What I'd like to see is yourself, but preferably multiple people, taking ownership of the FEN support.

The Solaris group would work with myself and hopefully some others on cross-platform API considerations (rename to, recursive watching, filtering, etc.). If there are bugs in the FEN support, I need to know there are people familiar with FEN that will triage issues and provide fixes. If/when we do a v2 API, I need to know there are people familiar with FEN that will help us move fsnotify forward on all platforms.

Honestly, I'd like to "recruit" small teams for each platform, but Solaris is a good place to start.

[nathany]

@isaacdavis In the near term I'd like to break fsnotify up into a few packages within the same repository. I've already begun some work in that direction with kqueue, by isolating the syscall wrappers.

This would allow someone to import "github.com/fsnotify/fsnotify/fen" and use it directly if they really want. My idea with the wrapper is to provide a very low-level platform-specific Go-ish API that provides documentation and testing (improving our means of testing fsnotify will be part of this effort). Also, it could provide all the things for the platform, whereas fsnotify is lowest common denominator.

Not saying this needs to happen before the PR is merged, but letting you know the direction I'm thinking of taking things.

The big question is how low-level is very low level. For example, the kqueue API takes file handles, not names of files to watch, so the kqueue wrapper may operate at that low level with very little in the way of amenities.

Part of my goal with this is to associate ownership to each subpackage. A small team managing each of fen, inotify, kqueue, fsevents -- possibly some people on multiple teams, and everyone participating in the common API design as well.

[isaacdavis]

@nathany Thanks for your thorough responses. I'm happy to take ownership of FEN support and get involved in this project!

Regarding CI: After talking with some others at Joyent, we think the easiest solution would be to set up a Jenkins instance on a VM in the Joyent Public Cloud, and run the build and tests there. This would integrate well with Github. I would be your point of contact for administering this. Does that sound reasonable to you?

One other note: I only have the resources to provide illumos support - the build and tests would run on illumos, not Oracle Solaris. illumos diverged from Oracle Solaris in 2010, and there is no guarantee that the two operating systems share a common FEN API.

If this all sounds reasonable, let me know and I'll go ahead with addressing the code review and setting up CI. Thanks!

[nathany]

@isaacdavis Thank you. That all sounds good to me. Use whatever CI works for you [though I will mention Drone.io may be worth looking into, as an open source alternative to Jenkins written in Go]

I think we'll document in the README who the teams are for each platform with respective GitHub handles, and perhaps figure out how to do some @ lists [I can look into these things].

Hopefully we will see some other people commit to support FEN, and maybe even Solaris specifically, but we can move ahead either way if it's okay with you. /cc @gfrey @4ad @amitkris

[isaacdavis]

@nathany Sounds good. I'll get started setting everything up.

[houndci-bot]

comment on exported var ErrEventOverflow should be of the form "ErrEventOverflow ..."

[nathany]

note: if you rebase on master, this was resolved

[isaacdavis]

Hi @nathany - happy 2019! Just checking up on this. Joyent is still eager to get FEN support merged. I've updated my above comment about Jenkins slightly.

[nathany]

@isaacdavis An "fsnotify automation" account sounds good. I've also sent you an invite to grant write access and can bump your permissions further once accepted.

[isaacdavis]

@nathany great - I've accepted the invite.

[isaacdavis]

@nathany ok, I've created an automation account. This account will need to be given write access to the repo -- once that's done, I can set up the Jenkins webhook and build job.

[jubalskaggs]

@isaacdavis thank you for your work on this - I'm just chiming in to say that we're testing this pr on solaris 11 (also illumos) and it's working just fine. I'd love to see solaris-fen merged into mainline fsnotify :)

[nathany]

Apologies for the delay. It looks like this pull request has fallen out of date a bit and still needs a review by someone familiar with FEN.

[Smithx10]

@isaacdavis @nathany I'm not sure if go 1.13 came out yet in May of 2019, but with it came the "illumos" build tag. Are we waiting on someone from Solaris? If so perhaps @isaacdavis would only have to maintain the Illumos tag since they are different at this point.

[Smithx10]

I tried Cross Compiling from Linux to Illumos (go 1.13) https://github.com/papertrail/go-tail with this PR and received the following error:

../../../go-tail/follower/follower.go:55:12: undefined: fsnotify.Watcher

On Illumos it builds fine. Is this to be expected?

I'm assuming I am missing something to make the cross compiling work.

[isaacdavis]

@Smithx10 @nathany It's new information that this doesn't build on Solaris, though I'm not surprised given the divergence between the two operating systems. Now that the illumos build tag exists, I think it's a good idea for fsnotify to begin differentiating between illumos and Solaris to avoid these types of issues.

@nathany -- do you have an interest in introducing the illumos build tag to fsnotify? I'm happy to create a PR for that and maintain the illumos tag. I only have the resources to build and test on illumos, so we'd have to find someone else to maintain the Solaris tag going forward.

The remaining work of getting another reviewer and granting access to the Jenkins server still stands. I realize I've neglected this, but I'm happy to get back to it. @nathany just confirm that you're willing for fsnotify to begin using the illumos build tag before I go ahead with a PR.

Thank you both!

[Smithx10]

@nathany @isaacdavis any update on this, or perhaps some flags for the cgo cross compile?

[Smithx10]

@nathany Anything I can do to help move this along?

[Smithx10]

./inserts jeopardy music here.

[isaacdavis]

@Smithx10 I've played around with this a little bit -- my hunch is that cross-compilation of programs that use cgo is difficult, because you'd also need to set up the C toolchain to cross-compile. This would involve obtaining a linux-to-illumos C cross-compiler and setting it up to link with the illumos libraries, etc.

I'm not sure if this has ever been done -- all I could find via google was this: https://gist.github.com/AlainODea/6679613

[Smithx10]

For cross compilation support I think we can rewrite this just using go. I believe that’s what the other distros did.

[Toasterson]

Hello everyone,

I see that this PR is stuck. Is there anything we from the illumos community can do to advance this @nathany ?

[nshalman]

I'm working on getting these Event Ports primitives into x/sys/unix and then cleaning up this code to use those instead of cgo.
So far I have some basic bits ported (though not yet upstreamed) sitting in this branch:
nshalman/sys@master...nshalman:fsnotify-fen

I also have the beginnings of the port of this code to use those changes here:
nshalman#1

Unfortunately that code currently passes most but not all of the integration tests, so I'm currently a bit confused and temporarily blocked. Eventually when I get it all working and get the x/sys/unix changes through the review and upstreaming process, I'll inevitably further refactor my fsnotify code and then open a PR to supersede this one.

My goal is to remove all use of cgo and unsafe in the fsnotify code.

[nshalman]

For anyone checking in here, https://go-review.googlesource.com/c/sys/+/324630 is close to complete, and an updated PR to supersede this one will follow after that finally lands.

[nathany]

Closing in favour of #371. Thanks for working on this.