Need an interface to poll arbitrary kqueue filters in Tokio 0.3

[asomers]

Is your feature request related to a problem? Please describe.
With Tokio 0.1 and 0.2 I could create a custom implementation of mio::Evented that worked with any sort of kqueue filter, and build futures on that using tokio::io::PollEvented. However, PollEvented is no longer publicly exposed in Tokio 0.3. The suggested replacement, AsyncFd, only works for raw file descriptors. Basically, it's used for socket-like things. AFAICT, tokio 0.3 provides no facility to poll arbitrary kqueue filters.

Describe the solution you'd like
Ideally PollEvented would be public again.

Describe alternatives you've considered
I could create a new kqueue file descriptor for my future, registry my future to that, and poll that kqueue with AsyncFd. But that would require me to handle kevent(2) myself, and I would have to create a new kqueue for every single future I make. Either that, or basically create some kind of subreactor just for all futures using my kqueue filter. Basically, I would have to reimplement a large chunk of Tokio.

Additional context

• https://github.com/asomers/mio-aio POSIX AIO bindings for mio. Works with tokio-0.1 and tokio-0.2 (the latter in the std_future2 branch).
• https://github.com/glebpom/linux-aio-tokio Another crate that would likely have the same problem. This one works with Linux's libaio, and is also stuck in tokio-0.2.
• https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2 kqueue man page. Other filter types that would require PollEvented include:
  • EVFILT_VNODE: notifies about file system events like file creation and deletion
  • EVFILT_SIGNAL: notifies about reception of a signal. I'm not sure how tokio is currently handling signals, but it isn't doing it this way.
  • EVFILT_TIMER: the subject of issue AsyncFd doesn't allow to decide read/write interest making kqueue's fd fail in tokio 0.3 #3072 .
  • EVFILT_USER: user-triggered kqueue events.
  • EVFILT_SENDFILE: notification for completion of an asynchronous sendfile(2) operation.

[carllerche]

PollEvented is removed to avoid coupling all platforms to mio. Instead, we could explore a way to expose kqueue specific capabilities in a platform-specific way.

I believe this should be doable by exposing those filters on Interest and Ready?

[asomers]

PollEvented is removed to avoid coupling all platforms to mio. Instead, we could explore a way to expose kqueue specific capabilities in a platform-specific way.

I believe this should be doable by exposing those filters on Interest and Ready?

I don't think that would be sufficient, because for EVFILT_AIO you don't register with kevent(2), instead you register by setting some fields on the struct aiocb that you pass to aio_write(2) or similar. So at a minimum, using AIO with Tokio requires knowing that file descriptor of Tokio's kqueue, and the mio registration Token .

[asomers]

I don't understand the problem with mio. Is it that mio is Unix specific? Or that you think Tokio's public API would be too fragile if it depended on mio's public API?

[Darksonn]

The problem is that we are releasing Tokio 1.0 very soon at which point we guarantee no more breaking releases of Tokio for at least 3 years, and support of 1.0 for at least 5 years. If mio is visible in the public API, then we will not be able to upgrade Tokio in case of a breaking change in mio. This was a big problem in Tokio 0.2, which was stuck on mio 0.6 even though mio 0.7 had been released a long time ago.

[asomers]

What if Tokio reexports that interface? It doesn't even have to be a pass-through reexport. Tokio could create its own trait that basically does the same thing as mio's Source, but uses its own equivalents of Registry, Interest, and Token. That way, there wouldn't be any unexpected breakages if the mio version changes under the hood.

[carllerche]

It's a bit more complicated than that. IMO PollEvented is too broad of an abstraction to have at the core. This is fairly obvious w/ windows, but also on linux, there is no guarantee that there is a epoll poller. There could be uring.

It's definitely an interesting problem. I will have to think about it a bit more. Exposing Token and Registry (though you really want the fd it sounds like) is a fairly intrusive API to commit to.

Personally, I don't hate the sub reactor strategy.

[asomers]

What about exposing PollEvented for kqueue-based operating systems only, gated by a "beta" or "unstable" feature flag. That would pretty well indicate that it's not part of the stable API. That at least would allow me to make tokio-file work with Tokio 0.3. And once we integrate the read_at/write_at methods (#1529), then we can remove PollEvented, unless somebody has found a different need for it.

[asomers]

@carllerche now that 1.0 is out, are you willing to reconsider this request? I only see two decent options:

1. Expose PollEvented, or in some other way expose the kqueue file descriptor, to consumers
2. Move all of tokio-file into tokio. That wouldn't require publicly exposing PollEvented, but it would require publicly exposing the new file API. The file API is going to be a lot bigger, and will likely need some refinement. It would be easier to refine it if it weren't in the main tokio repo.

[carllerche]

It should be possible to expose kqueue specifics on AsyncFd. It might take a bit of design work, but we can use a similar strategy where we use Interest and Ready types defined in Tokio the same way we do in mio. We would conditionally define kqueue specific interest & readiness values on those types.

AsyncFd may need to be expanded a bit to support this. We currently have fns to await any readable and writable events. We can add a async fn ready(Interest) -> io::Result<AsyncFdReadyGuard> I think. We also can expose the readiness details from that guard.

The next steps would be to propose specific API additions in here or a new issue. We can discuss those, and then move forward w/ a PR once we reach something that works.

[asomers]

I don't see how AsyncFd would help. With POSIX AIO, you don't register the file's file descriptor with kqueue. Instead, you register each operation with kqueue. And you do that not by calling kevent(), but by writing the kqueue's file descriptor to one of the fields of the struct aiocb. Registering a regular file's file descriptor with kqueue would cause it to immediately return both readable and writable readiness, which wouldn't be helpful.

For POSIX AIO, I could create a type that implements mio::event::Source, but I can't do anything with that since Tokio no longer exposes PollEvented. If tokio::runtime::Handle had an as_raw_fd method, which called self.io_handle.inner.registry.as_raw_fd, that would allow me to set the kqueue in the struct aiocb. However, that wouldn't properly register the aiocb with Tokio, because it wouldn't associate any Token.

I still can't find any option better than exposing PollEvented. If Tokio provides some way to expose the kqueue's fd instead, it would also have to expose a way to do what PollEvented::poll_read does, and that means exposing Registration, which is a lot more intrusive than exposing PollEvented.

[asomers]

@carllerche what do you think about PR #3841 ? Instead of exposing PollEvented, I created a new, similar structure. It only has the minimum API needed for POSIX AIO. AFAIK, that makes it useless for any other application, so I named it PollAio. It's the smallest API addition I could come up with.