Context reactor hook

[jkarneges]

Proposal

Changelog

• 2024-03-03: Initial draft.
• 2024-03-10: Add PartialEq to WakerWaiter and LocalWakerWaiter. Add getters to TopLevelPoller. Define setter error types.

Problem statement

Future implementations are often coupled to specific executors. For example, tokio network objects require using tokio's executor. This coupling has led to siloed async environments, limiting interoperability. Ideally, it would be possible to run any Future to completion by following the Future interface alone. Further, it would be nice if main and test functions could be declared as async (e.g. async fn main), and having a general way to run futures could help solve this.

The root of the problem is the Future API is incomplete, at least for practical purposes. It is the executor's responsibility to supply wakers to futures and to poll those futures whenever their wakers are triggered, and it is each future's responsibility to trigger their waker (somehow). However, somewhat counter-intuitively, futures commonly rely on the executor to trigger their wakers. That is, not only will the executor poll futures until they return Pending, but the executor will then wait (or "park") on a blocking call that will trigger wakers. Futures from popular multithreaded runtimes do this to limit context switches (tokio does this, smol may do it opportunistically), and futures from single-threaded runtimes require this by design.

If pretty much everyone is comingling waker-triggering logic in the execution thread, and if the goal of the Future interface and its abstract Waker is to ensure executors and futures can be decoupled, then it is arguable that this interface is missing an important piece.

Motivating examples or use cases

Successfully decoupling executors and futures could yield the following benefits:

• Async runtimes could separate their executor and reactor parts, to allow developers to mix them. For example, it could be possible to use the executor from smol to run tasks containing network objects from tokio, and vice versa.
• The stdlib could offer a universal block_on function able to run any future. Such a function could, for example, execute tasks containing network objects from tokio, without being aware of tokio and without any prior configuration.
• The language could support async fn main, by simply wrapping block_on.
• The improved interoperability could make async easier for newcomers to understand and appreciate.

Solution sketch

As an exercise, let's imagine if executors were only ever responsible for running one future. In that case we could imagine solving this problem by adding another method to the Future trait to wait for (and trigger) its waker:

pub trait Future {
    type Output;

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
    fn wait_for_waker(&self);
}

This wait_for_waker method would allow the future to provide the waiting & waker-triggering behavior without the executor needing to know anything about it. An executor could simply alternate between calling poll and wait_for_waker, and it would be compatible with any future.

Of course, this would not be usable in real life, since the whole point of Future is to be able to run operations concurrently on the same thread. We can't allow one future to tie up a whole thread waiting to trigger its waker. Instead, we need a way for the executor to make a single call that will wait for the wakers of all of the futures it is managing.

The waiter object

To solve this, we can introduce a shared object to perform this waiting. Let's call it WakerWaiter. We'll also want a variant that is !Send for single-threaded use, which we'll call LocalWakerWaiter. Rough API below:

pub struct WakerWaiter { /* fields */ }

impl WakerWaiter {
    pub fn wait(&self);
    pub fn canceler(&self) -> WakerWaiterCanceler;
}

impl PartialEq for WakerWaiter {
    fn eq(&self, other: &Self) -> bool;
}

impl Send for WakerWaiter {}
impl Sync for WakerWaiter {}

pub struct LocalWakerWaiter { /* fields */ }

impl LocalWakerWaiter {
    pub fn wait(&self);
    pub fn canceler(&self) -> Option<WakerWaiterCanceler>;
}

impl PartialEq for LocalWakerWaiter {
    fn eq(&self, other: &Self) -> bool;
}

impl !Send for LocalWakerWaiter {}
impl !Sync for LocalWakerWaiter {}

pub struct WakerWaiterCanceler { /* fields */ }

impl WakerWaiterCanceler {
    pub fn cancel(&self);
}

The futures could provide a single WakerWaiter or LocalWakerWaiter to the executor, and the executor could call its wait() method to wait for (and trigger) any wakers associated with those futures.

Note: since it only makes sense for the executor to wait on a single waiter, an inherent caveat of this design is that all of the futures that desire such in-thread waiting behavior must either a) agree to use the same waiter, or b) be able to fall back to triggering their waker some other way (e.g. from another thread). This means this proposal is mainly solving interoperability between executors and futures with arbitrary in-thread waiting requirements, but it is not solving the problem of mixing multiple futures with different in-thread waiting requirements (i.e. from different runtimes) on the same executor. The latter will likely require some agreement on a common "reactor" that could act as the waiter, which the author considers to be an important orthogonal problem.

The canceler() method enables the blocking waits to be unblocked, by returning a WakerWaiterCanceler object that contains a single cancel() method. Canceling waits is useful for interoperating with futures that don't rely on in-thread waiting behavior, for example futures that trigger their wakers from separate threads (either because they always work this way, or perhaps as a fallback due to conflicting with whatever waiter was set up by other futures). In that case, triggering their wakers should unblock the waiter so that they can make progress.

The reason for having a separate object to handle cancellation instead of offering a cancel() method directly on the waiter is to allow the two objects to have different Send-ness. Cancellation must always happen from a separate thread, so the WakerWaiterCanceler is Send. However, we want the waiter itself to not have to be, as is the case with LocalWakerWaiter.

That said, LocalWakerWaiter also shouldn't be required to offer a canceler at all, since implementing cancellation relies on thread synchronization. To allow for that possibility, its API deviates slightly from WakerWaiter by having its canceler() method return an Option.

Waiter construction would be done using a raw pointer and vtable, similar to Waker, to enable use in no_std environments.

Waiters would implement PartialEq to allow comparison. These comparisons could be made cheap by simply comparing the pointer/vtable.

Thus, the waiter API would be able to support three main modes:

• Thread-safe waiting with WakerWaiter, expected to be useful for work stealing executors such as tokio.
• Thread-local waiting with LocalWakerWaiter with cancellation support, expected to be useful for single-threaded executors or thread-per-core executors with access to std.
• Thread-local waiting with LocalWakerWaiter without cancellation support, expected to be useful for single-threaded executors in no_std environments.

Conveying a waiter to the executor

To provide a way for an executor to receive a waiter from the futures, we could offer a trait that the executor could implement:

pub struct SetWaiterError;

pub enum SetLocalWaiterError {
    Conflict,
    Unsupported,
}

pub trait TopLevelPoller {
    fn set_waiter(&mut self, waiter: &WakerWaiter) -> Result<(), SetWaiterError>;
    fn set_local_waiter(&mut self, waiter: &LocalWakerWaiter) -> Result<(), SetLocalWaiterError>;

    fn waiter(&self) -> Option<&WakerWaiter>;
    fn local_waiter(&self) -> Option<&LocalWakerWaiter>;
}

The interface would support accepting up to one waiter (either thread-safe or local) and only once. Basically, whichever future is the first to set a waiter would win.

The interface would also support getting a reference to the currently-set waiter, if any. This would enable futures to check if a desired waiter is already set without having to blindly attempt to set one, mainly useful for futures that support more than one reactor and want to avoid constructing extra reactors unnecessarily.

The executor could expose its TopLevelPoller implementation via Context. Perhaps it could be set using a method on ContextBuilder:

impl<'a> ContextBuilder<'a> {
    // ...

    pub fn top_level_poller(self, &mut dyn TopLevelPoller) -> ContextBuilder<'a>;

    // ...
}

Futures could retrieve the value, if available, using a method on Context:

impl<'a> Context<'a> {
    // ...

    pub fn top_level_poller(&self) -> Option<&mut dyn TopLevelPoller>;

    // ...
}

Context inheritance

The TopLevelPoller is provided by, well, the top level poller, and it is primarily of interest to leaf futures that may be dealing with reactor registrations. However, if there are any other futures in between that create their own Contexts, such as combinators, then the assigned TopLevelPoller could get lost along the way. To solve this, we can enable inheriting one context from another via the builder:

impl<'a> ContextBuilder<'a> {
    // ...

    pub fn from(&mut Context) -> ContextBuilder<'a>;
    pub fn waker(self, &'a Waker) -> ContextBuilder<'a>;

    // ...
}

This would allow combinators do so something like:

let mut cx = ContextBuilder::from(parent_cx).waker(&my_waker).build();

Alternatives

Global runtime configuration

An abstract async runtime interface could be configured at the global level, either via a static global similar to the allocator API or via a cargo setting. The drawback to this approach is that it isn't very friendly to mixing different runtimes in the same application.

Separate crate

This interface could be provided in a separate crate, and in fact waker-waiter is an attempt at that. However, if we want to provide a universal block_on function in the stdlib, or to be able to make async fn main work, then we'll need this interface in the stdlib.

Provider API

Instead of adding a specific method to Context to enable looking up the TopLevelPoller, it is possible to imagine Context could offer the Provider API to support exposing arbitrary interfaces. However, the Provider API effort seems to have stalled.

Contexts and capabilities

Instead of adding a specific method to Context to enable looking up the TopLevelPoller, it is possible to imagine passing the interface around via some other context-like mechanism such as discussed in the Contexts and capabilities blog post. However, this ability may not be available for some time.

Links and related work

Blog post: Context reactor hook
Experimental crate: waker-waiter

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[bjorn3]

What about futures which can support multiple waiters? For example any waiter which polls an fd. If the poller is provided by the future it wouldn't know which waiter it should provide.

If two futures need a distinct waiters wouldn't the first future wins semantics cause a deadlock rather than a panic like currently happens?

The suggested API would still require a thread local to store the active poller as there is no way to get it from the Context, right? That is incompatible with no_std and TLS accesses are relatively slow, especially from inside a dylib as there the initial-exec tls model can't be used.

[jkarneges]

With the current API, if a future supported multiple waiters it would need to try to set each of them until one succeeds. I suppose that's not great as it might result in more waiters (reactors) being instantiated than necessary. Perhaps a getter method could be added to TopLevelPoller to inspect the current waiter.

How would a failure to set a waiter result in a deadlock? The intention is if a waiter cannot be set, and the waiter is required, then the future should not operate (return error or panic). This is similar to how the objects of async runtimes today will check for their expected runtime and fail otherwise.

You're correct that futures would still need to keep their waiter in a place that they all can find it, for example by using TLS, which is where most async runtimes keep their runtime today. This proposal is not trying to address that problem. However, maybe if there were a getter to get the current waiter then this proposal could end up addressing that problem too.

[bjorn3]

The intention is if a waiter cannot be set, and the waiter is required, then the future should not operate (return error or panic).

Missed the error return type. How would the current api distinguish between the same kind of waiter already being set and a different kind of waiter having been set though? The former is entirely fine while the latter is an error condition.

[jkarneges]

The intent is to allow very cheap comparison by data pointer and vtable, similar to Waker. Waiters could implement PartialEq to do this.

[m-ou-se]

cc @rust-lang/wg-async

[jkarneges]

Updated the proposal based on feedback so far.

Earlier I'd said that adding a way to get the current waiter could help futures find their reactor instead of having to use TLS, but it occurs to me that without a way to cast from the waiter to the reactor it won't help with that problem. So, the getter is mainly useful to avoid constructing multiple reactors.

[tmandry]

Have you done anything to quantify the performance impact of this proposal, in terms of both runtime cost and code size? I think that's the main thing that makes me hesitant about it: On one hand I want it to make everything just work together; on the other, I don't want highly constrained environments to pay a substantial cost for something they don't need.

[jkarneges]

@tmandry Great question.

At minimum it's opt-in. If the executor doesn't offer support for waiters and if futures don't construct waiters or try to interact with the executor about waiters, then there is no meaningful cost. Basically it would just be an Option<&mut dyn TopLevelPoller> field added to Context and ContextBuilder that never gets set.

As for the costs of actually using the feature, we could break this down into negotiation and usage.

For negotiation:

1. The executor must provide a TopLevelPoller reference to Context. This is expected to be cheap, similar to how the executor must provide a Waker reference. It ought to be pre-constructable with no meaningful cost at poll-time.
2. When evented objects (TcpStream, etc) are constructed, they must check/set a waiter. This would require getting the current waiter reference from the TopLevelPoller via the Context, and checking if it's an acceptable type (just realized we may need downcasting ability to do this). If it's an acceptable type, there is nothing else to do. If it's not an acceptable type, error/panic. If it's unset, instantiate a waiter and put it somewhere (global/TLS) and also set it on the TopLevelPoller. Essentially the cost would be a downcast in every evented object construction. Note that evented objects of runtimes today typically do a global/TLS get+unwrap to look up the runtime at the time of their construction. Adding a downcast may be negligible. (And I guess if we have a downcast then it could replace the existing lookups in which case going through the context could be faster than what is done today).

For usage:

1. The executor must use the waiter to park the thread if one is provided. For single threaded reactors, this ought to be fairly cheap (waiter virtual call). For multithreaded reactors, I would not be surprised if this requires a mutex somewhere. Either way, I suspect the costs would be very similar to how runtimes park today aside from maybe the virtual calls for wait/cancel.

In short, I would expect its presence to have no meaningful cost, and its negotiation and use to be similar in cost to how existing runtimes work.

[jkarneges]

After stewing on this a bit, I'm thinking it may be better to bootstrap this endeavor by simply adding an Any field to Context. There are a few reasons:

• It could provide just enough of a hook to enable experimentation outside of std, without needing consensus on the entire API yet.
• I believe we'll want to add other data to Context in the future beyond just TopLevelPoller (e.g. "Spawner", "Budgeter", etc), and an Any field could enable such experimentation too.
• Not only would this enable experimenting with what kinds of extra data we want to add, it would also enable experimenting with how such data is structured. For example, TopLevelPoller ought to be retained in sub-Contexts, but a "Spawner" might be overridden in a sub-Context to reduce the spawning scope.

Here's what we could do:

impl<'a> ContextBuilder<'a> {
    // ...

    // inherit from existing Context (same waker, local waker, and ext data)
    pub fn from(other: &'a mut Context) -> ContextBuilder<'a>;

    // replace current waker on the builder
    pub fn waker(self, &'a Waker) -> ContextBuilder<'a>;

    // replace current ext on the builder
    pub fn ext(self, &'a mut dyn Any) -> ContextBuilder<'a>;

    // ...
}

impl<'a> Context<'a> {
    // ...

    // get the ext data, if any
    pub fn ext(&mut self) -> &mut dyn Any;

    // ...
}

Internally, the extension data could be held in an Option<&mut dyn Any>.

If the inner value is None when Context::ext() is called, we could return &mut (). This way fetching the extension data could be infallible, so that the only fallible operation the caller has to worry about is the downcast.

Then the original proposal could be implemented on top, like so:

struct ContextExtensions<'a> {
    top_level_poller: Option<&'a mut dyn TopLevelPoller>,
}

let mut ext = ContextExtensions {
    top_level_poller: Some(&mut poller),
};

let mut cx = ContextBuilder::from_waker(&waker).ext(&mut ext).build();

if let Some(ext) = cx.ext().downcast_mut::<ContextExtensions>() {
    // ... do something with ext.top_level_poller ...
}

If there are not any objections, I can submit this as a separate proposal. The current proposal can remain, as I believe it belongs in std as well, but we could worry about it afterwards.

[programmerjake]

    pub fn ext(&mut self) -> &'a mut dyn Any;

i think you shouldn't return with lifetime 'a since that requires replacing the Option<&'a mut dyn Any> field with None, i presume you want to borrow it instead, returning with the &mut self lifetime.

[jkarneges]

Good catch. Updated.

[jkarneges]

One hiccup: Any impls cannot contain non-static references. This is unfortunate if the goal is for it to hold a struct of dyn references.

Since accessing ext requires exclusively borrowing through Context, I wonder if there are any lifetime games we can play to work around this. Maybe transmuting the lifetime to 'static when building and then transmuting again to the lifetime of self in the accessor.

[jkarneges]

Perhaps the implementer of the extension could control this by requiring all accesses to go through accessor methods, and having those accessors limit any returned lifetimes to that of self.

struct ContextExtensions<'a> {
    top_level_poller: Option<&'a mut dyn TopLevelPoller>,
}

impl<'a> ContextExtensions<'a> {
    fn top_level_poller(&mut self) -> Option<&mut dyn TopLevelPoller> {
        match self.top_level_poller.as_mut() {
            Some(v) => Some(*v),
            None => None,
        }
    }
}

let mut ext = ContextExtensions {
    top_level_poller: Some(&mut poller),
};

let mut cx = {
    // SAFETY: we can fake the lifetime as 'static here in order to enable casting to Any,
    // because all of ContextExtensions' accessors return limited lifetimes
    let ext = unsafe {
        std::mem::transmute::<&mut ContextExtensions<'_>, &mut ContextExtensions<'static>>(&mut ext)
    };

    ContextBuilder::from_waker(&waker).ext(ext).build()
};

if let Some(ext) = cx.ext().downcast_mut::<ContextExtensions>() {
    let poller = ext.top_level_poller();
    drop(ext);
    // error: cannot move out of `ext` because it is borrowed
    let poller = poller.unwrap();
}