rt: Allow concurrent block_on's with basic_scheduler

[LucioFranco]

This PR changes how Runtime::block_on works when the basic_scheduler is selected. Before this change, we used a mutex to guard when entering a block_on with a basic_scheduler which meant that we could never call block_on concurrently event hos this was totally possible. This PR changes that to allow us to call block_on concurrently. This also allows multiple block on's to "steal" the driver/parker. Doing this by sharing the ParkThread condvar and allowing us to notify that thread to check for the parker to allow moving it forward.

The stealing works by the first thread to call Runtime::block_on will acquire the driver. Every other call to block_on while the first block_on is still running, will not acquire the driver but will instead enter the runtime context and attempt to poll the passed in future. The parker is slightly modified for this case to allow us to provide our own Condvar that we will use once the initial block_on finishes to notify the second/other threads that they can now steal the driver.

Related to #2720

[hawkw]

The stealing works by the first thread to call Runtime::block_on will acquire the driver. Every other call to block_on while the first block_on is still running, will not acquire the driver but will instead enter the runtime context and attempt to poll the passed in future. The parker is slightly modified for this case to allow us to provide our own Condvar that we will use once the initial block_on finishes to notify the second/other threads that they can now steal the driver.

At one point when we talked about this, I thought you said that the solution was to just force the initial block_on to continue running the driver until all other block_ons finish, and then return that future's output — am i misremembering, or just wrong?

[hawkw]

I think this shouldn't be too tricky if we just ensure that the state is only changed while inside a lock.

[LucioFranco]

I would like to punt this to the next PR.

[hawkw]

sure, that's fine, i was just mentioning this for whenever you get to it.

[carllerche]

Not for this PR, but when we do this, we probably want to define some "cell" type in src/util.

[LucioFranco]

At one point when we talked about this, I thought you said that the solution was to just force the initial block_on to continue running the driver until all other block_ons finish, and then return that future's output — am i misremembering, or just wrong?

No, we wanted to actually move the driver between threads. I think that would be odd to have the block_on run longer than expected? That feels like an implicit dependency that would be pretty surprising.

[LucioFranco]

@carllerche and I had a chat last week, I have some tweeks I am going to change this. I want to see if we can get away with a LinkedList style approach for notifying other threads instead of going via the condvar/mutex pair.

[hawkw]

At one point when we talked about this, I thought you said that the solution was to just force the initial block_on to continue running the driver until all other block_ons finish, and then return that future's output — am i misremembering, or just wrong?

No, we wanted to actually move the driver between threads. I think that would be odd to have the block_on run longer than expected? That feels like an implicit dependency that would be pretty surprising.

Yeah, I think this is better, I just wasn't sure if I remembered our chat from earlier. Carry on!

[LucioFranco]

Okay, I've refactored this to use a VecDeque<Waker> instead of a Condvar/Mutex pair. I've also addressed issues around double panics.

[hawkw]

i think we need to rework the way we're handling mutex poisoning here. i think we just want to always put everything back regardless of whether the mutex is poisoned. we can use PoisonError::into_inner to get a useable guard out of a poisoned mutex, ignoring the poison. i think we should do that here.

otherwise, this all seems fine to me.

[carllerche]

Thanks for the work. I left comments.

I mentioned in the original issue that I thought this could be done mostly by working with a Park layer, but we can punt that to another PR.

[carllerche]

Looking good. I think we are almost there. Left some notes.

[carllerche]

Thanks 👍