Asyncio integration for eventlet

[itamarst]

Motivation

A migration path off of eventlet for libraries/applications that exclusively rely on eventlet

Eventlet has different users with different use cases. This issue covers one specific subset of users: those who deliberately choose eventlet as their exclusive networking/event loop solution. A notable and relevant example is OpenStack.

My basic assumption is that the monkey-patching approach used by eventlet is fundamentally non-viable without massive engineering resources. For example, consider all the effort being put into patching RLock, when patching it actually introduces bugs (e.g. https://bugs.python.org/issue13697). A correct solution would involve maintaining a custom C RLock, and C Queue, and so on. And the more the Python stdlib changes, the harder eventlet maintenance becomes.

As such, I believe eventlet-only users are likely better off switching to event-driven frameworks like asyncio.

Access to asyncio libraries

Even if you disagree with the above, or you're a user who doesn't fit the above criteria, access to asyncio libraries in eventlet code is useful. Eventlet used to have Twisted support for this reason.

In general I don't think new features should be added to eventlet (see #835), but given that this aids with migration I think it's worth it.

I also believe it's easier to do this inside of eventlet rather than as an external project (see eventlet/aiohub#2).

The work involved

Prior art for gevent suggests this all should be possible. Each of these should be done as its own issue.

• An asyncio hub for eventlet #869
• Support multiple readers in the asyncio hub #874
• Asyncio hub support (or nice error message) for Python 3.9 and earlier #885
• Support awaiting GreenThread in an async def context. #887
• Support calling asyncio-using async def functions from inside greenlets #873
• Figure out how to deal with Locks and Queues etc when doing asyncio+eventlet integration
• GreenPool-specific support for turning a future into a GreenThread
• Document asyncio integration and how to use it for migration path off of eventlet #891
• Make sure asyncio hub + dnspython work correctly together #936
• Maybe make the asyncio hub the default, after enough releases and real-world usage demonstrates correctness

Questions

@temoto @tipabu what is your opinion on this approach of integrating asyncio and eventlet?

[4383]

Hello,

I agree with this proposal.

Itamar's proposal, will provide an official migration path to asyncio. This is well aligned with the fact that we want to discourage new usages of eventlet, and this is well aligned with our agreement to move maintenance to a legacy mode (#850).

As I already said (#824), from an Openstack perspective, our long term plan is to retire from eventlet and if nobody volunteer to maintain eventlet again, then retire eventlet in the same time. But we want to be transparent. We want to officialize our position and the impact on the future of this project. This proposal lays the foundations of this planed retirement.

This thread was initiated more than 3 weeks ago (eventlet/aiohub#2) and nobody raised any concerns or objections, so I think we can now move forward. I think we can start updating our maintenance goals (#850) to reflect the fact that no new features will be added to eventlet or accepted in PR, apart, those related to the asyncio hub (so far named aiohub).

We could leave this thread opened, until the support of eventlet is fully implemented. This thread would be a place to track the discussion and the advancement.

The aiohub discussion can be now closed eventlet/aiohub#2. However, as we don't have the owner rights on this organization, we are not able to remove the aiohub repository. We could simply update its readme file to redirect here and leave this repo opened just in case we need a parallel repository.

I'm gonna start reviewing and merging the related pull requests (#870).

Thanks @itamarst for your proposal.