Support multiple readers in the asyncio hub #874
Comments
|
Apparently Openstack Swift relies at some point on this multiple readers mechanisms. For more context, please see:
So if we want to see Swift transitioned to Asyncio, either we have to introduce support for this feature on the Asyncio hub, or Swift should drop usages of this feature before starting its migration. |
|
Toogling Line 146 in e1afe7b |
|
From swift's code: def __init__(self):
# Usually, it's an error to have multiple greenthreads all waiting
# to write to the same file descriptor. It's often a sign of inadequate
# concurrency control; for example, if you have two greenthreads
# trying to use the same memcache connection, they'll end up writing
# interleaved garbage to the socket or stealing part of each others'
# responses.
#
# In this case, we have multiple greenthreads waiting on the same
# (logging) file descriptor by design. So, similar to the PipeMutex,
# we must turn off eventlet's multiple-waiter detection.
#
# It would be better to turn off multiple-reader detection for only
# the logging socket fd, but eventlet does not support that.
eventlet.debug.hub_prevent_multiple_readers(False)https://opendev.org/openstack/swift/src/branch/master/swift/common/utils/__init__.py#L6102-L6116 |
|
In all the case, the Swift use case is based on an Eventlet debug feature which is used to disable protection at runtime... IMO this "enabling" feature should be only used for debug purpose, and not to be used to allow race conditions between greenthreads by bypassing protection. |
From: |
raise RuntimeError("Second simultaneous %s on fileno %s "\
"detected. Unless you really know what you're doing, "\
"make sure that only one greenthread can %s any "\
"particular socket. Consider using a pools.Pool. "\
"If you do know what you're doing and want to disable "\
"this error, call "\
"eventlet.debug.hub_multiple_reader_prevention(False)" % (
evtype, fileno, evtype))Still from cb7c8c0 So definitely, I don't think we should rely on this feature at all. Concurrent accesses to resources and possible race conditions should be handled on user side. |
|
I suspect most networking frameworks don't allow this sort of feature, yes, it doesn't really make sense for the vast majority of users (and for those who want to do it, there's a workaround, I'm pretty sure: |
|
|
|
As this "mutliple reader" functionality seems heavily used (see the list of commits which refer to this feature in #432), I'd also suggest to amend our eventlet documentation with a dedicated section to give advices on how to refactor migrated code, to bypass "multiple readers". IMO it should be fixed by design at the user app level. |
The
asynciohub in #869 doesn't support multiple readers on a single file descriptor.The mechanism which implements this feature is very confusing to me; additional readers are added in a
BaseHub.secondariesdata structure, and then promoted to a primary listener if the main listener is removed for some reason. I don't understand how any of it works at all, and so I've failed to get it work.The text was updated successfully, but these errors were encountered: