Eventlet 35.2, and 36.1 both fail with dnspython2.6.1 - Python3.12 #957
Comments
sudhakar-datta
changed the title
|
Thanks for reporting this problem. Please can you downgrade dnspython to version 2.6.0 and then retry your scenario with this downgraded version? Downgrading dnspython could help to isolate the root cause. We are waiting for your feedback. Thanks in advance. |
|
Also, please can you provide a reproducer? |
|
Thanks so much for getting back so quick. Regards! Let me do that testing today. BTW, the producer is a keystone middleware (openstack) talking to keystone backend. Context is, any authN or authZ requests to keystone (openstack component). |
|
Which Openstack version are you using? As far as I know Python 3.12 is only supported by the current version of Openstack, still in development, aka Dalmatian 2024.2. For this reason I suppose you are facing this problem with the current development version, exact? |
|
Caracal. This also happens with any ancillary service that is written to use eventlet. Let me bisect the issue as you have highlighted. It is a good idea. I will try to recreate with 2.6.0. Also, 2.6.1 and forced or locked dependency for Cryptography for 41. I will let you know in next two hours :) |
Much appreciated, thank you! |
|
Thanks for pointing at the keystone changes. Looking at the diff, bit confused why a downgrade of dnspython would solve the issue. Just now tested with modified eventlet (dirty way, no monkey patching), but uses dnspython 2.6.1, for name resolution, it works. Confused at this point :). |
|
As the issue seems related to SSL, and regarding the nature of the problem I was thinking that cryptography recent changes could be the culprit. As this problem seems bound to a specific version of dnspython, I was thinking that downgrading the version of dnspython would also downgrade the version of cryptography. Allowing us to accuse or not cryptography. As Eventlet monkey patch the stdlib, it is possible that cryptography recently introduced changes bound to the ssl modules and which are not compatible with monkey patched modules from Eventlet... or some scenario like that. |
|
Makes total sense. Let me run through the tests by pinning crypto, and then go down the path of dnspython. |
|
Don't be disturbed by the other Eventlet links I posted earlier in this discussion. They were more side notifications than anything else. |
|
Okay, this changes breaks the eventlet - rthalley/dnspython#908 |
|
Is this the reoccurrence of - #726 Trace back reflects the same issue. |
|
Fails with Python3.11.9 with Antelope as well. |
|
I'm a bit surprised If rthalley/dnspython#908 ( This commit is now present in dnspython since many versions (at least version 2.4.0): I can see that dnspython versions equal or higher to version 2.4.0 are in use in Openstack upper-constraints since at least 7 months ago: For this reason, I'm a bit surprised that this dnspython commit is at the root cause of the error you reported. Indeed, if this problem found its origine in this commit, we should have already observed similar errors even with previous versions of Eventlet. |
|
I wonder if the observed behavior isn't a side effect of e2c874e |
|
These TLS changes are now present in Eventlet since version 0.35.2, versions which correspond the version you highlighted in the title of this github issue: e2c874e |
https://opendev.org/openstack/requirements/src/branch/master/global-requirements.txt, still packs the old combo for opendev. If you check canonical release, it is 0.33.1 and 2.3.0 for Jammy. Also, for most of the components in openstack, ssl termination happens at the LB. |
With dnspython >=2.4.0 <= 2.5.0, fails with connection reset error, and any higher fails with tls version issue. 24.04 will be released with caracal. Will see what they locked the dependencies and versions at :) |
|
Hervé, looks like you are also knee deep with openstack. What version of openstack are you working with? Also, do you use DOT, or DOH for DNS service? |
|
I'm core maintainer on Openstack shared libraries (Oslo) and one of the official release manager of Openstack, so I work with all the active versions of Openstack, and those maintained by my employer, Red Hat. However, as I'm a maintainer of common libs, I'm not really working with dnspython directly. I maintain libs like oslo.messaging, oslo.cache, etc... |
Ok, so it more or less point out that hot point is around from e2c874e |
global-requirements do not packs specific versions. global-requirements are more related to versions that we want to ignore: https://docs.openstack.org/project-team-guide/dependency-management.html#format Quoting the doc of Openstack requirements:
|
Wish openstack moves to a different package manager (poetry, pdm or even hatchling) :) e2c874e - This does not break. Without socket patching, it works with dnspython==2.6.1. ssl patch certainly breaking. In most of the openstack components, it user internal url, which talking over http instead of https. Lot of these dependencies does not have a bearing though. Is there any other way to bisect the issue? |
|
May dnspython is not a problem at all. May the problem is simply between Eventlet and the recent versions of the stdlib ssl module and would merely require a patching update on the Eventlet side. As you pointed out, it failed with Python 3.12 and Python 3.11, maybe this is an other bisect path... Bisecting from Python 3.11 to 3.10, don't know... In all case it would useful to have a tiny reproducer script to do not have to install all Openstack to reproduct. Would you mind to try to isolate a bit of logic of this Keystone context to see if we are able to reproduce it in a lightweight fashion? |
import eventlet
import urllib3
urls = [
'https://google.com',
'https://opendev.org',
'https://github.com/eventlet/eventlet'
]
def url_request(url):
resp = urllib3.request('GET', url)
return(resp.json())
eventlet.patcher.monkey_patch(socket=True)
pool = eventlet.GreenPool(size=len(urls))
pile = eventlet.GreenPile(pool)
for url in urls:
pile.spawn(url_request, url)
resp = '\n'.join(pile)
print(resp)Just a quick one :) ! Environment: Pip freeze details:
|
|
Awesome, thank you. |
|
FYI, I'm able to reproduce the problem on my fedora with the given script and the given versions of package. Early next week I'll try to see if I find something interesting. |
|
At first glance it seems that the problem is between that version of urllib3 and the patching of ssl from eventlet. |
|
The compatibility problem seems located around these changes urllib3/urllib3@472109c |
Still related to the ssl context problem, dnspython introduced changes with version 2.5.0 which are also related to the ssl minimum_version rthalley/dnspython@609d6b2 That could confirm your previous claims about the dnspython versions which you identified with TLS problems. The patch I gave in the previous sentence was released with dnspython 2.5.0. They are also located in the same area that the urllib3 changes I shared in my previous comment (urllib3/urllib3@472109c). |
|
The reproducer fails even if I downgrade urllib3 to a version that do not contains urllib3/urllib3@472109c, by example urllib 2.0.0. |
|
It would be worth to bisect dnspython until reaching the commit just before rthalley/dnspython@609d6b2 |
|
This is the commit "6c5f0c9d8086c999357531c38b831efd24b6b5ac", when eventlet and dnspython don't love each other anymore. |
|
https://changelogs.ubuntu.com/changelogs/pool/main/o/openssl/openssl_3.0.13-0ubuntu3/changelog - openssl (3.0.5-2ubuntu1) kinetic; Security level changed there. Python3.12 is compiled with openssl 3.0.13, and Jammy with 3.0.3. Could be the ssl module? |
|
ok. Then will dive deep in 6c5f0c9d808 |
|
My fedora 39 is using openssl 3.1.1 and reproduce the problem. |
|
The openssl changelog speak about:
Which is close to the error we seen in logs: |
|
I see the same issue using eventlet, flask-socket-io and boto3: |
|
Well, I think the way the monkey patch is made is problematic. The monkey patch should be made early, just after the import of eventlet. import eventlet
eventlet.patcher.monkey_patch()
import urllib3
... |
|
Please ensure that your code is monkey patched as early as possible, else we can't guarantee the result. |
|
Now I get an unicode error but which I think is not related to the original problem: |
|
That will involve lot of changes in the openstack components. Ubuntu 24.04 got released with Caracal on py3.12. Not sure what canonical tested. It breaks all the tests, and the openstack services talking to keystone via keystone middleware :) Hervé Beraud, still a solid workaround. Thanks for debugging this one. |
|
The supported runtimes for Caracal (2024.1) are Ubuntu 22.04 and from Python 3.8 to Python 3.12. I'm a bit surprised that you previously claimed that Ubuntu 24.04 got released with Caracal and a Python 3.12, a Python runtime not officially supported by this Openstack series. https://governance.openstack.org/tc/reference/runtimes/2024.1.html |
Should I understand that you tested it and that you confirm that it also fix the problem on your side? |
If you are subscribed to canonical upgrade path, Ubuntu 24.04 is shipped with Python 3.12, and just 3.12. Also, if not opting for cloud archive, it would be Caracal. |
It does not complete fix the issue. Especially in the swift component, where patching is kind of deferred. |
IMO its a bit problematic as all the components are tested with versions of Python lower than 3.12, and so the versions of requirements too... |
I see, thanks for details. |
Agreed. Only swift-proxy is entangled with ssl (keystone interface), the plan is to go with the podman container route for swift-proxy and rest is all py3.12, and finger crossed :) |
|
@sudhakar-datta: Thanks for all these details and thanks for your time. Much appreciated! |
@4383 Hervé Beraud, I will keep you posted about what route we end up with 💯 |
|
Please do! |
Eventlet 35.2, and 36.1 both fail with dnspython2.6.1. Works fine with dnspython2.3.0
Fails with the following traceback
urllib3/util/ssl_.py", line 292, in create_urllib3_context
context.minimum_version = TLSVersion.TLSv1_2
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/ssl.py", line 546, in minimum_version
super(SSLContext, SSLContext).minimum_version.set(self, value)
File "/usr/lib/python3.12/ssl.py", line 546, in minimum_version
super(SSLContext, SSLContext).minimum_version.set(self, value)
File "/usr/lib/python3.12/ssl.py", line 546, in minimum_version
super(SSLContext, SSLContext).minimum_version.set(self, value)
[Previous line repeated 968 more times]
File "/usr/lib/python3.12/ssl.py", line 544, in minimum_version
if value == TLSVersion.SSLv3:
^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded
The text was updated successfully, but these errors were encountered: