Add rudimentary support for Python 3.10 #715

tipabu · 2021-06-11T21:40:10Z

There's still work to be done:

dnspython<2.0.0 has a reference to collections.MutableMapping which was removed in favor of collections.abc.MutableMapping. Our dependency will need to be updated. See also: eventlet is incompatible with dnspython 2.0.0rc1 #619.
nose has a reference to collections.Callable which was similarly removed for collections.abc.Callable. Our test infrastructure will need to be updated to support testing py310 in CI. See also: WIP: Remove dependency on nose #638.

Manually fixing those references in my local Python 3.10 env, though, I see

`EVENTLET_HUB`	Tests	Status	Skipped
`selects`	710	OK	118
`poll`	710	OK	118
`epolls`	710	OK	119

Looks like the skips were largely DB- and ZMQ-related tests -- not sure what the extra skip was about when not explicitly specifying a hub.

Feedback is welcome; much of this seems... less than ideal:

The greenio._open requires additional locking, and callers in a mixed greenthread/pthread-enabled process may unexpectedly use GreenFileIO when calling _pyio.open.
The TLS version cap seems undesirable and the error when allowing tests to use TLS 1.3 likely warrants further investigation. Even if we keep it, the PROTOCOL_TLSv1_2 constant used was deprecated in py36 and may later be removed.
The dict/module floppiness of __builtins__ in is_timeout is quite strange and also warrants further investigation.

eventlet/greenio/py3.py

temoto · 2021-06-11T22:01:19Z

tests/wsgi_test.py

@@ -579,7 +579,8 @@ def wsgi_app(environ, start_response):
        sock = eventlet.wrap_ssl(
            eventlet.listen(('localhost', 0)),
            certfile=certificate_file, keyfile=private_key_file,
-            server_side=True)
+            server_side=True,
+            ssl_version=ssl.PROTOCOL_TLSv1_2)


Please show which tests fail without this TLS cap.

It's this test -- the assert success at the end fails, and the server prints a traceback like

Traceback (most recent call last): File "/home/tburke/code/eventlet/tests/wsgi_test.py", line 566, in server serv.process_request([addr, client_socket, wsgi.STATE_IDLE]) File "/home/tburke/code/eventlet/eventlet/wsgi.py", line 815, in process_request proto.__init__(conn_state, self) File "/home/tburke/code/eventlet/eventlet/wsgi.py", line 348, in __init__ self.finish() File "/home/tburke/code/eventlet/eventlet/wsgi.py", line 729, in finish BaseHTTPServer.BaseHTTPRequestHandler.finish(self) File "/usr/lib64/python3.10/socketserver.py", line 811, in finish self.wfile.close() File "/usr/lib64/python3.10/socket.py", line 723, in write return self._sock.send(b) File "/home/tburke/code/eventlet/eventlet/green/ssl.py", line 193, in send return self._call_trampolining( File "/home/tburke/code/eventlet/eventlet/green/ssl.py", line 161, in _call_trampolining return func(*a, **kw) File "/usr/lib64/python3.10/ssl.py", line 1203, in send return self._sslobj.write(data) ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2365)

Curiously, the test passes on py39, even when using TLS 1.3. On both versions of Python, I confirmed that TLSv1.3 was in fact negotiated by adding a print(client.version()) after sending the X.

Maybe we miss close_notify in green/ssl. https://datatracker.ietf.org/doc/html/rfc8446#section-6.1

tipabu · 2021-06-12T05:05:46Z

Good news! pyzmq seems to Just Work -- down to 94/95 skips locally. I'll see what I can do about the DB tests; historically I've not had great luck bringing up an env that hits them, so I'd usually just let CI exercise them.

temoto · 2021-06-14T06:15:26Z

@tipabu for DB tests, you'd have both mysql and postgresql running and this

export EVENTLET_DB_TEST_AUTH='{"psycopg2": {"host": "127.0.0.1", "port": 5432, "user": "postgres", "password": "secret"}, "MySQLdb": {"host": "127.0.0.1", "port": 3306, "passwd": "secret", "user": "root"}}'

hroncok · 2021-06-16T09:57:18Z

nose has a reference to collections.Callable which was similarly removed for collections.abc.Callable. Our test infrastructure will need to be updated to support testing py310 in CI. See also: #638.

Just out of curiosity. From the three distributions available at https://pypi.org/project/nose/#files only nose-1.3.7-py3-none-any.whl seem to have collections.Callable. Installation from sdist might be a temporary solution (but there might be new problems, since apparently, the code is different).

thedrow · 2021-06-27T13:50:07Z

FYI this also happens on 3.10b3.

  _____________ ERROR collecting t/unit/concurrency/test_eventlet.py _____________
  t/unit/concurrency/test_eventlet.py:17: in <module>
      pytest.importorskip('eventlet')
  .tox/3.10-unit/lib/python3.10/site-packages/eventlet/__init__.py:17: in <module>
      from eventlet import convenience
  .tox/3.10-unit/lib/python3.10/site-packages/eventlet/convenience.py:7: in <module>
      from eventlet.green import socket
  .tox/3.10-unit/lib/python3.10/site-packages/eventlet/green/socket.py:4: in <module>
      __import__('eventlet.green._socket_nodns')
  .tox/3.10-unit/lib/python3.10/site-packages/eventlet/green/_socket_nodns.py:7: in <module>
      eventlet.patcher.slurp_properties(__socket, globals(), ignore=__patched__, srckeys=dir(__socket))
  E   AttributeError: partially initialized module 'eventlet' has no attribute 'patcher' (most likely due to a circular import)

eventlet/timeout.py

tests/backdoor_test.py

tipabu

Good news: I can get away without most of the changes in greenio.

Bad news: somewhere between 3.10.0b1 and 3.10.0rc1, test_patcher_existing_locks_locked started failing with RuntimeError: cannot release un-acquired lock around

eventlet/tests/isolated/patcher_existing_locks_locked.py

Line 19 in c6c350e

lock.release()

I'm still scratching my head over that one. It looks like gc.get_referrers(old) is mostly returning lists and dicts, though? I wonder if it's somehow related to the trouble I was seeing with __builtins__ showing up as a dict in socket_test.test_error_is_timeout...

tests/backdoor_test.py

eventlet/greenio/py3.py

eventlet/timeout.py

temoto · 2021-08-19T09:09:12Z

@tipabu think we can merge parts of this work that are clearly working and backward compatible, like timeout?

tipabu · 2021-09-10T18:07:27Z

Sorry for the late reply -- I'm perfectly happy to have this merge in parts; whatever you think is ready.

temoto · 2021-10-07T12:05:38Z

The greenio._open requires additional locking, and callers in a mixed greenthread/pthread-enabled process may unexpectedly use GreenFileIO when calling _pyio.open.

@tipabu is this fixed by __wrapped__?

temoto · 2021-10-07T12:10:21Z

@tipabu moved TLS cap commit into separate branch in hope we can fix it with close_notify.
Recover with git pull https://github.com/eventlet/eventlet tipabu-tls-cap

tipabu · 2021-10-07T22:11:24Z

Yeah, all of the locking I'd tried in greenio went away when I found __wrapped__; it was no longer necessary. I wouldn't worry about keeping a branch for the TLS version cap -- it was a tiny change and not something we want to pursue in earnest anyway.

On py310, socket.timeout is TimeoutError, which our is_timeout() helper func already knows is a timeout. Note that this doesn't get us to py310 support (not by a long shot), but it's a step along the way. Closes eventlet#687

_pyio.open is now a staticmethod, so we've got to go down to _pyio.open.__wrapped__ to get to the python function object.

...rather than requiring an is_timeout attribute on errors. TimeoutErrors (which are covered by is_timeout) can't necessarily have attributes added to them.

Python 3.10 started including build info on the version line, so the expectation in tests had to change. Also, start printing the banner as we read it to aid in future debugging.

I'm still not sure how this happens, but somehow it does in socket_test.test_error_is_timeout. As a result, is_timeout wouldn't get a reference to TimeoutError, so the socket error would not be correctly identified as a timeout.

temoto

LGTM

temoto · 2021-10-08T10:18:29Z

@tipabu thank you very much, great job.

rahuledb · 2021-11-16T07:54:55Z

Hi @temoto ,

Can you mention specific date & version (or revision) of your next release where this fix will be included?

temoto · 2021-11-16T19:56:14Z

@rahuledb v0.33.0

update: released

roedoejet · 2021-11-18T17:43:23Z

I'm still getting this with eventlet==0.33.0 and python 3.10.0 - using with gunicorn

[Traceback (most recent call last):
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/gunicorn/util.py", line 99, in load_class
    mod = importlib.import_module('.'.join(components))
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/gunicorn/workers/geventlet.py", line 10, in <module>
    import eventlet
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/eventlet/__init__.py", line 17, in <module>
    from eventlet import convenience
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/eventlet/convenience.py", line 7, in <module>
    from eventlet.green import socket
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/eventlet/green/socket.py", line 21, in <module>
    from eventlet.support import greendns
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/eventlet/support/greendns.py", line 66, in <module>
    setattr(dns, pkg, import_patched('dns.' + pkg))
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/eventlet/support/greendns.py", line 61, in import_patched
    return patcher.import_patched(module_name, **modules)
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/eventlet/patcher.py", line 132, in import_patched
    return inject(
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/eventlet/patcher.py", line 109, in inject
    module = __import__(module_name, {}, {}, module_name.split('.')[:-1])
  File "/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dns/namedict.py", line 35, in <module>
    class NameDict(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'
]

hugovk · 2021-11-18T18:08:54Z

@roedoejet Please can you check what version of dnspython you have installed?

I guess eventlet needs to require dnspython>=2.

roedoejet · 2021-11-18T18:19:06Z

OK, I updated dnspython to 2.1.0, but now get this error:

ImportError: cannot import name 'ALREADY_HANDLED' from 'eventlet.wsgi' (/Users/pinea/.pyenv/versions/3.10.0/lib/python3.10/site-packages/eventlet/wsgi.py)

Which seems related to #702 but I am using gunicorn 20.1.0

temoto · 2021-11-18T20:24:44Z

@roedoejet Benoit has accepted fix in gunicorn/master but didn't release. pip (requirements) works with git links to specific commit.

fixes #740 fixes #780 related to #715

…0, drop non-essential CI against 2.7 to 3.6 fixes #740 related to #715

This comment has been minimized.

Sign in to view

tipabu mentioned this pull request Jun 11, 2021

Only wrap socket.timeout on Python < 3.10 #714

Closed

temoto reviewed Jun 11, 2021

View reviewed changes

tipabu force-pushed the py310 branch from 8bdbcbc to 900c8e1 Compare June 11, 2021 23:27

temoto reviewed Aug 15, 2021

View reviewed changes

eventlet/timeout.py Outdated Show resolved Hide resolved

tests/backdoor_test.py Outdated Show resolved Hide resolved

tipabu commented Aug 19, 2021

View reviewed changes

tests/backdoor_test.py Outdated Show resolved Hide resolved

eventlet/greenio/py3.py Outdated Show resolved Hide resolved

eventlet/timeout.py Outdated Show resolved Hide resolved

tipabu force-pushed the py310 branch from 900c8e1 to a5d9e9e Compare August 19, 2021 05:51

tipabu mentioned this pull request Oct 7, 2021

Compatibility with Python 3.10: TypeError: can't set attributes of built-in/extension type 'TimeoutError' #687

Closed

temoto force-pushed the py310 branch from 158f918 to ca28708 Compare October 7, 2021 12:01

tipabu added 5 commits October 8, 2021 13:03

Only wrap socket.timeout on Python < 3.10

3fde87d

On py310, socket.timeout is TimeoutError, which our is_timeout() helper func already knows is a timeout. Note that this doesn't get us to py310 support (not by a long shot), but it's a step along the way. Closes eventlet#687

Get a working greenio._open on py310

ccf6db2

_pyio.open is now a staticmethod, so we've got to go down to _pyio.open.__wrapped__ to get to the python function object.

Test using eventlet.is_timeout

5641a81

...rather than requiring an is_timeout attribute on errors. TimeoutErrors (which are covered by is_timeout) can't necessarily have attributes added to them.

Fix backdoor tests on py310

2ea6991

Python 3.10 started including build info on the version line, so the expectation in tests had to change. Also, start printing the banner as we read it to aid in future debugging.

temoto force-pushed the py310 branch from ca28708 to bad82b6 Compare October 8, 2021 10:04

temoto approved these changes Oct 8, 2021

View reviewed changes

temoto merged commit 15d3c24 into eventlet:master Oct 8, 2021

arthurzam mentioned this pull request Oct 16, 2021

python 3.10 issue with RLock patcher existing locks #730

Closed

foutrelis mentioned this pull request Dec 9, 2021

test_017_ssl_zeroreturnerror raises SSLEOFError on Python 3.10 #739

Open

Miezhiko mentioned this pull request Jan 18, 2022

dev-python/eventlet: python 3.10 support gentoo/gentoo#23853

Closed

dpage mentioned this pull request Sep 15, 2022

With Python 3.10 eventlet==0.31.0 breaking the application (RM #7006) pgadmin-org/pgadmin4#4766

Closed

temoto added a commit that referenced this pull request Jan 22, 2023

Python 3.10+ support, drop non-essential CI against Python 2.7 and 3.5

a9d3223

fixes #740 fixes #780 related to #715

temoto added a commit that referenced this pull request Jan 22, 2023

Python 3.10 support, drop non-essential CI against Python 2.7 to 3.6

f62f86c

fixes #740 fixes #780 related to #715

temoto mentioned this pull request Jan 22, 2023

Python3.10 support #780

Closed

temoto added a commit that referenced this pull request Jan 22, 2023

Python 3.10 support, drop non-essential CI against Python 2.7 to 3.6

1d8937e

fixes #740 fixes #780 related to #715

temoto added a commit that referenced this pull request Jan 22, 2023

green.http: collections.Iterable alias removed in Python 3.10, CI 3.1…

0e46f04

…0, drop non-essential CI against 2.7 to 3.6 fixes #740 related to #715

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add rudimentary support for Python 3.10 #715

Add rudimentary support for Python 3.10 #715

tipabu commented Jun 11, 2021

This comment has been minimized.

temoto Jun 11, 2021

tipabu Jun 11, 2021

temoto Oct 7, 2021

tipabu commented Jun 12, 2021

temoto commented Jun 14, 2021

hroncok commented Jun 16, 2021

thedrow commented Jun 27, 2021

tipabu left a comment

temoto commented Aug 19, 2021

tipabu commented Sep 10, 2021

temoto commented Oct 7, 2021 •

edited

temoto commented Oct 7, 2021 •

edited

tipabu commented Oct 7, 2021

temoto left a comment

temoto commented Oct 8, 2021

rahuledb commented Nov 16, 2021 •

edited

temoto commented Nov 16, 2021 •

edited

roedoejet commented Nov 18, 2021

hugovk commented Nov 18, 2021

roedoejet commented Nov 18, 2021

temoto commented Nov 18, 2021

Add rudimentary support for Python 3.10 #715

Add rudimentary support for Python 3.10 #715

Conversation

tipabu commented Jun 11, 2021

This comment has been minimized.

temoto Jun 11, 2021

Choose a reason for hiding this comment

tipabu Jun 11, 2021

Choose a reason for hiding this comment

temoto Oct 7, 2021

Choose a reason for hiding this comment

tipabu commented Jun 12, 2021

temoto commented Jun 14, 2021

hroncok commented Jun 16, 2021

thedrow commented Jun 27, 2021

tipabu left a comment

Choose a reason for hiding this comment

temoto commented Aug 19, 2021

tipabu commented Sep 10, 2021

temoto commented Oct 7, 2021 • edited

temoto commented Oct 7, 2021 • edited

tipabu commented Oct 7, 2021

temoto left a comment

Choose a reason for hiding this comment

temoto commented Oct 8, 2021

rahuledb commented Nov 16, 2021 • edited

temoto commented Nov 16, 2021 • edited

roedoejet commented Nov 18, 2021

hugovk commented Nov 18, 2021

roedoejet commented Nov 18, 2021

temoto commented Nov 18, 2021

temoto commented Oct 7, 2021 •

edited

temoto commented Oct 7, 2021 •

edited

rahuledb commented Nov 16, 2021 •

edited

temoto commented Nov 16, 2021 •

edited