Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Potential issue with asyncio hub + mysql #878

Open
itamarst opened this issue Jan 12, 2024 · 3 comments
Open

Potential issue with asyncio hub + mysql #878

itamarst opened this issue Jan 12, 2024 · 3 comments
Assignees
@itamarst
Labels
asyncio bug

Comments

@itamarst
Copy link
Contributor

In https://github.com/eventlet/eventlet/actions/runs/7503102994/job/20427156525?pr=789, tests/mysqldb_test.py::TestMySQLdb::test_run_query just froze and never finished.

May not be related to asyncio, but haven't seen this sort of freezing happen in any other circumstances.
@itamarst itamarst added the bug label Jan 12, 2024
@itamarst itamarst self-assigned this Jan 12, 2024
@itamarst
Copy link
Contributor Author

I got a segfault!

platform linux -- Python 3.11.7, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/itamarst/devel/eventlet
plugins: cov-4.1.0, repeat-0.9.3
collected 9 items                                                                                                                                                                            

tests/mysqldb_test.py .......Fatal Python error: Segmentation fault

Thread 0x00007f0dad9bf640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0daf66f640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0dc902f640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0dc99bf640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0dca34f640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0dcacdf640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0dcb66f640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0dcbfff640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0de102f640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0de234f640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0de2cdf640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0de366f640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0de3fff640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0df0cdf640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0df166f640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0df1fff640 (most recent call first):
  File "/home/itamarst/devel/eventlet/.tox/py311/lib/python3.11/site-packages/MySQLdb/connections.py", line 185 in __init__
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 82 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Current thread 0x00007f0daffff640 (most recent call first):
  File "/home/itamarst/devel/eventlet/.tox/py311/lib/python3.11/site-packages/MySQLdb/connections.py", line 259 in query
  File "/home/itamarst/devel/eventlet/.tox/py311/lib/python3.11/site-packages/MySQLdb/cursors.py", line 319 in _query
  File "/home/itamarst/devel/eventlet/.tox/py311/lib/python3.11/site-packages/MySQLdb/cursors.py", line 206 in execute
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 82 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0dae34f640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0de19bf640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0daecdf640 (most recent call first):
  File "/usr/lib/python3.11/threading.py", line 327 in wait
  File "/usr/lib/python3.11/queue.py", line 171 in get
  File "/home/itamarst/devel/eventlet/eventlet/tpool.py", line 74 in tworker
  File "/usr/lib/python3.11/threading.py", line 982 in run
  File "/usr/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f0df4ccb000 (most recent call first):
  File "/home/itamarst/devel/eventlet/eventlet/hubs/epolls.py", line 31 in do_poll
  File "/home/itamarst/devel/eventlet/eventlet/hubs/poll.py", line 80 in wait
  File "/home/itamarst/devel/eventlet/eventlet/hubs/hub.py", line 362 in run

Extension modules: greenlet._greenlet, __original_module__thread, __original_module_select, __original_module_time, MySQLdb._mysql (total: 5)

Running with slow 5 pytest tests/mysqldb_test.py, where slow is the following script:

#!/bin/bash
set -euo pipefail
PERCENTAGE="$1"
shift
systemd-run --uid=$USER --gid=$USER --scope --property CPUQuota="${PERCENTAGE}%" -- "$@"

@itamarst
Copy link
Contributor Author

Looks like it segfaulted in test_yields, which may've hit a timeout. So potentially unrelated.

@itamarst
Copy link
Contributor Author

Unable to reproduce the original issue, however, guess will keep an eye out to see if it recurs.

@4383 4383 added the asyncio label Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@itamarst itamarst

Labels
asyncio bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@4383 @itamarst