New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_multiprocessing_pool causes deadlock with Python 3.8a4 #295
Comments
Does the deadlock still happen with the master of pytest-cov? |
I am trying to find where the problem comes from but I am not successful. I'm working with the latest master. The strange thing is that it works when I run it in a virtual environment but not when I use system packages. With virtual environment:
without virtual environment:
As you can see, the main difference is the version of pytest so I tried to run multiprocessing tests with the pytest version 4.4.1 inside virtual environment and it works. There are several different versions of packages so I'll try to investigate them more. |
Ok, I am lost :( I've updated all packages on my system one by one via Do you know about any difference between these two environments other than dependencies versions?
|
Well first off: don't trust pip :) it's buggy, been very buggy and it only lists possibly broken metadata - which is not the actual code installed. To get at that stuck test:
About running the suite, you can use this neat lil trick to use tox with your custom python, eg:
Another thing: why coverage 5.0alpha? 4.5 is the stable version. |
Unfortunately, I don't have that much time to dig into it deeper but I got something. This is the output of strace command for all processes while the deadlocked one is still running.
Process 2103 (the main one) is waiting for process 2105 which is waiting for process 2107. Processes command line:
When I kill the process 2107 another one appears with the same last syscal and then process 2105 is waiting for them. By the last syscal I mean:
That's all I have now. By the way,
|
One more thing that might be important is that
|
It's a bit messy ... don't you have pip there? Use Since you don't have a compiler I suppose you also use a python3.8 image from somewhere? |
We discovered this problem while rebuilding RPM packages for Fedora in COPR where my Python 3.8a4 comes from. https://copr.fedorainfracloud.org/coprs/g/python/python3.8/ |
Ok, you can forget what I said before because it seems that the deadlock problem is not deterministic and you have to run multiprocessing tests multiple times to actually discover it. With this fact in mind, it seems that the problem is in the coverage itself because I can reproduce it with coverage version 4.5.3 but I cannot reproduce it with version 5.0a5. The first virtual environment with coverage 5.0a5
The second virtual environment with coverage 4.5.3
I am able to run multiprocessing tests in the first virtual environment without any troubles over a hundred times. But in the second one, it needs only 10 runs to discover the issue. Especially it appears from 3 to 8 times in 10 runs. I see in the tests source code that they are skipped on windows (because of broken multiprocessing) and on PyPy (because of deadlock problems) which might mean something. The next step, at least from my point of view, is to create a simple reproducer which will prove that the problem is in coverage and help me report an issue. |
I think this fix to coverage.py fixes this problem: nedbat/coveragepy@72b21f1. It hasn't been released for 4.5.x yet. |
This is now fixed in coverage.py 4.5.4. |
Hmm, still hangs for me?! (updated the c45 factor to use 4.5.4)
|
With Coverage 4.5.4, this is still an issue. |
I'm not sure what is different between our setups. When I run that environment (adjusted for the changes on master), the test fails, with a ton of output, finishing in 41 seconds:
|
I've done some debugging with Python 3.8.0b4 and logged https://bugs.python.org/issue38227 - it appears there's a different issue in b4. |
Hello.
Tests test_multiprocessing_pool, test_multiprocessing_pool_terminate and test_multiprocessing_pool_close cause deadlock with Python 3.8a4.
I don't know whether it's related to #294 or not and I haven't had time to investigate it more.
The text was updated successfully, but these errors were encountered: