New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug: concurrency causes Premature destruction of containers and networks (e.g. w/pytest-xdist) #567
Comments
What do you mean by
Give some time to prematurely stop
…On Thu, May 9, 2024, 2:57 PM Fabian Haenel ***@***.***> wrote:
*Describe the bug*
Testcontainers behaves incredibly flaky when running tests in parallel
with pytest-xdist. Specifically containers and networks are destroyed
prematurely resulting in network connections being interrupted, ports being
reassigned to different containers resulting in calls to the wrong
application or instance.
*To Reproduce*
Install the packages used for the tests.
pip install pytest==8.2 pytest-xdist==3.6 testcontainers==4.4 requests==2.31
Create a file named example.py
from time import sleep
import pytestimport requestsfrom testcontainers.core.network import Networkfrom testcontainers.postgres import PostgresContainerfrom testcontainers.vault import VaultContainer
@pytest.mark.parametrize("run", range(30))def test_healthcheck(run: int) -> None:
with Network() as network:
with PostgresContainer().with_network(network) as postgres_container:
with VaultContainer().with_network(network).with_exposed_ports(8200) as vault_container:
sleep(1) # Give some time to prematurely stop
response = requests.get(f"{vault_container.get_connection_url()}/v1/sys/health")
assert response.status_code == 200
Run pytest with parallel test execution on a multicore CPU (at least 2
cores).
pytest -n auto example.py
*Runtime environment*
Linux 6.8.0-76060800daily20240311-generic
Python 3.10 3.11 3.12
CPU i7-1165G7
testcontainers 4.4
—
Reply to this email directly, view it on GitHub
<#567>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACECGJEDDYQAU64UDVEMKRLZBPBJNAVCNFSM6AAAAABHPKJ4OOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4DQMRXGI3DQNY>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
This is a minimal example to showcase the buggy behavior, in my actual tests the sleep is other operations that just take some time, in which case the container sometimes have been stopped and disposed already, when they should not have been (the context has not been left at that point). Additional information: |
Sometimes I see errors as follows, which are caused by the vault container in my example being stopped mid-test execution. It did not crash, it was stopped and removed before it should have been. The issue is hard to reproduce, due to the issues flaky nature, sometimes no tests fail, sometimes only one, sometimes many.
|
So I'm guessing something in the Ryuk class or something else is failing to start the containers under parallelism. I guess we could start surrounding parts of the code with |
I'm going to go a bit off road here, @skeletorXVI are you familiar with Note Taken from https://pytest-xdist.readthedocs.io/en/latest/distribution.html
@alexanderankin I know this is not a proper fix, just trying to have some sort of workaround. |
Describe the bug
Testcontainers behaves incredibly flaky when running tests in parallel with
pytest-xdist
. Specifically containers and networks are destroyed prematurely resulting in network connections being interrupted, ports being reassigned to different containers resulting in calls to the wrong application or instance.To Reproduce
Note due to the flakiness you might need to tune to number of runs or repeat the test execution to see the errors.
Install the packages used for the tests.
Create a file named
example.py
Run pytest with parallel test execution on a multicore CPU (at least 2 cores).
Runtime environment
Linux
6.8.0-76060800daily20240311-generic
Python
3.10
3.11
3.12
CPU
i7-1165G7
testcontainers
4.4
The text was updated successfully, but these errors were encountered: