Different tests were collected (since 1.30.0 release), with pytest-randomly

[therefromhere]

I've been seeing this a lot today. I'm fairly sure it's related to the 1.30.0 release not playing nice with pytest-randomly.

Running

pytest --reuse-db -n 8

Hits this error almost every time:

pytest --reuse-db -n 8
Test session starts (platform: linux, Python 3.7.3, pytest 5.2.0, pytest-sugar 0.9.2)
Django settings: wolcenweb.settings.dev (from ini file)
Using --randomly-seed=1569981723
rootdir: /home/johnc/Projects/wolcenweb, inifile: pytest.ini, testpaths: wolcenweb
plugins: xdist-1.30.0, forked-1.0.2, django-webtest-1.9.7, cov-2.7.1, sugar-0.9.2, django-3.5.1, env-0.6.2, randomly-3.1.0
gw0 [246] / gw1 [246] / gw2 [246] / gw3 [246] / gw4 [246] / gw5 [246] / gw6 [246] / gw7 [246]

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― gw0 ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Different tests were collected between gw5 and gw0. The difference is:
--- gw5

+++ gw0

@@ -1,246 +1,246 @@

Any of the following seems to fix it:

• downgrading to pytest-xdist==1.29.0
• removing pytest-randomly
• setting the randomly see with --randomly-seed=123456789

For what it's worth this is with pytest-django, other pytest stuff in play is:

pip freeze | grep pytest
pytest==5.2.0
pytest-cov==2.7.1
pytest-django==3.5.1
pytest-env==0.6.2
pytest-forked==1.0.2
pytest-randomly==3.1.0
pytest-runner==5.1
pytest-sugar==0.9.2
pytest-xdist==1.30.0

FWIW I'm not using pytest.mark.parametrize in my tests.

[mmehan1]

I am using pytest.mark.parametrize and pytest-random-order, a test-suite that was passing yesterday is failing today. I have downgraded to 1.29.0 and it is working again. will check back soon 👍

[blueyed]

Are you using $PYTEST_ADDOPTS by chance? (#478)

[therefromhere]

@blueyed I'm not, no.

[blueyed]

@therefromhere
Ok. Can you git-bisect it, please?

[therefromhere]

@blueyed done, I think it's caused by 953a3f0

[bashtage]

Seeing same types of random collection-based failures in statsmodels. Pinning to < 1.30 in statsmodels/statsmodels#6205

[blueyed]

Might be worth checking if pytest-dev/pytest#6008 fixes it (in pytest).

[therefromhere]

@blueyed doesn't seem to, I'm still seeing it with pytest==5.2.2.dev84+g978c7ae1b (from install with
pip install git+https://github.com/pytest-dev/pytest.git@features )

[liiight]

I encountered this issue as well but found a weird workaround.
Here is an example of a failed run:

gw3:None (gw3)
Different tests were collected between gw0 and gw3. The difference is:
--- gw0

+++ gw3

@@ -5,30 +5,30 @@

 tests/casb/rules/test_component_rules_engine_v2.py::TestRulesEngineV2::test_rule_context_granularity_include_predicate[vendor:google_google_casb_license-context_with_users]
 ...

The long test name is derived from a custom param ID I pass. When I remove the custom ID, the node name changes to:
tests/casb/rules/test_component_rules_engine_v2.py::TestRulesEngineV2::test_rule_context_granularity_include_predicate[vendor:google_google_casb_license-include_predicate_context0] and xdist runs just fine.

[eberg23]

I am also seeing this issue with some data-driven tests (using ddt).

--- gw0

+++ gw9

@@ -1,745 +1,745 @@

pip freeze looks like:

ddt==1.2.0
flaky==3.6.1
py==1.8.1
pytest==5.3.2
pytest-cache==1.0
pytest-cov==2.8.1
pytest-django==3.5.1
pytest-forked==1.1.3
pytest-pep8==1.0.6
pytest-random-order==1.0.4
pytest-socket==0.3.3
pytest-timeout==1.3.3
pytest-xdist==1.29.0```
This is not a complete list, just the pytest-relevant stuff. As you can see, we reverted to `1.29.0` to get builds passing.

[gt-novelt]

Also got this error with:

• pytest == 5.3.5
• pytest-xdist == 1.31.0
• pytest-randomly == 3.2.1

[bodharma]

Same issue with
Python 3.7.1
Python 3.7.5

"allure-pytest==2.8.11", "pytest==5.3.5", "pytest-xdist==1.31.0", "requests==2.23.0", "python-dateutil==2.8.1", "grappa==0.1.12", "tox==3.14.5", "virtualenv==20.0.10", "pytest-sugar==0.9.2", "click", "lxml", "pytest-split-tests"

OR

"allure-pytest==2.8.11", "pytest==5.3.5", "pytest-xdist==1.29.0", "requests==2.23.0", "python-dateutil==2.8.1", "grappa==0.1.12", "tox==3.14.5", "virtualenv==20.0.10", "pytest-sugar==0.9.2", "click", "lxml", "pytest-split-tests"

How to deal with that?

UPD.
If I remove pytest-split-tests everything works fine

[adamchainz]

🎲 pytest-randomly 3.3.0 has fixed this issue 🎲

[adamchainz]

This issue has collected some reports for other plugins:

• @mmehan1 @eberg23 you reported with using pytest-random-order.
• @velocyno you reported with pytest-split-tests

If you are experiencing these, please open new issues and on the corresponding plugins too. They may want to do the same as pytest-dev/pytest-randomly#245 .