pytest-xdist is not reporting the same nodeid as pytest does #445

elyezer · 2019-06-21T14:43:58Z

Running pytest and pytest -n <executors> does not produce the same nodeid for the tests and therefore this is preventing lastfailed from running properly. Since the nodeids in the cache don't match any test collected, it ends up running tests that are not meant to run.

I have the following workflow:

Run the tests that can run in parallel using xdist
Run the tests that need to run serially without using xdist's -n option
Run pytest with the --last-failed option to try rerunning the failures to check if they are flaky. This is again without using xdist's -n option

Here is the reproducer project structure:

$ ls -1R
.:
pytest.cfg
tests

./tests:
api

./tests/api:
__pycache__
test_bar.py
test_foo.py

Here are the file contents:

$ cat tests/api/test_bar.py 
def test_bar_1():
    assert True


def test_bar_2():
    assert True


def test_bar_3():
    assert False

$ cat tests/api/test_foo.py 
import os


def test_foo_fail():
    assert os.environ.get('FOO_PASS', 'false') == 'true'


def test_foo_ok():
    assert True

$ cat pytest.cfg 
[tool:pytest]
python_files = *.py
testpaths = tests/api

Considering the project described above, if you run pytest -n <executors> it shows the following (mind the tests' nodeids):

$ pytest -v --tb=no -c pytest.cfg -n 2
================================ test session starts ================================
platform linux -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- /home/elyezer/.local/venvs/pytest/bin/python3
cachedir: .pytest_cache
rootdir: /home/elyezer/code/xdist-reproducer, inifile: pytest.cfg, testpaths: tests/api
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/elyezer/code/xdist-reproducer
[gw1] linux Python 3.7.3 cwd: /home/elyezer/code/xdist-reproducer
[gw0] Python 3.7.3 (default, May 11 2019, 00:38:04)  -- [GCC 9.1.1 20190503 (Red Hat 9.1.1-1)]
[gw1] Python 3.7.3 (default, May 11 2019, 00:38:04)  -- [GCC 9.1.1 20190503 (Red Hat 9.1.1-1)]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

test_bar.py::test_bar_1 
test_bar.py::test_bar_2 
[gw1] [ 20%] PASSED test_bar.py::test_bar_2 
[gw0] [ 40%] PASSED test_bar.py::test_bar_1 
test_bar.py::test_bar_3 
test_foo.py::test_foo_fail 
[gw0] [ 60%] FAILED test_bar.py::test_bar_3 
test_foo.py::test_foo_ok 
[gw0] [ 80%] PASSED test_foo.py::test_foo_ok 
[gw1] [100%] FAILED test_foo.py::test_foo_fail 

======================== 2 failed, 3 passed in 0.32 seconds =========================

But if we run without the -n option here is the output:

$ pytest -v --tb=no -c pytest.cfg     
================================ test session starts ================================
platform linux -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- /home/elyezer/.local/venvs/pytest/bin/python3
cachedir: .pytest_cache
rootdir: /home/elyezer/code/xdist-reproducer, inifile: pytest.cfg, testpaths: tests/api
plugins: xdist-1.29.0, forked-1.0.2
collected 5 items                                                                   

tests/api/test_bar.py::test_bar_1 PASSED                                      [ 20%]
tests/api/test_bar.py::test_bar_2 PASSED                                      [ 40%]
tests/api/test_bar.py::test_bar_3 FAILED                                      [ 60%]
tests/api/test_foo.py::test_foo_fail FAILED                                   [ 80%]
tests/api/test_foo.py::test_foo_ok PASSED                                     [100%]

======================== 2 failed, 3 passed in 0.04 seconds =========================

As you can see both calls has the same rootdir but produce different nodeids for the tests. And if we check the cache for lastfailed:

$ pytest -c pytest.cfg --cache-show=cache/lastfailed
================================ test session starts ================================
platform linux -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: /home/elyezer/code/xdist-reproducer, inifile: pytest.cfg, testpaths: tests/api
plugins: xdist-1.29.0, forked-1.0.2
cachedir: /home/elyezer/code/xdist-reproducer/.pytest_cache
------------------------ cache values for 'cache/lastfailed' ------------------------
cache/lastfailed contains:
  {'test_bar.py::test_bar_3': True,
   'test_foo.py::test_foo_fail': True,
   'tests/api/test_bar.py::test_bar_3': True,
   'tests/api/test_foo.py::test_foo_fail': True}

=========================== no tests ran in 0.00 seconds ============================

pytest-xdist test nodeid generation is not consistent with plain pytest nodeid generation and this inconsistency is breaking plugins such as the lastfailed that relies on the nodeids to do its job.

The text was updated successfully, but these errors were encountered:

nicoddemus · 2019-07-01T22:39:10Z

Hi @elyezer,

Really sorry for the delay in answering this!

Thanks a lot for the detailed report, we appreciate it.

The problem is related to the testpaths option; commenting that out produces the correct node ids:

 λ pytest -v --tb=no -c pytest.cfg -n 2
======================== test session starts ========================
platform win32 -- Python 3.6.3, pytest-4.6.4.dev15+g09dee292c.d20190630, py-1.5.2, pluggy-0.12.0 -- x:\pytest-xdist\.env36\scripts\python.exe
cachedir: .pytest_cache
rootdir: x:\pytest-xdist\.tmp\node-bug, inifile: pytest.cfg
plugins: forked-0.2, xdist-1.28.1.dev9+g407d6e1
[gw0] win32 Python 3.6.3 cwd: x:\pytest-xdist\.tmp\node-bug
[gw1] win32 Python 3.6.3 cwd: x:\pytest-xdist\.tmp\node-bug
[gw0] Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
[gw1] Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

tests/api/test_bar.py::test_bar_2
tests/api/test_bar.py::test_bar_1
[gw0] [ 20%] PASSED tests/api/test_bar.py::test_bar_1
[gw1] [ 40%] PASSED tests/api/test_bar.py::test_bar_2
tests/api/test_bar.py::test_bar_3
tests/api/test_foo.py::test_foo_fail
[gw0] [ 60%] FAILED tests/api/test_bar.py::test_bar_3
tests/api/test_foo.py::test_foo_ok
[gw0] [ 80%] PASSED tests/api/test_foo.py::test_foo_ok
[gw1] [100%] FAILED tests/api/test_foo.py::test_foo_fail

================ 2 failed, 3 passed in 1.29 seconds =================

Definitely a bug, thanks for the report.

nicoddemus · 2019-07-01T23:30:54Z

Actually the underlying cause is the -c flag. Unfortunately we have a technical debt regarding config initialization (tracked in https://github.com/pytest-dev/pytest/projects/2).

For now a workaround is to rename your pytest.cfg to setup.cfg (the standard name), and then the node ids are generated as expected:

λ pytest -v --tb=no -n2
======================== test session starts ========================
platform win32 -- Python 3.6.3, pytest-4.6.4.dev15+g09dee292c.d20190630, py-1.5.2, pluggy-0.12.0 -- x:\pytest-xdist\.env36\scripts\python.exe
cachedir: .pytest_cache
rootdir: x:\pytest-xdist\.tmp\node-bug, inifile: setup.cfg, testpaths: tests/api
plugins: forked-0.2, xdist-1.28.1.dev9+g407d6e1
[gw0] win32 Python 3.6.3 cwd: x:\pytest-xdist\.tmp\node-bug
[gw1] win32 Python 3.6.3 cwd: x:\pytest-xdist\.tmp\node-bug
[gw0] Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
[gw1] Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

tests/api/test_bar.py::test_bar_2
[gw1] [ 20%] PASSED tests/api/test_bar.py::test_bar_2
tests/api/test_bar.py::test_bar_1
[gw0] [ 40%] PASSED tests/api/test_bar.py::test_bar_1
tests/api/test_foo.py::test_foo_fail
tests/api/test_bar.py::test_bar_3
[gw0] [ 60%] FAILED tests/api/test_bar.py::test_bar_3
[gw1] [ 80%] FAILED tests/api/test_foo.py::test_foo_fail
tests/api/test_foo.py::test_foo_ok
[gw1] [100%] PASSED tests/api/test_foo.py::test_foo_ok

================ 2 failed, 3 passed in 1.05 seconds =================

nicoddemus · 2019-07-01T23:34:26Z

Related: #6

elyezer · 2019-07-02T13:50:52Z

@nicoddemus thanks for digging into that. I was thinking if it would make sense to have some sort of helper on pytest that would provide the nodeid since I assume the logic to deal with the config file is there already and therefore we could avoid having to somewhat repeat the same logic here.

I am willing to contribute, so if you could give me some pointers I could try getting this done. This is going to be the first time that I will be contribute and I still learning more about the source code of both pytest and pytest-xdist.

Thank you

While investigating a recent node id bug (pytest-dev#445) I decided to play with config initialization around the idea of initializing the config on the remote *exactly* we initialize the config object on pytest.main() One thing that immediately broke all tests was inprocess pytester runs, but other than that just a few tests failed which I find encouraging. While this patch is in no way even remotely ready, I would like to get opinions if the overall idea of initializing the config object like this is sound.

nicoddemus · 2019-07-03T18:24:28Z

Hi @elyezer,

Great to hear that you are considering contributing, we definitely appreciate it. 👍

But this topic in particular is not really recommended for newcomers to the project, as it involves a lot of complex iteration during the config initialization between pytest and pytest-xdist.

Good news is that while I was debugging your report I decided to take the approach shown in #448 to fix the config initialization problem, and it looks promising. Hopefully I will find the time required to move this forward in the next few weeks.

I suggest to take a look at other issues in the tracker that might be easier to get started. 👍

elyezer · 2019-07-05T00:02:41Z

@nicoddemus I will keep a eye on it. Thank you for working on it.

I will check other issues and hopefully no can start contributing sooner rather than later.

Thank you very much

Decided to keep the old way still working for now. Fix pytest-dev#6 Fix pytest-dev#445

nicoddemus added the bug label Jul 1, 2019

nicoddemus mentioned this issue Jul 2, 2019

Use Config.invocation_params for consistent worker initialization #448

Merged

nicoddemus added a commit to nicoddemus/pytest-xdist that referenced this issue Jul 10, 2019

Use Config.invocation_params for consistent worker initialization

953a3f0

Decided to keep the old way still working for now. Fix pytest-dev#6 Fix pytest-dev#445

nicoddemus closed this as completed in #448 Jul 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pytest-xdist is not reporting the same nodeid as pytest does #445

pytest-xdist is not reporting the same nodeid as pytest does #445

elyezer commented Jun 21, 2019 •

edited

nicoddemus commented Jul 1, 2019

nicoddemus commented Jul 1, 2019

nicoddemus commented Jul 1, 2019

elyezer commented Jul 2, 2019

nicoddemus commented Jul 3, 2019

elyezer commented Jul 5, 2019

pytest-xdist is not reporting the same nodeid as pytest does #445

pytest-xdist is not reporting the same nodeid as pytest does #445

Comments

elyezer commented Jun 21, 2019 • edited

nicoddemus commented Jul 1, 2019

nicoddemus commented Jul 1, 2019

nicoddemus commented Jul 1, 2019

elyezer commented Jul 2, 2019

nicoddemus commented Jul 3, 2019

elyezer commented Jul 5, 2019

elyezer commented Jun 21, 2019 •

edited