Parallelize the test runs via pytest-xdist

[webknjaz]

Summary of changes

This is a demo showing how using pytest-xdist affects the test runs.

Resolves #2458

Performance improvements

Comparing before/after, this patch makes the jobs run 2x faster in the CI and 4x faster locally.

	main	this PR
py36/ubuntu-latest	240.99s (0:04:00)	137.93s (0:02:17)
pypy3/macos-latest	1642.34s (0:27:22)	732.48s (0:12:12)

Pull Request Checklist

• Changes have tests
• News fragment added in changelog.d/.
  (See documentation for details)

[webknjaz]

This PR demonstrates both points raised in #2458:

1. There's some race condition supposedly because of insufficient test isolation
2. It's faster with xdist. Compare 233.77s (0:03:53) on master vs 105.42s (0:01:45) in this PR for the GHA ubuntu py3.8 job. (https://github.com/pypa/setuptools/runs/1419466762#step:17:1076 vs https://github.com/pypa/setuptools/pull/2459/checks?check_run_id=1419764437#step:17:2199). That's x2 improvement over the current state of things!

[webknjaz]

setuptools/tests/test_build_meta.py::TestBuildMetaBackend::test_build_sdist_relative_path_import

This test seems to fail because pytest-xdists injects '' into sys.path (it's not there when I add -n 0 to disable it). But I've also noticed a chdir() call that is not thread-safe + a deprecated tmpdir fixture is used (it's recommended to use tmp_path instead).

[jaraco]

I've already stumbled onto a use-case this breaks. I added a breakpoint() to the code to inspect something. Before adding xdist, such a breakpoint would enable me to begin inspecting the code at that point. With xdist, that causes the test suite to fail with a BdbQuit exception. The tests aren't just running in parallel, but they're being forked to other processes that don't have a terminal with which to interact.

[pganssle]

I've already stumbled onto a use-case this breaks. I added a breakpoint() to the code to inspect something.

I don't have any particular insight into this, but I've recently been working on Django and they have a test runner that defaults to running the tests in parallel and explicitly throws an error if you run --pdb. Might be that you simply have to disable running in parallel if you want breakpoints. Seems like a fair trade to me, TBH. I run test suites much more frequently than I set breakpoints in them, and I usually only make the "can't debug under X condition" mistake once or twice per debugging session, for a large net savings of time.

[webknjaz]

@jaraco

I'd prefer if this could use pytest-enabler, because otherwise, it becomes difficult to turn this feature off or customize the setting for a particular run. e.g.:
[...]
I've already stumbled onto a use-case this breaks. I added a breakpoint() to the code to inspect something. Before adding xdist, such a breakpoint would enable me to begin inspecting the code at that point. With xdist, that causes the test suite to fail with a BdbQuit exception. The tests aren't just running in parallel, but they're being forked to other processes that don't have a terminal with which to interact.

In this cause, I normally use

$ tox -- -n0 -s path/to/test.py::test_something

-n0 disables xdist and makes it sequential + -s allows to use breakpoint() without pytest stealing stdin/stdout streams from you.

@pganssle

I don't have any particular insight into this, but I've recently been working on Django and they have a test runner that defaults to running the tests in parallel and explicitly throws an error if you run --pdb. Might be that you simply have to disable running in parallel if you want breakpoints.

I believe that it's a bit better with pure pytest: if you actually use --pdb and not breakpoint(), xdist will automatically disable the concurrency and turn the test run into sequential.
It'll error out, though, if you will attempt to force the concurrency:

$ pytest -n2 --pdb  # fails
[...]
ERROR: --pdb is incompatible with distributing tests; try using -n0 or -nauto.
$ pytest --pdb  # works

Seems like a fair trade to me, TBH. I run test suites much more frequently than I set breakpoints in them, and I usually only make the "can't debug under X condition" mistake once or twice per debugging session, for a large net savings of time.

My experience is about the same. I also occasionally do PYTHONBREAKPOINT=pudb.remote.set_trace and connect there via telnet in a separate terminal emulator tab.

[jaraco]

Good points. I'm inclined to accept the detriments in favor of the benefits. I've also confirmed that simply passing --pdb to pytest disables xdist, so works to bypass the behavior. Also, with the use of pytest-enabler, -p no:xdist also works.