Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[good first issue][CI breakage] pytest-xdist == 3.6.0 makes the CI fail #4317

Closed
webknjaz opened this issue Apr 19, 2024 · 10 comments
Closed

[good first issue][CI breakage] pytest-xdist == 3.6.0 makes the CI fail #4317

webknjaz opened this issue Apr 19, 2024 · 10 comments
Labels
blocker critical good first issue Needs Implementation Issues that are ready to be implemented. task

Comments

@webknjaz
Copy link
Member

webknjaz commented Apr 19, 2024 •
edited

Since the release of new pytest-xdist roughly an hour ago, all CI jobs fail as follows:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/_pytest/main.py", line 273, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/_pytest/main.py", line 327, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 181, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_result.py", line 99, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 166, in _multicall
INTERNALERROR>     teardown.throw(outcome._exception)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/_pytest/logging.py", line 796, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/xdist/dsession.py", line 138, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/xdist/dsession.py", line 163, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/xdist/dsession.py", line 201, in worker_workerfinished
INTERNALERROR>     self.config.hook.pytest_testnodedown(node=node, error=None)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pytest_cov/plugin.py", line 316, in pytest_testnodedown
INTERNALERROR>     self.cov_controller.testnodedown(node, error)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pytest_cov/engine.py", line 342, in testnodedown
INTERNALERROR>     rinfo = node.gateway._rinfo()
INTERNALERROR>             ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/execnet/gateway.py", line 87, in _rinfo
INTERNALERROR>     self._cache_rinfo = RInfo(ch.receive())
INTERNALERROR>                               ^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/execnet/gateway_base.py", line 934, in receive
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR> execnet.gateway_base.RemoteError: concurrent remote_exec would cause deadlock for main_thread_only execmodel

(https://github.com/pypa/setuptools/actions/runs/8759251151/job/24041851385?pr=4092#step:9:114)

I also filed the upstream report here: pytest-dev/pytest-xdist#1071.

Immediate action item

Temporarily restrict the version of this dependency by adding , != 3.6.0 to the dependency specifier in tox.ini and maybe elsewhere, this needs investigation.
It looks like execnet is what needs to be capped below v2 per pytest-dev/pytest-xdist#620 (comment).
@MorganWeiss
Copy link

MorganWeiss commented Apr 20, 2024 •
edited

I am experiencing the same issue as well. Is there a way of changing the version to have it work? I do not see a dependency in my packages so I am not sure where it is coming from.

@abravalheri
Copy link
Contributor

If I understood correctly, as per pytest-dev/pytest-xdist#1071 (comment) pytest==3.6.0 has been yanked, and probably new versions of pytest-xdist, execnet and pytest will be coordinated to fix the problem, right?

So my understanding is that for the time being we don't have to pin any dependency as a workaround. (Also happy to add a workaround if I understood things wrong).

@webknjaz
Copy link
Member Author

@abravalheri I haven't tested it but I'd expect that it's execnet that needs to be yanked, not pytest-xdist. So my guess is that it's still possible to get a broken version. You can restart any GHA workflow in the repo and see if it breaks..

@abravalheri
Copy link
Contributor

@abravalheri I haven't tested it but I'd expect that it's execnet that needs to be yanked, not pytest-xdist. So my guess is that it's still possible to get a broken version. You can restart any GHA workflow in the repo and see if it breaks..

Possibly working? (https://github.com/pypa/setuptools/actions/runs/8759517556/job/24055273503, although cygwin seems to have trouble on finding the dependencies).

@webknjaz
Copy link
Member Author

Possibly working? (https://github.com/pypa/setuptools/actions/runs/8759517556/job/24055273503

Yep

although cygwin seems to have trouble on finding the dependencies).

No, that's an incorrect assessment. This is a classical case of "who knows what broke in the CI step" — if you look closer, you'll see that pip install was successful and coverage xml is what actually failed. This is one of the reasons I always insist on having a single command per CI step (it actually enables more confusing behaviors when there's several invocations inside, like a "false success").

@MorganWeiss
Copy link

Any progress on fixing this issue?

@abravalheri
Copy link
Contributor

abravalheri commented Apr 21, 2024 •
edited

@MorganWeiss I don't see any immediate action to be taken by setuptools. Maybe it is relevant for you to follow the issues in pytes-xdist and co and possibly contributing towards the fix there?

@MorganWeiss
Copy link

MorganWeiss commented Apr 21, 2024 via email

@abravalheri
Copy link
Contributor

abravalheri commented Apr 21, 2024 •
edited

You can start with the link shared in the first message of this thread, and then you can follow the discussions (they point to more links there).

@webknjaz
Copy link
Member Author

v3.6.1 has just been released. Hopefully, this is enough to address this problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocker critical good first issue Needs Implementation Issues that are ready to be implemented. task
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@abravalheri @webknjaz @MorganWeiss