TST: all CI is broken by pytest 8.1.0

[neutrinoceros]

Running pytest astropy crashes at startup with

pluggy._manager.PluginValidationError: Plugin 'pytest_filter_subpackage' for hook 'pytest_ignore_collect'
hookimpl definition: pytest_ignore_collect(path, config)
Argument(s) {'path'} are declared in the hookimpl but can not be found in the hookspec

Full Traceback

Traceback (most recent call last):
  File "/Users/clm/.pyenv/versions/astropy.devdeps/bin/pytest", line 8, in <module>
    sys.exit(console_main())
             ^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/_pytest/config/__init__.py", line 195, in console_main
    code = main()
           ^^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/_pytest/config/__init__.py", line 153, in main
    config = _prepareconfig(args, plugins)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/_pytest/config/__init__.py", line 335, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/pluggy/_callers.py", line 138, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/_pytest/helpconfig.py", line 105, in pytest_cmdline_parse
    config = yield
             ^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1141, in pytest_cmdline_parse
    self.parse(args)
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1490, in parse
    self._preparse(args, addopts=addopts)
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1377, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/pluggy/_manager.py", line 415, in load_setuptools_entrypoints
    self.register(plugin, name=ep.name)
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/_pytest/config/__init__.py", line 497, in register
    plugin_name = super().register(plugin, name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/pluggy/_manager.py", line 167, in register
    self._verify_hook(hook, hookimpl)
  File "/Users/clm/.pyenv/versions/astropy.devdeps/lib/python3.12/site-packages/pluggy/_manager.py", line 342, in _verify_hook
    raise PluginValidationError(
pluggy._manager.PluginValidationError: Plugin 'pytest_filter_subpackage' for hook 'pytest_ignore_collect'
hookimpl definition: pytest_ignore_collect(path, config)
Argument(s) {'path'} are declared in the hookimpl but can not be found in the hookspec

Example Logs

pytest 8.1's release notes read that some internal changes may break user plugins, which seems to be what we're seeing here, however this exact change isn't mentioned, so it's not 100% clear whether this should be fixed in pytest_filter_subpackage or pytest itself yet. Will report upstream when I know more. In the mean time, an obvious mitigation we can apply on our side is to pin pytest.

[matteobachetti]

In affiliated packages too, as expected https://github.com/StingraySoftware/stingray/actions/runs/8136691328/job/22233514077

[MridulS]

Just for ref: this is across the pytest plugin infrastructure pytest-dev/pytest#12069

Let's quickly get #16147 in to make our CIs happy.

[effigies]

xref scientific-python/pytest-doctestplus#239

[pllim]

How is this possible? We test against pytest-dev and it never came up. 😩

[effigies]

They consider the failure to remove in the 8.0 release a bug: pytest-dev/pytest#12069

Looks like they're reverting, though.

[MridulS]

It's being yanked, @pllim we need a new release of doctest-plus

[pllim]

I released pytest-doctestplus anyway, so this should no longer be a problem going forward.

[pllim]

For completeness, I also released pytest-filter-subpackage .