[FR] Loading/disabling plugins conditionally

[webknjaz]

Is your feature request related to a problem? Please describe.

I'm integrating Cython.Coverage in the @aio-libs projects. It works fine for CPython but crashes on PyPy. This is because CoverageWarning is emitted and pytest is configured with filterwarnings = error (coveragepy is integrated through pytest-cov).

In projects, where we use Cython, we normally maintain a fallback pure-python implementation. So in CI we run tests with and without C-extensions to cover these cases. Cython.Coverage is added to .coveragerc statically and gets picked up even when we test pure-python builds.
That works fine on CPython, there's just no matches for PYX files since they aren't measured.
It explodes on PyPy, though. For PyPy, we always disable building C-extensions, so it's always pure-python.

But since the config is static, and the plugin is needed for other envs, we're stuck with coveragepy picking it up when it's undesired.

Describe the solution you'd like

I think, there's a need for dynamically enabling and disabling plugins:

• a CLI arg
• a public API interface arg (so things like pytest-cov could integrate it seamlessly)
• an environment variable

Describe alternatives you've considered

• Putting a no-op module with the same import signature shadowing the actual plugin into $PYTHONPATH and auto-injecting that on PyPy, using some wrapper.
• Using sed -i 's/^plugins = .*//g' in CI.

Additional context
The failing CI job: https://github.com/aio-libs/yarl/actions/runs/6971527253/job/18972275042?pr=961#step:9:82

I extracted the traceback since GHA logs get garbage-collected over time:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/runpy.py", line 200, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 192, in console_main
    code = main()
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 150, in main
    config = _prepareconfig(args, plugins)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 331, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 130, in _multicall
    teardown[0].send(outcome)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/helpconfig.py", line 104, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 1075, in pytest_cmdline_parse
    self.parse(args)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 1425, in parse
    self._preparse(args, addopts=addopts)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 1327, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pytest_cov/plugin.py", line 152, in pytest_load_initial_conftests
    plugin = CovPlugin(options, early_config.pluginmanager)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pytest_cov/plugin.py", line 203, in __init__
    self.start(engine.Central)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pytest_cov/plugin.py", line 225, in start
    self.cov_controller.start()
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
    return meth(self, *args, **kwargs)
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pytest_cov/engine.py", line 241, in start
    self.cov.start()
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/control.py", line 631, in start
    self._init_for_start()
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/control.py", line 562, in _init_for_start
    self._warn(
  File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/control.py", line 434, in _warn
    warnings.warn(msg, category=CoverageWarning, stacklevel=2)
coverage.exceptions.CoverageWarning: Plugin file tracers (Cython.Coverage.Plugin) aren't supported with PyTracer

[webknjaz]

I just noticed that there's a feature to suppress warnings that have unique identifiers @ https://coverage.readthedocs.io/en/latest/cmd.html#warnings. But this specific warning isn't listed there.

I located the _warn() call @ https://github.com/nedbat/coveragepy/blob/184dc78/coverage/control.py#L562-L569 and realized that it doesn't pass a slug= argument. I suppose it could be useful to set it as well.