pytest coverage reports active even when it makes no sense

[ssbarnea]

Example where pytest-cov produces console spam: when you call pytest --collect-only. Obviously that doing a coverage report when you do not run even one test makes no sense.

Also when using markers of filers (-m/-k) to run specific tests, pytest-cov when enabled in pytest.ini will continue to produce reports.

Even if I manually add --no-cov to the CLI, I will still get some warnings:

============================================================================= warnings summary (final) ==============================================================================
/Users/ssbarnea/os/pytest-cov/src/pytest_cov/plugin.py:254
  /Users/ssbarnea/os/pytest-cov/src/pytest_cov/plugin.py:254: PytestWarning: Coverage disabled via --no-cov switch!
    warnings.warn(pytest.PytestWarning(message))

It what once a projects adopts pytest-cov and configures it, you are no longer able to silently bypass coverage unless you modify the source code.

As I stated previously, pytest-cov needs an option, controlable via environment variables (as defining them does not require you to alter the project codebase) for disabling itself. Please keep in mind that disabling the plugin via pytest option would produce errors because the pytest.ini from inside the repository would likely have arguments that are specific to pytest-cov.

Mainly what we need is a --no-cov-and-shut-up-about-it option ;)

PS. There is another interesting issue related to adding export PYTEST_ADDOPTS= --no-cov to your shell profile: it may work for projects that have pytest-cov but it may break projects that do not have pytest-cov, even if you do install pytest-cov on your system. The moment you will invoke tox, it will create an isolated venv without pytest-cov and the pytest call will fail when encountering an option that it is not aware about.

I am not sure how to address that last issue but there is a decent workaround: you can use direnv or similar tool to declare environment variables that are loaded only for certain paths, so you can use project specific PYTEST_ADDOPTS values.

[nedbat]

@ssbarnea this sounds like a good example of why pytest-cov should do less: it's hard to control the reporting options when you consider all the different ways pytest runs. Would you be in favor of #337 (Pytest-cov should do less)?

[ssbarnea]

@nedbat I am one of the 👍 on that ticket but this is almost one year old without any tangible result. I guess that I will have to propose a PR soon about it. I doubt the linked ticket will be addressed soon.

[ionelmc]

The way the arguments were created, and why that warning exists, is that pytest-cov is "opt-in". Normally it's disabled and you enable it by adding to PYTEST_ADDOPTS or whatever. Thus the warning exists so users don't wonder why coverage don't work when they add --cov (iow: avoid the silent failure). How do you propose to solve both this problem and your zero warning preoccupation?