Proposal: pytest-cov should do less

[nedbat]

I've been working in the pytest-cov code recently to get it to support coverage.py 5.0 (pull request: #319), and also reviewing #330. The problems that have come up have brought me to a conclusion: pytest-cov does too many things.

The current code attempts to be a complete UI for coverage. I think this is a mistake. Pytest-cov should limit itself to those functions that can only be done within a pytest plugin. In particular, the reporting options of pytest-cov add no value over just using coverage directly to generate reports.

Pytest-cov is no one's passion project. There are a few of us that would like to move it forward, but want to do it efficiently. Removing functionality from pytest-cov would make it easier to maintain.

If we keep to the current model of pytest-cov being a complete UI for coverage.py, then every time something is added to coverage.py, the pytest-cov plugin needs to be updated. As an example, we just added a new report type to coverage.py (JSON). Why do we need to add options to pytest-cov to support it?

Pytest is a test runner. Its job is to run tests. The pytest-cov plugin should limit itself to integrating the run phase of coverage into the test running process. Reporting is a separate step that is better separately.

The current design leads to tortured option syntax in an attempt to squeeze everything into the pytest command line. There's no need. Remove options that aren't related to running, and let some options be specified in configuration files. Sophisticated test suites already have configuration files. Let's simplify pytest-cov.

Thoughts?

[nedbat]

/cc @ionelmc @graingert

[nedbat]

@jezdez Is there something i can do to clear up the confusion (if that's what the emoji means!)? I may have written too fast and left out some context.

[jezdez]

I absolutely understand the "simple is better than complex" idea in this proposal and I support you in making maintenance easier. 👍🏻

I definitely worry about having to know the coverage CLI tool as well as setting up pytest-cov to generate terminal output though. At the moment it's one step in a very pytest-typical config setup (updating addopts in pytest.ini) and you're proposing two steps to get the same result, addopts and another way to run coverage report. Given how difficult some CI environments are (e.g. path problems, incompatible dependencies etc) I think this change would basically trade lower maintenance friction on your side with a higher barrier of ease of use. Not saying that's a bad trade, just that it would make sense to maybe find a way to alleviate that higher barrier for users.

Alternatively to your suggestion would there be a way to instead implement a simplified version of the coverage API in pytest-cov? Only cover terminal reporting in pytest-cov for example and document how to do more complex reporting via the coverage CLI?

[graingert]

I'd be in favour of the coverage API providing a method like .summary() such that this code would pass the tests: 5493d82

[graingert]

would be some wiggle room in how cov_fail_under would be passed to summary of course, or it could handle the cov_fail_under entirely and raise an exception after having finished all the writing

[nedbat]

1. A few people have mentioned the convenience of having one command (pytest --cov-etc) that will run tests and produce coverage reports. I understand that convenience, but a simple shell script would do the same job. I don't think it makes sense for pytest-cov to implement a new but worse UI for coverage.py to get that convenience.

I could compromise on @jezdez's idea of keeping text reporting, and leaving other reporting to the coverage.py command line. How would other people feel about that?

2. I don't understand how a new API method in coverage.py will solve the problem? The UI would still be a set of baroque options in pytest-cov, that would need to be maintained, extended, and tested.

BTW: I'm very glad for the discussion! :)

[somacdivad]

As a frequent user of Pytest-cov, I'm strongly in favor of reducing its features.

And I agree with @jezdez that keeping the console reports is worth thinking about. It's literally the only reporting feature I use with Pytest-cov, if I use it for reporting at all, and I can see how it's handy enough to keep around.

The benefits of the Pytest-cov reporting UI always felt marginal to me, at best. At worst they just add bloat to my Pytest commands.

[okken]

simplifying the plugin would be great. Allowing text reporting would be preferable to not.

[gaborbernat]

You get my vote to go down this path with pytest cov 3 👍

[maurobaraldi]

What about two "versions" of plugin: pytest-cov and pytest-cov (enhanced).Ok, it does not much sense for me too, an enhanced version from a plugin, but the version could be like a branch from vcs project.

Maybe, this would be like a experiment, to analyse how many people miss this "enhanced features".

[nedbat]

@maurobaraldi I'm not sure who would maintain the enhanced one. If I only contribute to the plain version (to add support for new coverage.py features), then the enhanced won't be a superset of the plain version. That sounds extra-confusing. :(

[loechel]

It may be good for the discussion to talk about the affected API and options and common use cases and configurations.

I think it should be differentiated between three major useages:

• direct call of pytest
• usage of tox
• usage of CI/CD with coveralls

The direct useage of pytest is for simplicity and speedup of concurrent testing / subprocesses

Usage of tox may follow two mayor config styles:

as described in coverage docs:

[testenv]
...
commands =
    pytest --cov=src --cov=tests --cov-report=xml

setenv =
  COVERAGE_FILE=.coverage.{envname}

deps =
    pytest
    pytest-cov

[testenv:coverage]
skip_install = true

deps =
    coverage

setenv =
  COVERAGE_FILE=.coverage

commands =
    coverage erase
    coverage combine
    coverage html
    coverage xml
    coverage report --fail-under=100.0

or as documentet in pytest-cov docs:

[testenv]
commands = pytest --cov --cov-append --cov-report=term-missing
deps =
    pytest
    pytest-cov
depends =
    {py27,py36}: clean
    report: py27,py36

[testenv:report]
deps = coverage
skip_install = true
commands =
    coverage report
    coverage html

[testenv:clean]
deps = coverage
skip_install = true
commands = coverage erase

My question is, if cov-report options are removed, would both approches still work. Are other options as terminal and xml actually used?
May it be possible to delegate the whole reporting to the coverage framework itself?

Could it be an option make pytest-cov just a commad wrapper around coverage and do noting itself anymore?

[nedbat]

Could it be an option make pytest-cov just a commad wrapper around coverage and do noting itself anymore?

@loechel, there are important things pytest-cov does that only it can do: managing the distribution and re-collection of coverage data under xdist; signaling when tests begin and end for who-tests-what; probably a few other things. It should continue to do those things. I don't see the value in pytest-cov being a wrapper around coverage.py's other functionality.

[nedbat]

There were 26 +1's on the top description. I don't know how to see the names in the GitHub UI. I had to use the API to get them: blueyed, davidism, sethmlarson, gvoysey, styvane, boxed, CodeMouse92, yeraydiazdiaz, ionelmc, RazerM, timofurrer, ivoflipse, somacdivad, obestwalter, okken, dcoles, otrenav, Stranger6667, gaborbernat, mblayman, eddieantonio, sivy, merwok, fschulze, jab, gkapfham.

[ionelmc]

Mkay so I'll add my feedback:

• If we do a pytest-cov 3.0 that removes shitton of features then we'd have more things to do: spending time on arguing what to remove, explaining why was it removed and ensuring that removal doesn't make the plugin completely unusable. So the questions is who offers time to fight the bulk of the user complaints.
• It's a good idea architecturally to avoid implementing features that should be, could be or are in coveragepy.
• I think we should start with something less radical: drop compatibility with weird or old combos or dependencies: only latest pytest, only latest coveragepy, only latest python and so on.

I do have the same "things are not optimal" feeling, that's why I +1.

[nedbat]

I hear what you are saying about dealing with the fallout.

I think probably everyone is on-board with dropping support for older everything (though I would like to keep Python 2.7 in the mix for now). Let's assume that is happening no matter what.

Then we have to decide what else to do. Here are some possibilities, a gradient of severity:
A. Nothing more than that. (This is your third bullet.)
B. Keep everything working as-is, but add warnings to features that will be dropped. So if you use --cov-report=html, you will get a warning, and an HTML report.
C. Drop features we don't want any more, but print helpful messages about what happened. If you use --cov-report=html, you get a message saying, "This feature has been removed, use 'coverage html' instead."
D. Drop features completely. If you use --cov-report=html, you get an error message, "--cov-report: unrecognized option"

(Note: in the above, I used --cov-report=html as a proxy for any feature that has been dropped.)

While D would mean the leanest resultant code, I would lean toward option C, to help people transition.

[fschulze]

+1 on C

[offbyone]

I'd personally want the current reporting to be available as an API then; I use pytest to drive coverage extensively for a variety of reasons, and if pytest-cov loses those features I'll need to reimplement them. Wrapping a command line when I'm already in Python is not ideal; I'd like to be able to invoke them as code.

If so, that'd be fine; I'd be happy to have consistent coverage reporting control between test frameworks.

[nedbat]

@offbyone I'm not sure what you mean by an API. Coverage.py has an API. Would that work? Keeping all the code in pytest-cov but exposing it in an API from the plugin only adds to the complexity, it doesn't reduce it.

[meejah]

coverage should inherit pytest-cov's ability to get coverage out of sub-proceses easily. Every time I (attempt) to do subprocess coverage, I re-discover a bunch of gotcha's with atexit, signals, environment and .pth files ;) that mean it doesn't "just work".

[blueyed]

@meejah re subprocesses see: https://pypi.org/project/coverage_enable_subprocess/ - nedbat/coveragepy#367

[nedbat]

@meejah I am in favor of exploring the parts of pytest-cov that could be moved into coverage.py itself. To keep the process a little simpler now, I'm only talking here about dropping code from pytest-cov that is already in coverage.py.

The .pth/subprocess/etc stuff is gross and scary however it is done, but I understand that people find it frustrating.

[rixx]

I'm a frequent user of pytest and pytest-cov. I use the various reporting options (both terminal in various versions, and HTML) through pytest-cov, currently, as a matter of my regular development cycle. Thank you for soliciting opinions and making the discussion so open for everybody.

I, as a user, would be happy enough to deal with option C. I frequently work offline or on very bad internet, so being confronted with an error after an already tedious update woud be unpleasant, whereas having explicit instructions available would be an unexpectedly positive experience. (I assume I'm not speaking for a majority here, but maybe for a subset of devs that tend to be forgotten, occasionally.)

I'd understand if you choose option D instead, but I'd feel less happy about it. If adding instructions for the different flags is hard, then please make that fact very clear in the changelog – it would definitely reduce my annoyance if I ran into it by accident.

[offbyone]

@nedbat yes, probably. I wasn't saying it didn't, only that I need one to replace pytest-cov's reporting for in-process work :)

[ionelmc]

@meejah re subprocesses see: https://pypi.org/project/coverage_enable_subprocess/ - nedbat/coveragepy#367

pytest-cov does much more than that, and much better. It supports all packaging and installation forms (coverage_enable_subprocess does not).

Someone has suggested that there's no need for pytest-cov (ignoring subprocess stuff) since you can just run coverage run -mpytest, however that has some problems:

• slower runs since coverage is active during pytest's initialization (and not really needed there, since you don't care about coverage of pytest as a user of pytest)
• lack of API to control coverage from within hooks (conftest) or tests (https://pytest-cov.readthedocs.io/en/latest/markers-fixtures.html)

[merwok]

FWIW: I always use coverage run form, because I knew it before I started using pytest, I have the (incorrect?) notion that coverage needs to run first to setup its tracing hooks, and I like memorizing a generic way that works with any test runner or other command. I also visit the coverage doc regularly to see settings and news, so I guess in my mind coverage is its own tool, not just a test runner plugin.

I thought pytest-cov was sugar (only a shortcut) until I read #337 (comment), and tox already gives me a shortcut for coverage run + coverage report. Now I see that it’s useful for test suites with subprocesses or pytest-xdist.

Also, I also really enjoy the possibility of running bare pytest without virtualenv or coverage for maximum feedback speed; on client projects configured with pytest-cov it’s a little cumbersome to type pytest --no-cov=project to disable coverage report when you’re in a «many tests broken, fix one at a time» phase.

@rixx If you have pytest-cov installed, you also have coverage, and coverage help gives offline help about commands and options. Does that address your needs?

[rixx]

@rixx If you have pytest-cov installed, you also have coverage, and coverage help gives offline help about commands and options. Does that address your needs?

Not exactly. My use case is that I'm used to running pytest with --cov-report flags of various kinds. If those suddenly just throw errors at me, I won't know where to look. (Please note that I'd understand if that option is the one that wins out, I just think the other one will make a bunch of people pretty happy when upgrading.)

[merwok]

Throwing an idea out there: what if pytest-cov was not changed, and the new ideas implemented in a new plugin with another name? That way no automated upgrade can break command lines, maintainers will have to see warnings or announcements and decide to switch (the docs could help with translation of pytest-cov options to coverage.py commands).

[nedbat]

@merwok I think I like the idea of changing the name. We could make one more release of pytest-cov, where it prints a warning that there will be no more updates to it, and that you should switch to pytest-coverage. Then pytest-coverage can simply remove all the obsolete options.

This has advantages: people have to explicitly switch over, and we don't need an awkward middle phase where the code is still complex, but doesn't do what people want.

[gaborbernat]

I think I like the proposal minus the warning part 👍

[gpfreitas]

I am a user, and the name change @merwok suggested is my favorite option if the UI or functionality changes significantly, for the reasons he and @nedbat explained.

I support any effort to notify the user of the changes and give clear directions about what to do. Seeing a warning about future changes would be a a nice touch and wouldn't bother me.

[hugovk]

For comparison, pep8 was renamed to pycodestyle in June 2016, with a runtime warning added to pep8 in October 2017.

In September 2019, pep8 had nearly a million downloads, pycodestyle had 5.5m.

category	percent	downloads
2.7	46.19%	451,264
3.7	21.71%	212,100
3.6	20.77%	202,906
3.5	8.97%	87,607
3.4	1.07%	10,445
null	0.77%	7,486
3.8	0.25%	2,438
3.3	0.17%	1,695
2.6	0.09%	882
3.2	0.00%	48
3.9	0.00%	2
Total		976,873

Source: pip install -U pypistats && pypistats python_minor pep8 --last-month

category	percent	downloads
3.6	33.61%	1,849,079
3.7	31.72%	1,745,178
2.7	26.43%	1,454,039
3.5	5.94%	326,832
3.4	1.07%	58,727
null	0.81%	44,587
3.8	0.38%	20,789
2.6	0.04%	2,147
3.3	0.01%	288
3.9	0.00%	67
Total		5,501,733

Source: pip install -U pypistats && pypistats python_minor pycodestyle --last-month

Not sure what to conclude exactly! People still use the old one a lot. But renaming would be a clear signal and let you concentrate only on the new project and retire the old.

[ssbarnea]

I don't think that rename is necessary here but I cant help to notice that there is not even one downvote on the original issue. Rarely I see so much agreement. Go and and start removing stuff from it, I am myself busy with similar tasks on other projects. Less is more! ...especially for older projects.

[nramirezuy]

I only use pytest-cov for the ease of use to run coverage and get the reports out, a single command line that gives me everything that I need. I even was here looking for answers on why isn't this process already automatic when you already have a .coveragerc telling coverage what to do.

Having all this removed, why should I keep using pytest-cov or migrate to pytest-coverage?

[graingert]

I disagree. I believe pytest-cov should integrate more closely with coverage and expose more of the new features - for example: #377

[nedbat]

@graingert Can you explain why it should? What is difficult about using coverage reporting commands to do reporting? Why do those operations need tight integration with pytest?

The only reason I've heard is, "I like just having one command to do everything." That command should be "tox" or "make" or "dostuff.sh", not "pytest".

[graingert]

reply from #330

@graingert We should come to an agreement. This issue (#337) had overwhelming support for removing reporting features from pytest-cov. There's no reason for it to implement them. Whatever people are using to run pytest, they can use that thing to also run coverage reporting commands afterward.

This just adds needless complexity and awkard interfaces.

pytest-cov's job should be integrating pytest with coverage where that integration is needed. It should not be a one-stop shop for everything people want to do with coverage.

---

In my view pytest-cov's core feature is reporting coverage immediately after running the tests (potentially failing the suite if the coverage isn't high enough). Without this feature what is left? To me it seems a few workarounds for covering code in pytest-xdist and other subprocesses.

What benefit would pytest-coverage have over coverage run pytest ... && coverage report (html) && coverage report (xml)?

I see the pytest-cov.pth workaround for subprocess as something that could be moved into coveragepy directly or another package coverage-pth?

[nedbat]

@graingert You make a good point: it is easy to misunderstand what pytest-cov does that is essential. In my mind, pytest-cov's core features are those things that can be done best (or only) within a pytest plugin. This may not be a complete list but those features include:

• Coordinating with xdist to transfer coverage data back for combination
• Setting up subprocess measurement
• Switching dynamic contexts for each test run

You say:

What benefit would pytest-coverage have over coverage run pytest ... && coverage report (html) && coverage report (xml)?

This is exactly my point: pytest-cov reports have no benefit over coverage commands. Pytest-cov does nothing different or better when reporting than the coverage commands you can already run. Except the command-line interfaces are tortured (because the switches from "coverage report" have to all be crammed into --cov-report), and it's extra code to maintain and debug.

It's fine for pytest-cov to only do a few small things. That's what plugins should do.

If you want to add features like per-directory fail-under, let's add them to coverage. It's got nothing to do with pytest, and could benefit people using other test runners.

[ofek]

Any update on this?

[ssbarnea]

I think that @ionelmc is the one that can make that decision. I personally started to remove pytest-cov from most of my projects and replace it with direct use of coverage-py, as I hit some several bugs/limitations.

Considering the very limited maintenance time for the plugin, if pytest-cov wants to survive, it should redefine its scope as far more limited than before.

Even so, in my case it was impossible to correctly record the coverage with pytest-cov because I has a pytest plugin inside my own codebase, and the imports was happening before even pytest-cov was fully initialize. I tried all the hacks that @ionelmc suggested but still failed. In the end I was able to use coveragepy while also enabling both subprocess and xdist support as I needed both, and apparently it did work well.

I still wonder what could pytest-cov provide me that I cannot get from coveragepy? Or at least can it do to simplify the entire coverage configuration?

Now, there is still one issue that I want to resolve on coverage part, navigation of the coverage reports online. At this moment I am using codecov but I am unpleased about their ability to fix reported bugs. For example they are not able recognize pragma: no branch or pragma: no cover, or to recognize setting from pyproject.toml. Looking at how they thickets are handled there is very little hope that the situation will improve. The big dilemma is that I am not aware of any similar web based, free for open-source sass service that is better or more reliable and i have no plans to host the reports myself. If anyone know such a service please let me know.

I also want to thank to all those involved in improving the coverage experience with Python!

[ofek]

I was able to use coveragepy while also enabling both subprocess and xdist support

Link? 🙂

[ssbarnea]

https://github.com/ansible/ansible-lint/blob/main/pyproject.toml#L13-L24 also check tox.ini for other settings related to coverage. Nothing in the code itself.

[ionelmc]

Even so, in my case it was impossible to correctly record the coverage with pytest-cov because I has a pytest plugin inside my own codebase, and the imports was happening before even pytest-cov was fully initialize. I tried all the hacks that @ionelmc suggested but still failed. In the end I was able to use coveragepy while also enabling both subprocess and xdist support as I needed both, and apparently it did work well.

If there's ever going to be a clean new pytest-cov without all the old cruft (that apparently can be easily replaced with coverage run -mpytest) that has to be solved. I guess this project needs to stop being a pytest plugin and be more of a pytest shim. Basically a coverage run-pytest-and-report-and-fail-if-cover command. Some cli sugar basically. What do you think of this idea?

[RonnyPfannschmidt]

A basic integration that will help with coverage requirements and early initialization might be really nice, i believe too much that's better served by coverage these days was added in the beginning

I vaguely recall that when Pytest-cov started coverage was not nearly as nice and feature packed

The fact that these days gutting Pytest-cov to a small shim seems like a good idea is a testament to the hard and tireless work @nedbat and others put into coverage

[ssbarnea]

There is also another option, but I am not sure what @nedbat thinks about it. What it coveragepy would also expose a coverage pytest plugin, one that would be used to add the very small useful integration with pytest. The only thing where an integration with pytest could help is related to discovery.

If that is true, we could end-up retiring pytest-cov.

I would love to see a way to simplify coverage setup on python projects, especially for pytest+tox+gha+codecov users, as I seen myself forced to add several hacks in order to make it work fine, most of them being not really obvious when you look at the config. If the others are interested about this subject, I could probably spend some time to document them, so we can review them and decide what can we do to minimize the amount of config someone would have to do do, to have a just-works approach.

In fact I was wondering if a coverage plugin for tox would not be more suitable, because tox is already the wrapped the testing commands, so clearly in a better suited situation for inject coverage call. On this path, my only dilemma is that I do not know how to decide which tox commands should be run with coverage or not as from time to time I run multiple commands and some of them do not need to be covered. Maybe that bit should remain manual.

[nedbat]

Thanks for picking up the conversation. I am still interested in seeing this move forward.

Ionel said:

Basically a coverage run-pytest-and-report-and-fail-if-cover command. Some cli sugar basically. What do you think of this idea?

I don't understand why a pytest plugin would do anything beyond what is needed to run the tests. Why should it do any coverage reporting? All of that can be handled by a coverage command after the pytest command. The coverage command can also handle the --fail-under=90 logic as well. Pytest should run tests, and fail if the tests fail. That's it.

I think the urge to do more comes down to this idea that "people just want to type pytest", and we should resist that. There are lots of tools for combining commands into one command. Pytest plugins don't have to do that.

About including a plugin with coverage.py: I'm open to exploring that idea. Would people be willing to help me build and maintain it?

[gaborbernat]

I don't understand why a pytest plugin would do anything beyond what is needed to run the tests. Why should it do any coverage reporting?

I want to point out that python has a startup cost; if pytest-cov does all in the same process, I get around 0.5s cost save as opposed to splitting up reporting separately. So from a developer experience point of view, a single tool gives a more fluent experience. This is one of the primary reasons I prefer using pytest-cov over coverage invocations. Plus, it's also less output in my tox runs because instead of displaying 5 coverage invocations, now I have a single command displayed with some extra args.