-
-
Notifications
You must be signed in to change notification settings - Fork 416
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fail coverage based on absolute number of uncovered lines #815
Comments
I understand your logic, but this is starting to get into very specific policies. Maybe the best way to support you use is to give you an easy way to get all the statistics in a machine-readable form? For example, #720 suggests a "json report" would would let you get at all the details. |
I'd also find both IMO "the amount of uncovered code should be at most X" is actually less specific than "the percentage of uncovered code should be at most X", because it doesn't have to account for the effect of covered code. In the end it's your call though, and it would be possible to build this from a nice machine-readable format. I'd also be happy to implement these options in a PR, if you'd accept them! |
Maybe this is something to add to a separate coverage-goals checker? https://nedbatchelder.com/blog/202111/coverage_goals.html |
Is your feature request related to a problem? Please describe.
When improving the test coverage of a legacy code base, I would like to have a ratchet based on the absolute number of uncovered lines.
There are two aspects to this:
I want test coverage because untested code is very likely to be buggy. The number of bugs is not a function of the percentage of uncovered code, but rather the amount of uncovered code. Saying I've got 98% coverage might impress my friends, but if that means I've got 20kLoC of untested code then it's somewhat meaningless.
More seriously, deleting tested code reduces the test coverage percentage even though coverage is not objectively worse. Working around this takes all sorts of hijinx that are unnecessary when using absolute numbers.
This means I would like
coverage
to fail not only if the coverage has got worse but also if it has got better. Failing if it's worse is obvious—I don't want coverage to get worse. Failing if it's better means that contributors will have to update the number of uncovered lines in the build scripts to lock in the newer, lower number.Neither of these are a problem with greenfields projects, because then I start with
--fail-under=100
immediately, which means I don't need a ratchet, and I have an absolute number of uncovered lines: zero.Describe the solution you'd like
A new option to
coverage report
called--num-uncovered-lines=N
that would cause it to exit with status 2 if there were more than N uncovered lines, and exit with status 3 if there were fewer than N uncovered lines.The error message should clearly indicate whether coverage got worse or better. In the case where it's better, it should emit the new number of uncovered lines so it is easy to update build machinery.
Describe alternatives you've considered
If the ratchet aspect is distasteful, having
--max-uncovered-lines=N
would also be very useful.Additional context
Here's the script that I'm using for this right now: https://gist.github.com/jml/27d21c9322d112a3b9b5e94508c93928
I run it in CI with something like:
The text was updated successfully, but these errors were encountered: