[FEATURE] configurable coverage levels

[esatterwhite]

Is there an existing issue for this?

• I have searched the existing issues

Have you read the CONTRIBUTING guide on posting issues, and CODE_OF_CONDUCT?

• yes I read the things

Description

In previous versions of tap, it was possible to specify a minimum level of coverage

lines: 95
functions: 95
statements: 95
branches: 95

These options have all been removed, and it doesn't see like there is a way to replicate this behavior. There is allow-incomplete-coverage, but as far as I can tell, this will allow 0% coverage - there is no in between.

I see this in the docs:

A module with 99% code coverage can be considered as two modules; a large one that is tested, and a smaller one with no tests at all. If code is worth testing, it's worth testing completely. For that reason, the tap runner instruments code using the built in V8 coverage API, and considers incomplete coverage to be a test failure. If code coverage is complete, no coverage report is generated. If it is incomplete, or if no coverage is generated at all, then a report is printed and the process exits with an error status code.

I understand the intention around this, and I generally agree with the sentiment, but this isn't really practical. I would like to be able to enforce a minimum standard of quality - and enforcing that through code coverage in automated CI is a great place to start. Fail a build if coverage drops below a certain level, But the all or nothing approach makes that a high barrier to entry.

For example, inheriting large codebases with no tests. Going from 0 to 100% coverage in a reasonable amount of time isn't practical for many companies. But getting to a reasonable level of test coverage, and enforcing a new minimum, while working to improve that minimum incrementally over time is.

Example

lines: 95
functions: 95
statements: 95
branches: 95

[isaacs]

I wonder if there isn't a better feature hiding here for that use case. One thing I never liked about the levels configs is just how arbitrary and manual they feel.

Really, it seems like, if you're not at full coverage (or even if you are, I guess) then thing you want is to never reduce coverage.

What if, instead of setting levels explicitly, there was an option to track the current coverage level on each run, and fail if it ever goes below that limit?

So you'd run tap --increase-coverage, which would imply --allow-incomplete-coverage and write the current levels to the config if they're not there. If they are there, and the current levels are below, fail. If they're at or above, pass, and write the new values back.

[esatterwhite]

So you'd run tap --increase-coverage, which would imply --allow-incomplete-coverage and write the current levels to the config if they're not there. If they are there, and the current levels are below, fail. If they're at or above, pass, and write the new values back.

Ya, I could get behind this. I think this is usually what people want when trying to enforce coverage levels. It goes up, not down. And this would remove the manual maintenance and human error from trying to keep up with it.

[esatterwhite]

I'm not too familiar with how tap manages configuration, but if those coverage values are stored in the existing tap config - that could be .taprc, package.json or one or more shared config packages. Which one gets updated?

If the intent is for this to not be manage by a human, maybe a separate top level file?

[isaacs]

It'd be the same logic as using tap config set key=value, ie, update .taprc if found, otherwise package.json if found, otherwise create a .taprc.