Configure GitHub to require codecov checks

[pquentin]

Since the codecov reports are noisy, we frequently merge pull requests that lower coverage or even completely break coverage upload.

Indeed, we're trained to look at status checks, but not at the coverage report. But codecov is able to report its results with a status check too:

But how can we enable that? I think the trick is to add a branch protection rule in Settings > Branches > Add rule and require the travis and appveyor statuses.

Here's an example from pquentin/urllib3:

For python-trio/trio and python-trio/urllib3, it looks like setting this was enough to see codecov checks. We can try to enforce the codecov checks too, and decide to sometimes ignore them when merging. Trio does not do that because they're actually quite noisy.

[Zac-HD]

It looks like you're really close to full coverage - have you considered failing the build if it's less than 100%? We've found knowing there are zero lines not covered really helpful for Hypothesis. (and judicious use of # pragma: no cover doesn't mess with this; not least because it's greppable)

[pquentin]

@Zac-HD We had 100% coverage for some time now, and we'll actually be back at 100% coverage with #1726. The problem we face is that we have OS-specific code, so there's no one build at 100% coverage, it's only the combination that gets us to 100%. Which is why we have to rely on codecov.

[Zac-HD]

Right! To be explicit: I think using a status check is a great idea, and the only one you need is "does the project have 100% coverage" (of non-pragma'd branches) 😄

[pquentin]

Ah, I see! You're not suggesting failing individual jobs using say a pytest plugin, you're suggesting that we require the codecov GitHub status check before allowing merging pull requests?

Well, the problem here is that codecov is not always reliable, so requiring this check can sometimes block pull requests for bad reasons. But we could try and see how much this is a problem in practice. Sometimes it's enough to just close/reopen the pull request to fix this kind of issues.

[pquentin]

Works as expected in #1806, yay! Thanks @shazow for setting this up. (It turns out @shazow does a lot of thankless but yet invaluable work behind the scenes.)

Forgetting about coverage will no longer be an option. :) We may need to relax some requirements if codecov fails at some point, but let's wait until this actually happens, as I believe codecov is more reliable these days.

[shazow]

Very welcome, happy to help. ❤️ 😄