Codecov failures on untracked code / non-deterministic tests

[oliverbarnes]

Several PRs have CIs failing on Codecov coverage, even though the files changed aren't even tracked and all the other tests pass. Coverage changes seem to be reported on files completely unrelated to the PR.

Here's an example of a PR changing docker-compose.yml, which shouldn't affect any testing at all as far as I know, and the Codecov report on their site showing coverage changed on files within three different modules:

https://github.com/decidim/decidim/pull/7082/files
https://github.com/decidim/decidim/pull/7082/checks?check_run_id=1652992360
https://codecov.io/gh/decidim/decidim/pull/7082/changes

Googling I found thread on codecov.io's forum about non-deterministic tests throwing it off:

https://community.codecov.io/t/reported-change-in-coverage-when-no-code-changes-made/1346/7

I don't see a simple fix, and this seems to ask for a bigger discussion on testing practices, hence my posting this here rather than an issue.

I've also found a related issue (#3060) and PR (#3095) moving test coverage tracking to Code Climate:

But it seems like we went back to Codecov. I haven't dug enough to figure out what happened since then, and I was hoping maybe @mrcasals or @oriolgual could give some context about it?

Update

#7088 increases the coverage change threshold for failure to 0.1%, to bypass inconsistencies while a proper fix isn't available.

[mrcasals]

What we found a while ago is that the coverage upload fails for some reason even though the job is marked as green.

[oliverbarnes]

Thanks @mrcasals, let me check if I understood correctly: the coverage upload on Codeclimate failed, and so it was dropped in favor of Codecov again?

[mrcasals]

No, sorry. We used to have the CI on CircleCI, which gives 4 parallel jobs for free, and used CodeClimate as a coverage reporter. Some companies (mainly Codegram at the time) paid for more concurrency, but it wasn't enough. When GitHub Actions appeared, it offered 20 jobs in parallel for free for OSS repos. We moved from CircleCI to GitHub Actions.

For how those CI systems work, we were able to use CodeClimate on CircleCI but not on GitHub Actions, so we looked for an alternative to CodeClimate because it seemed less critical than having a faster CI suite. We found CodeCov, which worked great with the parallelization system we currently use on GitHub Actions, but the uploader sometimes seems to fail for unknown reasons, even though all the tests passed.

I still think I'd rather have the current amount of parallelization, so CodeCov sounds like a good compromise to me. Also, the reality is that we have more than 20 parallel jobs. I suspect GitHub Actions has a pool of runners and it ensures at least 20 jobs/repo, but if more are available then they use it. Since we're in a different timezone than the US, then we get more available runners so we have a de facto total parallelization.

[mrcasals]

Apart from that, we used to suspect simplecov (the gem we use to generate the coverage reports and then upload them to whatever service) had some troubles identifying coverage for some files. Eg, this PR:

simplecov-ruby/simplecov#885
#6240

The issue seemed to be fixed and I haven't detected any other similar issue, I'm just mentioning this for context

[oliverbarnes]

This is excellent context, thanks. The problem is making more sense now

[oliverbarnes]

@mrcasals this indeed doesn't sound like an immediately solvable issue, and not super high priority for investing more time into it :) would it make sense to increase the change threshold a little bit, so these inconsistencies don't break the CI? So far I've been seeing changes of 0.05%, 0.07% triggering the failure.

[mrcasals]

We could explore that, yeah!

[mrcasals]

Related PR by @oliverbarnes #7088