Bug: Inconsistent code coverage between builds and jobs (parallel mode)

[yann-combarnous]

We are running our tests in parallel, which each test group testing specific files.

We use coveralls to consolidate these tests. However, we have seen inconsistent reporting between the parallel jobs code coverage and the consolidate code coverage

For instance, we have a file that is reporting a a test group 74.29% coverage (/src/components/PolicyForm/PolicyForm.tsx). But this same file is reporting no data when looking at the consolidated report from parallel tests.

Coverage report from test group:
https://coveralls.io/jobs/97559458/source_files/19538899624

Coverage report from consolidated parallel test jobs:
https://coveralls.io/builds/48116126/source?filename=src%2Fcomponents%2FPolicyForm%2FPolicyForm.tsx

[afinetooth]

Hi, @yann-combarnous.

I think there's an issue with your build that's illustrated by the fact that you're seeing two (2) copies of the same file in your consolidated view (or "build" view):
https://coveralls.io/builds/48116126

Below, I'm on that build view (your build page), looking at the LIST tab of your SOURCE FILES table, searching for the filename, PolicyForm.tsx.

You can see that two files are returned with the same path:

There should only ever be one (1) file at a given path.

FWIW, you can also see this in the TREE view:

This made me think there was a failure somewhere while processing your jobs, and perhaps while performing your aggregate coverage calculation. So I re-processed your jobs, and also re-triggered your aggregate coverage calculation, but I'm still seeing the same thing.

I will have to submit this to our engineers and see what else we can determine as a potential root cause and fix.

In the meantime, I would like to know if you see this in any of your other builds (or if it's limited to this one). Also, please let me know if it happens in your next build, or any future builds.

I'll get back with updates.

[jjochen]

Any news on this @afinetooth?

I think we are seeing the same issue.
The combined coverage is wrong most of the times for pull requests, but it seems to work fine for the main branch.

Here the coverage of the individual jobs decreased or stayed the same, but the combined coverage increased by a lot.
Same here and here.

[afinetooth]

Hi @jjochen. I think you're issue would be different from this one, mainly because you're not seeing duplicate source files in your SOURCE FILES table.

That said, it is possible your PRs aren't completing with correct coverage, which is something we've been seeing with some repos. We have not identified the root cause yet, but I can verify whether re-running the jobs in your build results in a different coverage %, in a moment.

First, just a look at your for build, a PR build (RIGHT), alongside its base build (LEFT):

The key details in terms of your final, aggregate coverage calculation can be found in RUN DETAILS. And as you can see here, while the coverage % are not far off, the number of relevant lines covered are significantly different between the two builds. The difference seems unlikely with only two files modified, according to the DIFFS section ("NEW MISSED LINES IN DIFF").

I re-ran the jobs in your build to see if anything changed and...

It did change:
https://coveralls.io/builds/49630352

(Again, this is the build in the RIGHT, above.)

Note that the RUN DETAILS look much more similar to the base build's.

So, while I can confirm you're having the issue that some other repos are having, where PR builds are completing with incorrect aggregate coverage calculations, your problem doesn't seem related to the specific one in this issue. Also, unfortunately, we don't have a fix ATM.

I have added your case to the open issue card in our backlog, so if something changes I will update you here, or, better, would be to email us at support@coveralls.io, mentioning this issue, so we can keep you updated directly.

[afinetooth]

@yann-combarnous @jjochen

Workaround for this issue: the Rerun Build Webhook

While we've not yet identified a fix for this issue, we released a workaround today that should resolve it for you: the Rerun Build Webhook.

Since the nature of the issue appears to be that, for some repos with parallel builds:

1. The coverage % of pull_request builds is calculated incorrectly (on the first build), but
2. Re-running the build (recalculating coverage) corrects the coverage %.

A Rerun Build Webhook, similar to the (Close) Parallel Build Webhook, fixes the issue by triggering your build to re-calculate itself.

Instructions

Call this at the end of your CI config, after calling the (Close) Parallel Build Webhook.

Call it like this:

curl --location --request GET 'https://coveralls.io/rerun_build?repo_token=<YOUR REPO TOKEN>&build_num=<YOUR BUILD NUMBER>'

But substitute your repo_token, and your build_num (the same value you used for build_num in your (Close) Parallel Build Webhook).

Please note a few differences between the Rerun Build Webhook and the (Close) Parallel Build Webhook:

1. The /rerun_build endpoint will accept a GET or a POST, and
2. The only two required URL params are your repo_token and the build_num, and build_num is a regular URL param and not part of a JSON body called "payload" as required by the (Close) Parallel Build Webhook:

curl -k https://coveralls.io/webhook?repo_token=<YOUR REPO_TOKEN> -d "payload[build_num]=<YOUR BUILD NUMBER>&payload[status]=done"

[afinetooth]

NOTE: In case you're having trouble determining what build_num is for your project, I posted some follow-up here.

If you're using a different Coveralls integration and/or are still having trouble determining the correct values for either build_num or repo_token let me know here, or in the context of your issue, or at support@coveralls.io.