GitHub Actions: Unavailable with parallel jobs. Ones except last disappears.

[ghost]

Problem

• Sending coverage reports through actions/coveralls-github-action (that's based on nickmerwin/node-coveralls) (that's based on Official API)
• With parallel: true
• Get response with url, that is available first, but disappears after next parallel response.

Logs

Here is full log. (If expired, please let me know to re-run.)

Request is like below.

1st send

{
  run_at: '2020-02-25T04:28:39.319Z',
  service_name: 'github',
  service_job_id: '85458e6769d7f413fe2a77a3946e365a95d567f0',
  parallel: true,
  repo_token: '***',
  git: {
    head: {
      id: '85458e6769d7f413fe2a77a3946e365a95d567f0',
      committer_name: 'Luma',
      committer_email: 'tomorinao.info@gmail.com',
      message: 'test',
      author_name: 'Luma',
      author_email: 'tomorinao.info@gmail.com'
    },
    branch: 'refs/heads/coverage-playground',
    remotes: [ [Object] ]
  }
}

2nd send

{
  run_at: '2020-02-25T04:29:24.983Z',
  service_name: 'github',
  service_job_id: '85458e6769d7f413fe2a77a3946e365a95d567f0',
  parallel: true,
  repo_token: '***',
  git: {
    head: {
      id: '85458e6769d7f413fe2a77a3946e365a95d567f0',
      committer_name: 'Luma',
      committer_email: 'tomorinao.info@gmail.com',
      message: 'test',
      author_name: 'Luma',
      author_email: 'tomorinao.info@gmail.com'
    },
    branch: 'refs/heads/coverage-playground',
    remotes: [ [Object] ]
  }
}

Will get like....

{"message":"Job #85458e6769d7f413fe2a77a3946e365a95d567f0.1","url":"https://coveralls.io/jobs/59357703"}

{"message":"Job #85458e6769d7f413fe2a77a3946e365a95d567f0.2","url":"https://coveralls.io/jobs/59357728"}

---

Before the 2nd send, 1st responded url (https://coveralls.io/jobs/59357703) is available.
After the 2nd response, 1st responded url (https://coveralls.io/jobs/59357703) is unavailable.
This time, and also now, 2nd responded url (https://coveralls.io/jobs/59357728) is available.

This feels obviously strange.

My attempts

• Trigger webhook one by one.
• Trigger webhook after all parallel jobs.
• Not to trigger webhook.
• Not to set parallel: true
• Use numerical id for service_job_id

---

Possibly duplicate of #1399 , but I hope logs in this issue may be helpful.

Thanks.

[Charles-Gagnon]

We seem to be hitting the same (or a similar issue) using the Coveralls Github action. We have Azure Pipelines CI setup to upload multiple files in parallel and then end with calling the action with parallel-finished: true

https://github.com/microsoft/azuredatastudio/blob/71e72de792a67d4b70a653219c2df49281817560/.github/workflows/ci.yml#L55

This works fine for PRs : https://coveralls.io/builds/29375466

But for pushes it only seems to pick up the last file :

https://coveralls.io/builds/29347724

[Charles-Gagnon]

A bit hacky - but I ended up just manually merging the lcov files after the tests were all done running and then uploading that single file. That worked fine for our use case.

https://github.com/microsoft/azuredatastudio/pull/9635/files

[jrfnl]

Might just be me, but this sounds like a case of a missing flag-name - which is a required field for parallel builds.
See: https://github.com/marketplace/actions/coveralls-github-action#inputs

[afinetooth]

@Charles-Gagnon I agree with @jrfnl in that if you haven't supplied a unique value for flag-name for each of your parallel jobs, each successive job will overwrite the previous one.

See the Complete Parallel Job usage example in the Coveralls Github Action README.

For a working example project, see this demo project. It demonstrates configuring a parallel build for several different CI services, including Github Actions. The relevant Github Actions workflow file is here.

[vtjnash]

I am seeing possibly a similar issue? Even with parallel: true and setting flag-name as per the Github Action README, I am seeing only the last build on branch builds for master in coveralls (e.g. https://coveralls.io/builds/48419874) while PRs seem to work okay (showing all builds). This leads to significant coverage discrepancies between master and PRs against master, as seen at #1514 (comment) for example, which is a PR to run exactly the same code as master across the same set of machine configurations, but computes a 10.6% difference in coverage.

[jrfnl]

@vtjnash Sounds like you may be experiencing a similar issue as we're discussing in #1631 ?

[afinetooth]

Hi, @vtjnash. That is possible, but it's a slightly different context since in #1631 all the parallel jobs were coming in and being processed. Unfortunately, I can't test your case by re-running the jobs because we don't have all the ones you sent to the Coveralls API stored in the database. (I did re-run the one that was stored, just for good measure, and it doesn't report any changes in coverage % so no problems in processing there.)

What's happening, usually, when a repo with parallel builds only reports one job (especially when that job has the final, expected index number—in your case, some_build_number.7), is that each subsequent job the Coveralls API receives after the first, replaces the previous one. And that's usually because the incoming jobs share some identical data.

In your case, I believe this originates with the value you're setting for flag-name in your GHA config YAML, specifically flag-name, here:
https://github.com/vtjnash/Pidfile.jl/blob/master/.github/workflows/CI.yml#L52

There, you are naming each flag with an appended value, test_number (per the Coveralls Github Action parallel job example, I presume):

flag-name: run-${{ matrix.test_number }}

However, you don't declare test_number prior in your matrix-related config, so it looks like Github Actions is just substituting env.GITHUB_RUN_ID:

Further up in your workflow file, it looks like the unique value you've declared is version:
https://github.com/vtjnash/Pidfile.jl/blob/master/.github/workflows/CI.yml#L17

      matrix:
        version:
          - '1.0'
          - '1.6'
          - '1.7'
          - 'nightly'

So maybe try that instead of test_number where you're declaring flag-name:
https://github.com/vtjnash/Pidfile.jl/blob/master/.github/workflows/CI.yml#L52

In other words:

flag-name: run-${{ matrix.version }}

Or, perhaps you'll need to include more values to fully represent each job in your matrix build uniquely, like os, arch or include (seems like include is the only one that has multiple values).

In any case, try modifying flag-name there to be unique for each job and see if that fixes it.

It is indeed strange that this is only affecting your push builds, and not your pull_request builds. I would like to see if the above works before we dig deeper on that. The good news is that, once the above is done, that difference should provide our next clue.

[vtjnash]

Thanks! I have now made them unique and made a PR to the documentation to make that step clearer coverallsapp/github-action#119, so we will see how that goes with future builds.

[afinetooth]

@vtjnash, great, thank you.

Your last PR looks good:

So I'm hopeful you're all set.

Thanks for helping others who may face this in the future.

[vtjnash]

I reran a new PR build, and the comparison is still looking very confused. The PR status reports that coverage increased by over 10%, despite the individual reports being the same: https://coveralls.io/builds/48497048

[afinetooth]

I think the issue there is that the base build (LEFT) is 10% under the normal %. Making the the PR build (RIGHT) correct relative to the base build (+10%):

Assuming the base build was wrong, I re-ran the jobs we have for it and the coverage changed to 93% (LEFT). Then, to accommodate the change of comparison, I re-ran the PR build as well (RIGHT), which didn't change its aggregate %, but it now reflects the correct comparison (no change):

I don't know what the issue with the base build could have been in real-time, but the calculation reflects the data now.

Please let me know if it happens again.