coveralls --finish not working as expected in Circle-CI

[stephenroller]

Thanks for this nice client.

It looks like since the github api was introduced, the Circle-CI version seems broken. When I try putting having a run that finishes up a parallel go, the wrong build number is being used to notify the service

(venv) circleci@b3cd35dd8048:~/project$ coveralls debug -v --finish
Missing .coveralls.yml file. Using only env variables.
Testing coveralls-python...
{"source_files": [], "git": {"branch": "coveralls", "head": {"id": "fad6a2cc6ab84ef45d06bf2150a4d103c16e1a76", "author_name": "Stephen Roller", "author_email": "roller@fb.com", "committer_name": "Stephen Roller", "committer_email": "roller@fb.com", "message": "Hm."}, "remotes": [{"name": "origin", "url": "git@github.
com:facebookresearch/ParlAI.git"}]}, "config_file": ".coveragerc", "parallel": true, "repo_token": "[secure]", "service_name": "circle-ci", "service_job_id": "48529", "service_pull_request": "2950"}
==
Reporting 0 files
==

Here you see that the CIRCLE_BUILD_NUM value has been put into "service_job_id" field of the config. In

However, the call to finish uses the "service_number" field instead:

https://github.com/coveralls-clients/coveralls-python/blob/30e4815169b3db2616981939d55d2f4495816821/coveralls/api.py#L217-L218

As such, when you try calling coveralls --finish you get the following error:

$ coveralls -v --finish
Missing .coveralls.yml file. Using only env variables.
Finishing parallel jobs...
Parallel finish failed: No build matching CI build number  found
Traceback (most recent call last):
  File "/home/circleci/venv/lib/python3.7/site-packages/coveralls/cli.py", line 80, in main
    coverallz.parallel_finish()
  File "/home/circleci/venv/lib/python3.7/site-packages/coveralls/api.py", line 236, in parallel_finish
    raise CoverallsException('Parallel finish failed: {}'.format(e))
coveralls.exception.CoverallsException: Parallel finish failed: No build matching CI build number  found

Note that Circle always puts None into the service_number field:

https://github.com/coveralls-clients/coveralls-python/blob/30e4815169b3db2616981939d55d2f4495816821/coveralls/api.py#L87

https://github.com/coveralls-clients/coveralls-python/blob/30e4815169b3db2616981939d55d2f4495816821/coveralls/api.py#L52-L59

[stephenroller]

My current work around is to just directly call this, rather than using coveralls --finish:

curl -k https://coveralls.io/webhook?repo_token=$COVERALLS_REPO_TOKEN -d "payload[build_num]=$CIRCLE_BUILD_NUM&payload[status]=done"

[stephenroller]

Hmm, it seems that using CIRCLE_BUILD_NUM is not right :(

[TheKevJames]

Shoot, happy to update this if we can figure out what the correct value is. By any chance have you figured out what value we need to be using, if CIRCLE_BUILD_NUM is not correct?

According to @caleb15's PR here it might be CIRCLE_WORKFLOW_ID -- could you please try to see if that works for you @stephenroller?

[stephenroller]

Sure, here's my WIP attempting to use it facebookresearch/ParlAI#2950

Indeed, CIRCLE_WORKFLOW_ID mostly works well. I think the one problem I have is that our build system is rather large: something like 92 builds for every commit. (We heavily parallelize). Coveralls seemed to be choking on it (only realizing 62 of them: https://coveralls.io/builds/32619558) and vastly underestimating our coverage:

codecov.io puts us at ~38% right now. This coveralls build think we're at ~11%. Browsing the file tree, large parts that I know are covered aren't currently being caught.

[TheKevJames]

Not sure if I'm just missing it in your CircleCI config, but it seems you might need to set COVERALLS_PARALLEL=true in your environment? See our docs:

If you are running multiple jobs in parallel and want coveralls.io to merge those results, you should set COVERALLS_PARALLEL to true in your environment:

COVERALLS_PARALLEL=true coveralls

Later on, you can use coveralls --finish to let the Coveralls service know you have completed all your parallel runs:

coveralls --finish

Does everything work out when you use both COVERALLS_PARALLEL and the CIRCLE_WORKFLOW_ID hack?

[caleb15]

Indeed, CIRCLE_WORKFLOW_ID mostly works well. I think the one problem I have is that our build system is rather large: something like 92 builds for every commit. (We heavily parallelize). Coveralls seemed to be choking on it (only realizing 62 of them: https://coveralls.io/builds/32619558) and vastly underestimating our coverage:

Coveralls has been having peformance issues recently so that might be related: https://status.coveralls.io/

[stephenroller]

I did set the COVERALL_PARALLELS environmental variable to true, but I did it in the CircleCI environment rather than in the script.

Anyway, the parallelism mostly worked, but it did seem to not like me sending 100 builds to a single branch. Since then, I haven't had time to follow up at all.

[stephenroller]

I feel somewhat confident at this point that using CIRCLE_WORKFLOW_ID is the right thing, and that there is another bug further down the line I haven't caught yet.

[TheKevJames]

Alrighty -- I'll go merge and deploy #233 then, which should let you nuke the CIRCLE_WORKFLOW_ID hack. I'll leave this issue open to help track whatever other issue is going on here.

[stephenroller]

Alright, I'm coming back to this now, thanks for helping before.

I feel like I'm very close now. You can see a report here: https://coveralls.io/builds/33679104

After #233, one discussion above is fixed, but I still had to make another change for our workflow (which consists of many different jobs, each run in parallel; see here). Before calling coveralls I have:

export CIRCLE_BUILD_NUM=$CIRCLE_BUILD_NUM-$CIRCLE_NODE_INDEX

This avoids the different parallel workers stomping on each other, I believe.

When I do my final call to the webhook, it seems to report the correct number

curl -k https://coveralls.io/webhook?repo_token=$COVERALLS_REPO_TOKEN -d "payload[build_num]=$CIRCLE_WORKFLOW_ID&payload[status]=done"
{"done":true,"url":"https://coveralls.io/builds/33679104","jobs":49}

I see all the jobs on the corresponding coveralls page. The only frustrating thing is the final build coverage says 21.99, when it should be closer to 42-45. When I look at the tree at the bottom, that all looks correct (the two main folders have 39.9 and 44.0 coverage), just the final total looks wrong. Comparing the tree with what codecov outputs, the per-folder totals all look right.

[TheKevJames]

I see all the jobs on the corresponding coveralls page. The only frustrating thing is the final build coverage says 21.99, when it should be closer to 42-45. When I look at the tree at the bottom, that all looks correct (the two main folders have 39.9 and 44.0 coverage), just the final total looks wrong. Comparing the tree with what codecov outputs, the per-folder totals all look right.

I'm pretty lost on this one -- as far as I can tell, it seems like coveralls.io is getting all the right data but reporting the aggregate stats incorrectly? Unfortunately, I don't think there's anything I'll be able to help with, then; data presentation is strictly out of our hands haha! The only thing which comes to mind is the settings for what sort of coverage you're reporting (eg. line coverage vs branch coverage, that sort of thing). Maybe playing with those settings might help?

Closing this issue -- sorry we couldn't help more!

[stephenroller]

Thank you!