(Internal) Coverage changes based on how the commit is pushed

[mitchell-codecov]

codecov-action/src/buildExec.ts

Lines 132 to 139 in e3c5604

	if (overrideCommit) {
	execArgs.push('-C', `${overrideCommit}`);
	} else if (
	`${context.eventName}` == 'pull_request' ||
	`${context.eventName}` == 'pull_request_target'
	) {
	execArgs.push('-C', `${context.payload.pull_request.head.sha}`);
	}

Coverage changed in #690 for seemingly no reason. The reports say otherwise:

• https://codecov.io/api/gh/codecov/codecov-action/download/build?path=v4/raw/2022-04-05/EFB5334892C4A701EAE431729013DA8F/e3c560433a6cc60aec8812599b7844a7b4fa0d71/59998752-6b3b-4c2d-a0b6-65c849e92c19.txt
• https://codecov.io/api/gh/codecov/codecov-action/download/build?path=v4/raw/2022-04-05/EFB5334892C4A701EAE431729013DA8F/2c83f35c201827cc1b2a250b2546ca9378d632f9/15f439bd-41a5-4a48-a47f-e4d3193e932a.txt

diff --git a/1.json b/2.json
index 9c0d047..ef411ff 100644
--- a/1.json
+++ b/2.json
@@ -2157,7 +2157,7 @@
             "81": 2,
             "82": 1,
             "83": 1,
-            "84": 0,
+            "84": 1,
             "85": 2,
             "86": 1,
             "87": 1,
@@ -2198,7 +2198,7 @@
             ],
             "1": [
                 2,
-                2
+                0
             ],
             "2": [
                 2
@@ -2256,11 +2256,11 @@
                 1
             ],
             "20": [
-                0
+                1
             ],
             "21": [
                 1,
-                1
+                0
             ],
             "22": [
                 1,

[jbergler]

I think we're running into this problem too.

When github actions triggers on pull_request rather than push github sets the ref to refs/remotes/pull/:id/merge which is the test merge of the PR.

In our case when code changed between the PR's merge base and the current head of the target branch, codecov doesn't seem to figure out that the coverage data being submitted wasn't actually for the HEAD of the PR branch, and creates really funky results.

I've tried overriding the commit when calling the action with github.sha which points to the code being checked out, but this doesn't seem to have any effect.

      - uses: codecov/codecov-action@v3
        with:
          token: "${{ secrets.CODECOV_TOKEN }}"
          override_commit: "${{ github.sha }}"

For us, in a busy repo this makes developers mistrust the codecov reports because they see things that are wrong and shouldn't be possible, eg: