E2e tests improvements

[laurentsimon]

• Re-use existing issue instead of creating new one (Disable gostaging until this is done: we're inundated with issues otherwise)
• Verify push/dispatch_workflow work properly
• Auto-close issues when e2e tests pass
• Fix the content which contains \n instead of displaying a new line
• Links to workflow files are broken
• Provide the token type in the issue description (PAT or GITHUB_TOKEN)

[laurentsimon]

/cc @naveensrinivasan @azeemsgoogle

[laurentsimon]

I looked at the e2e2 workflows (only the non-gostaging ones). Findings:

1. https://github.com/ossf-tests/scorecard-action/blob/main/.github/workflows/Scorecards-with-default-GH-Token-main.ym is triggering the main https://github.com/ossf-tests/scorecard-action/blob/main/.github/workflows/scorecards.yml via https://github.com/ossf-tests/scorecard-action/blob/main/.github/workflows/workflowdispatch.yml#L32. The code in https://github.com/ossf-tests/scorecard-action/blob/main/.github/workflows/workflowdispatch.yml#L47 is only run if the dispatch event is not delivered, but not if the Action result fails. We need to update the issue messages in https://github.com/ossf-tests/scorecard-action/blob/main/.github/workflows/scorecards.yml#L68 with the correct trigger events, etc

2. We don't have tests for the push event. We need something similar to https://github.com/ossf-tests/scorecard-action/blob/main/.github/workflows/workflowdispatch.yml#L32 but to trigger a push. See this code as example https://github.com/slsa-framework/example-package/blob/main/.github/workflows/scripts/e2e-push.sh: you need to commit some random data to the repo to simulate a push.

@naveensrinivasan WDUT?

[laurentsimon]

Follow-up: I think the first item Re-use existing issue instead of creating new one is important because we get drowned into the number of created issues atm.

[azeemshaikh38]

@laurentsimon you had mentioned you have a script for re-using existing issues? Could you point me to that?

[azeemshaikh38]

Some more e2e tests needed given recent bugs:

• Create a single issue per workflow instead per-matrix value to reduce noise.
• Add tests for a repository with Branch-Protection settings enabled.
• Add tests for Branch-Protection event trigger.
• Add tests for PAT based workflows.

[azeemshaikh38]

@spencerschrock when you add e2e for #856 this issue can be resolved too. 2 things I'm not implementing and skipping: (i) Re-use existing issue instead of creating new one and (ii) Auto-close issues when e2e tests pass.

If we really need that capability, lets re-assess and open new issues for those.

[laurentsimon]

See https://github.com/slsa-framework/example-package/blob/main/.github/workflows/scripts/e2e-report-failure.sh and https://github.com/slsa-framework/example-package/blob/main/.github/workflows/scripts/e2e-report-success.sh for reporting / closing issues