Coveralls-style reporting on PRs

[danepowell]

Is your feature request related to a problem? Please describe.

MSI is a great metric to complement code coverage, but mutation tests take so long to run (at least on our project) that we can only run a full suite after a pull request is merged. We get a nice badge with our MSI via Stryker integration, but that doesn't help us to gauge the quality of a pull request before merging.

Describe the solution you'd like

I love the way that Coveralls reports changes in code coverage on a pull request, and I'd love something similar for MSI reporting:

The two obstacles I see are, again, the time required to run the full test suite, and the lack of any system to surface the MSI on the PR. Stryker doesn't integrate with PRs AFAICT.

To speed up tests: #1085

To report MSI on the PR: maybe Infection could call the GitHub Status API to report MSI? Something like this should work, but it lacks the additional context (i.e. % change from base branch) of Coveralls: gh api /repos/{owner}/{repo}/statuses/{sha} -f state="success" -f description="23%" -f context="MSI"

[maks-rafalko]

IMO to produce such report, we will still need to run the whole MT on a PR, so I can't see how it will speed up your particular case, while I agree that this feature can be still implemented.

but mutation tests take so long to run (at least on our project) that we can only run a full suite after a pull request is merged

did you consider using --git-diff-lines?

[danepowell]

Thanks, I'll definitely use --git-diff-lines. Though that's still only half of the use case: it can tell you when a PR introduces new mutations, but it can't tell you if it fixes existing mutations.

Maybe this is satisfactory for now. I'm happy to close this unless other people voice interest. Fundamentally, the limitation will be the time required to run the full test suite, it can never work on PRs.

[maks-rafalko]

If you use --git-diff-lines together with --min-covered-msi=100, that would work like a baseline feature in PHPStan/Psalm - you will not have any new escaped mutations in a touched code, but you agree with already existing ones (in not touched code).

I'm happy to close this unless other people voice interest

Let's close for now. We can always reopen if there will be any interest in it. Thank you.