test: Stryker Jest runner with config workarounds

[brodybits]

updated: I reconfigured by removing stryker.conf.json from my work area, running npx stryker init again, and re-patching stryker.conf.json with some of the configuration from the master branch. The reconfiguration gave me the coverageAnalysis entry changed to perTest.

I was hoping to improve the speed on GitHub. I tried with and without this update on cloud instances with 2 vCPU, for 2 test runner processes, which seems to reflect what happens on GitHub. But it took about half minute longer with the Jest runner, which is insignificant considering that it takes almost 2 hours on GitHub with 2 test runner processes.

I had to update jest.config.js to configure testMatch instead of testRegex. From the debug tracing command, as directed when Stryker fails, it looked to me like with the Jest runner it was messing up the configuration of testRegex. I suspect that this is why the jest-runner did not work in PR #140 (update to Stryker 4). I think it would be ideal if we could get this fixed on Stryker at some point. My changes may need cleanup, and I am not sure if we need to configure testPathIgnorePatterns or not.

Test time was almost the same, maybe half a minute longer with the Jest runner. See stryker-mutator/stryker-js#1514 (comment):

Mutation switching by itself will not have a significant impact on Jest projects unfortunately. You might have some performance improvement because Jest would be able to cache build results, but definitely not an order of magnitude. However, it does open up the way for coverage analysis in jest. Hot reload should theoretically also be possible with Jest, but not with their current public api alone. We'll be focussing on jest next, after we've released v4.

End results with command runner

Mutation testing  [==================================================] 100% (elapsed: ~1h 53m, remaining: n/a) 2014/2014 tested (695 survived, 41 timed out)
[...]
Ran 0.98 tests per mutant on average.
---------------|---------|----------|-----------|------------|----------|---------|
File           | % score | # killed | # timeout | # survived | # no cov | # error |
---------------|---------|----------|-----------|------------|----------|---------|
All files      |   65.49 |     1278 |        41 |        695 |        0 |       0 |
 dom-parser.js |   80.00 |      156 |         0 |         39 |        0 |       0 |
 dom.js        |   65.81 |      590 |        24 |        319 |        0 |       0 |
 entities.js   |    7.85 |       19 |         0 |        223 |        0 |       0 |
 sax.js        |   82.30 |      513 |        17 |        114 |        0 |       0 |
---------------|---------|----------|-----------|------------|----------|---------|
[...]
23:51:59 (1131) INFO MutationTestExecutor Done in 113 minutes 37 seconds.

End results with Jest runner

Mutation testing  [==================================================] 100% (elapsed: ~1h 53m, remaining: n/a) 1832/1832 tested (513 survived, 92 timed out)
[...]
Ran 183.07 tests per mutant on average.
---------------|---------|----------|-----------|------------|----------|---------|
File           | % score | # killed | # timeout | # survived | # no cov | # error |
---------------|---------|----------|-----------|------------|----------|---------|
All files      |   65.42 |     1223 |        92 |        513 |      182 |       4 |
 dom-parser.js |   80.00 |      149 |         7 |         24 |       15 |       0 |
 dom.js        |   65.81 |      585 |        29 |        168 |      151 |       0 |
 entities.js   |    7.85 |       19 |         0 |        223 |        0 |       0 |
 sax.js        |   82.19 |      470 |        56 |         98 |       16 |       4 |
---------------|---------|----------|-----------|------------|----------|---------|
[...]
23:55:19 (1135) INFO MutationTestExecutor Done in 114 minutes 11 seconds.

I would also like to investigate the 4 errors on sax.js at some point.

[brodybits]

As I said in PR #220 Stryker 4.6.0 update seems to run much slower, proposing to use Jest runner with Stryker 4.5.1 for now at least.

[brodybits]

In terms of status: I have tested this with Stryker 4, not sure if it will work with Stryker 5 or not. I am also not sure whether or not these workarounds are needed with Stryker 5.x as well.

In case we do still need these workarounds with Stryker 5.x, I think it would be good to investigate why.

[karfau]

We are no longer using stryker mutation testing, so this PR is obsolete