Race condition in karma can result in a mutant being tested as a 'runtimeError'

[nicojs]

Summary

There is a race condition in karma that can result in a mutant being reported as a runtime error. The race condition is explained in karma-runner/karma#3424

I've found the issue in a gh workflow (on linux). I haven't been able to reproduce it on windows. As you can see in the stack trace: clearly the onbeforeunload is triggered from karma code (see karma-runner/karma#3424 for more info)

ERROR: 'Window reload!!', 'Error
2020-02-15T23:56:13.4716280Z     at window.onbeforeunload (http://localhost:9877/base/test/AddSpec.js?cc8af791b57fdffcc1474a86f80f049403495bde:21:40)
2020-02-15T23:56:13.4716758Z     at navigateContextTo (http://localhost:9877/karma.js:135:18)
2020-02-15T23:56:13.4717186Z     at clearContext (http://localhost:9877/karma.js:161:5)
2020-02-15T23:56:13.4717823Z     at http://localhost:9877/karma.js:250:9'

Stryker config

This happens in our e2e test suite sometimes with this karma config:

{
    mutate: ['src/*.js'],
    mutator: 'javascript',
    testFramework: 'mocha',
    testRunner: 'karma',
    reporters: ['clear-text', 'html', 'event-recorder'],
    karma: {
      config: {
        frameworks: ['mocha', 'chai'],
        files: ['src/*.js', 'test/*.js']
      }
    },
    logLevel: 'trace',
    timeoutMS: 60000,
    maxConcurrentTestRunners: 2,
    coverageAnalysis: 'perTest'
  }

Stryker environment

See our e2e tests. Both karma-mocha and karma-jasmine

A possible solution

We can implement the workaround proposed in karma-runner/karma#3424 by forcing client.clearContext to false