[Windows] Mutating the source fails the tests but still Stryker shows no kills

[sincerekamal]

Summary

I have recently configured stryker as per the "Getting started". When I tried stryker run there are no mutants which were killed. So, I doubted myself and tried mutating(changing) as per the mutation statements shown in console. Now, I tried npm test and the tests were failed. So, why the mutants aren't killed even when the tests are failing ? Is there any config issue ? or is it due to the jasmine or Windows OS ?

Stryker config

{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "packageManager": "npm",
  "reporters": ["html", "clear-text", "progress", "dashboard"],
  "mutate": ["src/modules/users/route.js"],
  "tempDirName": "stryker-tmp",
  "coverageAnalysis": "perTest",
  "testRunner": "jasmine",
  "jasmineConfigFile": "test/jasmine-unit.json"
}

Test runner config

{
  "spec_dir": "test",
  "spec_files": ["unit/**/*[sS]pec.js"],
  "helpers": ["helpers/**/*.js"],
  "stopSpecOnExpectationFailure": false,
  "random": false
}

Stryker environment

+-- @stryker-mutator/core:5.3.0
+-- @stryker-mutator/jasmine-runner:5.3.0

"jasmine": "^3.8.0"
"jasmine-core": "^3.8.0"

Test runner environment

# Test command
jasmine --config=./test/jasmine-unit.json

{
  "spec_dir": "test",
  "spec_files": ["unit/**/*[sS]pec.js"],
  "helpers": ["helpers/**/*.js"],
  "stopSpecOnExpectationFailure": false,
  "random": false
}

Your Environment

software	version(s)
node	14.6.0
npm	6.14.11
Operating System	Windows 10 x64

Add stryker.log

stryker.log

[Edgpaez]

Could this be related to #3012(comment) ?

How do you know "the mutants aren't killed"?
If you think that based on the Test tab, then you might see that tests only covered although applying the mutant by hand does make the test fail.

Also, the docs link about a possible problem where all mutants survive and known workarounds: https://stryker-mutator.io/docs/stryker-js/troubleshooting#all-mutants-survive---jest-runner

Have you tried that?

[sincerekamal]

@Edgpaez If you see my "stryker config", I have given "tempDirName": "stryker-tmp", but still no luck. And I'm using jasmine, not jest.

[nicojs]

@sincerekamal Are you using native ECMA script modules by any chance? If so, then that's the issue. You will need to run with --maxTestRunnerReuse 1. This is bad for performance, but the only way to make it work at the moment, see #2922 for more info.

If that isn't the issue, could you please create a small reproduction repository (or point us to the source code if your project is open source)? The jasmine runner should work as expected, as per our jasmine example in our e2e tests: https://github.com/stryker-mutator/stryker-js/tree/master/e2e/test/jasmine-javascript

[nicojs]

@sincerekamal do you think you could find some time to test my hypothesis and report back here? 😇

[sincerekamal]

Sure @nicojs , I will try it and update here.

[sincerekamal]

@nicojs I tried it, it's still not working. You can try it here, https://github.com/sincerekamal/stryker-test

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sincerekamal]

@nicojs did you get chance to look at that ?

[nicojs]

Hi, @sincerekamal, thanks for pinging me. I forgot all about this issue, sorry 😢.

I found what the culprit is: you're using module-alias:

// register modules that are aliased (see package.json)
require("module-alias/register");

This registeres the module aliases from your package.json:

{
  "_moduleAliases": {
    "@config": "./src/config",
    "@middlewares": "./src/middlewares",
    "@models": "./src/models",
    // ...
  }
}

This allows you to do magic imports like these:

const { TokenError } = require("@errors");
const { Meals } = require("@models");
const { getCalories } = require("@services/nutrition");

But it doesn't play nice with StrykerJS's sandboxing, as alias imports like this always import from your local project instead of the sandbox as Stryker is counting on. This line is the culprit: https://github.com/ilearnio/module-alias/blob/dev/index.js#L166

I would advice you to move away from module-alias for 2 reasons:

1. It works by overriding a private node API so it might break at any major node release.
2. It won't ever work for native ESM

If you want to keep using module-alias, there are 2 workaround:

1. Mark the node_modules as part of your sandbox:

   {
     "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
      "packageManager": "npm",
      "reporters": ["html", "clear-text", "progress", "dashboard"],
      "coverageAnalysis": "perTest",
      "testRunner": "jasmine",
   -  "jasmineConfigFile": "test/jasmine-unit.json"
   +  "jasmineConfigFile": "test/jasmine-unit.json",
   +  "ignorePatterns": ["!node_modules"],
   +  "symlinkNodeModules": false
   }

2. Let StrykerJS mutate your files in place: npx stryker run --inPlace

Result for both workarounds:

Ran 14.49 tests per mutant on average.
---------------------------|---------|----------|-----------|------------|----------|----------|
File                       | % score | # killed | # timeout | # survived | # no cov | # errors |
---------------------------|---------|----------|-----------|------------|----------|----------|
All files                  |   42.07 |      434 |         9 |         89 |      521 |       15 |
 config                    |    0.00 |        0 |         0 |         30 |        1 |        1 |
  index.js                 |    0.00 |        0 |         0 |         30 |        1 |        1 |
 middlewares               |    0.00 |        0 |         0 |          0 |       87 |        0 |
  auth                     |    0.00 |        0 |         0 |          0 |       23 |        0 |
   index.js                |    0.00 |        0 |         0 |          0 |       23 |        0 |
  requestInitializer       |    0.00 |        0 |         0 |          0 |       29 |        0 |
   index.js                |    0.00 |        0 |         0 |          0 |       29 |        0 |
  responseHandler          |    0.00 |        0 |         0 |          0 |       29 |        0 |
   index.js                |    0.00 |        0 |         0 |          0 |       29 |        0 |
  routeNotFoundHandler     |    0.00 |        0 |         0 |          0 |        6 |        0 |
   index.js                |    0.00 |        0 |         0 |          0 |        6 |        0 |
 models                    |    0.00 |        0 |         0 |          0 |      141 |        0 |
  index.js                 |    0.00 |        0 |         0 |          0 |       30 |        0 |
  Meals.js                 |    0.00 |        0 |         0 |          0 |       49 |        0 |
  User.js                  |    0.00 |        0 |         0 |          0 |       62 |        0 |
 public_modules            |   94.12 |       30 |         2 |          1 |        1 |        0 |
  main                     |   94.12 |       30 |         2 |          1 |        1 |        0 |
   controllers             |   92.86 |       13 |         0 |          0 |        1 |        0 |
    createToken.js         |  100.00 |        6 |         0 |          0 |        0 |        0 |
    get.js                 |  100.00 |        3 |         0 |          0 |        0 |        0 |
    index.js               |    0.00 |        0 |         0 |          0 |        1 |        0 |
    signup.js              |  100.00 |        4 |         0 |          0 |        0 |        0 |
   payload_validators      |  100.00 |        3 |         2 |          0 |        0 |        0 |
    schema                 |  100.00 |        0 |         2 |          0 |        0 |        0 |
     createToken.schema.js |  100.00 |        0 |         1 |          0 |        0 |        0 |
     signup.schema.js      |  100.00 |        0 |         1 |          0 |        0 |        0 |
    createToken.js         |  100.00 |        1 |         0 |          0 |        0 |        0 |
    index.js               |  100.00 |        1 |         0 |          0 |        0 |        0 |
    signup.js              |  100.00 |        1 |         0 |          0 |        0 |        0 |
   handler.js              |   90.91 |       10 |         0 |          1 |        0 |        0 |
   route.js                |  100.00 |        4 |         0 |          0 |        0 |        0 |
 services                  |   96.67 |       29 |         0 |          1 |        0 |        0 |
  nutrition.js             |   96.67 |       29 |         0 |          1 |        0 |        0 |
 utils                     |   32.97 |      153 |         0 |         52 |      259 |       14 |
  errors                   |   31.98 |       79 |         0 |         23 |      145 |        0 |
   BaseError.js            |   40.00 |        4 |         0 |          5 |        1 |        0 |
   DatabaseError.js        |   46.15 |        6 |         0 |          7 |        0 |        0 |
   errorHandler.js         |    0.00 |        0 |         0 |          0 |       89 |        0 |
   index.js                |  100.00 |        1 |         0 |          0 |        0 |        0 |
   JoiSchemaError.js       |   90.00 |       18 |         0 |          2 |        0 |        0 |
   JsonSchemaError.js      |   38.64 |       34 |         0 |          7 |       47 |        0 |
   NoRecordFoundError.js   |    0.00 |        0 |         0 |          0 |        5 |        0 |
   NotFoundError.js        |  100.00 |        2 |         0 |          0 |        0 |        0 |
   RedirectionError.js     |    0.00 |        0 |         0 |          0 |        1 |        0 |
   ServerError.js          |  100.00 |        8 |         0 |          0 |        0 |        0 |
   TokenError.js           |   55.56 |        5 |         0 |          2 |        2 |        0 |
   ValidationError.js      |  100.00 |        1 |         0 |          0 |        0 |        0 |
  joiValidation            |   50.00 |       33 |         0 |         24 |        9 |       13 |
   common.schema.js        |   16.67 |        1 |         0 |          5 |        0 |        1 |
   index.js                |   85.71 |        6 |         0 |          1 |        0 |        0 |
   joiExtensions.js        |   49.06 |       26 |         0 |         18 |        9 |       12 |
  appConstants.js          |    0.00 |        0 |         0 |          4 |        0 |        1 |
  helpers.js               |   97.62 |       41 |         0 |          1 |        0 |        0 |
  logger.js                |    0.00 |        0 |         0 |          0 |      105 |        0 |
 v1                        |   97.03 |      222 |         7 |          5 |        2 |        0 |
  modules                  |   97.03 |      222 |         7 |          5 |        2 |        0 |
   meals                   |   99.15 |      113 |         3 |          0 |        1 |        0 |
    controllers            |   98.68 |       75 |         0 |          0 |        1 |        0 |
     create.js             |  100.00 |       15 |         0 |          0 |        0 |        0 |
     delete.js             |  100.00 |        4 |         0 |          0 |        0 |        0 |
     get.js                |  100.00 |       15 |         0 |          0 |        0 |        0 |
     index.js              |    0.00 |        0 |         0 |          0 |        1 |        0 |
     list.js               |  100.00 |       13 |         0 |          0 |        0 |        0 |
     update.js             |  100.00 |       28 |         0 |          0 |        0 |        0 |
    payload_validators     |  100.00 |       15 |         3 |          0 |        0 |        0 |
     schema                |  100.00 |        2 |         3 |          0 |        0 |        0 |
      create.schema.js     |  100.00 |        1 |         0 |          0 |        0 |        0 |
      delete.schema.js     |  100.00 |        0 |         1 |          0 |        0 |        0 |
      get.schema.js        |  100.00 |        0 |         1 |          0 |        0 |        0 |
      list.schema.js       |  100.00 |        1 |         0 |          0 |        0 |        0 |
      update.schema.js     |  100.00 |        0 |         1 |          0 |        0 |        0 |
     create.js             |  100.00 |        8 |         0 |          0 |        0 |        0 |
     delete.js             |  100.00 |        1 |         0 |          0 |        0 |        0 |
     get.js                |  100.00 |        1 |         0 |          0 |        0 |        0 |
     index.js              |  100.00 |        1 |         0 |          0 |        0 |        0 |
     list.js               |  100.00 |        1 |         0 |          0 |        0 |        0 |
     update.js             |  100.00 |        1 |         0 |          0 |        0 |        0 |
    handler.js             |  100.00 |       17 |         0 |          0 |        0 |        0 |
    route.js               |  100.00 |        6 |         0 |          0 |        0 |        0 |
   users                   |   94.96 |      109 |         4 |          5 |        1 |        0 |
    controllers            |   92.94 |       79 |         0 |          5 |        1 |        0 |
     create.js             |  100.00 |       17 |         0 |          0 |        0 |        0 |
     delete.js             |  100.00 |        8 |         0 |          0 |        0 |        0 |
     get.js                |   80.00 |        8 |         0 |          2 |        0 |        0 |
     index.js              |    0.00 |        0 |         0 |          0 |        1 |        0 |
     list.js               |  100.00 |        7 |         0 |          0 |        0 |        0 |
     update.js             |   92.86 |       39 |         0 |          3 |        0 |        0 |
    payload_validators     |  100.00 |        7 |         4 |          0 |        0 |        0 |
     schema                |  100.00 |        1 |         4 |          0 |        0 |        0 |
      create.schema.js     |  100.00 |        0 |         1 |          0 |        0 |        0 |
      delete.schema.js     |  100.00 |        0 |         1 |          0 |        0 |        0 |
      get.schema.js        |  100.00 |        0 |         1 |          0 |        0 |        0 |
      list.schema.js       |  100.00 |        1 |         0 |          0 |        0 |        0 |
      update.schema.js     |  100.00 |        0 |         1 |          0 |        0 |        0 |
     create.js             |  100.00 |        1 |         0 |          0 |        0 |        0 |
     delete.js             |  100.00 |        1 |         0 |          0 |        0 |        0 |
     get.js                |  100.00 |        1 |         0 |          0 |        0 |        0 |
     index.js              |  100.00 |        1 |         0 |          0 |        0 |        0 |
     list.js               |  100.00 |        1 |         0 |          0 |        0 |        0 |
     update.js             |  100.00 |        1 |         0 |          0 |        0 |        0 |
    handler.js             |  100.00 |       17 |         0 |          0 |        0 |        0 |
    route.js               |  100.00 |        6 |         0 |          0 |        0 |        0 |
 server.js                 |    0.00 |        0 |         0 |          0 |       30 |        0 |
---------------------------|---------|----------|-----------|------------|----------|----------|

[sincerekamal]

Thanks a lot @nicojs , let me try it and will update you. Thanks for spending your time on this !!

[nicojs]

I think this should be documented in our troubleshooting guide, so re-opening :).