How about a stryker-mutator plugin?

[geoffreytools]

I have been using stryker-mutator with AVA recently.

Although it works, it is very slow because Stryker can't do a coverage analysis with AVA and thus it runs every test for each mutant regardless of them being covered by a given test.

This could be improved with an AVA plugin.

They currently support Jasmine, Jest, Karma and Mocha and I guess adding AVA to the list could increase its adoption.

[novemberborn]

What would this plugin need to do?

[geoffreytools]

I myself can only speculate about how this problem is solved in other plugins, and there is an overlap in responsibilities, so which of Stryker or AVA should do what is not clear in my head.

Pragmatically, when I look up one of the test suites for their mocha runner I see things like:

• "collect mutant coverage",
• "active the given mutant"
• "report killed mutant when a test fails"
• "report a survived mutant when all test succeed"
• etc.

How I picture it myself, someone needs to

• read Stryker's configuration to filter which test files should be run and which source files should use mutants,
• run a coverage analysis to see what piece of source code is covered by which test,
• pilot the creation of mutants in the source files,
• potentially add a build step for babel plugins,
• selectively run the tests which concern the piece of source code each mutant was inserted in,
• report which mutants survived, were killed, timed out because their related tests succeeded, failed or timed out

[novemberborn]

Our t.try() assertion was built for part of this use case. You can try a set of assertions and discard the results. This way you could try a variety of inputs.

I don't know how you'd integrate that with realtime code coverage.

[Adam-Schlichtmann]

Has anything been done to move this along?

[novemberborn]

Not that I'm aware of. You're welcome to give it a try @Adam-Schlichtmann.

[nicojs]

Hi 🙋‍♂️. I'm one of the StrykerJS maintainers. I would be happy to help 😄

@geoffreytools put a nice list together. Let me clarify some parts

A test runner plugin in StrykerJS needs to implement this interface

export interface TestRunner {
  init?(): Promise<void>;
  dryRun(options: DryRunOptions): Promise<DryRunResult>;
  mutantRun(options: MutantRunOptions): Promise<MutantRunResult>;
  dispose?(): Promise<void>;
}

See https://github.com/stryker-mutator/stryker-js/blob/master/packages/api/src/test-runner/test-runner.ts

The main a test runner should be able to do is to run tests whenever it is asked to do so. StrykerJS takes care of mutant generation, coverage bookkeeping etc, parallel worker creation, etc. A test runner doesn't need any of that.

Some examples:

collect mutant coverage
This means reporting the __stryker__.mutantCoverage variable back to Stryker. This is pretty straightforward, as long as you can access global variables in your test environment.

Active the given mutant
Again, if you have access to the global scope of the test environment, it is as simple as setting the __stryker__.activeMutant variable.

report killed mutant when a test fails
Straight forward stuff, when a test fails -> mutant killed.

I'm thinking of abstracting some of this work away in a "NodeTestRunner" class. It would mean that you could extend this class and implement even less. For example: setting and retrieving the global variables would be taken care of. We could use this class to implement our mocha, jasmine, and CucumberJS test runners. Would you be interested in this as well?

[novemberborn]

All of that should still be possible but it's not the most elegant way of doing things (from AVA's perspective).

[nicojs]

but it's not the most elegant way of doing things

It usually isn't 😅, but mutation testing is messy and a lot of work. In order to make it perform, we need to do coverage analysis and test filtering.

If someone of the AVA team wants to meet with me or another StrykerJS maintainer, let me know. That way we can discuss design in-depth and explain both our philosophies. Maybe the plugin could be created without much effort? 🤷‍♂️

[geoffreytools]

As someone unfamiliar with the codebases and not a specialist of the domain, I find that the more you two speak together, the more the task seems approachable! I would encourage any further discussion ;) It is something the community could implement but a clearer roadmap would help overcome the blank page syndrome.

[novemberborn]

I can't make any time commitments, but I'd happy to provide ideas on how to make each of the integration points possible.

[sculpt0r]

As I see - there is no movement around that plugin :) Could you tell me how hard it will be for the beginner (in terms of stryker repo) to build such a plugin based on an already existing one?