Support for Web Test Runner

[jensk-dev]

Is your feature request related to a problem? Please describe.
With the deprecation of karma, I suspect Web Test Runner, amongst others, will increasingly be used to run in-browser tests. Therefore, adding support for web test runner to Stryker seems like a logical step.

Describe the solution you'd like
I suspect that support for Web Test Runner can be added by using @web/test-runner-core in combination with @web/test-runner-commands. Browser instances can be spawned through @web/test-runner-puppeteer or @web/test-runner-chrome

Describe alternatives you've considered
Not applicable (or maybe jasmine-browser-runner

Additional context
N/A

[nicojs]

It's fantastic that you're willing to pick this one up 😁!

An excellent place to start is the docs here:
https://stryker-mutator.io/docs/stryker-js/guides/create-a-plugin/#high-level-stryker-workings

There is a lot of info there, but the only mandatory thing that a test runner plugin must do is to enable the correct mutant in the runMutant method by setting a global variable. I would suggest to start with that.

Note: i'd suggest keeping out of instantiating specific browser instances. Instead leaving that to the WRT to deal with.

[jensk-dev]

I'll look into it. WTR defers browser instantiation to the user config, thereby allowing a multitude of browsers to run through various means like puppeteer, selenium, etc. The default behaviour of @web/test-runner is chrome headless. Would you like to expose browser configuration to the overlying application or would you rather restrict access to browser instantiation by selecting a predefined browser launcher?

[nicojs]

I hope there is a way for the WRT to choose the browser based on the user's config file.

In short:

1. We want to load the WRT config
2. We want to change the config slightly (enable bail, add a beforeEach hook, activate a mutant, communicate mutation coverage back to the main process, etc)
3. We want to be able to run in quick succession.

As mentioned before, the only must-have is that we can activate a mutant.

[nicojs]

I've made some more progress. It seems that wrt supports middlewares. This would make something like this possible:

Code example

  middleware: [
    async (context, next) => {
      await next();
      if (context.path === "/" && typeof context.body === "string") {
        context.body = context.body.replace(
          "</body>",
          `
          <script type="module">
          import { executeServerCommand } from '${webUrl(
            "@web/test-runner-commands",
          )}';
          import { getConfig } from '${webUrl(
            "@web/test-runner-core/browser/session.js",
          )}';

          const ns = globalThis.__stryker__ = globalThis.__stryker__ ?? {};

          // Example of test filtering:
          mocha.grep(/add/);

          // Grab currently executing test
          beforeEach(function() {
            ns.currentTestId = this.currentTest?.fullTitle();
          });
          after(async() => {
            // Report mutant coverage
            const { testFile } = await getConfig();
            await executeServerCommand('stryker-report', {
              mutantCoverage: ns.mutantCoverage,

              // Also report the test file that was executed
              testFile,
            });
          });
          </script>
          </body>`,
        );
        console.log(`👽 ${context.body}`);
      }
    },
  ],

We can do anything we want that way.

This would help us with 2, but not 1.

@jensk-dev are you still planning to pickup this issue?

[jensk-dev]

I've made some more progress. It seems that wrt supports middlewares. This would make something like this possible:

Code example

  middleware: [
    async (context, next) => {
      await next();
      if (context.path === "/" && typeof context.body === "string") {
        context.body = context.body.replace(
          "</body>",
          `
          <script type="module">
          import { executeServerCommand } from '${webUrl(
            "@web/test-runner-commands",
          )}';
          import { getConfig } from '${webUrl(
            "@web/test-runner-core/browser/session.js",
          )}';

          const ns = globalThis.__stryker__ = globalThis.__stryker__ ?? {};

          // Example of test filtering:
          mocha.grep(/add/);

          // Grab currently executing test
          beforeEach(function() {
            ns.currentTestId = this.currentTest?.fullTitle();
          });
          after(async() => {
            // Report mutant coverage
            const { testFile } = await getConfig();
            await executeServerCommand('stryker-report', {
              mutantCoverage: ns.mutantCoverage,

              // Also report the test file that was executed
              testFile,
            });
          });
          </script>
          </body>`,
        );
        console.log(`👽 ${context.body}`);
      }
    },
  ],

We can do anything we want that way.

This would help us with 2, but not 1.

@jensk-dev are you still planning to pickup this issue?

Hi Nico,

I've briefly looked at it a bit more in my free time. However, currently I do not have the time to meaningfully pick up this issue in the foreseeable future. Your example looks interesting and is relatively comparable with what I've come up with for communicating back to the wtr server. How are you planning on handling beforeEach/afterEach if a test suite defines those lifecycle hooks themselves?

[igomezal]

Not sure if this can help, but wtr also support configuring the HTML environment where the test run in using the testRunnerHtml property, some time ago I used it to integrate wtr and stryker together, here you can take a look https://dev.to/igomezal/setting-up-mutation-testing-with-stryker-and-web-test-runner-3c9p

It was somewhat slow, as it used the command test runner and it didn't support incremental mode to make it faster on new runs.