Automated testing strategies

[damontgomery]

We are using Web Test Runner yarn wtr to run tests.

This uses (among other things)

• Mocha
• Chai

Discussion topics

What do we want to test?

What limitations have we encountered?

References

Open PR to add testing: #63

Storybook has an overview of how they recommend testing: https://storybook.js.org/docs/web-components/workflows/testing-with-storybook

They break it down into

• Unit
• Visual
  • Recommend their tool Chromatic
• Interactive
  • Recommend end-to-end tools like Enzyme or Cypress
• Snapshot
  • Have an official addon Storyshots

[damontgomery]

I created a test POC branch here: https://github.com/phase2/outline/blob/poc/mdx-stories/src/components/base/outline-alert/test/outline-alert.test.ts

Open questions

• Is there a better way to load default CSS?
• Is there a better way to load the necessary JS for each component?
• Is there a better way to handle render time for components?

Loading CSS

The interactive test runner yarn wtr --watch provides a nice way to open the browser and see what's happening during tests. For this to be more useful, it would be nice to load the default CSS.

The outline.theme.css isn't loaded by default.

I got around this with this

before(() => {
    const defaultCSS = document.createElement('link');
    defaultCSS.setAttribute('rel', 'stylesheet');
    defaultCSS.setAttribute('src', '/outline.theme.css');

    document.head.append(defaultCSS);
  })

Loading component JS

If we don't use the Outline* object from the component, the test won't load it by default. If we import the module, but don't use it, TypeScript complains...

I got around this with adding a meaningless test.

it('is defined', () => {
    const alertElement = document.createElement('outline-alert');
    alertElement.setAttribute('statusType', 'error');

    // If we don't use OutlineAlert at least once, it's JS will never be loaded and the component won't render. :(
    assert.instanceOf(alertElement, OutlineAlert);

    document.body.append(alertElement);

    const alertElementInDOM = document.querySelectorAll('outline-alert');
    expect(alertElementInDOM).to.exist;
  });

Waiting for the render of components

If we don't wait for components to render, their properties / slots / etc won't be accurately tested.

I got around this by waiting a small amount of time.

setTimeout(() => {
  expect(messageElement.assignedSlot, 'Message is not slotted.').is.not.null;
 }, 10);

I tried to see if Lit elements emit any events that we could respond to, but I couldn't find anything about that.

I tried wait-for-expect, but had issues. See TheBrainFamily/wait-for-expect#31.

Could we use CypressJS which automatically retries to avoid these types of brittle tests?

Other feedback

I think this test works OK. It doesn't check anything about the implementation so far other than that the slots are slotted. Not sure how to check if text shows up on the screen given that all the text checkers in chai-dom will show true since the content is actually in the light DOM even if it's not visible.

[damontgomery]

Tried this with CypressJS. Not sure how I feel about it.

describe('Outline Alert', () => {
  beforeEach(() => {
    cy.visit('http://localhost:8000/cypress/fixtures/empty-html');
  });

  it('Add simple alert', () => {
    // This is done inside this wrapper so we are using the document from the application iFrame. 
    cy.document().then( document => {
      const alertElement = document.createElement('outline-alert');

      const testMessage = 'Test message';
      const messageElement = document.createElement('p');
      messageElement.classList.add('message');

      messageElement.innerText = testMessage;
      alertElement.append(messageElement);

      const body = document.querySelector('body') as HTMLBodyElement;
      body.append(alertElement);

      // If we use a cy.* method, we will let Cypress retry the `expect` a few times.
      // This is helpful because we wait for the element to render.
      cy.get('.message').should(($messageElement) => {
        expect($messageElement.prop('assignedSlot'), 'Message is not slotted.').is.not.null;
      });
    });
  });
});

I saw some examples of trying to load components, but mostly for React, or a complex one for Web Components that I didn't try yet.

It's not too easy to interact with the DOM since to keep the benefits of Cypress, you end up using jQuery instead of plain javascript.

I do like the test runner in some ways, but in others, it wasn't working according to the documentation. Not sure why, but debugging and pausing was giving me issues.

I would probably use our current setup with Web Test Runner and maybe accept the time-out brittleness. :( It's a shame since Cypress has a lot of nice features if we could get it to work like the GUI and snapshots.

[damontgomery]

I did some more testing and tried some interactions with the modal component.

Storybook documentation suggest Cypress as an option for end-to-end testing which I think makes sense for unit tests that involve interaction.

Benefits

• Storybook integration works well where we can load the generated story and run tests directly against those starting states / configurations
• It seems to find elements consistently and retry a few times which makes the tests less brittle
• The user interactions are nice, such as clicks and typing
• Easy enough to access the shadow DOM with .shadow()

Drawbacks / Challenges

• The jQuery syntax is a bit weird and makes accessing the actual DOM a little trickier
• The DOM snapshot doesn't seem to reflect the state completely, such as opening the modal (always looks closed)
• Key presses need to be on specific elements
• TAB is only supported in a contributed plugin
• Some of the interactions are not so easy to read since they are CSS selectors, etc, unsure if there is a way to replace with message

Here is an example of what it looks like:

And a code sample:

import { OutlineModalElementType } from '../../src/components/base/outline-modal/outline-modal';

describe('Outline Modal', () => {
  beforeEach(() => {
    // This URL is slightly different that the storybook page.
    // Swap `?path=/story/` with `iframe.html?id=`.
    // This loads just the component iframe.
    cy.visit(
      'http://localhost:6006/iframe.html?id=molecules-modal--custom-focus-element'
    );
  });

  it('Default slot (message) before and while interacting with modal.', () => {
    // Assigning the results of `cy.get()` seems to result in changes as we use other selectors.
    const defaultSlottedElementsSelector = 'outline-modal > :not("[slot]")';

    // If we use a cy.* method, we will let Cypress retry the `expect` a few times.
    // This is helpful because we wait for the element to render.
    cy.get(defaultSlottedElementsSelector).should($messageElements => {
      expect(
        $messageElements.length,
        'Elements in the default slot'
      ).to.be.greaterThan(0);

      // We use `function() {}` instead of `() => {}` so we can access `this`.
      $messageElements.each(function () {
        expect(this.assignedSlot, 'Message is not yet slotted.').is.null;
      });
    });

    cy.get('[slot="outline-modal--trigger"]').click();

    cy.get(defaultSlottedElementsSelector).should($messageElements => {
      $messageElements.each(function () {
        expect(
          this.assignedSlot,
          'Message is now slotted.'
        ).is.not.null.is.not.undefined;
      });
    });

    cy.get('outline-modal').shadow().get('body').type('{esc}');

    cy.get(defaultSlottedElementsSelector).should($messageElements => {
      $messageElements.each(function () {
        expect(this.assignedSlot, 'Message is no longer slotted.').is.null;
      });
    });
  });
});

[damontgomery]

I tried to get the Storyshots addon to work, but ran into a number of issues getting it started.

• After adding the package, yarn tsc gives errors
• The example storybook.test.js file when run with Jest complains about using an import in a js file, but changing that to require or trying to use a ts file were both causing issues for me.

[damontgomery]

I did more work with Web Test Runner and was able to get waitUntil to work. Here is an example of the modal test using this process:

import { OutlineModal as OutlineComponent } from '../outline-modal';
import { expect, waitUntil } from '@open-wc/testing';

// We are using these generic names so we can hopefully generalize this process or at least make copy & paste easier.
const componentElementName = 'outline-modal';

describe(componentElementName, () => {
  // We don't get base styles automatically. Is there a better way?
  const addBaseStyles = () => {
    const defaultCSS = document.createElement('link');
    defaultCSS.setAttribute('rel', 'stylesheet');
    defaultCSS.setAttribute('href', '/outline.theme.css');

    document.head.append(defaultCSS);
  };

  before(() => {
    addBaseStyles();
  });

  it('Make sure to load necessary component JS', () => {
    // If we don't use the component at least once, its JS will never be loaded and the component won't render. :(
    const componentElement = document.createElement(componentElementName);

    expect(componentElement).is.instanceOf(OutlineComponent);
  });

  it('Slot elements while interacting with the modal.', async () => {
    const modalElement = document.createElement(
      componentElementName
    ) as OutlineComponent;

    /**
     * <button slot="outline-modal--trigger">
     *  Open small modal.
     * </button>
     */
    const triggerElement = document.createElement('button');
    triggerElement.setAttribute('slot', 'outline-modal--trigger');
    triggerElement.innerText = 'Open small modal';
    modalElement.append(triggerElement);

    /**
     * <h2 slot="outline-modal--header">Modal header text</h2>
     */
    const headerElement = document.createElement('h2');
    headerElement.setAttribute('slot', 'outline-modal--header');
    headerElement.innerText = 'Modal header text';
    modalElement.append(headerElement);

    /**
     * <p>Test message</p>
     */
    const messageElement = document.createElement('p');
    messageElement.innerText = 'Test message';
    modalElement.append(messageElement);

    document.body.append(modalElement);

    await waitUntil(
      () => triggerElement.assignedSlot !== null,
      'Trigger is slotted'
    );

    expect(triggerElement.assignedSlot, 'Trigger is slotted.').is.not.null;

    expect(headerElement.assignedSlot, 'Header is not yet slotted.').is.null;

    expect(messageElement.assignedSlot, 'Message is not yet slotted.').is.null;

    modalElement.open();

    await waitUntil(() => modalElement.isOpen === true, 'Modal is open');

    expect(headerElement.assignedSlot, 'Header is slotted.').is.not.null;

    expect(messageElement.assignedSlot, 'Message is slotted.').is.not.null;

    modalElement.close();

    await waitUntil(() => modalElement.isOpen === false, 'Modal is closed');

    expect(headerElement.assignedSlot, 'Header is no longer slotted.').is.null;

    expect(messageElement.assignedSlot, 'Message is no longer slotted.').is
      .null;
  });
});

[damontgomery]

I tried using @testing-library/user-event and @testing-library/dom according to the documentation at https://testing-library.com/docs/ecosystem-user-event. These are supposed to provide ways of simulating user behavior within Web Test Runner.

I keep getting an error which I think is from trying to include a CommonJS module in our TypeScript ESModule world.

Tried:

• import userEvent from '@testing-library/user-event';
• import userEvent from '@testing-library/user-event/dist/index.js'
• import userEvent = require('@testing-library/user-event') (and some other variations).

I get errors in the browser like

🚧 Browser logs on Chromium:
      SyntaxError: The requested module './../../../../../node_modules/@testing-library/user-event/dist/index.js' does not provide an export named 'default'
 🚧 Browser logs on Firefox:
      SyntaxError: import not found: default
 🚧 Browser logs on Webkit:
      SyntaxError: Importing binding name 'default' cannot be resolved by star export entries.

To get set up I added these to package.json

"@testing-library/dom": "^8.1.0",
"@testing-library/user-event": "^13.2.1",

I also could not get fireEvent() to work. I couldn't find how to include it.

These seem like nice tools if we can get them to work. :(