Add interaction tests

[ericedem]

I've noticed in slate that a large majority of bugs that creep into codebase are specifically around events, interaction, cross browser, and slate-react. These bugs are often caught by people running through examples on different browsers, and happening to stumble across them.

What I wanted to do is start a discussion to talk about how we could better test for these kinds of issues, hoping someone has experience in this area. I'd like to get higher level testing, like DOM interaction and event triggers, without resorting to something like selenium which is generally error prone.

I haven't dug super deep, but a promising library that might help with this is https://github.com/kentcdodds/react-testing-library.

We could probably automate running this on something like sauce labs or browserstack, or something that has a free open source plan.

[ianstormtaylor]

Hey @ericedem thanks for opening this! I completely agree with you.

As for react-testing-library, I haven't looked into it myself. I can't figure out how they're arriving at a "real DOM". If they are using JSDOM for it, I'm not sure it will work for our case because JSDOM doesn't attempt to handle anything to do with contenteditable, or at least last I checked.

Another potential option might be using Chrome Headless? I think @zhujinxuan might have investigated this or was hoping to. It would be awesome if we can make it as low maintenance as possible, and headless Chrome might help us with this.

Either way, I totally agree. And it's great to have an issue for it!

[dmitrizzle]

FWIW it seems Slate could use end-to-end testing framework. There are some listed here. The downside, I suppose is that they are mostly browser-specific and we need to test for browser faults, meaning having one just won't cut it.

I'd say automated testing via BrowserStack would make the most sense. It probably isn't cheap, unless they'd be willing to be a "sponsor," which would make perfect sense, seeing that Slate is an excellent candidate with fantastic adoption and community.

[ericedem]

Slate could use end-to-end testing framework

I'd really prefer to avoid selenium testing if possible. They are very high overhead, and often require a full time team to maintain since there are many points of failure, and they are time consuming to debug. I think we can get most bang for our bug with CLI testing.

For cross browser testing, what I had in mind was using something like sauce labs which has an open source program. Browserstack has something similar. And using something like zuul for automating the cross browser CLI testing. Though I haven't used those in a few years, there might be something cooler and more hip these days 😄

[ianstormtaylor]

FWIW, 90% of the testing needed isn't cross-browser and just needs to test that selections work and that rendering gets updating, etc. Lots of this could be Chrome-only and still be a huge improvement over the current situation.

[ojassvi]

Yes you can use react-testing-library with headless Chrome. I am using Karma as the test runner and headless Chrome to render my components in real DOM.

I am using slate in my project and I am able to render it correctly using react-testing-libaray but not able to test the inputs so far. I can add here my findings and how I am testing it, if that is helpful.

[zhujinxuan]

@ojassvi Thank you. I did not know that react-testing-library can work with headless Chrome.

@ianstormtaylor Sorry, I did not have time to look into headless Chrome before next Jan. I think I will look into it by next Jan.

[dmitrizzle]

I've had some good success with Cypress, though I'm not 100% confident it can provide the level of testing needed for all the cases. It's easy to write tests for (i.e. go to this url, click this button, enter this text, find this text in this DOM node).

[brendancarney]

+1 for Cypress. Also, they have a nice dashboard service that's free for open source.

The Dashboard Service is free to use for non-commercial open source, public projects and has no limitations on the number of public recorded tests per month.

[ianstormtaylor]

Anyone who can help add this would be greatly appreciated!

[Jmaharman]

Some of the details in this thread may help whoever does come to work on getting this testable in Cypress

https://slate-js.slack.com/archives/C1RH7AXSS/p1579110773288500

[dmitrizzle]

I can add a few basic Cypress tests to examples and have them run in GH Actions. Haven't figured it all out but perhaps something that can catch basic crashes in Electron/Chrome browsers could be a good start.

Cypress is far from perfect but I've been using it extensively in the past year on a few projects and it does the job most of the time. One thing to note is that it adds considerable amount of time to the pipeline checks, around 10 minutes. I intend to add a "retry" plugin that counters the flaky nature of tests like this, which may extend wait time further by re-running failing tests x times until fail.

If this still sounds reasonable, I'm gonna start by copying this set of tests and have them run on the example pages.

Future:

We can also use BrowserStack's engine (it's free to open source projects but someone has to send a request). They have a Selenium driver which I understand some people would like to avoid. Though I think it can still be given a chance to at least see if an example page explodes some browser immediately and/or after very simple input like typing text.

BrowserStack also has what seems to be JavaScript engine models, which may be something we could run our builds through to see if we're adding Android/iOS/IE regressions with new builds. Though I haven't tried this tool.

[gztomas]

The issue with Cypress is that it does not support native browser events yet. Which seems like a deal-breaker here given how Slate relies on beforeinput.
I have had a very good experience testing things built on top of Slate with the new playwright library.

[dmitrizzle]

Do you think this approach can be used to get a head start?

I don't have extensive experience setting up Cypress with Slate (the tests I built are for a much older version of Slate). But I have been using Cy extensively with other projects and feel that it could be helpful with catching some basic regression issues I see pop up on my feed.

[saschahofmann]

https://stackoverflow.com/a/61360140 suggests an approach which enabled our Cypress tests to work with Slate still a little bit hacky.

[marcusstenbeck]

I don’t like having to run a local server to run my tests. I ended up creating a few test helpers around Puppeteer and pptr-testing-library so I can write my tests like below.

The code in the render literal is injected as the root component of a react app straight onto the puppeteer page object. So no server needed.

test('onChange gives us the editor content', async () => {
  const onChange = jest.fn();

  const page = await render`<TextEditor onChange={${onChange}} />`;

  const $document = await page.getDocument();

  /**
   * Partial code from TextEditor.js to show `data-testid`
   * 
   * <Slate editor={editor} value={value} onChange={handleOnChange}>
   *   <Editable data-testid="text-editor" />
   * </Slate>
   */
  const $editor = await $document.getByTestId('text-editor');

  // type something
  await $editor.type('hello everyone');

  // blur the editor
  await page.focus('body');

  expect(onChange).toHaveBeenLastCalledWith([
    { children: [{ text: 'hello everyone' }], type: 'paragraph' },
  ]);
});

It could possibly work also for playwright since their API:s are very similar.

Here's the code: https://github.com/marcusstenbeck/puppeteer-react-testing-template/

[beorn]

Cypress seems to support beforeinput events since Oct 2020: cypress-io/cypress#8411 - https://docs.cypress.io/api/commands/type#Events

[silviubogan]

I have opened this related issue: #4589.

Does anyone know how can I use Jest & JSDOM with Slate at this time?