RFC: E2E testing on the React template

[jaredpalmer]

Current Behavior

(React) e2e testing must be setup by hand.

Desired Behavior

TSDX has an e2e testing solution for React libraries.

Suggested Solutions

Add to or make a new react template

• We add cypress to the react project.
• We point cypress by default at the example on localhost:1234
• Or, we make another folder called like app or fixture or something (that's identical to the example) but specifically for e2e tests. I just did this in Formik and it's a really really nice development experience.
• Add e2e github action that runs cypress

New command or flag
We abstract the above steps behind tsdx e2e command. This is nice because we then we wouldn’t install a dep. however, will still leak into code as cypress is based on mocha/chai under the hood (and not jest). The describe and it API’s are the same, but cypress basically has its own custom assertion engine at this point.

Use Jest + Playwright or Puppeteer
I haven’t looked at the difference between what Cypress (without the dashboard) does vs. puppeteer or playwright in a while. The beauty of using Puppeteer or Playwright is that we wouldn’t actually need to do anything because they are just node.js packages. would work out of the box with our existing jest setup. Only thing we might have to do is potentially write a GitHub action for the e2e test that has the proper binary installed.

Who does this impact? Who is this for?

React users. Even if you don't do anything other than use Cypress to check that your that your React library actually renders I still think its worth it. It also helps people fall into a pit of success with testing instead of overdoing unit tests.

Describe alternatives you've considered

Between this and storybook and size limit and github actions, I think it may be time to consider some kind lazy initialization of features. Perhaps a new command for tsdx init storybook tsdx init cypress where these run in a way that evaluates the current project and morphs it accordingly.

Additional context

Formik Cypress PR: jaredpalmer/formik@af425b4
TSDX Monorepo Example PR: jaredpalmer/tsdx-monorepo@4402358

[bahmutov]

@jaredpalmer related question - is there a way to expose the bundler you make for bundling code is TSDX? Then we could point cypress-react-unit-test at that bundler and be able to do component testing directly without spinning example playground.

[jaredpalmer]

You mean expose the build and watch commands in a node.js API that can be used by scripts?

I don't see why we can't, all we would need to do is abstract the current commands into functions and export them. Right @agilgur5?

[agilgur5]

@jaredpalmer part of the rationale for #407 is also to eventually expose a programmatic API (which is also something microbundle has had for a bit).

But I don't think that's what @bahmutov had in mind however. Looking at cypress-react-unit-test's CRA integration, it doesn't run react-scripts build directly as a preprocessor, it runs the webpack preprocessor with some configuration for CRA.
I'm also not sure there would be much difference in exposing that vs just using the dist/ directory and running tsdx watch in a separate terminal.

@bahmutov could you please clarify and be specific as to what you mean? TSDX uses Rollup and has a fairly complex config there and some code on top of it (that I've mentioned a lot I'd like to rewrite them as Rollup plugins, but there's a ways to go get there)

[agilgur5]

Template option

Add to or make a new react template
[...]

• We point cypress by default at the example on localhost:1234

Would prefer using the example directory as opposed to a new directory -- this ensures you're testing what you're advertising.

Command option

New command or flag
We abstract the above steps behind tsdx e2e command. This is nice because we then we wouldn’t install a dep.

sade has subcommands, so one could use tsdx test e2e instead. But I'm not a fan of adding more commands to core, as I've said elsewhere, this drastically increases the API surface and makes a breaking change in one command / dependency a breaking change to all of TSDX. So if Cypress makes a breaking change and TSDX upgrades, TSDX must make a breaking change. As it already has to do for Prettier, ESLint, Jest, Rollup, etc. This makes breaking changes more frequent, which is not a good DX.

Some TSDX users also don't like the bulkiness and inability to opt-out of certain pieces either, so adding another large dependency makes that more complicated. Have mentioned elsewhere that I'd much prefer to support things like @tsdx/cypress (or @tsdx/mocha, etc). Also supporting apps has been a frequent ask as well (e.g. #654)

however, will still leak into code as cypress is based on mocha/chai under the hood (and not jest). The describe and it API’s are the same, but cypress basically has its own custom assertion engine at this point.

Yea, Cypress is its own test runner and has its own assertions. I'm not a huge fan of Cypress because of this and a large configuration surface with directories, plugins, preprocessors, config, etc (which goes against TSDX's purpose), which I've mentioned in agilgur5/physijs-webpack#16 (comment). (Also an instrumentation issue I filed cypress-io/code-coverage#102 was never responded to, though neither was one in Jest jestjs/jest#9233)

Alternative e2e automation frameworks

Use Jest + Playwright or Puppeteer
I haven’t looked at the difference between what Cypress (without the dashboard) does vs. puppeteer or playwright in a while.

Also without the test runner and assertion bit of Cypress.
Previously the main difference between Cypress and Puppeteer was the API, usage, and configuration (including plugins, etc). Utilities like puppeteer-recorder / daydream and jest-puppeteer among many others were more prevalent with Puppeteer. At least that's what it was in my experience. @bahmutov would obviously be able to speak to it more.

Unlike Puppeteer, Playwright is cross-browser. Cypress was not previously cross-browser, but seems to now support a few browsers. Playwright is also a pretty new player too and I'm guessing used internally at Microsoft.

I wouldn't suggest using Puppeteer because of its lack of cross-browser support. Libraries by definition should support at least as many browsers as the apps that use them. If those run cross-browser tests, so should the libraries.

would work out of the box with our existing jest setup

That's why I would prefer it as well, though that's not entirely accurate. jest-playwright specifically recommends a separate jest.e2e.config.js. tsdx test supports the --config flag (as of #526) that could be used for that, however I'm not sure if there's anything that may need changing in the TSDX config to support this.

Karma and @jest-runner/electron

I'd be remiss if I didn't mention karma and @jest-runner/electron that both let you run your existing tests in a browser, instead of running browser tests with browser events. Can think @jest-runner/electron like a replacement for JSDOM in that sense -- I've also experienced it running as fast as JSDOM.

Both can have their uses, but if your goal is to "just render" or to run unit tests in a browser, it is much faster to get started with @jest-runner/electron than start using something like Cypress or even Playwright.

Unfortunately, when I last checked there was still no way of running unit tests in browsers other than Electron with Jest (@jest-runner/electron itself was experimental). Karma is also its own test runner (but not assertions) and somewhat legacy, with no Jest integration. In recent months though karma-jest was released, but it's not a Jest runner for Karma, it's a Karma plugin for Jest. Though theoretically one could import your Jest config into it with jest: require('./jest.config.js') (though that would require #634 / #270 )

It also helps people fall into a pit of success with testing instead of overdoing unit tests.

In my experience, I've much more frequently seen too many e2e tests (over unit tests) or e2e tests that should really be unit tests as opposed to the opposite. Test Pyramid metaphor is a thing after all, though lack of testing is a much bigger problem in general. TSDX itself has issues with this (another reason for #407 ).

Lazy Template Init

Between this and storybook and size limit and github actions, I think it may be time to consider some kind lazy initialization of features.

Yes that duplicates #253 which I've been looking to implement at some point (though tsdx add sounds like a better name) and have each piece be an independent mrm plugin or preset as well that doesn't necessarily need to be installed with TSDX (since things like, e.g. size-limit don't rely on it).

[tony]

It would be nice to make jest dependencies and stuff optional and have support for mocha

(On projects already depending on it)