system-tests

@tooling/system-tests

This package contains Cypress's suite of system tests.

These tests launch the Cypress server process for each test and run different specs and projects under specific environment conditions, to get tests that can be as close to "real world" as possible.

These tests run in CI in Electron, Chrome, and Firefox under the system-tests job family.

Running System Tests

yarn test <path/to/test>
yarn test test/async_timeouts_spec.js
## or
yarn test async_timeouts ## shorthand, uses globbing to find spec

To keep the browser open after a spec run (for easier debugging and iterating on specs), you can pass the --no-exit flag to the test command. Live reloading due to spec changes should also work:

yarn test test/go_spec.js --browser chrome --no-exit

To debug the Cypress process under test, you can pass --cypress-inspect-brk:

yarn test test/go_spec.js --browser chrome --no-exit

Developing Tests

System tests cover the entire Cypress run, so they are good for testing features that do not fit into a normal integration or unit test. However, they do take more resources to run, so consider carefully if you really need to write a system test, or if you could achieve 100% coverage via an integration or unit test instead.

There are two parts to a system test:

1. A test written using the systemTests Mocha wrapper that lives in ./test, and
2. A matching Cypress test project that lives in the ./projects directory.

For example, if you initialized a new project in ./projects/my-new-project, and you wanted to assert that 2 tests fail and take a snapshot of the stdout, you'd write a test like this:

// ./test/my-new-project.spec.ts
import systemTests from '../lib/system-tests'
import Fixtures from '../lib/fixtures'

describe('my new project', () => {
    // scaffold projects
    systemTests.setup()

    systemTests.it('fails as expected', {
        project: 'my-new-project',
        snapshot: true,
        spec: '*',
        expectedExitCode: 2
    })
})

From here, you could run this test with yarn test my-new-project.

There are many more options available for systemTests.it and systemTests.setup. You can massage the stdout, do pre-run tasks, set up HTTP/S servers, and more. Explore the typedocs in ./lib/system-tests for more information.

These tests run in the system-tests-* CI jobs.

Developing Docker-based tests against built binary

Specs in the ./test directory are run against an unbuilt Cypress App. They don't test cypress NPM package installation or other prod app behavior. This is done so that they can run as fast as possible in CI, without waiting for a full build of the Cypress App.

Specs in ./test-binary are run against the built Cypress App. They also run inside of their own Docker containers to give a blank slate environment for Cypress to run in. Before each test, the prod CLI is npm installed along with the built Cypress .zip, and real cypress run commands are used to run the tests. There should be no functional difference between running a project in these tests and running real prod Cypress inside of Docker in CI.

The purpose of these tests is to test things that we normally can't inside of regular system-tests, such as testing Cypress with different Node versions, with/without Xvfb, or inside of different operating system versions.

An example of using dockerImage and withBinary to write a binary system test:

// ./test-binary/node-versions.spec.ts
import systemTests from '../lib/system-tests'
import Fixtures from '../lib/fixtures'

describe('node versions', () => {
    systemTests.it('runs in node 12', {
        dockerImage: 'cypress:node/12',
        project: 'todos',
        withBinary: true,
    })
})

Running yarn test node-versions would spin up a local Docker container for cypress:node/12, install Cypress from ../cypress.zip and ../cli/build, and then call the regular cypress run command within the container. Other options for systemTests.it such as onRun and expectedExitCode still function normally.

These tests run in the binary-system-tests CI job.

Updating Snaphots

Prepend SNAPSHOT_UPDATE=1 to any test command. See snap-shot-it instructions for more info.

SNAPSHOT_UPDATE=1 yarn test go_spec

Test Projects

Every folder in ./projects represents a self-contained Cypress project. When you pass the project property to systemTests.it or systemTests.exec, Cypress launches using this project.

If a test project has a package.json file, the systemTests.exec helper will attempt to install the correct node_modules by running yarn install against the project. This is cached in CI and locally to speed up test times.

systemTests.exec copies the project directory to a temporary folder outside of the monorepo root. This means that temporary projects will not inherit the node_modules from this package or the monorepo. So, you must add the dependencies required for your project in dependencies or devDependencies.

The exception is some commonly used packages that are scaffolded for all projects, like lodash and debug. You can see the list by looking at scaffoldCommonNodeModules in ./lib/fixtures.ts These packages do not need to be added to a test project's package.json.

You can also set special properties in a test project's package.json to influence the helper's behavior when running yarn:

package.json Property Name	Type	Description
_cySkipYarnInstall	boolean	If true, skip the automatic yarn install for this package, even though it has a package.json.
_cyYarnV311	boolean	Run the yarn v3.1.1-style install command instead of yarn v1-style.
_cyRunScripts	boolean	By default, the automatic yarn install will not run postinstall scripts. This option, if set, will cause postinstall scripts to run for this project.

Run yarn projects:yarn:install to run yarn install for all projects with a package.json.

Use the UPDATE_YARN_LOCK=1 environment variable with yarn test or yarn projects:yarn:install to allow the yarn.lock to be updated and synced back to the monorepo from the temp dir.