feat: support test retries (#3968)

* add retries e2e test

* restore runner/test/eslintrc

* use mocha pass event, move runner.spec to runner package

* fix .eslintignore

* remove npmInstall logic in helper/e2e script, force custom reporters to use our mocha

* temp 04/09/20 [skip ci]

* add retries output to server/reporter, fix mocha pass event order, cleanup

* e2e tests - dont run electron tests in other browsers

* Update readme to reflect how to start server for cypress tests

* fix after merge

* fix .coffee -> .js after merge

* fix attempt.tsx

* fix runnable titles emitted to terminal reporters

* fix more tests: update snapshots, fix 7_record_spec, 8_reporters_spec

* remove styling for 'attempt-error-region' so it's not indented

- This was the older styling before error improvements and is no longer
necessary.

* try 2: fix rerun before/after hooks

* fix runner with only, runner snapshots, lint fixes

* temp 04/29/20 [skip ci]

* backport changes from test-retries

* change logic to rerun before hooks after top navigation

* fix windowSize for browser e2e test

* fix windowSize for xvfb chrome in e2e test

* ok fine, just disable screenshots

* fix after merge: decaffed navigation.js

* update server/unit test snapshots

* fix after merge: decaffed aliases.js

* fix usage of cypress --parallel flag in circle.yml

* fix circle.yml integration-tests jobs

* fix decaf related typo

* fix circle.yml separate command for runner-integration-tests

* update runner/integration tests + snapshot after error improvements

* fix runner/integration snapshots for chrome/ff stacktrace differences

* rerun ci

* fix passing --parallel to runner-integration tests

* perf: faster lookup for hooks without runnables

* fix afterAll hook switch logic

* simplify mocha prototype patches

* fix decaf utils.coffee after merge

* backport to before/after fix

* backport to before/after fix 2

* cleanup from decaf, fix ui/package.json

* update helpers, simplify runner.spec

* fix lint-types errors, flaky spec

* fix noExit passed to e2e test inline options

* cleanup snapshot utility - refactor to use util file

* remove before/after changes

* make cy obj a class instance

* cleanup/unmerge before/after fixes PR...

* more cleanup

* add comment

* fix runner.spec

* cleanup snapshot utility more, cleanup reporter.spec

* fix after merge

* minor rename variable

* fix after merge: decaffed files

* fix specName in reporterHeader, spec_helper require

* replace reporter specPath usages with spec object from config

* cleanup, fix specs, fix types tests

* fix config spec paths in isolated runner, fix snapshot plugin button

* combine runner.spec.js and runner_spec.js

* fix incorrect merge

* minor minor cleanup

* rename driver/test/cypress to driver/test

* use yarn workspace over lerna for individual package commands

* add error message to driver start

* remove usage of wait-on

* update <reference types/>, import string

* fix driver/readme

* fix readmes after regex replace

* revert wait-on changes

* Revert "revert wait-on changes"

This reverts commit 6de684c.

* update yarn.lock

* fix broken path in spec

* fix broken paths in specs with @packages/driver

* move runner/test/cypress into runner/cypress

* start server in pluginsFile in runner/cypress tests

* fix more broken spec paths

* fix broken paths after runner/cypress folder move

* move type definition loading for driver/cypress into dedicated file

* move internal-types to "types" folder, fix driver/index.d.ts

* fix type-check in packages/runner. not exactly sure why

* fix runner type-check by excluding test folder in tsconfig

* bump timeout on e2e/8_error_ui_spec

* update snapshot utility, rename tests in runner/runner.spec, fix README yarn commands

* delete old spec

* fix snapshot naming, remove redundant test in reporter_spec

* fix file renames after merge

* rename runner/ snapshot

* update server/unit/reporter_spec snapshot

* update runner/runner_spec snapshot

* rename runner snapshot file

* address feedback: move server reporter snapshot specs out

* address feedback: add comment about exposing globals

* fix test-retries after merging isolated-runner

* fix runner/test helper, update snapshot

* address feedback: split out runner/retries spec, move reporter/ui tests to runner/ui spec (mostly done), various cleanup

* fix scrolling, attempt opening, update snapshots

* fix e2e support file

* fix 5_spec_isolation

* fix mislabeling attempt screenshots

* only add test results prevAttempts if exists

* fix reporter/unit tests, server/unit tests

* remove dead code, fix test isOpen

* update snapshots for retries.mochaEvents, fix snapshot error in state hydration test, remove dead snapshots

* new moduleAPI schema using attempts array, fix wrapping errors from hook retries, update snapshots

* add displayError, null out fields in moduleAPI schema

* change default retries to {runMode:2, openMode:0}

* fix reporter type-check

* upgrade json-schemas, update snapshots

* reformat error.stack to be only stacktrace, update snapshots

* fix stacktrace replacing in 5_spec_isolation

* fix navigation error causing infinite reloading, bump timeout on e2e/8_error_ui

* fix server/unit tests for new schema

* fix reporter/unit tests

* fix reporting duplicate screenshots using cy.screenshot during test retry

* update snapshot for 6_uncaught_support_file_spec

* bump x-route-version: 3

* fix test.tsx collapsible content, css, fix e2e/8_error_ui, e2e projects excluding retries

* fix css, fix padding in runnable-instruments, fix runner/integration tests

* fixup after merge

* fix reporter/runner to work with split hooks

* update api tests, runner/cypress tests, reporter

* fix 5_spec_isolation snapshots, fix runner/cypress errors.spec, fix null reference in test.tsx

* fix e2e/non_root spec, fix type_check, fix reporter/unit tests

* setup percy snapshots in runner/cypress, fix driver/runner test:after:run event, add tests for only,skip in runner/cypress, fix retried css

* add customPercySnapshot

* fix circle.yml

* fix circle.yml 2

* fix circle.yml 3

* add warning for incompatible retries plugin

* add more percy snapshots

* fix firefox screenshot resolution in e2e test

* Fix testConfigOverrides not affecting viewport (#8006)

* finish adding percy snapshots to runner/cypress retries spec, update error msgs, add tests to be fixed

* remove .only

* fixing missing repo argument

* fix testConfigOverrides usage with retries, fix test

* fix issues from previous merge

* add script that can query CircleCI workflow status

* add circleci job to poll

* add retries

* try yarn lock

* retry, percy finalize

* check for current running job

* do not swallow request error

* better print

* use job name from circle environment

* use debug instead

* renamed circle polling script

* refactor circle to conditionally run percy-finalize when env var is available

- pass job-names to wait on as an argument

* use multi-line strings and quote --job-names

- rename —circle-jobs to —job-names

* add comment

* only poll until the jobs to wait for are blocked or running

* fix running hooks at correct depth after attempt fails from hook and will retry, update e2e snapshots

* fix reporter/unit tests, remove unused toggleOpen code

* move custom percy command into @packages/ui-components and apply them to desktop-gui

* halt percy finalize job if env variable is not set

* if only I could code

* update runner/cypress mochaEvent snapshots, fix e2e firefox resolution

* fix css for attempt border-left, fix attempt-tag open/close icon, add color to attempt collapsible dot

* try percy set viewport width

* set default retries back to {runMode:0, openMode:0}

* formatting: add backticks to warning message

* write explicit test for screenshot overwriting behavior, fix snapshots after changing retries defaults

* fix e2e.it.only`

* cleanup whitespace

* update snapshots

* fix cypress module API types for new result schema

* build and upload binary for test-retries branch too (linux)

* add pre-release PR comment

* fix pre-release commit comment

* rename runner/cypress test

* update retries.ui.spec test titles

* fix after merge: use most recent attempt for before/after hooks

* add suite title to hook error in runner/cypress tests

Co-authored-by: Jennifer Shehane <jennifer@cypress.io>
Co-authored-by: Brian Mann <brian.mann86@gmail.com>
Co-authored-by: Gleb Bahmutov <gleb.bahmutov@gmail.com>

.eslintrc.json

@@ -6,7 +6,8 @@
     "plugin:@cypress/dev/general"
   ],
   "rules": {
-    "prefer-spread": "off"
+    "prefer-spread": "off",
+    "prefer-rest-params": "off"
   },
   "settings": {
     "react": {

circle.yml

@@ -129,14 +129,22 @@ commands:
       browser:
         description: browser shortname to target
         type: string
+      percy:
+        description: enable percy
+        type: boolean
+        default: false
     steps:
       - attach_workspace:
           at: ~/
       - run:
           command: |
+            cmd=$([[ <<parameters.percy>> == 'true' ]] && echo 'yarn percy exec --') || true
+
             CYPRESS_KONFIG_ENV=production \
             CYPRESS_RECORD_KEY=$PACKAGES_RECORD_KEY \
-            yarn workspace @packages/runner cypress:run --record --parallel --group runner-integration-<<parameters.browser>> --browser <<parameters.browser>>
+            PERCY_PARALLEL_NONCE=$CIRCLE_WORKFLOW_ID \
+            PERCY_PARALLEL_TOTAL=-1 \
+            $cmd yarn workspace @packages/runner cypress:run --record --parallel --group runner-integration-<<parameters.browser>> --browser <<parameters.browser>>
       - store_test_results:
           path: /tmp/cypress
       - store_artifacts:
@@ -211,6 +219,7 @@ commands:
           command: |
             git clone --depth 1 --no-single-branch https://github.com/cypress-io/<<parameters.repo>>.git /tmp/<<parameters.repo>>
             cd /tmp/<<parameters.repo>> && (git checkout $NEXT_DEV_VERSION || true)
+
   test-binary-against-repo:
     description: |
       Takes the built binary and NPM package, clones given example repo
@@ -337,6 +346,17 @@ commands:
                 path: /tmp/<<parameters.repo>>/cypress/videos
       - store-npm-logs
 
+  wait-on-circle-jobs:
+    description: Polls certain Circle CI jobs until they finish
+    parameters:
+      job-names:
+        description: comma separated list of circle ci job names to wait for
+        type: string
+    steps:
+      - run:
+          name: "Waiting on Circle CI jobs: <<parameters.job-names>>"
+          command: node ./scripts/wait-on-circle-jobs.js --job-names="<<parameters.job-names>>"
+
 jobs:
   ## code checkout and yarn installs
   build:
@@ -400,6 +420,37 @@ jobs:
           command: node cli/bin/cypress info --dev
       - store-npm-logs
 
+  # a special job that keeps polling Circle and when all
+  # individual jobs are finished, it closes the Percy build
+  percy-finalize:
+    <<: *defaults
+    executor: cy-doc
+    parameters:
+      required_env_var:
+        type: env_var_name
+    steps:
+      - attach_workspace:
+          at: ~/
+      - run:
+          # if this is an external pull request, the environment variables
+          # are NOT set for security reasons, thus no need to poll -
+          # and no need to finalize Percy, since there will be no visual tests
+          name: Check if <<parameters.required_env_var>> is set
+          command: |
+            if [[ -v <<parameters.required_env_var>> ]]; then
+              echo "Internal PR, good to go"
+            else
+              echo "This is an external PR, cannot access other services"
+              circleci-agent step halt
+            fi
+      - wait-on-circle-jobs:
+          job-names: >
+            desktop-gui-integration-tests-2x,
+            desktop-gui-component-tests,
+            cli-visual-tests,
+            runner-integration-tests-chrome,
+      - run: npx percy finalize --all
+
   cli-visual-tests:
     <<: *defaults
     parallelism: 1
@@ -701,6 +752,7 @@ jobs:
     steps:
       - run-runner-integration-tests:
           browser: chrome
+          percy: true
 
   runner-integration-tests-firefox:
     <<: *defaults
@@ -879,13 +931,6 @@ jobs:
           command: node index.js
           working_directory: packages/launcher
 
-  percy-finalize:
-    <<: *defaults
-    steps:
-      - run:
-          name: "finalizes percy builds"
-          command: npx percy finalize --all
-
   build-binary:
     <<: *defaults
     shell: /bin/bash --login
@@ -1369,6 +1414,11 @@ jobs:
           command: npm run test:ci
           pull_request_id: 515
           folder: examples/fundamentals__typescript
+      - test-binary-against-repo:
+          repo: cypress-example-recipes
+          command: npm test
+          pull_request_id: 513
+          folder: examples/fundamentals__module-api-wrap
 
   "test-binary-against-kitchensink":
     <<: *defaults
@@ -1493,6 +1543,11 @@ linux-workflow: &linux-workflow
         name: Linux lint
         requires:
           - build
+    - percy-finalize:
+        context: test-runner:poll-circle-workflow
+        required_env_var: PERCY_TOKEN # skips job if not defined (external PR)
+        requires:
+          - build
     - lint-types:
         requires:
           - build
@@ -1624,12 +1679,6 @@ linux-workflow: &linux-workflow
         requires:
           - build
 
-    - percy-finalize:
-        requires:
-          - desktop-gui-integration-tests-2x
-          - desktop-gui-component-tests
-          - cli-visual-tests
-
     # various testing scenarios, like building full binary
     # and testing it on a real project
     - test-against-staging:
@@ -1666,6 +1715,7 @@ linux-workflow: &linux-workflow
           branches:
             only:
               - develop
+              - test-retries
         requires:
           - build-npm-package
     - build-binary:
@@ -1677,6 +1727,7 @@ linux-workflow: &linux-workflow
           branches:
             only:
               - develop
+              - test-retries
         requires:
           - build-binary
     - test-npm-module-on-minimum-node-version:
@@ -1733,6 +1784,7 @@ linux-workflow: &linux-workflow
           branches:
             only:
               - develop
+              - test-retries
         requires:
           - upload-npm-package
           - upload-binary

cli/schema/cypress.schema.json

@@ -243,6 +243,18 @@
       "type": "boolean",
       "default": false,
       "description": "Polyfills `window.fetch` to enable Network spying and stubbing"
+    },
+    "retries": {
+      "type": [
+        "object",
+        "number",
+        "null"
+      ],
+      "default": {
+        "runMode": 0,
+        "openMode": 0
+      },
+      "description": "The number of times to retry a failing. Can be configured to apply only in runMode or openMode"
     }
   }
 }

cli/types/cypress-npm-api.d.ts

@@ -7,6 +7,11 @@
 // but for now describe it as an ambient module
 
 declare namespace CypressCommandLine {
+  interface TestError {
+    name: string
+    message: string
+    stack: string
+  }
   /**
    * All options that one can pass to "cypress.run"
    * @see https://on.cypress.io/module-api#cypress-run
@@ -166,14 +171,16 @@ declare namespace CypressCommandLine {
     title: string[]
     state: string
     body: string
-    /**
-     * Error stack string if there is an error
-     */
-    stack: string | null
-    /**
-     * Error message if there is an error
+     /**
+     * Error string as it's presented in console if the test fails
      */
-    error: string | null
+    displayError: string | null
+    attempts: AttemptResult[]
+  }
+
+  interface AttemptResult {
+    state: string
+    error: TestError | null
     timings: any
     failedFromHookId: hookId | null
     wallClockStartedAt: dateTimeISO
@@ -199,6 +206,7 @@ declare namespace CypressCommandLine {
     name: string
     testId: testId
     takenAt: dateTimeISO
+    testAttemptIndex: number
     /**
      * Absolute path to the saved image
      */

cli/types/cypress.d.ts

@@ -334,6 +334,11 @@ declare namespace Cypress {
      */
     getFirefoxGcInterval(): number | null | undefined
 
+    /**
+     * @returns the number of test retries currently enabled for the run
+     */
+    getTestRetries(): number | null
+
     /**
      * Checks if a variable is a valid instance of `cy` or a `cy` chainable.
      *
@@ -2563,6 +2568,13 @@ declare namespace Cypress {
      * the `includeShadowDom` option to some DOM commands.
      */
     experimentalShadowDomSupport: boolean
+    /**
+     * Number of times to retry a failed test.
+     * If a number is set, tests will retry in both runMode and openMode.
+     * To enable test retries only in runMode, set e.g. `{ openMode: null, runMode: 2 }`
+     * @default null
+     */
+    retries: Nullable<number | {runMode: Nullable<number>, openMode: Nullable<number>}>
   }
 
   interface TestConfigOverrides extends Partial<Pick<ConfigOptions, 'baseUrl' | 'defaultCommandTimeout' | 'taskTimeout' | 'animationDistanceThreshold' | 'waitForAnimations' | 'viewportHeight' | 'viewportWidth' | 'requestTimeout' | 'execTimeout' | 'env' | 'responseTimeout'>> {

package.json

@@ -136,6 +136,7 @@
     "fs-extra": "8.1.0",
     "gift": "0.10.2",
     "globby": "10.0.1",
+    "got": "11.5.1",
     "gulp": "4.0.2",
     "gulp-awspublish": "4.0.0",
     "gulp-debug": "4.0.0",

packages/desktop-gui/cypress/support/index.js

@@ -1,4 +1,4 @@
-require('@percy/cypress')
+require('@packages/ui-components/cypress/support/customPercyCommand')
 require('cypress-react-unit-test/dist/hooks')
 
 const BluebirdPromise = require('bluebird')

packages/driver/cypress/integration/commands/screenshot_spec.js

@@ -19,6 +19,7 @@ describe('src/cy/commands/screenshot', () => {
       takenAt: new Date().toISOString(),
       name: 'name',
       blackout: ['.foo'],
+      testAttemptIndex: 0,
       duration: 100,
     }
 
@@ -49,7 +50,7 @@ describe('src/cy/commands/screenshot', () => {
 
       Cypress.action('runner:runnable:after:run:async', test, runnable)
       .then(() => {
-        expect(Cypress.action).not.to.be.calledWith('cy:test:set:state')
+        expect(Cypress.action).not.to.be.calledWith('test:set:state')
         expect(Cypress.automation).not.to.be.called
       })
       .finally(() => {
@@ -68,7 +69,7 @@ describe('src/cy/commands/screenshot', () => {
 
       Cypress.action('runner:runnable:after:run:async', test, runnable)
       .then(() => {
-        expect(Cypress.action).not.to.be.calledWith('cy:test:set:state')
+        expect(Cypress.action).not.to.be.calledWith('test:set:state')
         expect(Cypress.automation).not.to.be.called
       })
     })
@@ -89,7 +90,7 @@ describe('src/cy/commands/screenshot', () => {
 
       Cypress.action('runner:runnable:after:run:async', test, runnable)
       .then(() => {
-        expect(Cypress.action).not.to.be.calledWith('cy:test:set:state')
+        expect(Cypress.action).not.to.be.calledWith('test:set:state')
         expect(Cypress.automation).not.to.be.called
       })
     })
@@ -137,6 +138,7 @@ describe('src/cy/commands/screenshot', () => {
           waitForCommandSynchronization: true,
           disableTimersAndAnimations: true,
           blackout: [],
+          testAttemptIndex: 0,
         })
 
         expect(Cypress.action).to.be.calledWith('cy:after:screenshot', {
@@ -147,6 +149,7 @@ describe('src/cy/commands/screenshot', () => {
           waitForCommandSynchronization: true,
           disableTimersAndAnimations: true,
           blackout: [],
+          testAttemptIndex: 0,
         })
       })
     })
@@ -183,6 +186,7 @@ describe('src/cy/commands/screenshot', () => {
           testFailure: true,
           blackout: [],
           scaled: true,
+          testAttemptIndex: 0,
         })
       })
     })
@@ -225,6 +229,7 @@ describe('src/cy/commands/screenshot', () => {
             simple: false,
             scaled: true,
             blackout: [],
+            testAttemptIndex: 0,
           })
         })
       })
@@ -264,6 +269,7 @@ describe('src/cy/commands/screenshot', () => {
           testFailure: true,
           scaled: true,
           blackout: [],
+          testAttemptIndex: 0,
         })
       })
     })
@@ -406,6 +412,7 @@ describe('src/cy/commands/screenshot', () => {
             waitForCommandSynchronization: false,
             disableTimersAndAnimations: true,
             blackout: ['.foo'],
+            testAttemptIndex: 0,
           })
         })
       })
@@ -425,6 +432,7 @@ describe('src/cy/commands/screenshot', () => {
             waitForCommandSynchronization: false,
             disableTimersAndAnimations: true,
             blackout: ['.foo'],
+            testAttemptIndex: 0,
           })
         })
       })
@@ -446,6 +454,7 @@ describe('src/cy/commands/screenshot', () => {
             waitForCommandSynchronization: true,
             disableTimersAndAnimations: true,
             blackout: [],
+            testAttemptIndex: 0,
           })
         })
       })
@@ -466,6 +475,7 @@ describe('src/cy/commands/screenshot', () => {
             waitForCommandSynchronization: false,
             disableTimersAndAnimations: true,
             blackout: ['.foo'],
+            testAttemptIndex: 0,
           })
         })
       })