chore(driver): fix integration test retry configuration #18643

emilyrohrbough · 2021-10-26T14:38:05Z

Test flakiness in packages/driver has been observed and tests have not been retried as expected. This change enables the retry configuration in the integration tests via the cypress configuration.

cypress-bot · 2021-10-26T14:38:08Z

Thanks for taking the time to open a PR!

Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

cypress · 2021-10-26T14:56:47Z

Test summary

452 • 0 • 6 • 0 • 0

Run details


Project	cypress
Status	Passed
Commit	`11f5c14`
Started	Nov 9, 2021 9:31 PM
Ended	Nov 9, 2021 9:40 PM
Duration	09:49 💡
OS	Linux Debian - 10.9
Browser	Electron 93

View run in Cypress Dashboard ➡️

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

flotwig

I can't observe this actually retrying like it's supposed to. I see flaky driver tests still in CI. Locally, I tried to validate these changes by adding this spec to packages/driver and using yarn cypress:run:

let i = 0; let j = 0; let k = 0
const N = 2

it('fails twice then succeeds', () => {
  if (i++ < N) throw new Error(`failed ${i}`)
})

it('fails twice then succeeds', () => {
  if (j++ < N) throw new Error(`failed ${j}`)
})

it('fails twice then succeeds', () => {
  if (k++ < N) throw new Error(`failed ${k}`)
})

It does retry the first test one time with these changes, but it does not retry twice like it's supposed to:

Also, none of the subsequent test failures are retried.

Changing N to 1 makes the first test pass (since it retries once) but the next 2 still fail and are not retried:

flotwig · 2021-10-29T13:48:49Z

packages/driver/cypress/support/defaults.js

-let isActuallyInteractive
-
-isActuallyInteractive = Cypress.config('isInteractive')
-if (!isActuallyInteractive) {
-  // we want to only enable retries in runMode
-  // and because we set `isInteractive` above
-  // we have to set retries here
-  Cypress.config('retries', 2)
-}


This was here as a workaround to the fact that we set isInteractive = true on line 19.

Cypress's getTestRetries switches on isInteractive to decide how many retries to use:

cypress/packages/driver/src/cypress.ts

Lines 147 to 159 in 2eb8ef1

this.getTestRetries = function () {

const testRetries = this.config('retries')

if (_.isNumber(testRetries)) {

return testRetries

}

if (_.isObject(testRetries)) {

return testRetries[this.config('isInteractive') ? 'openMode' : 'runMode']

}

return null

}

Locally, it was returning 0 for me in this PR, which makes sense since this workaround was removed.

flotwig · 2021-10-29T13:49:18Z

packages/driver/cypress.json

+  },
+  "retries": {
+    "runMode": 2,
+    "openMode": 0


I don't think this will work because of isInteractive being forced to true

flotwig · 2021-10-29T13:58:54Z

packages/driver/cypress/integration/commands/navigation_spec.js

@@ -163,6 +170,8 @@ describe('src/cy/commands/navigation', () => {
      })

      it('throws passing 2 invalid arguments', { defaultCommandTimeout: 200, retries: 1 }, (done) => {
+        expect(Cypress.config('retries')).to.eq(1)


testing the value of Cypress.getTestRetries() may be good to add somewhere

emilyrohrbough · 2021-11-08T21:36:45Z

Looking at the failures / flakey tests listed in the last Cypress comment, retries have been enabled.

emilyrohrbough · 2021-11-08T22:10:05Z

packages/driver/cypress/support/defaults.js

@@ -1,8 +1,7 @@
 const { $ } = Cypress

-let isActuallyInteractive
+const isActuallyInteractive = Cypress.config('isInteractive')


This is the change to enable retries

emilyrohrbough · 2021-11-08T22:10:28Z

packages/driver/cypress/support/utils.js

@@ -66,6 +66,12 @@ export const shouldBeCalledWithCount = (num) => (stub) => wrapped(stub).should('

 export const shouldNotBeCalled = (stub) => wrapped(stub).should('not.be.called')

+export const assertLogLength = (logs, expectedLength) => {


Helper to log the list of logs received to help debug when log length is different than expected.

packages/driver/cypress/integration/commands/navigation_spec.js

flotwig

My supplied test case from earlier works, but some stuff is still not retrying: https://app.circleci.com/pipelines/github/cypress-io/cypress/26345/workflows/825f5705-1c96-4ff5-b0d4-f43234e78612/jobs/994369/parallel-runs/1

There's no evidence of retrying on that failure in the logs.

You can observe this locally running yarn cypress:run --headed --no-exit, some tests will fail and not be retried, and some will be retried. Not sure what's happening, there's no per-test config in this file...

emilyrohrbough · 2021-11-09T18:07:08Z

@flotwig You linked to the CircleCI logs. If you look in the Cypress Dashboard results you will see retries are working.

I had a similar concern about the logs not showing a retry & when testing locally with the following example, I was able to observe retries when running yarn cypress:run --headed --not-exit & mocha did not log information to indicate a retry had occurred.

You can observe this locally running yarn cypress:run --headed --no-exit, some tests will fail and not be retried, and some will be retried. Not sure what's happening, there's no per-test config in this file...

I am currently running yarn workspace @packages/driver cypress:run --headed and have yet to observed this behavior , but there are a lot of tests here & haven't made it through the cypress/integration/commands/actions yet. It'll take while to get through them locally. Can you share which specs (or the video recording) where you observed this so I can dig in deeper?

Co-authored-by: Matt Henkes <mjhenkes@gmail.com>

flotwig · 2021-11-09T18:37:19Z

@flotwig You linked to the CircleCI logs. If you look in the Cypress Dashboard results you will see retries are working.

Some tests are retrying, some tests aren't retrying. That test code does work now, but some real specs in assertions_spec are still not retrying.

You can observe this locally running yarn cypress:run --headed --no-exit, some tests will fail and not be retried, and some will be retried. Not sure what's happening, there's no per-test config in this file...

I am currently running yarn workspace @packages/driver cypress:run --headed and have yet to observed this behavior , but there are a lot of tests here & haven't made it through the cypress/integration/commands/actions yet. It'll take while to get through them locally. Can you share which specs (or the video recording) where you observed this so I can dig in deeper?

Observe the results of yarn cypress:run --headed --no-exit --spec **/assertions_spec.js (to run the spec with the test that's not retrying in CI):

Peek.2021-11-09.13-33.mp4

Earlier tests in the file do retry, so I'm not sure what causes this yet. But, it's not retrying.

emilyrohrbough · 2021-11-09T19:15:59Z

@flotwig Thanks for the example. I'll take a look.

emilyrohrbough · 2021-11-09T20:32:42Z

packages/driver/src/cypress/mocha.ts

+        // user does an asynchronous `done`, so double-check
+        // that the test has not already passed before timing out
+        return
+      }


For anyone curious to reproduce this issue, you can comment this out and runt his test to consistently see the error shown in the video Zach linked:

it('should not be true', { defaultCommandTimeout: 1 }, (done) => { cy.on('fail', (err) => { expect(err.message).to.eq('expected false to be true') done() }) cy.noop(false).should('be.true') })

* develop: (329 commits) chore: Update Chrome (stable) to 96.0.4664.45 (#18931) fix: Loading of specs with % in the filename (#18877) chore: refactor `create` into class `$Cy` (#18715) chore: Update Chrome (beta) to 96.0.4664.45 (#18891) fix: flaky `system-tests-firefox` job (#18848) release 9.0.0 feat: ensure major release have conduit app wait on localhost:3000 fix install-required-node use --legacy-peer-deps feat: ensure major release fix darwin node install chore(driver): fix integration test retry configuration (#18643) feat(deps): update dependency electron to v15 🌟 (#18317) chore: Bind this correctly when setting response headers with cy.route() (#18859) feat: create config package for config validation (#18589) chore: patch `winston` to suppress `padLevels` warning (#18824) chore: test out major release build fix: remove outdated npm registry links (#18727) fix: Adding an existing command with `Cypress.Commands.add()` will throw an error (#18587) ...

test(driver): fix integration test retry configuration

d243a15

emilyrohrbough requested a review from a team as a code owner October 26, 2021 14:38

emilyrohrbough requested review from jennifer-shehane and removed request for a team October 26, 2021 14:38

emilyrohrbough marked this pull request as draft October 26, 2021 15:03

jennifer-shehane removed their request for review October 26, 2021 15:09

emilyrohrbough and others added 3 commits October 26, 2021 10:57

add assetions

973f354

Merge branch 'develop' into fix-driver-retries

f24ec80

move debug code to running test.

66c2fd8

emilyrohrbough marked this pull request as ready for review October 28, 2021 15:31

Merge branch 'develop' into fix-driver-retries

2eb8ef1

flotwig assigned emilyrohrbough Oct 28, 2021

flotwig requested review from flotwig and mjhenkes October 28, 2021 15:36

mjhenkes previously approved these changes Oct 29, 2021

View reviewed changes

Merge branch 'develop' into fix-driver-retries

e1cdde4

flotwig suggested changes Oct 29, 2021

View reviewed changes

flotwig reviewed Oct 29, 2021

View reviewed changes

pr feedback

891567a

emilyrohrbough dismissed mjhenkes’s stale review via 891567a October 29, 2021 15:17

BlueWinds requested review from flotwig and mjhenkes October 29, 2021 15:36

emilyrohrbough marked this pull request as draft October 29, 2021 16:03

emilyrohrbough and others added 5 commits November 2, 2021 12:21

add log context when failures occur

15ffd2f

revert change

f741ecb

.

55e70e1

try increasing retry just to see

f987fac

Update navigation_spec.js

14cc8a6

emilyrohrbough and others added 7 commits November 4, 2021 15:12

fix this test

f6bb5df

for fun, lets try these flaky tests

358b844

Merge branch 'develop' into fix-driver-retries

717c8dc

Update navigation_spec.js

5edf0ee

lint

e89ab54

lint

a9d0d81

Merge branch 'develop' into fix-driver-retries

37b8892

emilyrohrbough marked this pull request as ready for review November 8, 2021 21:36

emilyrohrbough commented Nov 8, 2021

View reviewed changes

mjhenkes reviewed Nov 9, 2021

View reviewed changes

packages/driver/cypress/integration/commands/navigation_spec.js Outdated Show resolved Hide resolved

Merge branch 'develop' into fix-driver-retries

952dda3

mjhenkes reviewed Nov 9, 2021

View reviewed changes

packages/driver/cypress/integration/commands/navigation_spec.js Outdated Show resolved Hide resolved

packages/driver/cypress/integration/commands/navigation_spec.js Outdated Show resolved Hide resolved

flotwig suggested changes Nov 9, 2021

View reviewed changes

emilyrohrbough and others added 2 commits November 9, 2021 12:07

Merge branch 'develop' into fix-driver-retries

c1f9ae1

Apply suggestions from code review

6c25eb5

Co-authored-by: Matt Henkes <mjhenkes@gmail.com>

don't timeout passing tests

e19d176

flotwig approved these changes Nov 9, 2021

View reviewed changes

emilyrohrbough commented Nov 9, 2021

View reviewed changes

Merge branch 'develop' into fix-driver-retries

11f5c14

chrisbreiding approved these changes Nov 9, 2021

View reviewed changes

chrisbreiding changed the title ~~test(driver): fix integration test retry configuration~~ chore(driver): fix integration test retry configuration Nov 9, 2021

chrisbreiding merged commit a5cf5c0 into develop Nov 9, 2021

chrisbreiding deleted the fix-driver-retries branch November 9, 2021 21:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(driver): fix integration test retry configuration #18643

chore(driver): fix integration test retry configuration #18643

emilyrohrbough commented Oct 26, 2021

cypress-bot bot commented Oct 26, 2021

cypress bot commented Oct 26, 2021 •

edited

flotwig left a comment •

edited

flotwig Oct 29, 2021

flotwig Oct 29, 2021

flotwig Oct 29, 2021

emilyrohrbough commented Nov 8, 2021

emilyrohrbough Nov 8, 2021

emilyrohrbough Nov 8, 2021

flotwig left a comment

emilyrohrbough commented Nov 9, 2021

flotwig commented Nov 9, 2021

emilyrohrbough commented Nov 9, 2021

emilyrohrbough Nov 9, 2021

	this.getTestRetries = function () {
	const testRetries = this.config('retries')

	if (_.isNumber(testRetries)) {
	return testRetries
	}

	if (_.isObject(testRetries)) {
	return testRetries[this.config('isInteractive') ? 'openMode' : 'runMode']
	}

	return null
	}

		@@ -66,6 +66,12 @@ export const shouldBeCalledWithCount = (num) => (stub) => wrapped(stub).should('

		export const shouldNotBeCalled = (stub) => wrapped(stub).should('not.be.called')

		export const assertLogLength = (logs, expectedLength) => {

chore(driver): fix integration test retry configuration #18643

chore(driver): fix integration test retry configuration #18643

Conversation

emilyrohrbough commented Oct 26, 2021

cypress-bot bot commented Oct 26, 2021

cypress bot commented Oct 26, 2021 • edited

Test summary

Run details

flotwig left a comment • edited

Choose a reason for hiding this comment

flotwig Oct 29, 2021

Choose a reason for hiding this comment

flotwig Oct 29, 2021

Choose a reason for hiding this comment

flotwig Oct 29, 2021

Choose a reason for hiding this comment

emilyrohrbough commented Nov 8, 2021

emilyrohrbough Nov 8, 2021

Choose a reason for hiding this comment

emilyrohrbough Nov 8, 2021

Choose a reason for hiding this comment

flotwig left a comment

Choose a reason for hiding this comment

emilyrohrbough commented Nov 9, 2021

flotwig commented Nov 9, 2021

emilyrohrbough commented Nov 9, 2021

emilyrohrbough Nov 9, 2021

Choose a reason for hiding this comment

cypress bot commented Oct 26, 2021 •

edited

flotwig left a comment •

edited