Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore(driver): fix integration test retry configuration #18643

Merged
merged 24 commits into from
Nov 9, 2021
Merged

chore(driver): fix integration test retry configuration #18643

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
d243a15
test(driver): fix integration test retry configuration
emilyrohrbough Oct 26, 2021
973f354
add assetions
emilyrohrbough Oct 26, 2021
f24ec80
Merge branch 'develop' into fix-driver-retries
emilyrohrbough Oct 26, 2021
66c2fd8
move debug code to running test.
emilyrohrbough Oct 26, 2021
2eb8ef1
Merge branch 'develop' into fix-driver-retries
emilyrohrbough Oct 28, 2021
e1cdde4
Merge branch 'develop' into fix-driver-retries
emilyrohrbough Oct 29, 2021
891567a
pr feedback
emilyrohrbough Oct 29, 2021
15ffd2f
add log context when failures occur
emilyrohrbough Nov 2, 2021
f741ecb
revert change
emilyrohrbough Nov 2, 2021
55e70e1
.
emilyrohrbough Nov 2, 2021
f987fac
try increasing retry just to see
emilyrohrbough Nov 3, 2021
14cc8a6
Update navigation_spec.js
emilyrohrbough Nov 4, 2021
f6bb5df
fix this test
emilyrohrbough Nov 4, 2021
358b844
for fun, lets try these flaky tests
emilyrohrbough Nov 4, 2021
717c8dc
Merge branch 'develop' into fix-driver-retries
emilyrohrbough Nov 4, 2021
5edf0ee
Update navigation_spec.js
emilyrohrbough Nov 4, 2021
e89ab54
lint
emilyrohrbough Nov 4, 2021
a9d0d81
lint
emilyrohrbough Nov 5, 2021
37b8892
Merge branch 'develop' into fix-driver-retries
emilyrohrbough Nov 8, 2021
952dda3
Merge branch 'develop' into fix-driver-retries
emilyrohrbough Nov 9, 2021
c1f9ae1
Merge branch 'develop' into fix-driver-retries
emilyrohrbough Nov 9, 2021
6c25eb5
Apply suggestions from code review
emilyrohrbough Nov 9, 2021
e19d176
don't timeout passing tests
flotwig Nov 9, 2021
11f5c14
Merge branch 'develop' into fix-driver-retries
emilyrohrbough Nov 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions packages/driver/cypress.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,9 @@
"reporter": "cypress-multi-reporters",
"reporterOptions": {
"configFile": "../../mocha-reporter-config.json"
},
"retries": {
"runMode": 2,
"openMode": 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this will work because of isInteractive being forced to true

}
}
10 changes: 10 additions & 0 deletions packages/driver/cypress/integration/commands/navigation_spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,13 @@ const helpers = require('../../support/helpers')
const { _, Promise, $ } = Cypress

describe('src/cy/commands/navigation', () => {
before(() => {
expect(Cypress.config('retries')).to.deep.eq({
runMode: 2,
openMode: 0,
})
})

context('#reload', () => {
before(() => {
cy
Expand Down Expand Up @@ -163,6 +170,8 @@ describe('src/cy/commands/navigation', () => {
})

it('throws passing 2 invalid arguments', { defaultCommandTimeout: 200, retries: 1 }, (done) => {
expect(Cypress.config('retries')).to.eq(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testing the value of Cypress.getTestRetries() may be good to add somewhere


cy.on('fail', (err) => {
expect(err.message).to.eq('`cy.reload()` can only accept a boolean or `options` as its arguments.')
expect(err.docsUrl).to.eq('https://on.cypress.io/reload')
Expand All @@ -174,6 +183,7 @@ describe('src/cy/commands/navigation', () => {
})

it('throws passing 1 invalid argument', (done) => {
expect(Cypress.config('retries')).to.deep.eq({ runMode: 2, openMode: 0 })
cy.on('fail', (err) => {
expect(err.message).to.eq('`cy.reload()` can only accept a boolean or `options` as its arguments.')
expect(err.docsUrl).to.eq('https://on.cypress.io/reload')
Expand Down
19 changes: 10 additions & 9 deletions packages/driver/cypress/integration/e2e/redirects_spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,14 @@ describe('redirection', () => {
.visit('/fixtures/meta-redirect-timeout.html')
.contains('timeout')
.then(function () {
// visit, contains, page load, new url
expect(this.logs.length).to.eq(4)
// visit, contains, page load, new url
const receivedLogs = this.logs.reduce((prev, curr, index) => `${prev}, ${index}: ${curr.get('name')}`, '')

expect(this.logs.length).to.eq(4, `received more logs than expected: ${receivedLogs}`)

expect(this.logs[0].get('name')).to.eq('visit')
expect(this.logs[1].get('name')).to.eq('contains')
expect(this.logs[2].get('name')).to.eq('page load')

expect(this.logs[3].get('name')).to.eq('new url')
})
})
Expand All @@ -31,13 +32,14 @@ describe('redirection', () => {
.visit('/fixtures/meta-redirect.html')
.get('a:first')
.then(function () {
// visit, get, page load, new url
expect(this.logs.length).to.eq(4)
// visit, get, page load, new url
const receivedLogs = this.logs.reduce((prev, curr, index) => `${prev}, ${index}: ${curr.get('name')}`, '')

expect(this.logs.length).to.eq(4, `received more logs than expected: ${receivedLogs}`)

expect(this.logs[0].get('name')).to.eq('visit')
expect(this.logs[1].get('name')).to.eq('get')
expect(this.logs[2].get('name')).to.eq('page load')

expect(this.logs[3].get('name')).to.eq('new url')
})
})
Expand All @@ -50,13 +52,12 @@ describe('redirection', () => {
.visit('/fixtures/js-redirect-timeout.html')
.contains('timeout')
.then(function () {
// visit, contains, page load, new url
// visit, contains, page load, new url
expect(this.logs.length).to.eq(4)

expect(this.logs[0].get('name')).to.eq('visit')
expect(this.logs[1].get('name')).to.eq('contains')
expect(this.logs[2].get('name')).to.eq('page load')

expect(this.logs[3].get('name')).to.eq('new url')
})
})
Expand All @@ -66,7 +67,7 @@ describe('redirection', () => {
.visit('/fixtures/js-redirect.html')
.get('a:first')
.then(function () {
// visit, get, page load, new url
// visit, get, page load, new url
expect(this.logs.length).to.eq(4)

expect(this.logs[0].get('name')).to.eq('visit')
Expand Down
14 changes: 4 additions & 10 deletions packages/driver/cypress/support/defaults.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,11 @@
const { $ } = Cypress

let isActuallyInteractive

isActuallyInteractive = Cypress.config('isInteractive')
if (!isActuallyInteractive) {
// we want to only enable retries in runMode
// and because we set `isInteractive` above
// we have to set retries here
Cypress.config('retries', 2)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was here as a workaround to the fact that we set isInteractive = true on line 19.

Cypress's getTestRetries switches on isInteractive to decide how many retries to use:

cypress/packages/driver/src/cypress.ts

Lines 147 to 159 in 2eb8ef1

this.getTestRetries = function () {
const testRetries = this.config('retries')
if (_.isNumber(testRetries)) {
return testRetries
}
if (_.isObject(testRetries)) {
return testRetries[this.config('isInteractive') ? 'openMode' : 'runMode']
}
return null
}

Locally, it was returning 0 for me in this PR, which makes sense since this workaround was removed.

before(() => {
expect(Cypress.config('retries')).to.deep.eq({ runMode: 2, openMode: 0 })
})

beforeEach(() => {
isActuallyInteractive = Cypress.config('isInteractive')
const isActuallyInteractive = Cypress.config('isInteractive')

// always set that we're interactive so we
// get consistent passes and failures when running
Expand Down
3 changes: 2 additions & 1 deletion packages/driver/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
"cypress:open": "node ../../scripts/cypress open",
"cypress:run": "node ../../scripts/cypress run",
"postinstall": "patch-package",
"start": "node -e 'console.log(require(`chalk`).red(`\nError:\n\tRunning \\`yarn start\\` is no longer needed for driver/cypress tests.\n\tWe now automatically spawn the server in the pluginsFile.\n\tChanges to the server will be watched and reloaded automatically.`))'"
"start": "node -e 'console.log(require(`chalk`).red(`\nError:\n\tRunning \\`yarn start\\` is no longer needed for driver/cypress tests.\n\tWe now automatically spawn the server in the pluginsFile.\n\tChanges to the server will be watched and reloaded automatically.`))'",
"test-integration": "yarn cypress:run"
},
"dependencies": {},
"devDependencies": {
Expand Down