Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky system-tests-firefox job #18644

Closed
flotwig opened this issue Oct 26, 2021 · 1 comment · Fixed by #18848
Closed

Fix flaky system-tests-firefox job #18644

flotwig opened this issue Oct 26, 2021 · 1 comment · Fixed by #18848
Assignees
Labels
process: flaky test Related to test(s) that have flake in our internal tests

Comments

@flotwig
Copy link
Contributor

flotwig commented Oct 26, 2021

system-tests-firefox seems to fail frequently on snapshot matches when it takes longer than normal to launch Firefox:

https://app.circleci.com/pipelines/github/cypress-io/cypress/25418/workflows/c6f39a26-4ae6-4c34-a562-730b7d5e9001/jobs/953049

Two ways to fix it:

  1. Support some env var like CYPRESS_CDP_CONNECTION_TIMEOUT in protocol.ts and pass it in system-tests.ts
  2. Filter out the noisy stdout in system-tests.ts like we do for video snapshot errors in FF:
    try {
    if (options.originalTitle) {
    systemTests.snapshot(options.originalTitle, str, { allowSharedSnapshot: true })
    } else {
    systemTests.snapshot(str)
    }
    } catch (err) {
    // firefox has issues with recording video. for now, ignore snapshot diffs that only differ in this error.
    // @see https://github.com/cypress-io/cypress/pull/16731
    if (!(options.browser === 'firefox' && isVideoSnapshotError(err))) {
    throw err
    }
    console.log('(system tests warning) Firefox failed to process the video, but this is being ignored due to known issues with video processing in Firefox.')
    }
@flotwig flotwig added the process: flaky test Related to test(s) that have flake in our internal tests label Oct 26, 2021
@cypress-bot cypress-bot bot added stage: needs review The PR code is done & tested, needs review stage: work in progress and removed stage: work in progress stage: needs review The PR code is done & tested, needs review labels Nov 9, 2021
@cypress-bot
Copy link
Contributor

cypress-bot bot commented Nov 23, 2021

Released in 9.1.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v9.1.0, please open a new issue.

@cypress-bot cypress-bot bot removed the stage: needs review The PR code is done & tested, needs review label Nov 23, 2021
@cypress-bot cypress-bot bot locked as resolved and limited conversation to collaborators Nov 23, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
process: flaky test Related to test(s) that have flake in our internal tests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants