Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repo: End-to-end tests are flaky (yet again) #5763

Closed
JoshuaKGoldberg opened this issue Oct 7, 2022 · 0 comments 路 Fixed by #5814
Closed

Repo: End-to-end tests are flaky (yet again) #5763

JoshuaKGoldberg opened this issue Oct 7, 2022 · 0 comments 路 Fixed by #5814
Labels
accepting prs Go ahead, send a pull request that resolves this issue repo maintenance things to do with maintenance of the repo, and not with code/docs tests anything to do with testing

Comments

@JoshuaKGoldberg
Copy link
Member

JoshuaKGoldberg commented Oct 7, 2022

Suggestion

馃檭 I hate this 馃檭

Example failure: https://github.com/typescript-eslint/typescript-eslint/actions/runs/3201746626/jobs/5230076005 in #5762:

  1) Index
       has no accessibility issues detected by aXe:
     CypressError: `cy.visit()` failed trying to load:

http://localhost:3000/

We attempted to make an http request to this URL but the request failed without a response.

We received this error at the network level:

  > Error: ESOCKETTIMEDOUT

These failures are happening pretty consistently now. They're making the end-to-end tests a useless signal for PR changes.

My primary theory is that we're not properly waiting for the Docusaurus server (yarn start) before running tests. Maybe the timeout is N seconds and the server often takes N+1 or more seconds to start.

My second theory is that this isn't the sole fault of any of the testing dependencies in our package.jsons, but rather GitHub Actions having a flaky network layer. And maybe one of those dependencies (Cypress?) not handling GitHub Actions well.

At this point, my steps to try would be:

  1. Bumping Cypress to the latest version again
  2. If we can identify how to, fixing Docusaurus server start-and-wait-ing
  3. Switching to Playwright

For each, if test pass on multiple re-runs, hooray! We're fixed!

Historical context: we have a long trail of going back and forth on test flake and Cypress versions. Previously we opted to pin the Cypress version to 8.3.0 because newer versions were flaky. I would hope that we can upgrade Cypress to the latest eventually. If the issue ends up being our waiting for the Docusaurus server, I'd still want to try bumping Cypress.

  1. chore: Bump cypress from 8.3.0 to 9.5.2聽#4710
  2. fix: revert "chore: Bump cypress from 8.3.0 to 9.5.2", and ignore cypress for now聽#4740
  3. Website tests flake: ESOCKETTIMEDOUT聽#4739
  4. chore: downgraded Cypress to 8.3.0 again聽#5006
  5. test: fix Cypress test flakes聽#5086
@JoshuaKGoldberg JoshuaKGoldberg added triage Waiting for maintainers to take a look repo maintenance things to do with maintenance of the repo, and not with code/docs accepting prs Go ahead, send a pull request that resolves this issue tests anything to do with testing and removed triage Waiting for maintainers to take a look labels Oct 7, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 13, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
accepting prs Go ahead, send a pull request that resolves this issue repo maintenance things to do with maintenance of the repo, and not with code/docs tests anything to do with testing
Projects
None yet
1 participant