Repo: End-to-end tests are flaky (yet again) #5763
Labels
accepting prs
Go ahead, send a pull request that resolves this issue
repo maintenance
things to do with maintenance of the repo, and not with code/docs
tests
anything to do with testing
Suggestion
馃檭 I hate this 馃檭
Example failure: https://github.com/typescript-eslint/typescript-eslint/actions/runs/3201746626/jobs/5230076005 in #5762:
These failures are happening pretty consistently now. They're making the end-to-end tests a useless signal for PR changes.
My primary theory is that we're not properly waiting for the Docusaurus server (
yarn start
) before running tests. Maybe the timeout is N seconds and the server often takes N+1 or more seconds to start.My second theory is that this isn't the sole fault of any of the testing dependencies in our
package.json
s, but rather GitHub Actions having a flaky network layer. And maybe one of those dependencies (Cypress?) not handling GitHub Actions well.At this point, my steps to try would be:
For each, if test pass on multiple re-runs, hooray! We're fixed!
Historical context: we have a long trail of going back and forth on test flake and Cypress versions. Previously we opted to pin the Cypress version to 8.3.0 because newer versions were flaky. I would hope that we can upgrade Cypress to the latest eventually. If the issue ends up being our waiting for the Docusaurus server, I'd still want to try bumping Cypress.
The text was updated successfully, but these errors were encountered: