[Testing:Developer] General Cypress test improvements #7695
Conversation
Codecov Report
@@ Coverage Diff @@
## master #7695 +/- ##
=========================================
Coverage 21.82% 21.82%
Complexity 7490 7490
=========================================
Files 198 198
Lines 24423 24423
Branches 54 54
=========================================
Hits 5331 5331
Misses 19038 19038
Partials 54 54
Flags with carried forward coverage won't be shown. Click here to find out more. |
site/cypress/support/index.js
Outdated
|
|
||
| afterEach(() => { | ||
| cy.logout(true); | ||
| }); |
There was a problem hiding this comment.
What happens if someone creates a test which doesn't log in, or logs out as part of the test? Would this fail if the test doesn't end in a logged-in state? Additionally, does leaving sessions hanging cause any issues? If not, do we need to be this explicit in making sure that each and every test logs out properly?
There was a problem hiding this comment.
Leaving tests in a logged-in state works most of the time but can occasionally be flaky; see #7695 (comment) for more details.
I added in a list that lets certain tests be excluded from logging out, though it only excludes based on the full path of the current test title and not the file name itself (Cypress.spec reports differently when running all tests versus running each test individually so I don't think there's a way to check the filename without a hacky workaround, and I don't think there will be a case where there will be two tests with the same title paths).
There was a problem hiding this comment.
Could we make the logout function have a conditional (activated via parameter) such that it can check if we're logged in first before triggering the action, else do nothing? I think that might be a bit cleaner than using text ignores that could be easy to inadvertently break.
There was a problem hiding this comment.
Thanks for this, reran the e2e tests a couple times and the cypress portion seems stable.
The manual login and logout were previously added because there was an issue where the existing user would remain logged in between tests - this is supposed to be prevented by cypress clearing the cookie/session state between each one. Maybe something was fixed in either cypress or github actions or submitty? Either way good to know its working and should save time
|
@DefinitiveAbove |
…ad bug w/section selection
What is the current behavior?
Some of the course material tests in Cypress were very flaky, and the
visitcommand is missing a return leading to undefined behavior.What is the new behavior?
Added return for
visitcommand overwrite (https://docs.cypress.io/api/cypress-api/custom-commands#Overwrite-visit-command) - breaks some of the tests requiring some of the fixes listed below.Added
waitPageChangehelper command - waits for the page to change once to a new page (navigation/refresh) before continuing code execution - used in login/logout and multiple other locations where appropriate to avoid race conditions.Added
forceonto multiple timestamp-related field entries (tests would fail depending on screen size because the date picker would be covering the typing box).Added
logoutto afterEach for test cases in general (visitfix led to Cypress resetting cookies - signing out most of the time, but not always (some sort of bug having to do when the test before the current test fails, leading to Cypress not signing out)).Fixes focused around the course materials tests to decrease inconsistencies/race conditions, removed waits/reloads where unneeded after fixes.
Other information?
Test that doesn't run in CI is failing -
Pre-autograding test-Should show newly added autograding jobs(autograding_status_1.spec.js); not sure what the correct numbers are supposed to be since the numbers keep changing on my local instance.