Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid random failures in system tests #5319

Draft
wants to merge 24 commits into
base: main
Choose a base branch
from
Draft

Conversation

jorg-vr
Copy link
Contributor

@jorg-vr jorg-vr commented Jan 22, 2024

This pull request is an effort to avoid occasional failures in the system tests. The global solution was increasing the maximal wait time for finding elements on a page to 30 seconds.
This does not slow down succeeding test. It will slowdown failing tests.

There were multiple points of failure encountered while working on this pr:

This finally led me to conclude that the issues were not test specific.

This pr also has multiple minor changes to help avoid these random failures:

  1. For the incorrect count:
    • annotation and questions tests work on different submissions. This avoids potential conflicts when tests run in simultaneously
    • Use the question count per submission
    • Wait for selector annotation-text count to change before checking backend change. This should avoid a timing issues if the frontend changes has not yet propagated to the backend.
  2. For the code tab not found: (all these changes were reverted)
    • The tests now use a json exercise with the normal feedback table renderer. The assumption here was that the code tab rendering might have infrequent errors for the code tab.
    • The code tab is clicked reference by id instead of by text. The idea was that a text search for the button may sometimes fails due to the number tag also in the button
    • The maximum waiting time for finding the code tab has been increased to 5 seconds (maybe the page is slow to load)
    • I have added a sanity check to first find the body element and the page wrapper before searching for the code tab. I added this to debug, not as a fix, but it is the only thing that worked as a fix.
  3. Papyros system test fails infrequently #4562 (all these changes were reverted)
    • Increase wait time for scratchpad test. This test deserves some additional worst case wait time for the setup of webworker.

I have tested this pr by rerunning the tests 10 times after each change.

Notes

I found it very odd, that although it varies in which test a failure happens, there is always only a single failing test.
Increasing the number of tests, by duplicating one of the failing tests 40 times, did not seem to increase the frequency of failures. (It took me 10 tries to reproduce a failure with the 40 added tests)

When I started working on this issue I mostly encountered the 'code tab not found' failure.
When that error stopped occurring I started seeing the 'question count value incorrect' failure.
Now that that that seems fixed, the #4562 failure reappeared after not having seen it for weeks.
Increasing the allowed waiting time for that test, let another totally unrelated test fail...

Except for the incorrect count the issue is always the same. An item that is present on the page is not found.
Always only a single test failing

@jorg-vr jorg-vr added the bug Something isn't working label Jan 22, 2024
@jorg-vr jorg-vr self-assigned this Jan 22, 2024
@jorg-vr jorg-vr changed the title Avoid race conditions in tests Avoid system tests impacting eachother Jan 22, 2024
@pdawyndt pdawyndt changed the title Avoid system tests impacting eachother Avoid system tests impacting each other Jan 22, 2024
@jorg-vr jorg-vr changed the title Avoid system tests impacting each other Avoid random failures in system tests Jan 29, 2024
@bmesuere bmesuere removed the bug Something isn't working label Feb 13, 2024
@bmesuere bmesuere added the chore Repository/build/dependency maintenance label Feb 13, 2024
@jorg-vr jorg-vr added the on hold Awaiting analysis on how to proceed label Feb 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore Repository/build/dependency maintenance on hold Awaiting analysis on how to proceed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants