[tracker] [tests] Python test suite reliability: striving for non-flaky, parallelizable, random-orderable tests.

[jayaddison]

This is an umbrella issue to track progress making the Sphinx test suite reliable. That means:

• No flaky tests.
• Tests can run in parallel.
• Tests are independent of each other.

Regarding the last item: test independence is important to ensure that each test is checking the behaviours it claims to, and does not inadvertently rely on the side-effects of some other test. Similarly, a test must not begin failing if any other test is run before it -- if it does, that could mean (but does not definitely mean) that the application's behaviour itself may vary based on the order in which code is evaluated.

It might seem like there are a large number of items here! However, please bear in mind that we have a total of more than 2000 individual pytest test cases that run.

Test flakiness

• test_build_linkcheck.test_connect_to_selfsigned_fails (pr: [tests] Increase timeouts for linkcheck build tests. #12166)
• test_build_linkcheck.test_defaults (pr: [tests] Increase timeouts for linkcheck build tests. #12166)
• test_build_linkcheck.test_linkcheck_exclude_documents (pr: [tests] Attempt to uncover and resolve test flakiness in 'test_linkcheck_exclude_documents'. #12189)
• ... report another?

Test parallelization

• tests.test_builders.test_build_linkcheck (pr: [tests] linkcheck: bind each test HTTP server to a unique port per-testcase. #12126)
• ... report another?

Test independence

• tests.test_builders.test_build_text.py (pr: none)
• tests.test_domains.test_domain_cpp (pr: [test] fix side-effect due to repeated fields #12187)
• tests.test_builders.test_build_html (pr: [tests] Make the 'test_html_copy_source' case agnostic of test evaluation order. #12118)
• tests.test_builders.test_build_html_download (pr: Remove test result-sharing between 'test_html5_output' and 'test_html5_download' #12119)
• ... report another?