Get a useful signal out of tests, minimize the noise

• Manning Twitch stream Feb 23, 2022
• Slides

Learn the difference between signal and noise when it comes to tests - it might not be what you expect! We'll walk through some examples of noisy passing tests, and noisy flakey tests, and talk about why retrying flakey tests will actually increase the noise in your tests.

Outline

1. Define signal versus noise
2. Describe how to handle test noise
3. Show examples of passing tests:
   1. Test that passes (success + signal) test_add_item_multiple
   2. Test that passes but shouldn’t (success but should fail) test_get_most_recent
4. Fix the test that passes but shouldn't
5. Show examples of failing tests:
   1. Failure that provides new information test_submit_orders (logic broken), test_get_total_orders (test broken)
   2. Failure that doesn’t provide new information (tests that are left broken, flakes)
6. Fix the failing tests
7. Describe flakey tests
8. Show example of a flakey test test_process_order
9. Fix the flakey test
   1. Using retry on the entire function @retry.retry(retries=3)
   2. Using retry on the connect function only retry_network_errors(mrf.connect, retries=3)
   3. Update the connect function itself to backoff and retry (using backoff lib)
10. If extra time: explore some real life flakey tests