Server parts of websocket tests are not affecting test outcome

[stefanw]

Checklist

• The bug is reproducible against the latest release and/or master.
• There are no similar issues or pull requests to fix it yet.

Describe the bug

While researching #757 I noticed that pytest assertions in server parts of websocket tests (with run_server(...)) are not working as intended in some cases.

Assertion failures are displayed (pytest -s) but do not affect the test run outcome. I suspect the reason lies in running the server in a loop in a thread and neither exceptions nor pytest failures are propagating to the main thread.

For example the pytest.raises in test_send_after_protocol_close is silently ignored.

To reproduce

I added an assert False in the server part of a test in #759 and all tests are passing.

Expected behavior

I expect the tests to fail when using assert False or pytest.fail("...") in the server parts of websocket tests.

Actual behavior

Exception or failures are printed to standard out, but do not affect the overall test outcome.

Debugging material

See test run for #759

Environment

• OS / Python / Uvicorn version: Running uvicorn 0.11.8 with CPython 3.8.2 on Darwin

Additional context

I suspect that the server running in a loop in a thread is not propagating exceptions or pytest failures.

[Kludex]

We are on uvicorn 0.20.0, and by now we don't run the server in a thread anymore. We run it as an async task, but we still have the same behavior.

[Kludex]

I think this is fine. When the request-response cycle fails in the server, we can have an exception, but that will only be valid for that cycle, and we can't actually retrieve that exception, unless we mock some object.

We need to be mindful that assertions need to happen outside the server on the test suite. 🙏