Possible race condition leading to a connection reset if worker is gracefully terminating

[rbagd]

We have encountered a relatively rare connection error which is probably due to a race condition as uvicorn worker is trying to shutdown.

Here is the setup

• Synchronous FastAPI endpoints
• uvicorn worker
• gunicorn with --max-requests for regularly restarting workers

I can reproduce it with Python 3.11, both uvloop and asyncio, but couldn't reproduce with asyncio and Python 3.12.

To reproduce I launch below app as

gunicorn app:app -w 2 -k uvicorn.workers.UvicornWorker --max-requests 5 --keep-alive 60

# app.py
import time
import random

from fastapi import FastAPI

app = FastAPI()

@app.post("/")
def test():
    time.sleep(random.uniform(0, 0.1))
    return {"status": "ok"}

Then I stress test the application with many concurrent users constantly hitting the API.

# hit.py
import requests
import random
import time

for _ in range(10000):
    requests.post("http://localhost:8000", json={})
    time.sleep(random.uniform(0.5, 1))

for i in {0..20}; do (python hit.py)& done

After some waiting I eventually hit Connection reset by peer errors.

I did some initial investigation into what is happening. Here's a tcpdump for one of these errors which I tried to correlate with some events in the code. It always happens around the time when max-requests is reached and worker is shutting down.

CONNECTION MADE: 2024-04-25T12:11:57.946005 ('127.0.0.1', 43826)     # end of connection_made in httptools_impl.py
SHUTD MAIN LOOP: 2024-04-25T12:11:57.946031                          # just before await self.shutdown(sockets=sockets) in server.py
CONNECTION SHUT: 2024-04-25T12:11:57.946105 ('127.0.0.1', 43826)     # just prior to connection.shutdown() in server.py
CLOSE TRANSPORT: 2024-04-25T12:11:57.946108                          # just before self.transport.close() in httptools_impl.py
CONNECTION LOST: 2024-04-25T12:11:57.946135 ('127.0.0.1', 43826)     # start of connection_lost in httptools_impl.py

It seems that in certain cases worker doesn't shutdown gracefully despite data having just arrived in TCP stack.
After some deep-dive I noticed that every time that the error happens it is true that self.cycle=None in HttpToolsProtocol.shutdown() whenever the error is triggered, and if I am correct the reverse is true as well. It seems to me that adding a block

if self.cycle is None:
    return

into httptools_impl.py or h11_impl.py solves the issue but I am not really sure what this means.