RuntimeError("No response returned") in BaseHTTPMiddleware

[jonathanslenders]

Previously, a "No response returned"-error has been discussed in this discussion: #1527, and was supposedly fixed in this PR: #1715

However, the bug is still present. It's a race condition, so quite tough to reproduce, but the following reproducer works.

Reproducer

Use anyio 4.2.0 It does happen also with anyio 3.0.0, but it happens much more regularly on 4.2.

First, apply the following patch to the Starlette codebase, so that we have a breakpoint in case it happens:

--- a/starlette/middleware/base.py
+++ b/starlette/middleware/base.py
@@ -134,7 +134,8 @@ class BaseHTTPMiddleware:
             async def send_no_error(message: Message) -> None:
                 try:
@@ -163,6 +164,7 @@ class BaseHTTPMiddleware:
             except anyio.EndOfStream:
                 if app_exc is not None:
                     raise app_exc
+                breakpoint()
                 raise RuntimeError("No response returned.")

             assert message["type"] == "http.response.start"

Then run this code:

import asyncio
from hypercorn.asyncio import serve
from hypercorn.config import Config

from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response
from starlette.middleware.base import RequestResponseEndpoint
from starlette.middleware import Middleware


class DummyMiddleware(BaseHTTPMiddleware):
    async def dispatch(
        self, request: Request, call_next: RequestResponseEndpoint
    ) -> Response:
        return await call_next(request)


async def homepage(request):
    return JSONResponse({})


app = Starlette(
    routes=[
        Route("/", homepage),
    ],
    middleware=[Middleware(DummyMiddleware) for i in range(100)],  # Yes, 100x the same middleware.
)

config = Config.from_mapping({})
config.bind = ["127.0.0.1:8000"]
asyncio.run(serve(app, config))

Open either Firefox or Chrome, navigate to http://localhost:8000 and hold down command-r (refresh) to repeatedly have the browser refresh the page. After a couple of seconds, we'll enter the breakpoint.

(Feel free to turn this issue into a discussion if that's more appropriate. I'm also looking for a fix but have a hard time understanding BaseHTTPMiddleware.)

edit: I updated the above snippet and added the missing imports.

This PR contains a possible fix: #2519

Important

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[mikkelduif]

versions:

python=3.11
starlette=0.37.1
uvicorn=0.27.1

@jonathanslenders this also happens for me, and seems to also be due to client disconnects.
it started to happen heavily after an upgrade of the Starlette version (previously we never got this exception).
I have just checked our upgrade, and that was from 0.27.0 to 0.36.3 - so quite a few versions though.

I have modified your example a bit, and interestingly in my example it consistently works when you have 3x middleware, and breaks when you have +4x middlewares.

import asyncio

from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response
from starlette.routing import Route


class DummyMiddleware(BaseHTTPMiddleware):
    def __init__(self, **kwargs) -> None:
        self._version = kwargs.pop("version")
        super().__init__(**kwargs)

    async def dispatch(self, request: Request, call_next) -> Response:
        print(self._version, id(self), "STARTED")
        try:
            res = await call_next(request)
            print(self._version, id(self), "COMPLETED")
            return res
        except Exception as exc:
            print(self._version, id(self), "FAILED", exc)
            return Response(b"")


async def sleepy(request: Request):
    sleep_ms = int(request.query_params.get("sleep", 200))
    await asyncio.sleep(sleep_ms / 1000)
    return Response(b"")


app = Starlette(
    routes=[Route("/", sleepy)],
    middleware=[Middleware(DummyMiddleware, version=_ + 1) for _ in range(4)],
)

Serve the app with Uvicorn

uvicorn app:app --reload

Make a CURL request where you can modify the sleep time that asyncio will run inside the app and the corresponding max execution time CURL will allow -m 0.2

curl http://127.0.0.1:8000?sleep=250 -m 0.2

OUTPUT

1 4392769744 STARTED
2 4392769616 STARTED
3 4392769232 STARTED
4 4392768208 STARTED
4 4392768208 COMPLETED
3 4392769232 COMPLETED
2 4392769616 COMPLETED
1 4392769744 FAILED No response returned.

Changing number of middlewares to 3

1 4368290960 STARTED
2 4368290768 STARTED
3 4368290576 STARTED
3 4368290576 COMPLETED
2 4368290768 COMPLETED
1 4368290960 COMPLETED

• It seems to make no difference whether you place the sleep inside the middleware or inside the endpoint function
• It seems like it consistently works with 3 middlewares and consistently breaks with +4 middlewares.
• It seems like the middlewares are initialised in the order they are passed, but resolved in the reverse way, it's consistently the first middleware passed that will also fail, though the other middlewares will pass through correctly before
• When using 3 middlewares, it seems to also not be possible to provoce the error with a combination of different sleep times

After some further debugging, this seems to appear exactly from starlette>=0.28.0 and seems to could be in relation to introduction of the _CachedRequest. I'ld have to dive a bit deeper into understanding it more specifically.
However, I have tried with combinations of anyio and that doesn't seem to impact anything, only whenever I downgrade to starlette 0.27.0 it works regardsless of anyio version.

Not surprisingly this change in Starlette would not cause the issue anymore, though I don't imagine this is actually the solution:

- message = await wrap(wrapped_receive)
+ message = await wrap(receive)

@Kludex would you have any clue why it happens at >=4 middlewares but not <=3? Or does it just seem to be a coincidence that this behaviour happens?

[XieJiSS]

Hi, seems like we encountered a similar issue here.

We are actually using fastapi and at first we suspect that we are facing tiangolo/fastapi#8187. However we noticed that the hang can only be reproduced when more than two people keep refreshing the page together. So, that looks more like a race condition, i.e. probably not related to tiangolo/fastapi#8187, which is a stream double-consume issue and hence should be reproducible by a single user's refresh, unconditionally, which is not our case.

Our starlette version is 0.36, and when two users are refreshing the page at the speed of ~one time per 2s, we can repro the hang issue in ~30 secs.

[mithaler]

👋🏻 We think we're seeing this too, with 5 middlewares, on:

fastapi==0.109.2
starlette==0.36.3
anyio==4.3.0