Weird memory leak related to number of middlewares and python version

[zhm9484]

Description

Memory leak happens when using starlette. And seems like the leak depends on number of middlewares and python version.

• With Python 3.7, the leak happens when #middleware>=3.
• With Python 3.8, the leak happens when #middleware>=5.

FYI -- tiangolo/fastapi#4649 (comment)

How To Reproduce

Server Code

from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.routing import Route
from starlette.middleware.base import (
    BaseHTTPMiddleware,
    RequestResponseEndpoint,
)
from starlette.requests import Request
from starlette.responses import PlainTextResponse


class TestMiddleware(BaseHTTPMiddleware):

    async def dispatch(self, req: Request, call_next: RequestResponseEndpoint):
        return await call_next(req)


def ping(request):
    return PlainTextResponse("pong")


app = Starlette(
    routes=[Route("/_ping", endpoint=ping)],
    middleware=[Middleware(TestMiddleware)] * 5,
)


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="127.0.0.1", port=14000)

Client Code

import requests
import multiprocessing as mp

def do():
    while 1:
        rsp = requests.get("http://127.0.0.1:14000/_ping")
        print(rsp.status_code)

for _ in range(20):
    p = mp.Process(target=do, daemon=True)
    p.start()

import time
time.sleep(1000000)

Dockerfile

# modify to python:3.7.12 if testing python3.7
FROM python:3.8.13

RUN pip install fastapi==0.81.0 \
                uvicorn==0.18.3 \
                requests

WORKDIR /home/root/leak

COPY client.py client.py
COPY server.py server.py

Command

docker build -t leak-debug:latest -f Dockerfile .
docker run -it leak-debug:latest bash

# in container
nohup python server.py &
nohup python client.py &
top

[agronholm]

I can repro this with Starlette 0.19.1 to which FastAPI is currently pinned, and also on 0.20.0 but it's gone in 0.20.1.

[zhm9484]

Maybe #1609 solved it?

Could you point me out why the leak disappears after modifying those lines? I'm just curious.

[Kludex]

Is this solved?

[zhm9484]

Yes. I just downgraded starlette to 0.14.2.

[aksh-gupta]

@zhm9484 Which fastapi version you are using along with starlette 0.14.2?

[zhm9484]

@aksh-gupta
fastapi==0.68.2

[kulievvitaly]

Is it fixed in starlette 0.26.1?

[Kludex]

Can you reproduce it?

[icfly2]

yes, starlette 0.27.0 still has this behaviour...
on Python 3.10.11

[nirupbbnk]

is this resolved ? with python 3.11 and starlette 0.27.0 , we are able to reproduce when we use middleware

[agronholm]

Can you reproduce it with the current version of Starlette? And what middleware?

[nirupbbnk]

BaseHTTPMiddleware

[agronholm]

Ok, can you reproduce it with the latest Starlette too?

[nirupbbnk]

I am not able to use latest version of starlette , as fastapi is depended on it , it uses starlette version 0.27.0 evenwith latest fastapi version.https://github.com/tiangolo/fastapi/blob/master/pyproject.toml#L43

[agronholm]

So what do you want us to do about Starlette 0.27? Go back in time and fix any memory leaks before it was released?

[nirupbbnk]

If you could do , we would love that 😄, currently I see people have raised PR for starlette upgrade in fastapi repo , but it's not merged as there are issues

…

On Fri, 22 Dec 2023, 6:09 pm Alex Grönholm, ***@***.***> wrote: So what do you want us to do about Starlette 0.27? Go back in time and fix any memory leaks before it was released? — Reply to this email directly, view it on GitHub <#1843 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEKKGUK54CST3UBKJKYIGDYKV5PXAVCNFSM6AAAAAAQEIXDLKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TSMRXGQ2DE> . You are receiving this because you commented.Message ID: ***@***.***>

[agronholm]

Ok, so what is it that you want us (members of the Starlette project) to do about this? We've asked numerous people to show us that there is still a leak in the current release, but nobody has provided any proof.

[thomasleveil]

TL;DR: To prevent further emotional debate, I conducted tests demonstrating the impact of using a certain number of middlewares. These revealed no memory leak in Starlette 0.34.0 associated with middlewares.

---

Individuals with Docker can readily replicate these tests by cloning the repository at this repo.

The test suite, which is configurable, includes:

• Starlette versions 0.34.0 and 0.19.1
• Python versions 3.7, 3.12, and 3.13
• Uvicorn 0.25.0 for Python >3.7
• Uvicorn 0.22.0 for Python 3.7

The testing procedure involves making HTTP requests to an endpoint while measuring the maximum Resident Set Size (RSS) memory used at regular interval. Tests are conducted with varying numbers of middlewares to ascertain their impact on memory usage.

This suite aims to facilitate the resolution of ongoing discussions, potentially leading to enhancements in Starlette or affirming its current efficiency.

Conclusions

• The data shows no memory leak in Starlette 0.34.0, whatever the number of middlewares used.
• The data confirms the memory leak in Starlette 0.19.1 when used with python 3.7.17 which is related to the number of middlewares used.

Another notable observation is the performance decline with additional middlewares. For example, with Python 3.12 and Starlette 0.34.0, the system processes approximately 1,300 queries per second without middleware, but this drops by 33% with just one middleware.

---

Note: I'm not a specialist in performance or memory related issues. If you find anything to improve this test suite, pull requests are welcome.

[thomasleveil]

Hi @agronholm, thank you for this review. Indeed that work cannot prove anything regarding memory leaks. I updated the tests to take regular measurements of the Max RSS used so that it can be compared to the number of HTTP queries served ; and I'll update my message with the new findings.
Regarding the performance drop, do you feel it is worrying enough to justify a new discussion and test project ?

[agronholm]

I just had a look at the middleware you used and it seems that you added memory allocation there which is freed immediately. Does the performance degradation still occur when you remove that?

[thomasleveil]

I'll take a look later next week, familly will soon be here for the week-end. Everyone needs to take a brake from open-source from time to time ;)

[thomasleveil]

@agronholm here is a test project for testing performances vs middlewares. Would you be able to tell if those results are worth putting any effort on the matter ? Middleware implementation is empty.

[agronholm]

I'm not sure what's the expected result. I should point out with an endpoint that simple, I would expect a significant amount of time to be spent in the middleware. It might be worth comparing to other ASGI frameworks (Litestar etc.) to see if the same thing causes significant performance degradation there.

[Kludex]

This whole discussion is about the BaseHTTPMiddleware, not "middlewares". The wording here is just not right. There are limitations on the BaseHTTPMiddleware, that we have been discussing for a long time.

If you want more performance, use a Pure ASGI Middleware. The BaseHTTPMiddleware provides an easy API, but there's a cost.

I do not wish to engage in another BaseHTTPMiddleware discussion, please look at the previous issues and discussions, there are plenty.

[nirupbbnk]

The issue is resolved in starlette 0.32.0 , fastapi has also upgraded the starlette to 0.32 in the release 0.108.0, Seems issue is solved now

[sany2k8]

No, I don't think it's solved, I am still facing the memory leak issue.

My Environment

Operating system/Architecture=Linux/X86_64
Python=3.10.9
Fastapi=0.108.0
Uvicorn=0.17.6
Starlette=0.32.0.post1
Pydantic=2.6.2

[insomnes]

I've found that anyio thread workers are not clean up until next call, and they hold the result of sync handlers (though I don't understand how). So on steady request rate with default executor settings (there are limit for 40 workers) it can eat a lot of mem and do not release it, if your handlers produce big chunk of data.
Cause worker cleanups happens only on request processing, and if worker is idle for longer than 10 seconds.

[agronholm]

Each worker thread is shut down automatically once it gets no more work in 10 seconds, so this would not cause a memory leak.

[insomnes]

You are not right, it only shuts down if there were new requests after he was idle for 10 seconds. If you stop sending request it would not free the memory until new request comes.

[agronholm]

Yeah, because the pruning is only done by another call to to the to_thread.run_sync(), right. But I fail to see your point still – we're still not talking about a memory leak here, as the number of worker threads is limited by a capacity limiter.

[insomnes]

Yeah, that's not leak, you are right, but it leaves huge memory foot print on high request rate in comparison to not sync handlers, but that could not be the root cause of rising memory, only may lead to earlier OOM in combo with leak. Will dig more on the leak.

[agronholm]

Huge memory footprint? In comparison to what?