How to limit the max number of threads with sync endpoints?

[mat105]

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from concurrent.futures import ThreadPoolExecutor
import asyncio

loop = asyncio.get_running_loop()
loop.set_default_executor(ThreadPoolExecutor(max_workers=1))

import fastapi
import time
import threading

app = fastapi.FastAPI()

@app.get("/")
def sleep_func():
    print(threading.active_count())
    time.sleep(5)
    return "ok"

Description

I want to limit the ammount of threads spawned by the framework when using sync endpoints. On my PC i have 4 cores, and when i try the following code i get up to 40-41 threads spawned. Tried to limit the workers with the answers found here https://github.com/tiangolo/fastapi/issues/603 but had no luck, it always spawns up to 41.

I'm using the latest version of fastapi, uvicorn, etc.

• Run the app (provided in example code) with uvicorn app_file:app
• Run this script:

import aiohttp
import asyncio
import json
import os
from aiohttp import ClientSession

url = "http://localhost:8000"

async def make_call():
    async with ClientSession() as session:
        response = await session.request(method='GET', url=url)
    return "ok"

async def main():
    tasks = [make_call() for x in range(100)]
    await asyncio.gather(*tasks)

asyncio.run(main())

• See the output of the app.

How can i apply a limit like 4 concurrent threads?

Operating System

Linux

Operating System Details

Ubuntu 20.0.4

FastAPI Version

0.70.0

Python Version

3.8.10

Additional Context

Tried the same on Windows, but no limit is applied there, like if an infinite ammount is allowed.

[mat105]

FastApi docs talk about ThreadPoolExecutor being used under the hood, but i didn't find that in the implementation. Instead, i found anyio, and in order to change the thread limit anyio uses, i have to replace an internal value called _default_thread_limiter with a new CapacityLimiter (default limit is 40).

from anyio.lowlevel import RunVar
from anyio import CapacityLimiter

app = FastAPI()

@app.on_event("startup")
def startup():
    print("start")
    RunVar("_default_thread_limiter").set(CapacityLimiter(2))

[raphaelauv]

fastAPI is based on starlette that use to control the ThreadPoolExecutor , but starlette is now using anyio , so I don't see a better way than your proposition :

RunVar("_default_thread_limiter").set(CapacityLimiter(2))

[SDAravind]

Does this block the main thread?

[Laksh47]

We can also set the variable directly, AnyIO's docs

[murthy95]

Hi I am facing a weird issue that when I run pod on Kubernetes I can't spawn more than 14 threads even if I set the limit to 200. The other usages like cpu and men are well beyond the limits. Any idea where the issue is ?
@raphaelauv @mat105

[anton-daneyko-ultramarin]

I came here, cause I was looking for a way to limit the amount of simultaneously processed requests and in the end it seems there's a way outside of FastAPI: use --limit-concurrency of the uvicorn (see, https://www.uvicorn.org/settings/#resource-limits).

[yaohongfenglove]

姚宏锋已收到您的来信，谢谢。