getting task attached to a different loop runtime error when using fastapi with gunicron and motor

[haseeb5i]

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# mongo
from motor import motor_asyncio

mongo_uri = "mongodb://mongoUser:***/?authSource=admin"
mongo_client = motor_asyncio.AsyncIOMotorClient(mongo_uri)

def get_users_db() -> motor_asyncio.AsyncIOMotorDatabase:
    return mongo_client.devdb

# user_routes
import fastapi
import mongo
from bson import objectid

db = mongo.get_users_db()
router = fastapi.APIRouter()


@router.get("/{user_id}")
async def get_user_by_id(user_id: str):
    user = await db.users.find_one({"_id": objectid.ObjectId(user_id)})
    if user:
        user["_id"] = str(user["_id"])
        return user
    return {"error": "no user found"}

# main.py
import fastapi
import user_routes

app = fastapi.FastAPI()

app.include_router(user_routes.router)

Description

i am trying to create an app with mongodb client motor and fastapi. I created seperate routes and a seperate file to setup mongodb client. Running this with uvicorn works fine and i get results after querying the database but when i deploy the app with gunicorn and again use the route, i get the following error.

I tried bunch of methods, like assigning a loop to motor client, changing uvworker to use asyncio loop etc. i don't know where to post this error but here i am

• expected: json of user with given id from database or no user found error
• got: Runtime error

Operating System

Linux

Operating System Details

Debian 11

FastAPI Version

0.68.1

Python Version

Python 3.9.2

Additional Context

motor==2.2.6
gunicorn==20.1

this issue only happens with gunicorn as server and only when accessing mongodb from a fastapi route (i checked running db query in the mongo file and it works)

this my first ever time creating an issue, so please guide me i did something wrong.

[frankie567]

This is a quite common issue when using Motor (unfortunately, their implementation doesn't always play well with some event loops patterns).

I think the problem comes from here:

def get_users_db() -> motor_asyncio.AsyncIOMotorDatabase:
    return mongo_client.devdb

I'm not really sure about Motor implements this underneath, but I guess it will give you different instances for the same database ; which may be attached to a different event loop. Maybe try something like this:

from motor import motor_asyncio

mongo_uri = "mongodb://mongoUser:***/?authSource=admin"
mongo_client = motor_asyncio.AsyncIOMotorClient(mongo_uri)
users_db = mongo_client.devdb

def get_users_db() -> motor_asyncio.AsyncIOMotorDatabase:
    return users_db

[DavidBord]

You should init Motor client as a startup event

[haseeb5i]

thanks @frankie567 and @DavidBord for taking a look into this problem.

solution by @frankie567 produces the same error, that is motor gets attached to a different loop than fastapi.
However if I init motor client in fastapi startup event, everyting works fine as pointed by @DavidBord.

This problem appears to be in newer version of either motor, gunicorn or fastapi as initializing motor outside startup was working fine prevesouly, also it works if I use uvicorn as a server instead of gunicorn (i don't know which one is better, i use gunicorn only because that is suggested in fastapi and uvicorn docs)

This might not be the place to ask, but can you suggest how to handle refrence to mongodb client or db when we init motor in app's startup event (using request.app.client seems unmanageable). Any help/pointer would be nice

[Kludex]

The problem is with motor.

[DavidBord]

@xcoder01 the startup event handler inits AsyncIOMotorClient and assigns it to a global variable.
Add another function that returns this client, or whatever else you need (db or collection).
When you need to use the DB in an endpoint handler, specify that other function with Depends

[miladva]

you should define in start up , like this :

app = FastAPI()
app.add_event_handler("startup", start_app) 


async def start_app():
    await Mongo().get_connection()


from motor.motor_asyncio import AsyncIOMotorClient
from config import CONNECTION_STRING, DB_NAME


class Mongo:
    client = None
    db = None

    async def get_connection(self) -> AsyncIOMotorClient:
        if not Mongo.client:
            Mongo.client = AsyncIOMotorClient(CONNECTION_STRING)

now you can use mongo client every where you want!!!

[CanD42]

I have the same problem but only for the tests which are running under pytest. The app under uvicorn works fine. The problem appears when I switch from fastapi 0.68.x to 0.69.x or higher. It seems that the motor event loop isn't initialized properly in the tests

print(asyncio.get_event_loop(), db.client.io_loop) under pytest:
<_UnixSelectorEventLoop running=True closed=False debug=False>
<_UnixSelectorEventLoop running=False closed=False debug=False>

print(asyncio.get_event_loop(), db.client.io_loop) under uvicorn:
<uvloop.Loop running=True closed=False debug=False>
<uvloop.Loop running=True closed=False debug=False>

I think it has to do with the breaking change introduced in 0.69.0:

• Add support for Trio via AnyIO, upgrading Starlette to 0.15.0

motor isn't for some reason using the existing event loop and might created it's own hence I get

print("Database", db)
print("get_client_with_pw()", asyncio.get_event_loop(), db.client.io_loop)
collection = db[config[f'db_client_collection']]
>       client = await collection.find_one({'_id': name})
E       RuntimeError: Task <Task pending name='starlette.middleware.base.BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.coro' coro=<BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.coro() running at /Users/dan/.pyenv/versions/3.9.2/lib/python3.9/site-packages/starlette/middleware/base.py:34> cb=[TaskGroup._spawn.<locals>.task_done() at /Users/dan/.pyenv/versions/3.9.2/lib/python3.9/site-packages/anyio/_backends/_asyncio.py:629]> got Future <Future pending cb=[_chain_future.<locals>._call_check_cancel() at /Users/dan/.pyenv/versions/3.9.2/lib/python3.9/asyncio/futures.py:384]> attached to a different loop

Database AsyncIOMotorDatabase(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=True, connect=False, driver=DriverInfo(name='Motor', version='2.5.1', platform='asyncio')), 'test'))
<_UnixSelectorEventLoop running=True closed=False debug=False> <_UnixSelectorEventLoop running=False closed=False debug=False>

I tried also
client = AsyncIOMotorClient(db_url, tz_aware=True, io_loop=asyncio.get_event_loop()
which don't help. I'll investigate further ...

[CanD42]

That is the problem: encode/starlette#1315

client = AsyncIOMotorClient()
client.get_io_loop = asyncio.get_event_loop

works for me

[brunorpinho]

@CanD42 I was facing this using fastapi + fsspec's adlfs and solved the same way. Thanks

abfs = AzureBlobFileSystem(**creds)
abfs._loop = asyncio.get_running_loop()

[tiangolo]

Thanks for the help here everyone! 👏 🙇

Thanks for reporting back and closing the issue @xcoder01 👍