Running async_to_sync in a forked process hangs indefinitely (if sync_to_async was called before forking)

[StefanCardnell]

Minimal Example

import multiprocessing
from asgiref.sync import async_to_sync, sync_to_async


async def async_process():
    print("async process running")


def sync_process():
    """Runs async_process synchronously"""
    async_to_sync(async_process)()


def fork_first():
    """Forks process before running sync_process"""
    fork = multiprocessing.Process(target=sync_process)
    fork.start()
    fork.join()


def schedule_on_loop():
    tasks = [
        async_process(),
        sync_to_async(sync_process)(),
        sync_to_async(fork_first)(),
    ]
    asyncio.get_event_loop().run_until_complete(asyncio.gather(*tasks))


schedule_on_loop()

Expected

Output of

async process running
async process running
async process running

Follow by process ending

Observed

async process running
async process running

Process hangs

Explanation

The third entry to tasks forks the process using multiprocessing.Process before calling async_to_sync. The synchronous call of async_process in the fork hangs indefinitely.

I looked at the asgiref code and I can see the reason... In AsyncToSync.__init__ the following code obtains an event loop stored at SyncToAsync.threadlocal.main_event_loop

       if force_new_loop:
            # They have asked that we always run in a new sub-loop.
            self.main_event_loop = None
        else:
            try:
                self.main_event_loop = asyncio.get_event_loop()
            except RuntimeError:
                # There's no event loop in this thread. Look for the threadlocal if
                # we're inside SyncToAsync
                self.main_event_loop = getattr(
                    SyncToAsync.threadlocal, "main_event_loop", None
             )

In my minimal example, sync_to_async was called before the process was forked, storing the event loop in threadlocal.main_event_loop. Once the process is forked, the event loop stored at SyncToAsync.threadlocal.main_event_loop becomes invalid, however it is still obtained and and used by async_to_sync after the fork, resulting in the hanging process. This might really be considered an issue with multiprocessing.Process seeming to keep all of the threadlocal state, even after forking.

(The call to asyncio.get_event_loop() threw RuntimeException I believe because there was no event loop in the forked process thread and the forked process is not the main thread. (https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_event_loop)).

Workarounds

1. Placing force_new_loop=True in async_to_sync works. However I'm not a fan as it spins a new thread and event loop every time, when instead it could create/use the event loop for the forked process instead.

2. Set spawn method of multiprocessing to forkserver using multiprocessing.set_spawn_method('forkserver') and create a suitable configuration to fork from that does not have event loops configured yet.

Use case

I use django channels which places our synchronous views behind an AsgiHandler, which uses sync_to_async on all views. These views may fork long running commands using multiprocessing.Process, these forked commands then may use async_to_sync where needed (for instance async_to_sync(get_channel_layer().group_send)). The conditions will result in this indefinite hanging.

[andrewgodwin]

Oh yes, this would happen, thinking about how we track parent threads. I can think of a fix for this we can do in the code - storing the current process PID and/or checking for existence of the parent - but writing a test for it is going to be tricky!

[andrewgodwin]

OK, so I've landed a fix for this on master - I don't actually think that the event loop from the parent process persists at all into the child process, so I instead just made it make a new event loop when this situation is detected - it's the safe approach.

[StefanCardnell]

Makes sense, thanks for the quick resolution.

[patrick91]

hi there, I think I'm getting something related to this, I'm doing a django db call inside a sync_to_async and it hangs, if I change to sync_to_async(thread_sensitive=False) it seems to work, but I guess that's not the way to go.

Do you have some pointers on how to debug this? I wasn't able to get to a minimal reproduction yet

[carltongibson]

@patrick91 Maybe see the discussion on #348. — if I change to sync_to_async(thread_sensitive=False) it seems to work — no, that's not the way to go, but it would explain the hang if you're getting the same (single worker) executor selected in SyncToAsync.__call__().

Do you have some pointers on how to debug this?

What's the executor at this point here at each level? The issue in #348 is that it's the same ThreadPoolExecutor(max_workers=1) and it deadlocks. Maybe that's happening to you?

This wouldn't explain your observation that it only happens on spawning OSs but... 🤔

[patrick91]

@carltongibson thanks for getting back to me! The issue in this comment is in another application so there's a possibility that's not related to the one from my mastodon post 😊

I really need to make some time and strip out all the unnecessary things from my app to get to a minimal reproduction and share it here! I'll try to do it again in future :)

And thanks for the pointers, I'll use them to share more informations when I try again! <3