Python 3.12 os.fork deprecation

[nishanthkarthik]

Python 3.12 added a DeprecationWarning when os.fork is called in a multithreaded process.

python/cpython#100228

PL servers throws an error with Python 3.12

error: Unexpected STDERR data: /source/PrairieLearn/apps/prairielearn/python/zygote.py:378: DeprecationWarning: This process (pid=19888) is multi-threaded, use of fork() may lead to deadlocks in the child.
  worker_pid = os.fork()
 {"callbackIsNull":true,"childIsNull":false,"lastCallData":null,"outputBoth":"","outputData":"{\"present\": true, \"val\": \"pong\"}\n","outputStderr":"","outputStdout":"","stack":"Error\n    at CodeCallerNative._errorData (/source/PrairieLearn/apps/prairielearn/src/lib/code-caller/code-caller-native.js:600:25)\n    at CodeCallerNative._logError (/source/PrairieLearn/apps/prairielearn/src/lib/code-caller/code-caller-native.js:617:26)\n    at CodeCallerNative._handleStderrData (/source/PrairieLearn/apps/prairielearn/src/lib/code-caller/code-caller-native.js:389:12)\n    at Socket.emit (node:events:518:28)\n    at Socket.emit (node:domain:488:12)\n    at addChunk (node:internal/streams/readable:559:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)\n    at Readable.push (node:internal/streams/readable:390:5)\n    at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)","timeoutIDIsNull":true}
error: Unexpected STDERR data: /source/PrairieLearn/apps/prairielearn/python/zygote.py:378: DeprecationWarning: This process (pid=19879) is multi-threaded, use of fork() may lead to deadlocks in the child.

[eliotwrobson]

Where are you seeing this error, are you running things locally? I didn't think that the PL servers were on 3.12 yet.

[nwalters512]

I was able to trace this back to pyarrow, which spins up a thread for jemalloc_bg_thd. How I was able to do this:

• Run a PL container
• Pop another shell into it
• Run top | grep python3 to identify the process PID
• Run top -H -p <PID> to see all the threads, which shows the following:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                          
 2015 root      20   0 1682688 304124  87808 S   0.0   1.9   0:01.92 python3                                          
 2046 root      20   0 1682688 304124  87808 S   0.0   1.9   0:00.00 jemalloc_bg_thd 

As Eliot points out, this isn't yet a problem, as we don't support Python 3.12. However, it will be a problem as soon as we decide to support that version. Probably the right thing to do is to disable this specific deprecation warning?

In principal pyarrow allows us to change the memory pool implementation that's used. I tried pa.set_memory_pool(pa.system_memory_pool()), but that didn't stop the jemalloc thread.

[eliotwrobson]

For now, I think disabling the warning is the easiest thing to do (this is pretty easy using a warning filter). In the past I've mentioned potentially using the asyncio framework in Python to replace this usage of os.fork() if we switch to TCP for communicating with the Python process, maybe worth considering if it avoids potential deadlocks? I'm not even completely sure why this fork gets used, but ripping out the usage might be kinda overkill in this case.

[nwalters512]

fork() is used to ensure a clean execution environment every time we execute code. We will not be getting rid of it.

[nishanthkarthik]

Where are you seeing this error, are you running things locally? I didn't think that the PL servers were on 3.12 yet.

Right, I am using the native development setup for PL. I use archlinux, so my packages are rolling. I can just ignore the warnings in Python but I wasn't sure if error: Unexpected STDERR data meant a hard stop somewhere.