Capturing stdout of child processes in prefect logger

[trahloff]

Hi joblib team,

It seems like the logger object in joblob results in side-effects like PrefectHQ/prefect#12428.

Do you have any idea why this is the case? I am also more than happy to contribute the fix if you point me to a potential root cause.

[fcharras]

AFAIK Parallel does not use Logger inheritance, relevant threads #1483 #1494 .

In your example:

from prefect import flow, task
from joblib import Parallel, delayed

def sqrt(x):
    print(x**0.5)


@task
def print_joblib():
    Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(4))


@task
def print_directly():
    print("This is a print statement.")


@flow(log_prints=True)
def main() -> None:
    print_joblib()


if __name__ == "__main__":
    main()

Child processes are just writing to stdout using their respective sys.stdout file handle (see print file argument)

Correct me if I'm wrong, I assume there that prefect captures outputs by wrapping sys.stdout in the processes that are spawned to run tasks, but the wrapper doesn't carry over to the child processes that joblib will in turn spawn from the task process. You most likely will see the same behavior if replacing joblib.Parallel with things like python built-in multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor, etc.

I know of a project that work around this by using file descriptors, see this documented implementation of capturing stdout of child processes using file descriptors, maybe this can help you ?

[trahloff]

Thank you for the quick reply @fcharras! They basically patch the logger object or patch the print function depending on how they capture the log stream.

@zzstoatzz, does that information help you?