Question: intercepting stdout / stderr / python logger on loky processes

[Sikerdebaard]

Hi, I've been using Loky to do some batch-processing on a supercomputer. So far this has worked well. This library has saved me a significant amount of time. However, I'm now running into an issue.

As far as my understanding on Loky goes each job submitted runs in a pool of worker processes outside of the main-process.

I'm wondering if the stdout / stderr of each job submitted to the reusable worker pool can be captured to a separate file. Currently it's being spewed out on the main process, either on stdout/stderr or the python logger, but this is making individual jobs hard to debug as I'm crunching through thousands of jobs over the course of a few days resulting in gigabytes of logging to be produced in a single file.

Here's a short code example on what I'm trying to achieve.

import sys

from loky import get_reusable_executor, wrap_non_picklable_objects
from loky.backend.context import get_context, set_start_method


class Job():
    def __init__(self, jid):
        self.id = jid

@wrap_non_picklable_objects
def _loky_run_job(job):
    # stdout / stderr produced when this code runs needs to be intercepted to separate files
    # and not dumped on the main process console or python logger

    errors = []
    try:
        print('job runs here')
    except Exception:
        # catch exceptions in case the job fails
        exc_type, _, trace = sys.exc_info()
        exc_info = traceback.format_exc()
        trace = traceback.extract_tb(trace, 1)[0]
        print('LokyExecution encountered exception ({}) during execution:\n{}'.format(exc_type.__name__, exc_info))
        errors.append((exc_type.__name__, exc_info, trace[0], trace[1]))

    return job.id, errors

nr_of_workers = 2
executor = get_reusable_executor(
    max_workers=nr_of_workers,
    timeout=3600,  # respawn workers after 1 hour of inactivity
    reuse=True,
)

jobs = [Job(jid) for jid in range(1, 1000)]

futures = [executor.submit(_loky_run_job, job) for job in jobs]