Allow using joblib/loky with `-P processes

[hadim]

Checklist

• I have checked the issues list
  for similar or identical feature requests.
• I have checked the pull requests list
  for existing proposed implementations of this feature.
• I have checked the commit log
  to find out if the if the same feature was already implemented in the
  master branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

• UserWarning: Loky-backed parallel loops cannot be called in a multiprocessing, setting n_jobs=1 #5830
• https://groups.google.com/g/celery-users/c/IGOBnp4n0gU

Possible Duplicates

• None

Brief Summary

Currently, the only way to use joblib and loky (in some extent multiprocessing too) is to use -P threads instead of -P processes.

Since -P threads is using ThreadPoolExecutor from the stdlib under the hood, the task in the same worker is not really running in parallel (but async). This is because of the Python GIL.

The problem becomes even more important if the workload is split between the main code and the subprocesses (executed by joblib).

Only the subprocesses are executed in parallel but not the main code. It adds a clear bottleneck that is not ideal.

Architectural Considerations

I am not comfortable enough with Celery to propose an implementation. Feel free to throw your ideas below in this ticket.

Potential workaround

A potential workaround is to have plenty of Celery worker replicas that execute only one task (concurrency of 1). So the task will have all the CPU time available for it and be free to use joblib at its convenience.

But having all the Celery workers with a concurrency of 1 also imply some overhead in the underlying infrastructure that is not really wanted ideally.

[open-collective-bot[bot]]

Hey @hadim 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

[auvipy]

if you are up to push it and it goes per our celery design we are OK with it

[hadim]

I am happy to help with it but I likely lack expertise in the area so any guidance might be helpful.

Also, it's worth bringing @linar-jether to the convo since he proposed a workaround recently at #4551 (comment) consisting of patching joblib with:

def _patch_joblib_loky_backend():
    import joblib._parallel_backends
    from joblib._parallel_backends import mp, cpu_count
    
    def effective_n_jobs(self, n_jobs):
        """Determine the number of jobs which are going to run in parallel"""
        if n_jobs == 0:
            raise ValueError('n_jobs == 0 in Parallel has no meaning')
        elif mp is None or n_jobs is None:
            # multiprocessing is not available or disabled, fallback
            # to sequential mode
            return 1
        elif n_jobs < 0:
            n_jobs = max(cpu_count() + 1 + n_jobs, 1)
        return n_jobs
    
    # Monkey-patch to allow daemonic thread to spawn processes
    joblib._parallel_backends.LokyBackend.effective_n_jobs = effective_n_jobs

_patch_joblib_loky_backend()

But it raises the following error:

  File "/home/hadim/local/conda/envs/circus/lib/python3.8/site-packages/joblib/externals/loky/reusable_executor.py", line 177, in submit
    return super(_ReusablePoolExecutor, self).submit(
  File "/home/hadim/local/conda/envs/circus/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 1122, in submit
    self._ensure_executor_running()
  File "/home/hadim/local/conda/envs/circus/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 1096, in _ensure_executor_running
    self._adjust_process_count()
  File "/home/hadim/local/conda/envs/circus/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 1087, in _adjust_process_count
    p.start()
  File "/home/hadim/local/conda/envs/circus/lib/python3.8/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'), \
AssertionError: daemonic processes are not allowed to have children

Note also that without this patch, the Celery worker works just fine, it's just that joblib specifically set n_jobs to 1 with the following warning:

[2021-08-21 13:17:11,114: WARNING/ForkPoolWorker-7] /home/hadim/local/conda/envs/circus/lib/python3.8/site-packages/joblib/parallel.py:733: UserWarning: Loky-backed parallel loops cannot be called in a multiprocessing, setting n_jobs=1
  n_jobs = self._backend.configure(n_jobs=self.n_jobs, parallel=self,

[hadim]

If you believe that issue is more joblib related than celery related. Let me know and I'll open a ticket on the joblib repo.

Edit: ticket opened on the joblib repo: joblib/joblib#1208

[linar-jether]

@hadim I believe the other issue you've encountered, is this one: #1709
Which can be "fixed" using a different workaround:

current_process()._config['daemon'] = False

[thedrow]

An actual solution to this problem would be to provide a Loky process pool since I'm assuming this will always work.
I'm willing to collaborate on such an effort.

[linar-jether]

@thedrow what would be required to implement a new worker pool class?

[thedrow]

Well, we'll need to integrate it with Celery's current event loop.

[hadim]

@hadim I believe the other issue you've encountered, is this one: #1709
Which can be "fixed" using a different workaround:

current_process()._config['daemon'] = False

I gave it a quick try but so far it seems to be working nicely. Thanks.

[hadim]

To be clear in my context, current_process()._config['daemon'] = False alone is sufficient without _patch_joblib_loky_backend().