Cannot allocate enough workers ("Too many open files")

[brettc]

Hi all. I want to use processes (not threads) in dask, and am trying to allocate as many workers as I have CPUS. But I have run into a "Too many open files..." error. I see this kind of error has been raised on StackOverflow, but here I have not even started to submit any work yet.

Is this a bug or a limitation, or do I just not understand something?

Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:23:20)
[Clang 12.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.cpu_count()
16
>>> from distributed import Client
>>> Client(n_workers=16)
2022-09-02 09:40:15,704 - distributed.nanny - ERROR - Failed to start process
Traceback (most recent call last):
  File "/usr/local/Caskroom/miniforge/base/envs/try-ray/lib/python3.9/site-packages/distributed/nanny.py", line 438, in instantiate
  File "/usr/local/Caskroom/miniforge/base/envs/try-ray/lib/python3.9/site-packages/distributed/nanny.py", line 663, in start
  File "/usr/local/Caskroom/miniforge/base/envs/try-ray/lib/python3.9/site-packages/distributed/process.py", line 81, in __init__
  File "/usr/local/Caskroom/miniforge/base/envs/try-ray/lib/python3.9/multiprocessing/context.py", line 63, in Pipe
  File "/usr/local/Caskroom/miniforge/base/envs/try-ray/lib/python3.9/multiprocessing/connection.py", line 532, in Pipe
OSError: [Errno 24] Too many open files
2022-09-02 09:40:15,706 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess
--------- CUT THOUSANDS MORE LINES OF ERRORS---------

[jrbourbeau]

The default number of files/connections on macOS is quite low (I think it's something like ~128). I recommend increasing that limit by, for example, doing something like

ulimit -n 5000

in your terminal before launching you Python REPL

See https://distributed.dask.org/en/stable/faq.html#too-many-open-file-descriptors for more details.

[brettc]

Thanks for the speedy reply. That fixes it.

[brettc]

A quick follow-up. If anyone stumbles across the same problem, you can programmatically do the equivalent of ulimit -n ... This worked for me:

    from resource import RLIMIT_NOFILE, getrlimit, setrlimit

    from dask.distributed import Client  # Start a Dask Client

    # without this, I get "Too many open files"
    soft, hard = getrlimit(RLIMIT_NOFILE)
    setrlimit(RLIMIT_NOFILE, (1024, hard))

    POOL_SIZE = 14
    client = Client(processes=True, threads_per_worker=1, n_workers=POOL_SIZE)

More discussion here: https://stackoverflow.com/questions/6774724/why-python-has-limit-for-count-of-file-handles