1 worker with n threads vs n workers with 1 thread?

[joyofdata]

Hello!

Let's say I have a worker node with 64GB RAM and 8 cores and I want to utilize 7 cores for Dask workers.

What would be likely the preferable setup?

• 1 worker with 7 threads
• something in between
• 7 workers with 1 thread each

Thanks

[jrbourbeau]

Thanks for the question @joyofdata. The optimal number of threads vs. processes depends on the particular computation you're doing. In practice, "something in between" often works well (though this isn't always the case). I recommend checking out the following documentation sections:

• "Thread Pool" worker docs
• "Local threads"
• "Local processes"

which outline some of the reasons why you might prefer more threads vs. more processes.

Additionally, you may find the nprocesses_nthreads utility function useful. This is what Dask's LocalCluster uses to determine it's default number of workers and threads-per-worker.