How do resource requirements of the Scheduler scale in relation to the task and workers? #7515

joyofdata · 2021-04-04T11:20:54Z

joyofdata
Apr 4, 2021

Hello!

Are resource requirements of Scheduler ...

... constant and hence independent of the task and number of workers?
... linearly growing with task and/or workers? (F.x. RAM ~ task and CPU ~ workers)

In other words - do I need more CPU/RAM for the Scheduler process if task is larger and distributed to more workers?

Thanks

Answered by chrisroat

In using the distributed scheduler, I have found that:

I need to allocate a full CPU. CPU fluctuates a lot during initialization, running, and completion... but often can be near 100%. Anecdotally (I haven't measured it precisely), the baseline CPU usage seems higher with more workers.
The RAM usage is low (sub GB) for small to moderate graphs. But it seems to grow super-linearly for large graphs (10k+ tasks). For 100k tasks, I need 1GB of RAM and for 200k tasks, I need 3.5GB of RAM.

chrisroat · 2021-04-07T13:36:36Z

In using the distributed scheduler, I have found that:

I need to allocate a full CPU. CPU fluctuates a lot during initialization, running, and completion... but often can be near 100%. Anecdotally (I haven't measured it precisely), the baseline CPU usage seems higher with more workers.
The RAM usage is low (sub GB) for small to moderate graphs. But it seems to grow super-linearly for large graphs (10k+ tasks). For 100k tasks, I need 1GB of RAM and for 200k tasks, I need 3.5GB of RAM.

0 replies