how does dask handle shuffling when num_partitions > num_workers #7223
Unanswered
nirandaperera
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi,
I was wondering how dask handles shuffling operation when the number of partitions is higher than the number of workers? Is there some sort of "mailbox" capability in the dask workers?
ex: num_partitions = 4 , num_workers = 2, and lets assume the case where shuffle_part0 and shuffle_part1 are running on workers. How are the following communications are facilitated?
shuffle_part0 --> shuffle_part2 or 3
shuffle_part1 --> shuffle_part2 or 3
Beta Was this translation helpful? Give feedback.
All reactions