A way to give scheduler hints on the size of unmanged memory? #8406
Unanswered
apiwat-chantawibul
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I'm a new user who just try to wrap some custom computation and
map_blocks
over some dask array.The scheduler seems unable to take into account how much larger temporary memory is needed for the custom computation relative to its output size.
This results in scheduler attempting start too many tasks in parallel instead of finishing pending tasks which would reduce the memory foot print to the smaller size of the output.
To solve this I'm thinking of a couple of ways:
Please help
Beta Was this translation helpful? Give feedback.
All reactions