AMM high level documentation #5456
Conversation
| distributed: | ||
| scheduler: | ||
| active-memory-manager: | ||
| start: true |
There was a problem hiding this comment.
Minor nit - is it too late to change these config fields? I'd find enable: true to be a clearer config name.
There was a problem hiding this comment.
IMHO I find enable to be ambiguous. You can call run_once while it's not started.
|
|
||
| .. note:: | ||
| This policy is incompatible with :meth:`~distributed.Client.replicate` and with the | ||
| ``broadcast`` parameter of :meth:`~distributed.Client.scatter`. |
There was a problem hiding this comment.
Can you comment more on what this incompatibility entails? What happens if AMM is enabled and scatter(..., broadcast=True) is called in user code?
| - Replicate a task that is not yet in memory | ||
| - Create more replicas of a task than there are workers | ||
| - Create replicas of a task on workers that already hold them | ||
| - Create replicas on paused or retiring workers |
There was a problem hiding this comment.
Is there a way to enable debug logs for issues like these? When developing a new policy, should you try to avoid generating unacceptable suggestions (in which case you'd want to be able to debug these issues), or is generating unacceptable suggestions (and relying on the AMM to ignore them) in a policy fine and intended if it makes the policy code simpler?
There was a problem hiding this comment.
The latter. Policies should be simple and readable. The subtle edge case management is left to the AMM.
There was a problem hiding this comment.
Can you add a note to this section clarifying that policies shouldn't worry about these cases?
| ipython | ||
| prometheus |
There was a problem hiding this comment.
those pages don't exist anywhere
There was a problem hiding this comment.
Good catch -- these are over in the dask/dask docs now (e.g. https://docs.dask.org/en/stable/setup/prometheus.html)
No description provided.