Executors: snooze on a lower level?

[sk1p]

Another follow-up of #1572:

With #1572, libertem-server can temporarily shut down the whole executor. This solves the issue of long-running processes eating memory and CPU time very well, but a similar issue exists for dangling notebooks. We might want to add the same feature on the executor level, at least for the default dask executor, such that notebooks also free up their resources after a while. Important for shared systems, and especially when a GPU is shared between multiple users.

[matbryan52]

That would definitely be helpful in our use case, we already have the problem for LT executors started from within Digital Micrograph.

I am happy to look at how to do this.

[sk1p]

Great to hear! I was thinking that this could be handled partially by the Context, which would spawn the background task/thread which checks for the timer expiration, and then asks the executor to snooze, if it is supported. The pipelined executor (edit: or LiveContext), for example, could opt out of this, as it is contra productive for live processing.

Maybe we can discuss this further tomorrow.

[sk1p]

I was thinking that this could be handled partially by the Context, which would spawn the background task/thread which checks for the timer expiration, and then asks the executor to snooze, if it is supported.

This also has the component that the executor needs to be kept alive across all those "running code/jobs/..." operations, which is best done in the executor itself (or maybe a wrapper kind of type), while, as noted above, high-level decisions about the timeout and concrete point of time of snoozing might be handled better by high-level code like Context or the executor state as we have it implemented now.