Improved tools for long running tasks

[finnhacks42]

Bootstrapping over model selection over multiple models is a very time consuming process (often > 8 hours on a single machine). This can make iterations on such projects very slow.

Approaches

• option for parallelization over multiple machines
• caching of results so that visualisation/summaries can be changed without re-running inference
• progress indication
• ability to pause/resume

[dsteinberg]

Integration with something like this may help:
https://luigi.readthedocs.io/en/stable/index.html