Replies: 1 comment 1 reply
-
I tried moving my analysis from snakemake to dask/prefect, but so far without much luck. I'm curious to learn what others are doing. The basic problem I have is that in HPC environment queues don't guarantee simultaneous availability of resources, so it's not uncommon for snakemake to wait a couple of days until a specific compute node is available. Also, implementing file-based workflow in dask and prefect is a bit cumbersome (or maybe I was doing it wrong). |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Bioinformatics pipelines tend to consist of chaining together calls to various tools that are often written in compiled languages and have a command-line interface for specifying input and output files. Because each tool can require vastly different resources I would like to be able to submit work to a Dask cluster consisting of heterogeneous workers. I think this is relatively straightforward if using the command-line interface to start workers, e.g. (to take an example from the
distributed
docs):and then I can specify what my required resources are when I submit a task:
Ideally I wouldn't have to manually start the workers but could instead do this programmatically. Indeed it looks like this is exactly what
SpecCluster
allows me to do. Since the clusters provided bydask_jobqueue
are now based onSpecCluster
this seems like something I should be able to do, although if so it seems like the way to do this is pretty well obscured by the cluster implementations. Has anybody had any luck getting something like this to work?Finally a quick digression: The (fairly small) overhead involved in starting Python processes to mostly call subprocesses is noted. The main advantage to me for using Dask for a bioinformatics workload is freeing myself from more cumbersome (IMHO) DSL-based tools like Snakemake, Nextflow, CWL, WDL, etc. and instead being able to mostly write code entirely in Python.
Beta Was this translation helpful? Give feedback.
All reactions