"module command not found" using Snakemake with Slurm #63
Comments
|
Not being a complete expert on many of the topics involved, and following a link to another, i've
while most of what we do on our cluster relies on Could that Happy to be the guinea pig testing things if someone is willing to guide me through the process, as I've been trying to solve that problem for a whole day now and I've been going in circle and losing track of what to focus on. |
|
Well,
Try, chances are they are the same. Even if not, I can think of two different causes here:
Could you use your minimal example, start with Snakemake will attempt to run the module command in the job context, if the module command is not present there, it is most likely an environment issue. Yet, we actively export the environment (which ought to be a default, anyway). Hence, a second thing to check would be running Also: Which module system are you using (output of |
|
Thanks for the quick reply! Interesting, they're different 👀 With Not sure why I see (and I haven't installed the slurm-jobstep plugin. |
That should be installed automatically, if you install this plugin via conda/mamba/etc. However, if it is really missing, please be sure to work with it: Snakemake submits itself and to work properly in the job context, you need the plugin. You can check this with or After installation, please try again and report the output, resp. log of |
|
Sorry for the confusion. I didn't mean that it was not installed. I meant that I hadn't installed it explicitly myself. To clarify, I've installed Snakemake and the Let me know if I need to reinstall anything (including the whole Conda environment), but in the meantime, the log of the adjusted rule you asked is (after adjusting the missing indentation space in front of cellranger.log Do you need the console log or the JOBID.log too? Seems to be that the |
|
ah, you are working with system c-modules. I am not sure how up-to-date your version is, so: What is - on the login node the output of:
And the same in job context. Get an interactive job like:
You might have |
I never paid attention what kind of module system we're asked to use 😬
Login node srun There's a typo (missing L) in your Also, I'm not sure what your Thanks again for your time so far! |
|
Anyway, the module command works in job context and the module path is the same. I need some time - I will be teaching next week. |
|
No worries. Thanks for the help. I've got my workaround for now, which is that I've got few enough samples to write and submit the job scripts myself using If only I could figure out what changed since I last used snakemake back in January. Really weird. |
|
FWIW and |
|
Stupid me! Did you run Snakemake with |
|
Thanks for coming back to me. I just did and I see no difference: A couple of thoughts:
Perhaps I should get you in touch directly with our IT team rather than acting as middle-man and potentially fudging the information I'm passing back and forth? |
|
Oh, this never ending not-invented-here syndrome, resulting in how-do-we-deviate-best-from-commuity-standards ... Yes, the Oh, wait: The plugin already does We strive to provide portable workflows, where the workflow should not be tinkered with when shipped (because every additional information is in profile and/or config files). Here, you could write a rule to load the module (a local rule) and then submit your job, but that result in a slightly odd software pattern, right? So, yes, talk to your admins. (Snakemake launches no shell at all upon submission. It assumes a working shell with an environment satisfying its needs, when in job context. You can see the launch, when running Snakemake with As to your other question about the job log: Snakemake submits itself. Hence, it needs to figure out what kind of rule it is dealing with. It will receive a part of the workflow (e.g. one rule), and subsequently interprets it to be local. |
|
Either I misundertood your suggestion, or it's also not working Still getting Anyway, at this point don't worry, I'll check with IT again if they want to offer a solution, but a simple workaround is to manually load the modules in my Bash terminal before running snakemake, and not using the |
|
What I meant is to write a rule which explicitly performs If the module command does not work on the compute nodes, there is no way to trigger it there - all you can do is to export the entire environment, which Snakemake does by default. |
I think I've initially posted in the wrong repo. Please see snakemake/snakemake#2802 (comment)
Especially as
--executor noneworks fine, but my profile using--executor slurmfails to load an environment module.Given my discussion with IT so far, we suspect that whatever shell is launched on the compute node isn't a login shell, which is a requirement to access the
modulefunction (at least on our cluster).The text was updated successfully, but these errors were encountered: