You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I get an error when running a job on with a slurm instance whose accounting storage is disabled (i.e. the sacct command just replies Slurm accounting storage is disabled). Here's the stack trace :
The job status query failed with command: sacct -X --parsable2 --noheader --format=JobIdRaw,State --starttime 2024-02-21T16:00 --endtime now --name ddcf5013-8004-418b-8832-9a563aaf5280
Error message: Slurm accounting storage is disabled
argument of type 'NoneType' is not iterable
Traceback (most recent call last):
File "/home/rosewood/venvs/aging_stearic/lib/python3.11/site-packages/snakemake_interface_executor_plugins/executors/remote.py", line 190, in _wait_thread
asyncio.run(self._wait_for_jobs())
File "/home/rosewood/stage/spack-0.21.1/opt/spack/linux-centos7-k10/gcc-8.3.1/python-3.11.6-aunbzdzafawzwjwh4wtq45ftn2zjmnzw/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/home/rosewood/stage/spack-0.21.1/opt/spack/linux-centos7-k10/gcc-8.3.1/python-3.11.6-aunbzdzafawzwjwh4wtq45ftn2zjmnzw/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rosewood/stage/spack-0.21.1/opt/spack/linux-centos7-k10/gcc-8.3.1/python-3.11.6-aunbzdzafawzwjwh4wtq45ftn2zjmnzw/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/rosewood/venvs/aging_stearic/lib/python3.11/site-packages/snakemake_interface_executor_plugins/executors/remote.py", line 180, in _wait_for_jobs
still_active_jobs = [
^
File "/home/rosewood/venvs/aging_stearic/lib/python3.11/site-packages/snakemake_interface_executor_plugins/executors/remote.py", line 180, in <listcomp>
still_active_jobs = [
^
File "/home/rosewood/venvs/aging_stearic/lib/python3.11/site-packages/snakemake_executor_plugin_slurm/__init__.py", line 263, in check_active_jobs
if j.external_jobid not in status_of_jobs:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
Thank you for this report. We definitively need to update the error message!
We had the fallback in the executor, but decided to drop it to be able to check the states in asynchronous mode with one command. A cluster without accounting db is pretty unusual. Re-introducing the fallback might not be so easy.
Is your particular cluster in an experimental stage?
I think the slurm cluster profile may be a workable fallback for me.
Perhaps. Then again, you might want to use storage plugins and/or other plugins. That would be a mess. Is there any chance your admins set up the cluster ... eh, properly?
I get an error when running a job on with a slurm instance whose accounting storage is disabled (i.e. the
sacct
command just repliesSlurm accounting storage is disabled
). Here's the stack trace :Looks like there's some error handling here:
snakemake-executor-plugin-slurm/snakemake_executor_plugin_slurm/__init__.py
Line 221 in 7e3de33
But after the loop over attempts to get job status the rest of the code assumes no error and treats
status_of_jobs
as a valid set.The slurm profile also uses
sacct
but falls back toscontrol
if that fails, might be a solution : https://github.com/Snakemake-Profiles/slurm/blob/c44315217d1ce36493dc7dccbd013528657747f9/%7B%7Bcookiecutter.profile_name%7D%7D/slurm-status.py#L40The text was updated successfully, but these errors were encountered: