Unknown image format/type in nextflow pipeline

[RosaDeSa]

I'm using singularity-ce version 4.0.3 on a HPC.

I'm trying to pull a docker container with singularity in a nextflow pipeline. After updating singularity to version 4.0.3 I'm having issues in pulling the container.

Command error:
ERROR: Unknown image format/type: /home/users/scratch/work/singularity/staphb-seqtk.img

In the singularity cachedir, I found the container with the .img extension, but it seems singularity doesn't recognize it. Maybe it is looking for a simg extension?
If I move in the singularity directory and run singularity shell or singularity exec on the img the container works perfectly.

[dtrudg]

Are you able to share the image file somewhere, so we can take a look?

If you cannot share it, the output of singularity sif list <image name> may contain some useful information.

If the problem only occurs within a Nextflow pipeline, but Singularity can run the container outside of a Nextflow pipeline... then perhaps the pipeline is actually using a different version of Singularity?

[RosaDeSa]

The docker container that I'm trying to pull in the pipeline is docker://staphb/seqtk:1.3. Everything worked well with the older version of singularity 3.5.
In the singularity cachedir, I have this downloaded image staphb-seqtk-1.3.img
Here is the output of the singularity sif list:

singularity sif list staphb-seqtk-1.3.img 
------------------------------------------------------------------------------
ID   |GROUP   |LINK    |SIF POSITION (start-end)  |TYPE
------------------------------------------------------------------------------
1    |1       |NONE    |32176-32218               |Def.FILE
2    |1       |NONE    |32218-35451               |JSON.Generic
3    |1       |NONE    |35451-35941               |JSON.Generic
4    |1       |NONE    |36864-80146432            |FS (Squashfs/*System/amd64)

I didn't specify the version of singularity in nextflow, it could detect automatically on the cluster. I've also updated Nextflow to the latest version to be sure of compatibility with the new version of singularity.

[dtrudg]

The image we need to know about is the one that is pulled during the Nextflow pipeline.

In Singularity's cachedir ~/.singularity/cache, images should be named according to a hash - they will not be visible with their name. It is likely that if you have an image in that directory named staphb-seqtk-1.3.img then it is perhaps one created by a much older version of singularity in the past. This cached image will not be used by a newer version of Singularity.

If you manually pull the image with the current version of singularity, and run it...

singularity --version

singularity pull docker://staphb/seqtk:1.3

singularity run staphb-seqtk-1.3.sif

... what happens?

If this works correctly then there is probably an issue with the Nextflow configuration, rather than Singularity itself.

[RosaDeSa]

Thank you for your help, I really appreciate it.

My singularity cachedir is defined in the .bashrc file, and this image staphb-seqtk-1.3.img is pulled from nextflow. Before having the unknown image error, I have this message during the pipeline running
Pulling Singularity image docker://staphb/seqtk:1.3 [cache /home/r.desantis/singularity/cachedir/staphb-seqtk-1.3.img].
So it seems that nextflow pulls the singularity container in exactly that directory.

If I manually pull the image with the current version of singularity, and run it, everything works:

singularity --version
singularity-ce version 4.0.3
singularity pull docker://staphb/seqtk:1.3
INFO:    Using cached SIF image
singularity run seqtk_1.3.sif
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Singularity> 

It seems like if I pull manually the container, singularity creates the .sif image instead if I pull the image inside the pipeline, singularity pull the image with the extension .img

[dtrudg]

I think this is going to need more debug information out of the Nextflow pipeline, given that manually pulling with Singularity works without issue.

It's likely that it would be better to ask in a Nextflow forum / chat, pointing out that Singularity works when run manually.

I'm afraid that I'm not too familiar with how images are being pulled via Nextflow, or how to get additional information about this. At this point it doesn't look like there's a problem with Singularity itself.