Wrappers for Google Batch

[vsoch]

I'm working on the batch executor, and there are common snippets (that in retrospect were non-trivial to figure out) for setting up MPI and similar that (I think) would work best as a snakemake wrapper. Adding a note here for myself todo. For some example context, here is what was needed for a barrier (setup) and then running script. I need to think through what belongs with the executor vs. as a wrapper and how to organize that logic.

run_script = """#!/bin/bash
export PATH=/opt/intel/mpi/latest/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/mpi/latest/lib:/opt/intel/mpi/latest/lib/release
find /opt/intel -name mpicc

# This is important - it won't work without sourcing
source /opt/intel/mpi/latest/env/vars.sh

if [ $BATCH_TASK_INDEX = 0 ]; then
  cd /mnt/share/{{outdir}}
  ls
  which mpirun
  mpirun -hostfile $BATCH_HOSTS_FILE -n {{tasks}} -ppn {{tasks_per_node}} /mnt/share/hello-world-mpi/hello_c
fi
"""

setup_script = """#!/bin/bash
export DEBIAN_FRONTEND=noninteractive
sleep $BATCH_TASK_INDEX

# Note that for this family / image, we are root (do not need sudo)
yum update -y && yum install -y cmake gcc tuned ethtool

# This ONLY works on the hpc-* image family images
google_mpi_tuning --nosmt
# google_install_mpi --intel_mpi
google_install_intelmpi --impi_2021
source /opt/intel/mpi/latest/env/vars.sh

# This is where they are installed to
# ls /opt/intel/mpi/latest/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/mpi/latest/lib:/opt/intel/mpi/latest/lib/release
export PATH=/opt/intel/mpi/latest/bin:$PATH

outdir=/mnt/share/{{outdir}}
mkdir -p ${outdir}
cd ${outdir}

if [ $BATCH_TASK_INDEX = 0 ]; then
    wget -O /tmp/ompi.tar.gz https://docs.it4i.cz/src/ompi/ompi.tar.gz
    cd /tmp
    tar -xzvf ompi.tar.gz
    rm ompi/Makefile
    cp -R ./ompi/* ${outdir}/
    cd ${outdir}/
    ls
    mpicc -g -lmpi -lmpifort hello_c.c -I/opt/intel/mpi/latest/include -I/opt/intel/mpi/2021.8.0/include -L/opt/intel/mpi/2021.8.0/lib/release -L/opt/intel/mpi/2021.8.0/lib -o hello_c
    ls
fi
"""

Note that I'm starting with the easier use cases with (e.g., not doing MPI!)

[vsoch]

Update: I am currently modeling these snippets within the executor. snakemake/snakemake-executor-plugin-googlebatch#2. This is because the design is specific to Batch, e.g., having different sub-steps (typically for setup, barrier, then run) so it would be hard to represent as a single snippet (traditional snakemake snippet). I'll keep this generic use case in mind in case we can still integrate here.

[github-actions]

This issue was marked as stale because it has been open for 6 months with no activity.