Running container in SLURM

Singularity / Apptainer

We support running containers via singularity or its successor apptainer.

Build a container image

In this simple example, we create a "hello-world" container image and let it run via SLURM.

First, create a definition file, like lolcow.def

BootStrap: docker
From: ubuntu:22.04

%post
   apt-get -y update
   apt-get -y install cowsay lolcat

%environment
   export LC_ALL=C
   export PATH=/usr/games:$PATH

%runscript
   date | cowsay | lolcat

%labels
   Author <your name>

We use docker as default repository and we select Ubuntu 22.04 as a base image.
Then we install cowsay and lolcat.
If we run the container image it will print out the current date in a speech bubble of a ascii cow. This ascii output will be then colorized by lolcat.

Let's build the cointair image:

apptainer build lolcow.sif lolcow.def

The lolcow.sif is the container image we want to build from the lolcow.def definition file.

We then can write a simple apptainer job file for SLURM, e.g. apptainer.sh

#!/bin/bash
#SBATCH -J Apptainer
#SBATCH -c 2
#SBATCH -p test
#SBATCH --gres=gpu:1
#SBATCH --mail-type=ALL
#SBATCH --mail-user=<your-email-address>

# In this scenario, we also want to use a GPU. It's not really needed for the lolcow example. 
# But here for documentation purposes how to include GPU support into apptainer.
export APPTAINERENV_CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES}
echo ${CUDA_VISIBLE_DEVICES}

# Add the --nv switch to activate GPU support
srun apptainer run --nv lolcow.sif

Submit the job via: sbatch apptainer.sh

Temporary build space and caches

Since building container images may need a lot of discspace, we changed the default storage path for temporary build space and apptainer caches.

You can find:

  • APPTAINER_TMPDIR in /data/<your group>/<your user>/.tmp/
  • APPTAINER_CACHEDIR in /data/<your group>/<your user>/.cache/

instead of the default locations /tmp/ and $HOME/.apptainer/cache.

If you get an error about non existing directories, it could be that they were not created automatically. You can create them by using the correspoding variables:

test -d $APPTAINER_TMPDIR || mkdir -p $APPTAINER_TMPDIR
test -d $APPTAINER_CACHEDIR || mkdir -p $APPTAINER_CACHEDIR

The $SINGULARITY_CACHEDIR and $SINGULARITY_TMPDIR are the same as the corresponding $APPTAINER_TMPDIR and $APPTAINER_CACHEDIR. You need to create them only once.

Mountpoints

/data and /home will be automatically mounted inside of your container image. So you can directly run command inside of the container with files in your specific /home or /data directory. For example

srun apptainer run --nv pytorch.sif $HOME/my_project/analysis.py /data/<your_group>/<user>/data/datasets

Clean up

Run from time to time 

apptainer cache clean

to get rid of old cache files, which take a lot of disc space.