SLURM example Jobfiles

You can ingore the trailing // comments in the examples

Basic Jobfiles

A jobfile is a text file where you define your job requirements. You can then submit the jobfile to SLURM and SLURM will run your code. Here is a basic jobfile:

File: simple_test.sh

#!/bin/bash
#SBATCH -J TestJob               // Name your job
#SBATCH -c 2                     // We want to use 2 cores
#SBATCH --mem=2G                 // Job needs 2GB of memory (default!)
#SBATCH -p test                  // Select test partion
#SBATCH --gres=gpu:2             // We need 2 GPU
#SBATCH --tmp=5G                 // We need 5G of /tmp space
#SBATCH --mail-type=ALL          // Send mail when job starts and finishes
#SBATCH --mail-user=<your-email-address>

# Run nvidia-smi to see if the GPUs are available (This is also a comment)
nvidia-smi -L

Submit your job afterwards via:

sbatch simple_test.sh

You will get some output like:

Submitted batch job 204690

The number is your job id. A slurm-<job id>.out file will be created in your $HOME directory with the output of your job. In our case, you will find the output of nvidia-smi -L in there:

cat slurm-204690.out 
GPU 0: NVIDIA L40 (UUID: GPU-0a12b672-97e5-3230-1865-e5d51e109d45)
GPU 1: NVIDIA L40 (UUID: GPU-37dba20f-ca0f-50bb-4b27-df6a4bc3afd8)

Run an interactive session

Use the srun command to run an interactive session. For setting up your environment and testing if your jobfile works, you can use the test partition. Be aware of the runtime limit of the test partition!

Schedule an interactive shell with 2 cores, 8GB of memory, one GPU and 5GB of /tmp space.

srun -c 2 -n 1 --mem=8G --gres=gpu:1 --tmp=5G -p standard --pty bash

Run a python program

For example: single_job.sh

#!/bin/bash
#SBATCH -J StandardJob           // Name your job
#SBATCH -c 16                    // We want to use 16 cores
#SBATCH --mem=16G                // Job needs 16GB of memory
#SBATCH -p standard              // Select standard partion
#SBATCH --gres=gpu:1             // We need 1 GPU
#SBATCH --tmp=5G                 // We need 5G of /tmp space
#SBATCH --mail-type=ALL          // Send mail when job starts and finishes
#SBATCH --mail-user=<your-email-address>

# Run pytorch via python environment
cd ~/work/pytorch
source bin/activate
srun python3 my_job.py

For running more complex code in parallel see Parallel jobs with MPI

Attach to a running job for debugging

Make sure you already created internal ssh-keys. If not check the section Setup your environment.

You can attach a interactive session to your running job with this command:

srun --pty --overlap --jobid <your_job_id> /bin/bash