Parallel jobs using MPI

CPU only jobs

Either your code is built including the provided OpenMPI library or you need wrapper like mpirun to run your code.

Example simple OpenMPI code

Thanks to Wes Kendall from www.mpitutorial.com for providing hello_world_mpi.c

// Author: Wes Kendall
// Copyright 2011 www.mpitutorial.com
// This code is provided freely with the tutorials on mpitutorial.com. Feel
// free to modify it for your own use. Any distribution of the code must
// either provide a link to www.mpitutorial.com or keep this header intact.
//
// An intro MPI hello world program that uses MPI_Init, MPI_Comm_size,
// MPI_Comm_rank, MPI_Finalize, and MPI_Get_processor_name.
//
#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
  // Initialize the MPI environment. The two arguments to MPI Init are not
  // currently used by MPI implementations, but are there in case future
  // implementations might need the arguments.
  MPI_Init(NULL, NULL);

  // Get the number of processes
  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);

  // Get the rank of the process
  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  // Get the name of the processor
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name, &name_len);

  // Print off a hello world message
  printf("Hello world from processor %s, rank %d out of %d processors\n",
         processor_name, world_rank, world_size);

  // Finalize the MPI environment. No more MPI calls can be made after this
  MPI_Finalize();
}

Compile it via

mpicc hello_world_mpi.c -o hello_world_mpi

Example: openmpi.sh

#!/bin/bash
#SBATCH -J OpenMPIJob             // Name your job
#SBATCH -n 128                    // 128 tasks in total on
#SBATCH -N 2                      // 2 nodes
#SBATCH -p small_cpu              // select small_cpu partition (small_cpu only supports up to 64 threads!)
#SBATCH --mail-type=ALL           // Send mail when job starts and finishes
#SBATCH --mail-user=<your-email-address>

# Disable MPI routing via CUDA framework. This will increase MPI speed.
export OMPI_MCA_btl="^smcuda"

srun ~/hello_world_mpi

srun will take care about all OpenMPI needed environment variables

If your code is not directly MPI aware, you can use mpirun or mpiexec. These wrappers are aware of the SLURM environment and do not require a hostfile. But if needed, you may also export some SLURM variables to your code:

[...]
#SBATCH --cpus-per-task=8

# set number of threads to requested cpus-per-task
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# for Slurm version >22.05: cpus-per-task has to be set again for srun
export SRUN_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK

mpirun -np $SLURM_NTASKS -x OMP_NUM_THREADS  my_mpi_program

Hybrid job

In this example, one H100 GPU is allocated. The executable will be run using 2 MPI processes with 8 OpenMP threads each. A common use case for that scenario is for example gromacs.

Example: gromacs.sh

#!/bin/bash
#SBATCH -J GromacsJob             // Name your job
#SBATCH --ntasks 2                // 2 tasks (OpenMPI main) with 
#SBATCH --cpus-per-task=8         // 8 OpenMP processes per task
#SBATCH --gres:gpu:H100:1         // Select 1 H100 GPU
#SBATCH -p h100                   // Select h100 partition
#SBATCH --mail-type=ALL           // Send mail when job starts and finishes
#SBATCH --mail-user=<your-email-address>

# Load Gromacs environment
source $HOME/bin/GMXRC

cd workdir
gmx mdrun -pme gpu -deffnm <input_file>