MPI Jobs and Infiniband

MPI is used for Distributed Memory Parallelization. Here, multiple instances of a binary are launched (possibly on different nodes) with some network communication between them to exchange data.

These multiple instances of your program are tasks in SLURM's sense, so use the --ntasks or, short, -n option in your job script.

On Julia HPC, the default MPI implementation is OpenMPI. Intel MPI is also available.

Code

Example hello_world_mpi.c by Wes Kendall from www.mpitutorial.com

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
  // Initialize the MPI environment. The two arguments to MPI Init are not
  // currently used by MPI implementations, but are there in case future
  // implementations might need the arguments.
  MPI_Init(NULL, NULL);

  // Get the number of processes
  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);

  // Get the rank of the process
  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  // Get the name of the processor
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name, &name_len);

  // Print off a hello world message
  printf("Hello world from processor %s, rank %d out of %d processors\n",
         processor_name, world_rank, world_size);

  // Finalize the MPI environment. No more MPI calls can be made after this
  MPI_Finalize();
}

OpenMPI

OpenMPI with GNU Compiler

Compile the example code with:

$ mpicc -o hello_world_mpi hello_world_mpi.c

OpenMPI with Intel Compiler

Using OpenMPI with the Intel compilers is currently not supported on Julia HPC.

Run OpenMPI Jobs

Set up your job script mpi_job.sh:

#!/bin/bash

#SBATCH -J MPIJob
#SBATCH -N 4
#SBATCH --ntasks-per-node=32

srun ./hello_world_mpi

This requests 4 machines (-N 4) each running 32 tasks, i.e. 128 tasks in total.

Submit your job:

$ sbatch mpi_job.sh

Intel MPI

Since Intel MPI is not the default implementation on Julia HPC, the usual MPI compiler commands need to be suffixed by .intel.

Intel MPI with GNU Compiler

Compile the example code with:

$ mpicc.intel -o hello_world_mpi hello_world_mpi.c

Intel MPI with Intel Compiler

The Intel implementation of mpicc accept a -cc switch to specify the desired underlying compiler. So compile the example code with:

$ mpicc.intel -cc=icc -o hello_world_mpi hello_world_mpi.c

Run Intel MPI Jobs

Make sure that the environment variable I_MPI_PMI_LIBRARY points to the correct location of SLURM's PMI library: /usr/lib/x86_64-linux-gnu/libpmi.so.0. This should already be set from the shell's profile file. In case it isn't, run

$ export I_MPI_PMI_LIBRARY=/usr/lib/x86_64-linux-gnu/libpmi.so.0

or include the above line in your job script.

Use a job script like the following mpi_job_intel.sh:

#!/bin/bash

#SBATCH -J MPIJob
#SBATCH -N 4
#SBATCH --ntasks-per-node=32

srun ./hello_world_mpi

and submit your job:

$ sbatch mpi_job_intel.sh

Like in the OpenMPI case above, this runs the job with a total of 128 tasks on 4 machines (-N 4) each running 32 tasks.

If you want to select a particular network fabric (TCP, for example), set the I_MPI_FABRICS environment variable either by running

$ export I_MPI_FABRICS=tcp

or including the above line in your job script.

Infiniband

Some nodes of Julia HPC have an Infiniband interconnect. They have, in SLURM terminology, a Feature named ib which you can request using the Constraint mechanism for your job: Use the option --constraint or -C (upper case C, not the lower case c from the well-known CPUs-per-task option!) like:

$ sbatch -C ib script.sh

MPI with Infiniband

To have MPI use Infiniband for communication, set a proper environment variable in you job script.

• On OpenMPI: export OMPI_MCA_btl=openib,self.
• On Intel MPI: export I_MPI_FABRICS=ofi for Infiniband and export I_MPI_FABRICS=tcp for Ethernet (default).

A note on mpirun

MPICH

MPICH is not supported on Julia HPC.