GPU Jobs

To use a GPU in your calculation, send the job to those nodes that have a GPU :-) These are the nodes in the gpu partition (and, same machines with other time limits, in short-gpu, long-gpu and ultra-gpu).

You will also need to explicitly request a GPU, otherwise you will not have access to it.

Code

Consider this CUDA example cudatest.cu (from An Even Easier Introduction to CUDA at the NVIDIA Developer Blog):

#include <iostream>
#include <math.h>

// Kernel function to add the elements of two arrays
__global__
void add(int n, float *x, float *y)
{
  for (int i = 0; i < n; i++)
    y[i] = x[i] + y[i];
}

int main(void)
{
  int N = 1<<20;
  float *x, *y;

  // Allocate Unified Memory – accessible from CPU or GPU
  cudaMallocManaged(&x, N*sizeof(float));
  cudaMallocManaged(&y, N*sizeof(float));

  // initialize x and y arrays on the host
  for (int i = 0; i < N; i++) {
    x[i] = 1.0f;
    y[i] = 2.0f;
  }

  // Run kernel on 1M elements on the GPU
  add<<<1, 1>>>(N, x, y);

  // Wait for GPU to finish before accessing on host
  cudaDeviceSynchronize();

  // Check for errors (all values should be 3.0f)
  float maxError = 0.0f;
  for (int i = 0; i < N; i++)
    maxError = fmax(maxError, fabs(y[i]-3.0f));
  std::cout << "Max error: " << maxError << std::endl;

  // Free memory
  cudaFree(x);
  cudaFree(y);

  return 0;
}

Compile with NVIDIA's CUDA compiler to create a cudatest binary:

$ nvcc cudatest.cu -o cudatest

When run successfully, it will look like this:

user@gpu1:~/GPUexample$ ./cudatest 
Max error: 0

Submit job

As mentioned above, you need to submit the job to one of the GPU partitions and request an actual GPU. For the latter, there are three ways to do this:

• Use the generic --gres option to sbatch: --gres=gpu:1 (or :2 if your job requires both GPUs in the node)
• Specify the number of GPUs per task: --gpus-per-task=1 (or =2)
• Specify the number of GPUs per node: --gpus-per-node=1 (or =2)

An job script gputest.sh for the above code example might therefore look like

#!/bin/bash

#SBATCH -J CUDAtest
#SBATCH -p short-gpu
#SBATCH -c 2
#SBATCH --gpus-per-task=1

srun ./cudatest

and is submitted the usual way:

$ sbatch gputest.sh