DEPRECATED

Check Julia 2 documentation instead!

Julia HPC SLURM Upgrade

June 2020

What has changed:

  1. No more Single-Core jobs
  2. Operating System Update
  3. Node-local scratch space
  4. Graphics Cards
  5. Intel Compiler and Libraries
  6. Infiniband and OpenMPI
  7. SSH
  8. Group Names

No more single-core jobs

Single-core jobs (in fact, any job with an odd number of CPUs) are no longer allowed! This is probably the change that most users will notice.

The compute nodes have hyperthreaded CPUs, and in SLURM, the two hyperthreads cannot be shared between two jobs. Hence it was never even possible to run a pure single-core (single-hyperthread, more specifically) job, and when such a job was submitted, it left a significant amount of computing power unusable. The system was just hiding that fact from the user.

We decided therefore to change the default behavior and reject jobs requesting an uneven number of CPUs right away. So when you submit a job, set -c/--cpus-per-task or -n/--ntasks to an even number.

Operating System Update

The job scheduling part of Julia HPC now runs on Debian 10, the SLURM version is 19.05. It is quite likely that the versions of compilers and shared libraries have changed. You will have to recompile your code and thereby link against the new library versions available on the system. Notably, newer versions of Python (now 3.7) and Singularity (now 3.1.1) are available

Node-local scratch space

Each job now gets its own node-local directory mounted to /tmp. This directory is deleted when the job ends. You can specify the maximum size of this directory using the sbatch/srun option --tmp. The default value is --tmp=1G, providing 1 GiB of node-local temporary disk space to the job.

Graphics Cards

Using the graphics cards is only possible if that resource is explicitly requested. You can do this by using on of the following sbatch options:

  • --gres=gpu:1 (or :2 if you want to use both GPUs in a node)
  • --gpus-per-node=1 (or 2)
  • --gpus-per-task=1 (or 2)

For some reason that is still unknown to the Julia Team, the general new options -G and --gpus (that should replace the --gres=gpu variant) do not work.

Additionally, you will still need to request one of the gpu-partions, e.g. -p gpu.

Intel Compilier and Libraries

We updated the INTEL Compiler suite to 19.05. Please recompile your code if necessary. This is also the case for INTEL libraries like MKL.

You do not need to source any shell scripts or set up environment variables any more to use the INTEL compilers or libraries. Binaries and libraries are found in the default paths. INTEL MPI compiler wrappers are available at mpicc.intel or mpifort.intel in the default path.

Infiniband and OpenMPI

Due to a bug in OpenMPI >3.1.2 [1], it is currently not possible to use Infiniband with OpenMPI. Therefore, we disabled Infiniband as OpenMPI provider globally. OpenMPI via TCP still works and is the default provider. An alternative is to use Infiniband with INTEL MPI instead of OpenMPI. We are investigating if an update to OpenMPI 4.x is feasible.

SSH

Users can login via SSH to a node on which they have a running job. This SSH session is subject to the same limits (CPU, Memory, GPU) that were requested for the job and is terminated when the job ends.

Group Names

The names of the "accounts" in SLURM terminology have changed to the pattern group<five-digit-number>. Most users will be unaffected by this.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941996