SLURM Partitions

Use the sinfo command to show an overview of available partitions. When you later want to submit a job to the cluster, you need to select one of these partitions. Partitions have different profiles on what ressources are available for your job.

$ sinfo
PARTITION               AVAIL  TIMELIMIT  NODES  STATE NODELIST
small_cpu*                 up 2-00:00:00      4    mix jn[001,006,008,010]
small_cpu*                 up 2-00:00:00      2  alloc jn[007,009]
small_cpu*                 up 2-00:00:00     14   idle jn[002-005,011-020]
large_cpu                  up 2-00:00:00      7    mix jn[101-102,104,116-117,119-120]
large_cpu                  up 2-00:00:00      2  alloc jn[103,118]
large_cpu                  up 2-00:00:00     11   idle jn[105-115]
standard                   up 1-00:00:00     43   mix- jn[001-006,008-020,101-120],jnfat[01-03,08-09]
standard                   up 1-00:00:00      1  alloc jn007
h100                       up 1-00:00:00      2   idle jnultra[01-02]
test                       up    1:00:00      1   idle jntest01
gpu_computervision         up 2-00:00:00      1   mix- jnfat06
gpu_computervision         up 2-00:00:00      3    mix jnfat[04-05,07]
gpu_computervision_long    up 4-00:00:00      1   mix- jnfat06
gpu_computervision_long    up 4-00:00:00      3    mix jnfat[04-05,07]
cpu_standard               up 2-00:00:00      9   mix- jncpu[03,05-12]
cpu_standard               up 2-00:00:00      3  alloc jncpu[01-02,04]
cpu_long                   up 7-00:00:00      1   mix- jncpu13
cpu_long                   up 7-00:00:00      2   plnd jncpu[17-18]
cpu_long                   up 7-00:00:00      2    mix jncpu[14,16]
cpu_long                   up 7-00:00:00      1   idle jncpu15

Overview

The following partitions are available:

  • small_cpu: CPU only: max 64 threads, 256 GB RAM
  • large_cpu: CPU only: max 128 threads, 256 GB RAM
  • standard: one to eight GPUs and 16 threads per GPU, 256-512 GB RAM
  • h100: one to eight H100 GPUs, if you really need one of those, 1 TB RAM
  • test: Test partition to test your jobs
  • gpu_computervision: Dedicated servers for the computervision group. You can run jobs here, but the computervision group can preempt your jobs here.
  • gpu_computervision_long: Dedicated servers for the computervision group only.
  • cpu_standard: CPU only: max 384 threads, 1 TB RAM
  • cpu_long: CPU only. Dedicated servers for the Computational Quantum Materials group only.

If you don't set a partion name for your job, small_cpu will be set as default. You can select an partition in your job file by using the -p <partition> option. A set of example job files can be found in the Basic Jobs chapter.

Default memory: 2G per thread. Change with --mem=xG

Runtime Limit: As you can see 1-00:00:0, right now one day is the maximum runtime limit for gpu jobs (WallTime) for your jobs. Make sure your job supports resuming. CPU only jobs have a maximum runtime limit of two days.

GPU selection

Use the standard-partition with --gres=gpu:<x>, where <x> can be 1 up to 8.

You can also select specific models like:

srun -p standard --gres=gpu:L40:<x>, where <x> can be 1 up to 3 for one single node or
srun -p standard --gres=gpu:L40s:<x>, where <x> can be 1 up to 8 for one single node

These commands will create an interactive session, where you can also test your code. It is not recommended to use interactive sessions for productive use, since the login node can be rebooted anytime, if there's an issue. Also if the cluster is full, your interactive shell may get scheduled and started any time. Usually its on the weekend at 04:53h in the morning, as an example.

Use the h100 partition for the special H100 cards.

srun -p h100 --gres=gpu:H100:<x>, where <x> can be 1 up to 8.