SLURM Introduction

The SLURM Nodes of Julia HPC are currently using Debian 10 (Codename "Buster") as operating system. Detailed documentation on SLURM can be found at slurm.schedmd.com

Login

Login via SSH (German SSH HowTo) to julia.uni-wuerzburg.de.

You will find yourself in your home directory of the cluster environment. Now you can start to:

• setup and compile your code
• write job scripts and run jobs through SLURM

The latter point is what this documentation is about.

A note on data transfer

If you need to copy a lot of data, e.g. simulation results or analysis data, please use julia-storage.uni-wuerzburg.de for the data transfer. User data is available the same way as on the regular cluster headnode. However, the storage machine has dedicated interfaces to both the storage backend and the university network.

You can login via SSH (just the same way as above), and transfer files using scp or rsync.

The cluster environment

Compute nodes within SLURM are generally referenced by name. On Julia HPC, the following nodes are available:

• cn10g-05 to cn10g-24, and cn10g-45 to cn10g-52
• cnib-01 to cnib-07
• gpu1 and gpu2
• fat2 to fat4
• cntest1 and cntest2 are virtual machines for running short jobs to see if your program runs properly

The Queue

SLURM builds a queue of all submitted jobs and uses a scheduling algorithm to decide which job can run at which time on which node.

The command squeue gives an overview of jobs that are currently running (state R) or pending in the queue (state PD). A normal user will only be able to see their own jobs here, administrators see all jobs. An example could look like this:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               110  standard  someJob  user-01 PD       0:00      1 (Resources)
                97  standard   foobar  user-02  R 1-01:00:50     16 cn10g-[05-20]
                96       gpu   barbaz  user-02  R    3:32:34      2 gpu[1-2]
                95  standard whatever  user-02  R 1-01:55:01      4 cn10g-[21-24]

Pending jobs are assigned a priority value which is frequently recalculated based on a number of factors, see the job priority page for details.

The job with the highest priority is supposed to run as soon as there are sufficient resources available. Jobs with lower priority have to wait unless they don't delay the start of a higher priority job, e.g. they use different nodes or are short and finish anyway before the other jobs begins.

Overall status of the cluster

Run sinfo to get a general overview of the current state of the cluster. Example:

$ sinfo 
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
standard*    up 2-00:00:00      4    mix cn10g-[23-24,51-52]
standard*    up 2-00:00:00     31  alloc cn10g-[05-22,45-50],cnib-[01-07]
gpu          up 2-00:00:00      2   idle gpu[1-2]
fat          up 2-00:00:00      2  alloc fat[2-3]
fat          up 2-00:00:00      1   idle fat4
short        up   12:00:00      4    mix cn10g-[23-24,51-52]
short        up   12:00:00     31  alloc cn10g-[05-22,45-50],cnib-[01-07]
short-gpu    up   12:00:00      2   idle gpu[1-2]
short-fat    up   12:00:00      2  alloc fat[2-3]
short-fat    up   12:00:00      1   idle fat4
long         up 7-00:00:00      4    mix cn10g-[23-24,51-52]
long         up 7-00:00:00     31  alloc cn10g-[05-22,45-50],cnib-[01-07]
long-gpu     up 7-00:00:00      2   idle gpu[1-2]
long-fat     up 7-00:00:00      2  alloc fat[2-3]
long-fat     up 7-00:00:00      1   idle fat4
ultra        up 30-00:00:0      4    mix cn10g-[23-24,51-52]
ultra        up 30-00:00:0     31  alloc cn10g-[05-22,45-50],cnib-[01-07]
ultra-gpu    up 30-00:00:0      2   idle gpu[1-2]
ultra-fat    up 30-00:00:0      2  alloc fat[2-3]
ultra-fat    up 30-00:00:0      1   idle fat4
test         up    1:00:00      2   idle cntest[1-2]

Interpreting the output (first five plus the last line in output, ignoring the partitions with short-, long- and ultra- prefixes for now):

• The nodes cn10g-05 to cn10g-22, cn10g-45 to cn10g-50 and cnib-01 to cnib-07 are fully allocated and currently in use
• The same goes for the nodes fat2 and fat3.
• The nodes cn10g-23, cn10g-24, cn10g-51 and cn10g-52 are in a "mixed" state, meaning they are partly occupied by jobs, but have some free CPUs left.
• Both GPU nodes gpu1 and gpu2 as well as the node fat4 are idle, ready to accept jobs.
• Both test nodes cntest1 and cntest2 are also idle.

Here, some small jobs could be run on the mixed-state nodes, and GPU and fat node jobs also have a chance of running. Large jobs, however, that require a full node or even multiple nodes will have to wait in the queue for free resources.

Partitions and time limits

Partitions (you will have noticed in the examples above) are a group of nodes together with a set of properties and limits. Compute nodes can belong to different partitions at the same time when limits or available features differ between those partitions. Most notably, the maximum time a single job is allowed to run on Julia HPC is different, depending on the partition the job is submitted to. If a user does not provide an explicit time limit for a job, the limit is set automatically (roughly a quarter of the partitions maximum time).

Partitions also affect the job's priority value: Jobs submitted to partitions with a higher time limit will have a lower priority.

See the partitions overview for a detailed description of the available partitions and their resources and limits.

Detailed job and node description

Nodes

Run scontrol with the arguments show node and the nodename to see a detailed description of the node's configured state and its configured and allocated resources:

$ scontrol show node cnib-01
NodeName=cnib-01 Arch=x86_64 CoresPerSocket=4
   CPUAlloc=2 CPUErr=0 CPUTot=32 CPULoad=0.02
   AvailableFeatures=ib
   ActiveFeatures=ib
   Gres=(null)
   NodeAddr=cnib-01 NodeHostName=cnib-01 Version=17.11
   OS=Linux 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u1 (2018-08-03) 
   RealMemory=384000 AllocMem=2048 FreeMem=380058 Sockets=4 Boards=1
   State=MIXED ThreadsPerCore=2 TmpDisk=0 Weight=2 Owner=N/A MCS_label=N/A
   Partitions=any,ib 
   BootTime=2018-08-09T16:31:35 SlurmdStartTime=2018-08-09T16:18:43
   CfgTRES=cpu=32,mem=375G,billing=32
   AllocTRES=cpu=2,mem=2G
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

As you can see from the CPUTot item (second line of output), this machine has 32 CPUs. The RealMemory item in line 8 tells you that it has 384000 MiB (= 375 GiB) of RAM, see also CfgTRES in line 12.

Additionally, this machine has a special feature listed at AvailableFeatures (line 2): The value of ib marks this node as a machine with enabled Infiniband support.

Jobs

Run scontrol with show jobid= and the numeric id of your job to get a detailed description of the job.

Storage

We provide a net total of 500TB for the home directories, employing a BeeGFS storage system. A disk usage quota is set on a per-group basis.

Compute nodes have local storage capacity for temporary use by currently running jobs.

Disk usage and quota

You can check your disk usage with the du command, e.g.:

$ du -sch $HOME
603M    /home/clusteruser
603M    total

Since a group level quota is set for the storage, you can compare your usage with the group quota.

First check your primary group:

$ id
uid=123456(clusteruser) gid=31798(42-julia-hpc-rz)

Look at the gid number. In this case it is 31798. Now you can check your group quota on the BeeGFS storage with your gid number:

$ /sbin/beegfs-ctl --getquota --gid 31798
      user/group      ||           size          ||    chunk files    
     name      |  id  ||    used    |    hard    ||  used   |  hard   
---------------|------||------------|------------||---------|---------
42-julia-hpc-rz| 31798||    2.07 GiB|  500.00 GiB||    38084|unlimited

In this example the group uses 2.07 GiB of a 500.00 GiB disk quota.

Scratch space

For each job, a private folder is created. It is available to the job at /tmp and gets deleted when the job ends.

A quota is set on this particular folder, depending the user's specification at submit time. Job will receive an error message such as "No space left on device" or "Quota exceeded" if their node-local output is larger than the quota.

The local disks of the compute nodes are about 1.7 TB in size. An exception are the FAT nodes with local disks of roughly 7 TB.