Job Priority

The job queue in SLURM is ordered by the jobs' priority value. This number is calculated considering three points:

• How long has it been waiting in the queue (the Age component)?
• To which partition has the job been submitted (the Partition component)?
• How much computation time has the user and their colleagues already used? This is referred to as their FairShare priority.

The total job priority is then the sum of those three values.

Age

The job's priority grows linear over time, up to some maximum. At Julia HPC, that maximum value is reached after three weeks, at a value larger than the other parts of the priority calculation.

Partition

Each job is assigned a priority offset based on the partition to which it is submitted. The longer the maximum walltime limit of this partition is, the smaller is the priority offset. Therefore, jobs that use the cluster for a longer time may enter the queue at a lower position compared to the shorter ones that get a bonus. The longer jobs can compensate this by waiting in the queue for a longer time, until their age priority surpasses the shorter jobs' bonus.

FairShare

This is the complicated one. A complete description of the Classic Fairshare Algorithm can be found at SchedMD.

In short, the FairShare calculation is based on an account tree. On Julia HPC this is roughly based on the organizational units of faculty, institute, chair, workgroup and individual user, although some levels have been left out to reflect to local conditions. At each level, the available computation time is broken down among all subdivision of that unit, employing some weighting if needed.

The actually used computing time is also broken down following that tree. Depending on whether a group or a user has used more or less computing time than their allocated share, the user (or the group as a whole) receive a bonus or malus on the priority of their jobs.

You can check your own fairshare value using sacct. Users or groups who used the cluster a lot, will have fairshare values close to 0; those who did not use it at all, have ~1.

The cluster usage that goes into this calculation decays over time so that the most recent jobs have the largest impact on the fairshare value.

On Julia HPC, there are basically three groups of users:

• Physics Department
• Biology Department
• everybody else

Physics and Biology contributed to the funding of the cluster, so they get shares of the computing time equivalent to their respective financial contribution.

Display a job's priority

To display a job's priority, you can add this field to the output of squeue.

• with the -o option use format specification %Q
• with the -O option (upper case O) use prioritylong

Example:

$ squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %.8Q %R"
             JOBID PARTITION     NAME     USER ST       TIME  NODES PRIORITY NODELIST(REASON)
            123456  standard   my_job  user123 PD       0:00      1   619521 (Resources)

The sprio command can be used to break down that priority number to the individual parts:

$ sprio -j 123456
          JOBID PARTITION   PRIORITY       SITE        AGE  FAIRSHARE        QOS
         123456  standard     619521          0      61158     558364          0