Job Priority

The job queue in SLURM is ordered by the jobs' priority value. This number is calculated considering three points:

• How long has it been waiting in the queue (the Age component)?
• To which partition has the job been submitted (the Partition component)?
• How much computation time has the user and their colleagues already used? This is referred to as their FairShare priority.

The total job priority is then the sum of those three values.

Age

The job's priority grows linear over time, up to some maximum. At Julia HPC, that maximum value is reached after three weeks, at a value larger than the other parts of the priority calculation.

Partition

Right now each partition have a assigned priority offset. Since we currently do not use partitions with different runtimes, all partitions have the dame offset. This might change when different runtime limits are introduced.

FairShare

This is the complicated one. A complete description of the Fair Tree Fairshare Algorithm can be found at SchedMD.

In short, the FairShare calculation is based on an account tree. On Julia HPC this is roughly based on the organizational units of faculty, institute, chair, workgroup and individual user, although some levels have been left out to reflect to local conditions. At each level, the available computation time is broken down among all subdivision of that unit, employing some weighting if needed.

The actually used computing time is also broken down following that tree. Depending on whether a group or a user has used more or less computing time than their allocated share, the user (or the group as a whole) receive a bonus or malus on the priority of their jobs.

You can check your own fairshare value using sacct. Users or groups who used the cluster a lot, will have fairshare values close to 0; those who did not use it at all, have ~1.

The cluster usage that goes into this calculation decays over time so that the most recent jobs have the largest impact on the fairshare value.

On Julia HPC, there are basically two groups of users:

• AI users
• everybody else

The AI users contributed massively to the funding of the cluster. So they are priorized for GPU scheduling. Since the AI users do not heavy rely on CPU computing, half of the CPU ressources are available for general computing

Display a job's priority

To display a job's priority, you can add this field to the output of squeue.

• with the -o option use format specification %Q
• with the -O option (upper case O) use prioritylong

Example:

$ squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %.8Q %R"
             JOBID PARTITION     NAME     USER ST       TIME  NODES PRIORITY NODELIST(REASON)
            123456  standard   my_job  user123 PD       0:00      1   619521 (Resources)

The sprio command can be used to break down that priority number to the individual parts:

$ sprio -j 206030 -l
          JOBID PARTITION     USER  ACCOUNT   PRIORITY       SITE        AGE      ASSOC  FAIRSHARE    JOBSIZE  PARTITION    QOSNAME        QOS        NICE                 TRES
         206030 standard  some_user account       2440          0       2438          0          3          0          0     normal          0           0