Job Scheduler

Job Scheduler#

A job scheduler, also known as a queueing system or workload manager, is software that manages jobs submitted to an HPC system. It allocates compute resources to jobs according to the system’s scheduling policies, allowing many users to run workloads efficiently and fairly.

A job scheduler is particularly useful for HPC workloads because jobs may require substantial resources or run for long periods. Rather than waiting for a job to finish in an interactive terminal, you can submit it to the scheduler and allow it to run when the required resources become available.

Why use a job scheduler?#

The job scheduler helps to:

Use resources efficiently: HPC systems are expensive to operate and consume significant amounts of energy. Scheduling jobs efficiently helps maximise system throughput and resource utilisation.
Save users time: Long running or repetitive workloads can run without requiring you to monitor them continuously.
Share resources fairly: The scheduler manages access to the available resources so that users and groups can share the system effectively.

Which job scheduler do Aire and Calder use?#

Both Aire and Calder use Slurm, a widely used workload manager in HPC systems.

Slurm is configured differently at different HPC centres to meet local requirements. However, the main commands and options are generally the same or similar between systems.

Used Slurm before?

If you have used Slurm on another HPC system, many of the commands and concepts will be familiar. However, always check the local documentation for the partitions, resource limits, and other configuration specific to Aire and Calder.

Submitting a job to the scheduler#

To submit a batch job, first create a job script containing the commands needed to run your application. The script also specifies the resources required by the job, such as the number of CPU cores, memory, GPUs, and the maximum run time.

You can then submit the script using Slurm’s sbatch command:

sbatch my_job.sh

Slurm will assign a job ID when the job is submitted. For example:

Submitted batch job 123456

The job ID can be used to monitor the job and inspect its accounting information after it has completed.

For interactive jobs, use Slurm’s srun command.

After a job is submitted#

After a job is submitted, Slurm determines when it can run based on the resources requested, the availability of those resources, and the scheduling policies in place.

If the required resources are available and the job can be scheduled, it will start running. Otherwise, it will remain in the queue until resources become available and the job reaches a position where it can run.

Checking the status of a job#

Use squeue to view jobs that are currently queued or running.

To see your own jobs:

squeue --user=$USER

You can also check a specific job by providing its job ID:

squeue --job=123456

The job’s state is shown in the ST or STATE column. Common states include:

PD (PENDING) - the job is waiting to run.
R (RUNNING) - the job is currently running.
CG (COMPLETING) - the job has finished its main work and is completing.
CD (COMPLETED) - the job completed successfully.
F (FAILED) - the job terminated unsuccessfully.
CA (CANCELLED) - the job was cancelled.

For pending jobs, the REASON column can provide useful information about why the job has not started.

Tip

If a job is waiting in the queue, do not assume that it is stuck. The scheduler may be waiting for the resources requested by the job to become available or for the job to reach a sufficiently high scheduling priority.

Checking completed jobs with `sacct`#

Once a job has finished, it will normally no longer appear in squeue. Use sacct to view accounting information for completed jobs and job steps.

For example:

sacct --job=123456

This provides information such as the job state, elapsed time, and exit code. You can request specific fields to make the output easier to interpret:

sacct --job=123456 --format=JobID,JobName,State,Elapsed,AllocCPUS,MaxRSS,ExitCode

Some useful fields include:

JobID - the job or job step ID.
JobName - the name of the job.
State - the final state of the job.
Elapsed - the amount of wall clock time for which the job ran.
AllocCPUS - the number of CPUs allocated to the job.
MaxRSS - the maximum resident memory used by the job or job step.
ExitCode - the exit code returned by the job.

sacct is particularly useful after a job has completed because it can help you understand how the job ran and whether the resources requested were appropriate. For example, MaxRSS can help identify whether you requested substantially more memory than the job required.

For more information about the fields available in sacct, see the Slurm sacct documentation.

Job output#

Unless otherwise specified in the job script, Slurm writes the standard output and standard error from a batch job to an output file in the directory from which the job was submitted.

You can specify separate output and error files using:

#SBATCH --output=output_%j.out
#SBATCH --error=error_%j.err

Here, %j is replaced by the job ID.

Cancelling a job#

If you need to stop a queued or running job, use scancel with the job ID:

scancel 123456

You can cancel all of your jobs with:

scancel --user=$USER

Note

Use scancel carefully. Cancelling a running job will terminate it, and any work that has not been saved may be lost.

Choosing resources for your job#

When submitting a job, request the resources that your application actually needs. These can include:

CPU cores
Memory
GPUs
Number of nodes
Wall time

Requesting substantially more resources than your application requires can increase the time you wait for the job to start and can reduce overall system utilisation. Conversely, requesting too few resources may cause the job to fail or perform poorly.

The available resources and limits differ between Aire and Calder. Refer to the system specific guidance when selecting a partition and requesting resources.