Job Submission

The submit nodes on the cluster should only be used for a limited range of tasks that do not have high computational demands. When in doubt it is better to complete a task as a job on a compute node.

The UCHC Computer clusters use SLURM for managing and scheduling jobs.

With SLURM there are two primary ways that a job can be run on a compute node.

1. Batch Submission
2. Interactive Jobs

Batch job submission

A batch job submission will take your job script, put it into a queue, and run it for you once the required resources are available. This is the most common way to run jobs on the cluster.

To submit a batch job, use the sbatch command. A basic command might look like:

sbatch \
  --job-name <name> \
  --cpus-per-task <cpus> \
  --mem <memory> \
  --time <time> \
  --qos <qos> \
  --output %x-%j.out \
  <script>

Where <script> is the name of the script that you want to run as a job.

Alternatively, the sbatch arguments can be specified as a header in your script:

#!/bin/bash
#SBATCH --job-name=<job name>
#SBATCH --cpus-per-task=<cpu number> 
#SBATCH --mem=<memory>
#SBATCH --time=<days-hrs:min:sec>
#SBATCH --qos=<partition name>
#SBATCH --output=%x-%j.out

<your script here>

This would be run with the command: sbatch <script>.sh

Example

Here is a more complete example which also demonstrates the use of SLURM environment variables which access the values of the --job-name and --cpus-per-task arguments in the job script. Note the use of the hostname command to show the node that the job is running on. This information can be useful for debugging jobs. Particularly if you request help from the CBC as we will ask for this information.

#!/bin/bash
#SBATCH --job-name=test-job
#SBATCH --cpus-per-task=1 
#SBATCH --mem=100mb
#SBATCH --time=1:00
#SBATCH --qos=debug
#SBATCH --output=%x-%j.out

hostname

echo "$SBATCH_JOB_NAME with $SLURM_CPUS_PER_TASK cpus"

Required arguments

The following arguments are required when running sbatch on the UCHC HPC system.

--qos

The qos (quality of service) argument constrains or modifies characteristics of a job such as scheduling priority, preemption, and resource limits. The qos argument must be specified for every job run on the Mantis cluster. This argument will often be the same as that given for the partition if specifying a partition. See the QOS section for more information about available QOS options.

Commonly used arguments

Below are some of the most useful and commonly used arguments for sbatch.

--job-name

The name of the job. Used for displaying job information when monitoring the job and for naming log files.

--cpus-per-task

The number of cpus to request. The default is 1. More can be requested when a multiple cpus can be used to parallelize tasks.

--mem

The amount of memory to allocate. This argument takes a number followed by a unit such as mb for megabytes or gb for gigabytes. If no unit is given, megabytes are used. If no --mem argument is used, a job defaults to 120 MB.

--time

Time limit for the job. Jobs not finished by the end of the allocated time will be killed. The value for this argument must specified with the format days-hrs:min:sec, days-hrs, hrs:min:sec, min:sec, or sec. The default time limit in non-debug partitions is 21 days. However, the time limit has an impact on the priority of jobs in the queue. Jobs with shorter time limits are likely to begin running sooner.

--output

This specifies the path of the log file where stdout is written. In the examples above %x is replaced with the job name and %j is replaced with the job ID. See the other file name pattern options here.

--partition

A group of nodes which the job will be allocated to. This defaults to general. See the partitions section for available partitions.

--mail-type

When to send emails. Options are NONE, BEGIN, END, FAIL, REQUEUE, ALL. Defaults to NONE.

--mail-user

Address to send mail to.

Additional commands

sbatch is a powerful tool with many other configuration options beyond the scope of this documentation.

See the sbatch documentation for more information about command arguments.

Interactive Jobs

An interactive session can be started on a compute node using the srun command. Most sbatch arguments can also be used with srun.

Required arguments

As with sbatch, the --qos argument is required for srun.

The --pty argument and a command to execute are also required to start an interactive session.

QOS

A special QOS (interactive) can optionally be used for starting interactive jobs which will give the job a higher priority in the queue. This QOS has the following limits:

• 1 concurrent job
• no more than 8 CPUs
• no more than 64 GB memory
• no more than 8 hours of wall time

Example

At a minimum, you would need to use these arguments when requesting an interactive session:

srun --qos interactive --pty bash

or if you don’t want to be bound by the limits of the interactive QOS:

srun --qos general --pty bash

Most sbatch arguments can also be used with srun. For example, --mem or --cpus-per-task if you needed to request additional resources.

Commands other than bash could be used to start an interactive session. But this is a topic is beyond the scope of this documentation. See the srun documentation for more information about command arguments.

Job status and monitoring

To monitor the status of your jobs you can run squeue -u <user> which will list the status of all jobs that belong to you. See the squeue documentation for additional arguments and output formatting options.

Canceling a job

If you need to cancel a job that is currently running or in the queue, you can use the scancel command. The basic syntax is:

scancel <job_id>

Request time extensions

If you have a job that may not finish within the allocated time limit you can submit a time extension request

Partitions

A partition is a group of compute nodes that share certain specifications and policies. Each partition is best suited to different types of tasks. The Mantis and Xanadu cluster have slightly different partitions summarized below.

Mantis

general

This partition contains nodes suitable for most tasks. These nodes have 64-128 AMD or Intel Xeon CPUs and 192 GB - 512 GB of ram. Most nodes in this partition have either one NVIDIA A100 or three NVIDIA L40S GPUs. Users can utilize up to 400 CPUs concurrently or with a limit of 400 concurrent jobs in this partition. Slurm defaults to this partition if no partition is specified.

Slurm Arguments:

#SBATCH --partition=general
#SBATCH --qos=general

himem

This parition contains nodes suitable for jobs that have very large memory requirements. These nodes have 128 AMD CPUs and 1 TB of ram. Each node in this partition has one NVIDIA A100 GPU. Jobs cannot run in this partition unless 500 GB of memory is requested. Only two jobs per user will run concurrently in this partition.

Slurm Arguments:

#SBATCH --partition=himem
#SBATCH --qos=himem

Xanadu

general

These nodes are suitable for most tasks.

Slurm Arguments:

#SBATCH --partition=general
#SBATCH --qos=general

xeon

Use this partition if you need to run on Intel Xeon CPUs.

Slurm Arguments:

#SBATCH --partition=xeon
#SBATCH --qos=general

amd

Use this partition if you need to run on AMD CPUs.

Slurm Arguments:

#SBATCH --partition=amd
#SBATCH --qos=general

himem

Use this partition if you have very large memory requirements > 500 GB.

Slurm Arguments:

#SBATCH --partition=himem
#SBATCH --qos=himem

gpu

Use this partition if you need to run on a GPU.

Slurm Arguments:

#SBATCH --partition=gpu
#SBATCH --qos=general

mcbstudent

This partition is available for students in courses or workshops who have been given access.

Slurm Arguments:

#SBATCH --partition=mcbstudent
#SBATCH --qos=mcbstudent

QOS (Quality of Service)

QOS (Quality of Service) is a SLURM feature that allows for the specification of job characteristics such as scheduling priority, preemption, and resource limits. The Mantis and Xanadu clusters have different QOS options available. See below for details.

Mantis

general

Suitable for most tasks.

himem

Required for jobs to run in the himem partition.

debug

The debug QOS is intended for testing and debugging jobs. Jobs with this QOS will have a time limit of only 5 minutes but will be given high priority and should start very quickly, allowing you to check for errors before submitting to a queue with potentially longer wait times.

Slurm Arguments:

#SBATCH --qos=debug

preemtible

The preemptible QOS is available on the Mantis cluster. Jobs submitted with this qos will have very high priority in the queue but may be preempted by higher priority jobs. This QOS is intended for users who are willing to have their jobs interrupted in exchange for potentially lower wait times when nodes are not in use by higher priority jobs. Tasks with short run times or tasks that can be restarted are good candidates for this QOS. This QOS can be used with any of the Mantis partitions.

Slurm Arguments:

#SBATCH --qos=preemptible

Xanadu

general

Suitable for most tasks.

himem

Required for jobs to run in the himem partition.

SLURM Environment Variables

SLURM provides a number of environment variables that can be used in job scripts to access information about the job and the resources allocated to it. See the SLURM environment variables documentation for a complete list of available variables.

GPUs

Allocation of a GPU differs on Mantis and Xanadu. See below for details on how to request a GPU on each cluster.

Mantis

A GPU will not be allocated to a job unless it is explicitly requested. In order to request a GPU, you can specify the --gres=gpu:[type:]<number> agrument where the required <number> is the number of GPUs to allocate to the job. The type is optional and specifies the type of GPU to allocate. The types of GPUs and the associated type arguments are listed in the table below. When requested under the general QOS, users will be limited to a maximum of 8 GPUs to be used concurrently. No limit is placed on the number of GPUs that can be used concurrently under the preemptible QOS. However, preemptible jobs can be interrupted by jobs from the general QOS at any time.

GPU Types

GPU	type	number per node
NVIDIA A100	A100	1
NVIDIA L40S	L40S	1-3

Example

#!/bin/bash
#SBATCH --job-name=test-gpu
#SBATCH --cpus-per-task 1
#SBATCH --mem=100M
#SBATCH --time=1:00
#SBATCH --qos=debug
#SBATCH --output=%x-%j.out
#SBATCH --gres=gpu:A100:1

# Detect GPU
nvidia-smi

Xanadu

Visit https://github.com/CBC-UCONN/CBC_Docs/wiki/xanadu-gpu-usage for information about requesting and using GPUs on the Xanadu cluster.

Feature Constraints

When a job requires specific hardware or software features on a compute node, a user can specify these requirements using the --constraint=<feature> argument where is a defined feature of a node. Multiple constraints can be specified with & (and) or | (or) operators. The defined features and description are listed in the table below. See the SLURM documentation for more information on using constraints.

The features defined for each node can be listed with the command sinfo -o "%N %f".

The defined features will differ between the Mantis and Xanadu clusters. See below for information about the defined features on each cluster.

Mantis

Defined Features

Feature	Description
cpu_amd	AMD CPUs
cpu_xeon	Intel Xeon CPUs
epyc_7352	AMD EPYC 7352 CPUs
xeon_platinum_8462Y+	Intel Xeon Platinum 8462Y CPUs

Xanadu

Some Defined Features

Feature	Description
cpu_amd	AMD CPUs
cpu_xeon	Intel Xeon CPUs

For others run sinfo -o "%N %f".

Example

Having software compiled for a specific CPU architecture is a common reason to need constraints. For example, if you have software that is optimized for AMD CPUs, you can use the following command on Mantis to ensure that your job runs on a node with AMD CPUs.

#!/bin/bash
#SBATCH --job-name=test-feature-constraint
#SBATCH --cpus-per-task 1 
#SBATCH --mem=100M
#SBATCH --time=1:00
#SBATCH --qos=debug
#SBATCH --output=%x-%j.out
#SBATCH --constraint=cpu_amd

lscpu

Job Arrays

If you have a large set of similar tasks that can be run in parallel, job arrays offer a convenient solution for managing these. Job arrays are created using the --array=<indexes>[%limit] argument where the required <indexes> is a range (e.g. 0-4) or list (e.g. 1,3,5,7) of indexes and an optional limit, separated by a % is the maximum allowed number of simultaneously running jobs (e.g. %20). Jobs which are part of a job array will have the environment variable SLURM_ARRAY_TASK_ID set to the array index value for that job. The maximum index allowed in a job array is 5001.

Example

Here is a basic example where the job array will run 5 jobs in parallel, each with a different index value from 1 to 5. A different param value is printed to stdout based on the job index. Only 2 jobs will run at the same time due to the %2.

#!/bin/bash
#SBATCH --job-name=test-array-job
#SBATCH --cpus-per-task 1 
#SBATCH --mem=100M
#SBATCH --time=1:00
#SBATCH --qos=debug
#SBATCH --output=%x-%A-%a.out
#SBATCH --array=1-5%2

params=("A" "B" "C" "D" "E")
param=${params[$SLURM_ARRAY_TASK_ID-1]}
echo $param

Parameter values will often be stored in a separate file. Below are some examples of ways to access these values in a job array script.

Accessing nth line from a text file

param=$(sed -n "${SLURM_ARRAY_TASK_ID}p" params.txt)

Accessing nth value and second column from a csv file

param=$(sed -n "${SLURM_ARRAY_TASK_ID}p" params.csv | cut -d ',' -f 2)

Accessing nth value and second column from a csv file with a header row

param=$(sed -n "1~${SLURM_ARRAY_TASK_ID}p" params.csv | cut -d ',' -f 2)

X11 Forwarding

Some applications, such as R, can produce graphical output. X Windows, a.k.a. X11, is an option for displaying that output in an interactive session.

MacOS

First you need to install XQuartz on your local machine.

Then, when connecting to Mantis, use the the ssh option -X (or possibly -Y if your application requires it) like this:

# external:
ssh -X <username>@login.hpc.cam.uchc.edu
# or when connected to the VPN:
ssh -X <username>@mantis-submit.cam.uchc.edu

Now, to run your software in an interactive session, start it using the option --x11. For example:

srun --x11 --partition=general --qos=general --mem=5G --pty bash

Now that you’re in an interactive session, you should be able to run your application.

Try launching R:

R

And then plot a histogram of random draws from a normal distribution:

hist(rnorm(10000))

The histogram should pop up in an XQuartz window.

Windows

In development

Linux

In development