Job Submission

The submit nodes on the cluster should only be used for a limited range of tasks that do not have high computational demands. See the best practices page for more information on appropriate use of the submit nodes. When in doubt it is better to complete a task as a job on a compute node.

The UCHC Computer cluster uses SLURM for managing and scheduling jobs.

With SLURM there are two primary ways that a job can be run on a compute node.

Batch Submission
Interactive Job

Batch job submission

A batch job submission will take your job script, put it into a queue, and run it for you once the required resources are available. This is the most common way to run jobs on the cluster.

To submit a batch job, use the sbatch command. A basic command might look like:

sbatch \
  --job-name <name> \
  --cpus-per-task <cpus> \
  --mem <memory> \
  --time <time> \
  --partition <partition> \
  --qos <qos> \
  --output %x-%j.out \
  <script>.sh

Alternatively, the sbatch arguments can be specified as a header in your script:

#/bin/bash

#SBATCH --job-name=<job name>
#SBATCH --cpus-per-task=<cpu number> 
#SBATCH --mem=<memory>
#SBATCH --time=<days-hrs:min:sec>
#SBATCH --partition=<partiion name>
#SBATCH --qos=<partition name>
#SBATCH --output=%x-%j.out

<your script here>

Example

Here is a complete example which also demonstrates the use of SLURM environment variables which access the values of the --job-name and --cpus-per-task arguments in the job script. Note the use of the hostname command to show the node that the job is running on. This information can be useful for debugging jobs. Particularly if you request help from the CBC as we will ask for this information.

#!/bin/bash
#SBATCH --job-name=test-job
#SBATCH --cpus-per-task=1 
#SBATCH --mem=100mb
#SBATCH --time=1:00
#SBATCH --partition=debug
#SBATCH --qos=debug
#SBATCH --output=%x-%j.out

hostname

echo $SBATCH_JOB_NAME with $SLURM_CPUS_PER_TASK cpus

Required arguments

The following arguments are required when running sbatch on the Mantis cluster. #### --partition A group of nodes which the job will be allocated to. A partition must be specified when running a job on the Mantis cluster. The available partitions are explained below.

--qos

The qos (quality of service) argument constrains or modifies characteristics of a job. A qos argument must be specified for every job run on the Mantis cluster. This argument will usually be the same as that given for the partition.

Commonly used arguments

Below are some of the most useful and commonly used arguments for sbatch. #### --job-name The name of the job. Used for displaying job information when monitoring the job and for naming log files.

--cpus-per-task

The number of cpus to request. The default is 1. More can be requested when a multiple cpus can be used to parallelize tasks.

--mem

The amount of memory to allocate. This argument takes a number followed by a unit such as mb for megabytes or gb for gigabytes. If no unit is given, megabytes are used. If no --mem argument is used, a job defaults to 120 MB?

--time

Time limit for the job. Jobs not finished by the end of the allocated time will be killed. The value for this argument must specified with the format days-hrs:min:sec, days-hrs, hrs:min:sec, min:sec, or sec. The default time limit in non-debug partitions is Infinite. However, the time limit has an impact on the priority of jobs in the queue. Jobs with shorter time limits are likely to begin running sooner.

--output

This specifies the path of the log file where stdout is written. In the examples above %x is replaced with the job name and %j is replaced with the job ID. See the other file name pattern options here

Additional commands

sbatch is a powerful tool with many other configuration options beyond the scope of this documentation.

See the sbatch documentation for more information about command arguments.

Interactive Job

An interactive session can be started on a compute node using the srun command.

At a minimum, you would use these arguments:

srun --partition=general --qos=general --pty=bash

Most sbatch arguments can also be used with srun. For --mem or --cpus-per-task if you needed to request additional resources.

Job status and monitoring

To monitor the status of your jobs you can run squeue -u <user> which will list the status of all jobs that belong to you. See the squeue documentation for additional arguments and output formatting options.

Canceling a job

If you need to cancel a job that is currently running or in the queue, you can use the scancel command. The basic syntax is:

scancel <job_id>

Request time extensions

If you have a job that may not finish within the allocated time limit you can submit a time extension request

Partitions

A partition is a group of compute nodes that share certain specifications and policies. Each partition is best suited to different types of tasks.

general

This partition contains nodes suitable for most tasks. These nodes have 64-128 AMD or Intel Xeon CPUs and 192 GB - 512 GB of ram. Most nodes in this partition have either one NVIDIA A100 or three NVIDIA L40S GPUs. Users can utilize up to 400 CPUs concurrently or with a limit of 400 concurrent jobs in this partition.

himem

This parition contains nodes suitable for jobs that have very large memory requirements. These nodes have 128 AMD CPUs and 1 TB of ram. Each node in this partition has one NVIDIA A100 GPU. Jobs cannot run in this partition unless 500 GB of memory is requested. Only two jobs per user will run concurrently in this partition.

debug

This partition is intended for testing and debugging jobs. It has a time limit of only 5 minutes but jobs should start very quickly, allowing you to check for errors before submitting to a queue with potentially longer wait times.

preemptible

Jobs run in this partition my be preempted by higher priority users. This partition is intended for users who are willing to have their jobs interrupted in exchange for potentially lower wait times when nodes are not in use by higher priority users. Jobs with short run times or jobs that can be restartd are good candidates for this partition.

SLURM Environment Variables

SLURM provides a number of environment variables that can be used in job scripts to access information about the job and the resources allocated to it. See the SLURM environment variables documentation for a complete list of available variables.

GPUs

A GPU will not be allocated to a job unless it is explicitly requested. In order to request a GPU, you can specify the --gres=gpu:[type:]<number> agrument where the required <number> is the number of GPUs to allocate to the job. The type is optional and specifies the type of GPU to allocate. The types of GPUs and the associated type arguments are listed in the table below.

GPU Types

GPU	type	number per node
NVIDIA A100	A100	1
NVIDIA L40S	L40S	1-3

Example

#!/bin/bash
#SBATCH --job-name=test-gpu
#SBATCH --cpus-per-task 1
#SBATCH --mem=100M
#SBATCH --time=1:00
#SBATCH --partition=debug
#SBATCH --qos=debug
#SBATCH --output=%x-%j.out
##SBATCH --gres=gpu:A100:1

# Detect GPU
nvidia-smi

Feature Constraints

When a job requires specific hardware or software features on a compute node, a user can specify these requirements using the --constraint=<feature> argument where is a defined feature of a node. Multiple constraints can be specified with & (and) or | (or) operators. The defined features and description are listed in the table below. See the SLURM documentation for more information on using constraints.

Feature	Description
cpu_amd	AMD CPUs
cpu_xeon	Intel Xeon CPUs
epyc_7352	AMD EPYC 7352 CPUs
xeon_platinum_8462Y+	Intel Xeon Platinum 8462Y CPUs

Example

Having software compiled for a specific CPU architecture is a common reason to need constraints. For example, if you have software that is optimized for AMD CPUs, you can use the following command to ensure that your job runs on a node with AMD CPUs.

#!/bin/bash
#SBATCH --job-name=test-feature-constraint
#SBATCH --cpus-per-task 1 
#SBATCH --mem=100M
#SBATCH --time=1:00
#SBATCH --partition=debug
#SBATCH --qos=debug
#SBATCH --output=%x-%j.out
#SBATCH --constraint=cpu_amd

lscpu

Job Arrays

If you have a large set of similar tasks that can be run in parallel, job arrays offer a convenient solution for managing these. Job arrays are created using the --array=<indexes>[%limit] argument where the required <indexes> is a range (e.g. 0-4) or list (e.g. 1,3,5,7) of indexes and an optional limit, separated by a % is the maximum allowed number of simultaneously running jobs (e.g. %20). Jobs which are part of a job array will have the environment variable SLURM_ARRAY_TASK_ID set to the array index value for that job. The maximum index allowed in a job array is 5001.

Example

Here is a basic example where the job array will run 5 jobs in parallel, each with a different index value from 1 to 5. A different param value is printed to stdout based on the job index. Only 2 jobs will run at the same time due to the %2.

#!/bin/bash
#SBATCH --job-name=test-array-job
#SBATCH --cpus-per-task 1 
#SBATCH --mem=100M
#SBATCH --time=1:00
#SBATCH --partition=debug
#SBATCH --qos=debug
#SBATCH --output=%x-%j.out
#SBATCH --array=1-5%2

params=("A" "B" "C" "D" "E")
param=${params[$SLURM_ARRAY_TASK_ID-1]}
echo $param

Parameter values will often be stored in a separate file. Below are some examples of ways to access these values in a job array script.

Accessing nth line from a text file

param=$(sed -n "${SLURM_ARRAY_TASK_ID}p" params.txt)

Accessing nth value and second column from a csv file

param=$(sed -n "${SLURM_ARRAY_TASK_ID}p" params.csv | cut -d ',' -f 2)

Accessing nth value and second column from a csv file with a header row

param=$(sed -n "1~${SLURM_ARRAY_TASK_ID}p" params.csv | cut -d ',' -f 2)

X11 Forwarding

Some applications, such as R, can produce graphical output. X Windows, a.k.a. X11, is an option for displaying that output in an interactive session.

First you need to install XQuartz on your local machine.

Then, when connecting to Mantis, use the the ssh option -X (or possibly -Y if your application requires it) like this:

# external:
ssh -X <username>@login.hpc.cam.uchc.edu
# or when connected to the VPN:
ssh -X <username>@mantis-submit.cam.uchc.edu

Now, to run your software in an interactive session, start it using the option --x11. For example:

srun --x11 --partition=general --qos=general --mem=5G --pty bash

Now that you’re in an interactive session, you should be able to run your application.

Try launching R:

And then plot a histogram of random draws from a normal distribution:

hist(rnorm(10000))

The histogram should pop up in an XQuartz window.

In development