Overview
User storage
The users have a quota of 25GB and 20000 files in their /home directory, they are NOT allowed to use more storage.
The users MUST run jobs only in /projects/$project, $project represents the project respective name.
Software in Deucalion
The software is available as loadable Modules. The modules allow a set of combinations of different versions of the software. Users can request additional module.
Info
Browsing and selection of available software is only available through command line, by default with SSH access, please check Connecting with SSH. If you are using HTTP access, you need to start a Shelll Access session to run the above commands.
List of available software modules:
module avail
Search for the desired software module:
module spider OpenMPI
Where OpenMPI is the name of software to search for.
Loading the default version of a software module:
module load OpenMPI
Loading a specific version of a software module:
module load OpenMPI/4.1.5-GCC-12.3.0
Display information of the selected software module:
module whatis OpenMPI/4.1.5-GCC-12.3.0
To change the software version of the module:
module switch OpenMPI OpenMPI/new-version>
List of loaded or currently active software modules:
module list
Remove all loaded software modules:
module purge
Remove and then load all loaded software modules:
module reload
Remove selected modules:
module unload OpenMPI
Help for the selected software:
module help OpenMPI/4.1.2.1
Additional commands can be found in the help section:
module help
System Environments ARM
Basic Information
| Compiler Name | FUJITSU Software Compiler Package |
| Compiler Version | V1.0L21 (cp-1.0.21.02a) |
| MPI Interconnect | InfiniBand (HDR100) |
| Job Scheduler Name | Slurm |
| Job Scheduler Version | 23.11.4 |
System Environments x86
Basic Information
| Compilers CPU | GCC 12.3.0, Intel oneAPI HPC Toolkit 2023.1.0 |
| Compilers GPU | CUDA 11.8: GCC 11.3.0, NVIDIA HPC SDK 22.9 |
| MPI Interconnect CPU | InfiniBand (HDR100) |
| MPI Interconnect GPU | InfiniBand (HDR200) |
| Job Scheduler Name | Slurm |
| Job Scheduler Version | 23.11.4 |
Partitions on Deucalion
| Partition | Architecture | Max Nodes | Time Limit |
|---|---|---|---|
| dev-arm | aarch64 | 2 | 4 hours |
| normal-arm | aarch64 | 128 | 48 hours |
| large-arm | aarch64 | 512 | 72 hours |
| dev-x86 | x86_64 | 2 | 4 hours |
| normal-x86 | x86_64 | 64 | 48 hours |
| large-x86 | x86_64 | 128 | 72 hours |
| dev-a100-40 | x86_64 | 1 | 4 hours |
| normal-a100-40 | x86_64 | 4 | 48 hours |
| dev-a100-80 | x86_64 | 1 | 4 hours |
| normal-a100-80 | x86_64 | 4 | 48 hours |
Slurm Basics
An HPC cluster is made up of a number of compute nodes, which consist of one or more processors, memory and in the case of the GPU nodes, GPUs. These computing resources are allocated to the user by the resource manager. This is achieved through the submission of jobs by the user. A job describes the computing resources required to run application(s) and how to run it. HPC Cluster uses Slurm as job scheduler and resource manager.
In the following, you will learn how to submit your job using the Slurm Workload Manager. If you're familiar with Slurm, you probably won't learn much. However, If you aren't acquainted with Slurm, the following will introduce you to the basics. If you would like to play around with Slurm in a sandboxed environment before submitting real jobs, we highly recommend that you try the interactive Slurm Learning tutorial.
The main commands for using Slurm are summarized in the table below.
| Command | Description |
|---|---|
sbatch |
Submit a batch script |
squeue |
View information about jobs in the scheduling queue |
scancel |
Signal or cancel jobs, job arrays or job steps |
sinfo |
View information about nodes and partitions |
srun |
Submit a job for execution in real time |
Creating a batch script
The most common type of jobs are batch jobs which are submitted to the
scheduler using a batch job script and the sbatch command.
A batch job script is a text file containing information about the job to be run: the amount of computing resource and the tasks that must be executed.
A batch script is summarized by the following steps:
- the interpreter to use for the execution of the script: bash, python, ...
- directives that define the job options: resources, run time, ...
- setting up the environment: prepare input, environment variables, ...
- run the application(s)
As an example, let's look at this simple batch job script (to see jobscripts for the gpu partition check here):
#!/bin/bash
#SBATCH --job-name=exampleJob
#SBATCH --account=exampleAccount
#SBATCH --partition=examplePartition
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
module load MyApp/1.2.3
myapp -i input -o output
In the previous example, the first line #!/bin/bash specifies that the script
should be interpreted as a bash script.
The line starting with #SBATCH are directives for the workload manager.
These have the general syntax
#SBATCH option_name=argument
Now that we have introduced this syntax, we can go through the directives one by one. The first directive is
#SBATCH --job-name=exampleJob
which sets the name of the job. It can be used to identify a job in the queue and other listings.
It is then necessary to select the account:
#SBATCH --account=exampleAccount
sacctmgr show Association where User=<username> format=Cluster,Account%30,User
<username> is your Deucalion username.
You also need to select the partition:
#SBATCH --partition=examplePartition
For GPU partitions you can request for any number (N) of GPU. For more specifics please go to our GPU documentation.
#SBATCH --gpus=N
The remaining lines specify the resources needed for the job. The first one is the maximum time your job can run. If your job exceeds the time limit, it is terminated regardless of whether it has finished or not.
#SBATCH --time=02:00:00
The time format is hh:mm:ss (or d-hh:mm:ss where d is the number of
days). Therefore, in our example, the time limit is 2 hours.
The next four lines of the script describe the computing resources that the job will need to run
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
In this instance we request one task (process) to be run on one node. A task corresponds to a process (or an MPI rank). One CPU thread (used, for example, with OpenMP) is requested for the one task as well as 2 GiB of memory should be allocated to the whole job.
Now that the needed resources for the job have been defined, the next step is to set up the environment. For example, copy input data from your home directory to the scratch file system or export environment variables.
module load MyApp/1.2.3
MyApp application is available to the batch job.
Finally, with everything set up, we can launch our program.
myapp -i input -o output
Submit a batch job
To submit the job script we just created we use the sbatch command. The
general syntax can be condensed as
$ sbatch [options] job_script [job_script_arguments ...]
The available options are the same as the one you use in the batch script:
sbatch --nodes=2 in the command line and #SBATCH --nodes=2 in a batch
script are equivalent. The command line value takes precedence if the same
option is present both on the command line and as a directive in a script.
For the moment let's limit ourselves to the most common way to use the
sbatch: passing the name of the batch script which contains the submission
options.
$ sbatch myjob.sh
Submitted batch job 123456
The sbatch command returns immediately and if the job is successfully
submitted, the command prints out the ID number of the job.
More details may be found on the dedicated [batch jobs][batch-jobs] page.
Examine the queue
Once you have submitted your batch script it won't necessarily run immediately.
It may wait in the queue of pending jobs for some time before its required
resources become available. In order to view your jobs in the queue, use the
squeue command.
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
123456 dev-arm exampleJ john.doe PD 0:00 1 (Priority)
The output shows the state of your job in the ST column. In our case, the job
is pending (PD). The last column indicates the reason why the job isn't
running: Priority. This indicates that your job is queued behind a higher
priority job. One other possible reason can be that your job is waiting for
resources to become available. In such a case, the value in the REASON column
will be Resources.
Let's look at the information that will be shown if your job is running:
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
123456 dev-arm exampleJ john.doe R 35:00 1 node-0123
The ST column will now display a R value (for RUNNING). The TIME column
will represent the time your job has been running. The list of nodes on which
your job is executing is given in the last column of the output.
In practice the list of jobs printed by this command will be much longer since
all jobs, including those belonging to other users, will be visible. In order
to see only the jobs that belong to you use the squeue command with the
--me flag.
$ squeue --me
The squeue command can also be used to determine when your pending job will
start.
$ squeue --me --start
JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON)
123456 batch Computat vananh PD 2021-06-01T16:10:28 1 node0012 (Priority)
123457 batch Computat vananh PD 2021-06-01T18:21:28 1 (null) (Priority)
In our example, both jobs listed will start June 1 at different times. You will
also notice that for the first job, the scheduler plan to run the job on
node0012 while for the second job, no node has been chosen yet.
Cancelling a job
Sometimes things just don't go as planned. If your job doesn't run as expected,
you may need to cancel your job. This can be achieved using the scancel
command which takes the job ID of the job to cancel.
$ scancel <jobid>
The job ID can be obtained from the output of the sbatch command when
submitting your job or by using squeue. The scancel command applies to
either a pending job waiting in the queue or to an already running job. In the
first case, the job will simply be removed from the queue while in the latter,
the execution will be stopped.
Batch jobs
This pages covers advanced topics related to running Slurm batch jobs on cluster. If you are not already familiar with Slurm, you should read the Slurm quickstart guide which covers the basics. You can also refer to the Slurm documentation or manual pages, in particular the page about sbatch. Because Deucalion slightly differs from other clusters with regards to GPU allocation, please visit this page.
Common Slurm options
Here is an overview of some of the most commonly used Slurm options.
Basic job specification
| Option | Description |
|---|---|
--time |
Set a limit on the total run time of the job allocation |
--account |
Charge resources used by this job to specified project |
--partition |
Request a specific partition for the resource allocation |
--job-name |
Specify a name for the job allocation |
Specify tasks distribution
| Option | Description |
|---|---|
--nodes |
Number of nodes to be allocated to the job |
--ntasks |
Set the maximum number of tasks (MPI ranks) |
--ntasks-per-node |
Set the number of tasks per node |
--ntasks-per-socket |
Set the number of tasks on each node |
--ntasks-per-core |
Set the maximum number of task on each core |
Request CPU cores
| Option | Description |
|---|---|
--cpus-per-task |
Set the number of cores per tasks |
Request memory
| Option | Description |
|---|---|
--mem |
Set the memory per node |
--mem-per-cpu |
Set the memory per allocated CPU core |
Email notifications
| Option | Description |
|---|---|
--mail-type |
Set when you want to receive emails (list of options) |
--mail-user |
Email to receive notifications |
Job Arrays
Job arrays let you submit many similar jobs (e.g. parameter sweeps, per-file
processing) as a single sbatch submission instead of writing a loop of
individual sbatch calls. Each element of the array is a distinct job with
its own job ID, but they all share the same script.
#!/bin/bash
#SBATCH --job-name=arrayJob
#SBATCH --account=exampleAccount
#SBATCH --partition=normal-x86
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --array=1-100
module load MyApp/1.2.3
myapp -i input_${SLURM_ARRAY_TASK_ID}.dat -o output_${SLURM_ARRAY_TASK_ID}.dat
The --array directive defines how many tasks make up the array and can be
expressed in a few different ways:
| Syntax | Description |
|---|---|
--array=1-100 |
Creates 100 array tasks, indices 1 through 100 |
--array=1,3,5,7 |
Creates 4 array tasks with the specified indices |
--array=1-100:5 |
Creates array tasks with a step of 5 (1,6,11,...,96) |
--array=1-100%10 |
Same as 1-100, but limits to 10 tasks running at once |
Inside the job, Slurm exposes the current task index through the
SLURM_ARRAY_TASK_ID environment variable, which you can use to select the
right input/output file, parameter set, etc. Other useful variables are
SLURM_ARRAY_JOB_ID (the "parent" job ID shared by all tasks) and
SLURM_ARRAY_TASK_COUNT (the total number of tasks in the array).
Submitting an array job returns the parent job ID, and squeue displays each
task using the <jobid>_<taskid> notation:
$ sbatch arrayjob.sh
Submitted batch job 123456
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
123456_1 normal-x86 arrayJob john.doe R 0:05 1 cn0012
123456_2 normal-x86 arrayJob john.doe R 0:05 1 cn0013
123456_[3-100%10] arrayJob john.doe PD 0:00 1 (JobArrayTaskLimit)
You can cancel the whole array, or a single task within it:
# cancel the whole array
$ scancel 123456
# cancel only task 5
$ scancel 123456_5
Array jobs can also be combined with dependencies (see Advanced Dependencies
and Pipelines below) and with
--array=...%N to throttle how many array tasks run concurrently, which is
particularly useful to avoid overwhelming shared filesystems or license
servers.
Node and Resource Selection
Beyond --nodes, --ntasks and --mem, Slurm provides several flags to give
you finer control over exactly where and how your job runs.
| Option | Description |
|---|---|
--exclusive |
Request exclusive use of the allocated nodes (no other job will share them) |
--exclude=<nodelist> |
Explicitly exclude a list of nodes from the allocation |
--nodelist=<nodelist> |
Request a specific set of nodes |
--constraint=<feature> |
Restrict the allocation to nodes that have a given feature/tag |
--reservation=<name> |
Run the job inside a pre-existing reservation |
# avoid a specific (misbehaving) node
#SBATCH --exclude=cn0045
# pin the job to specific nodes
#SBATCH --nodelist=cn0012,cn0013
# run inside a reservation, e.g. for a training session
#SBATCH --reservation=training2026
To check which reservations you have access to:
$ sinfo -T
Exclusive vs. Shared Node Allocation
By default, Slurm may pack multiple jobs from different users onto the same
node as long as there are enough free resources (CPUs, memory) to satisfy
each job's request. This is generally the case on the GPU (a100)
partitions, where nodes are not exclusive: you can request a single GPU
(--gpus=1) and share the rest of the node with other users' jobs.
If your job needs the whole node to itself — for example, to get consistent performance benchmarks, to use all the memory/local disk, or because it doesn't play well with other workloads — request it explicitly:
#SBATCH --nodes=2
#SBATCH --exclusive
With --exclusive, Slurm will not place any other job on the nodes
allocated to yours, even if your job doesn't use all the CPUs or memory
available. Keep in mind that this can increase queue time, since Slurm has to
wait until entire nodes are free rather than just enough resources for your
request.
Pipelining with dependencies
Job dependencies allow you to defer the start of a job until the specified
dependencies have been satisfied. Dependencies can be defined in a batch script
with the --dependency directive or be passed as a command-line argument to
sbatch.
$ sbatch --dependency=<type:job_id[:job_id]>
The type defines the condition that the job with ID job_id must fulfill
before the job which depends on it can start. For example,
$ sbatch job1.sh
Submitted batch job 123456
$ sbatch --dependency=afterany:123456 job2.sh
Submitted batch job 123458
will only start execution of job2.sh when job1.sh has finished. The available
types and their description are presented in the table below.
| Dependency type | Description |
|---|---|
after:jobid[:jobid...] |
Begin after the specified jobs have started |
afterany:jobid[:jobid...] |
Begin after the specified jobs have finished |
afternotok:jobid[:jobid...] |
Begin after the specified jobs have failed |
afterok:jobid[:jobid...] |
Begin after the specified jobs have run to completion |
The example below demonstrates a bash script for submission of multiple Slurm
batch jobs with dependencies. It also shows an example of a helper function
that extracts the job ID from the output of the sbatch command.
#!/bin/bash
submit_job() {
sub="$(sbatch "$@")"
if [[ "$sub" =~ Submitted\ batch\ job\ ([0-9]+) ]]; then
echo "${BASH_REMATCH[1]}"
else
exit 1
fi
}
# first job - no dependencies
id1=$(submit_job job1.sh)
# Two jobs that depend on the first job
id2=$(submit_job --dependency=afterany:$id1 job2.sh)
id3=$(submit_job --dependency=afterany:$id1 job3.sh)
# One job that depends on both the second and the third jobs
id4=$(submit_job --dependency=afterany:$id2:$id3 job4.sh)
Advanced Dependencies and Pipelines
In addition to after, afterany, afterok and afternotok above, Slurm
supports a few dependency types that are useful for more complex pipelines:
| Dependency type | Description |
|---|---|
singleton |
Begin after any previously launched job with the same job name and user has ended |
aftercorr:jobid |
For array jobs: start element N of this array once element N of the dependency array completes |
after:jobid+time |
Begin after the specified job has started, plus an optional delay (in minutes) |
singleton is handy when you want to guarantee that only one instance of a
named job runs at a time, e.g. for jobs that write to a shared output file:
$ sbatch --job-name=nightly-build --dependency=singleton nightly.sh
aftercorr is useful when chaining two array jobs together element-by-element,
instead of waiting for the entire array to finish:
# stage 1: process each input file
id1=$(sbatch --array=1-50 --parsable stage1.sh)
# stage 2: element N starts as soon as stage1's element N completes,
# rather than waiting for all 50 elements of stage1
sbatch --array=1-50 --dependency=aftercorr:$id1 stage2.sh
Note the use of --parsable on sbatch, which makes it print just the job
ID (instead of the full Submitted batch job <id> message), making it easier
to capture in shell variables for scripting pipelines:
id1=$(sbatch --parsable job1.sh)
id2=$(sbatch --parsable --dependency=afterok:$id1 job2.sh)
For pipelines with many stages and fan-in/fan-out dependencies, combining
--parsable with the submit_job helper function shown above keeps
submission scripts readable.
Job Checkpointing and Requeue
Because jobs are subject to the partition's --time limit (and can, in rare
cases, be preempted or affected by node failures), long-running applications
should periodically checkpoint their state so they can resume instead of
restarting from scratch.
Automatic requeue on failure
The --requeue flag tells Slurm to automatically resubmit your job if it
fails due to a node failure or is preempted (this does not apply to jobs
that fail due to exceeding the time limit or due to an application error):
#SBATCH --requeue
Inside the job script you can detect whether the current run is a restart
using SLURM_RESTART_COUNT, and branch your logic accordingly (e.g. resume
from the latest checkpoint instead of starting fresh):
if [ "${SLURM_RESTART_COUNT:-0}" -gt 0 ]; then
echo "Restart number ${SLURM_RESTART_COUNT}, resuming from checkpoint"
myapp --resume checkpoint.dat
else
myapp --start-fresh
fi
Warning signal before time limit
To checkpoint gracefully before hitting the walltime limit, use --signal to
ask Slurm to send a signal to your job some time before it's killed:
#SBATCH --time=12:00:00
#SBATCH --signal=B:USR1@300
This sends SIGUSR1 to the batch shell 300 seconds before the 12-hour limit
is reached, giving your application (if it traps the signal) a chance to
write a checkpoint and exit cleanly. Many applications and frameworks
(e.g. DMTCP for generic process checkpointing, or built-in checkpoint support
in tools like TensorFlow/PyTorch) can be combined with this pattern. Consult
your application's documentation for how it handles checkpoint/restart, since
the exact mechanism is application-specific.
Monitoring and Accounting
Once a job has started or finished, Slurm's accounting tools let you review how it performed and how much it cost against your allocation.
sacct
sacct queries the accounting database for information about past and
current jobs (by default, only jobs submitted today are shown):
$ sacct -j 123456
Some useful options:
# show more detail, including per-step resource usage
$ sacct -j 123456 --format=JobID,JobName,Partition,State,Elapsed,MaxRSS,MaxVMSize,ExitCode
# list all of your jobs from the last 7 days
$ sacct --starttime=now-7days --user=$USER
Billing and account usage
To check the overall usage and remaining hours for your project accounts, use
the billing command:
# summary for all your accounts
$ billing
# usage broken down per user for a specific account
$ billing -a <your_account>
Interactive Slurm jobs
Interactive jobs allow a user to interact with applications on the compute nodes. With an interactive job, you request time and resources to work on a compute node directly, which is different to a batch job where you submit your job to a queue for later execution.
You can use two commands to create an interactive session: srun and salloc.
Both of these commands take options similar to sbatch.
Using salloc
Using salloc, you allocate resources and spawn a shell that is then used to
execute parallel tasks launched with srun. For example, you can allocate 2
nodes for 30 minutes with the command
$ salloc --nodes=2 --partition=normal-<architecture> --account=<your_account> --time=00:30:00
salloc: Granted job allocation 123456
salloc: Waiting for resource configuration
<architecture> can be arm, x86, a100-40, and a100-80. To check all your accounts you can run sacctmgr show Association where User=<username> format=Cluster,Account%30,User, where <username> is your Deucalion username.
Once the allocation is made, this command will start a shell on the login
node. You can start parallel execution on the allocated nodes with srun.
$ srun --ntasks=32 --cpus-per-task=8 ./mpi_openmp_application
After the execution of your application ended, the allocation can be terminated
by exiting the shell (exit).
When using salloc, a shell is spawned on the login node. If you want to
obtain a shell on the first allocated compute node you can use srun --partition=normal-<architecture> --account=<your_account> --pty bash.
$ srun --cpu_bind=none --nodes=2 --pty bash -i
If you want to use an application with a GUI, you can use the --x11 flag with
srun to enable X11 forwarding.
Using srun
For simple interactive session, you can use srun with no prior allocation. In
this scenario, srun will first create a resource allocation in which to run
the job. For example, to allocate 1 node for 30 minutes and spawn a shell
$ srun --time=00:30:00 --partition=<partition> --account=<your_account> --nodes=1 --pty bash
Using srun to check running jobs
The srun command can
be used to check in on a running job in the cluster. In this case, you need to
give the job ID and possibly also the specific name of a compute node to srun.
This starts a shell, where you can run any command, on the first allocated node in a specific job:
$ srun --interactive --pty --jobid=<jobid> $SHELL
To check processor and memory usage quickly, you can run top directly:
$ srun --interactive --pty --jobid=<jobid> top
The -w cnXXXXX (cpu) or -w gnXXX option can be added to select a specific compute node to view (to check where your job is running you can run squeue --m to get a list of your submissions):
$ srun --interactive --pty --jobid=<jobid> -w cnXXXXX top
Enroot
What is Enroot?
- A tool to turn traditional container/OS images into unprivileged sandboxes.
- It uses the same underlying technologies as containers but removes much of the isolation they inherently provide while preserving filesystem separation.
- Enroot can be thought of as an unprivileged chroot that provides facilities to import well known container image formats (e.g. Docker)
Key concepts
- Standalone (no daemon)
- Fully unprivileged and multi-user capable (no setuid binary, cgroup inheritance, per-user configuration/container store…)
- Easy to use
- No isolation (no performance overhead, simplifies HPC deployements)
- Built-in GPU support with libnvidia-container
- Facilitate collaboration and development workflows (bundles, in-memory containers...)
Usage
Import and start an Ubuntu image from DockerHub Pull the image from docker hub
enroot import docker://ubuntu
Create the container
enroot create ubuntu.sqsh
Start the container
enroot start ubuntu
If you need to run something as root inside the container, you can use the --root option.
enroot start --root ubuntu
List the existing containers
enroot list -f
Remove a container
enroot remove ubuntu
Pyxis
Pyxis is a SPANK plugin for the Slurm Workload Manager.
It allows unprivileged cluster users to run containerized tasks through the srun command.
Pyxis requires Slurm and Enroot to work. Pyxis is only available in our accelerated nodes (gnx[501-533]).
Benefits
- Execute the user's task in an unprivileged container.
- Fast Docker image download.
- Simple command-line interface.
- Supports multi-node MPI jobs through PMI2 or PMIx (requires Slurm support).
- Allows users to install packages inside the container.
Usage
srun
Run a command on a gnx node
srun -A <slurm_account> -p normal-a100-40 -t 00:30:00 cat /etc/os-release
…
PLATFORM_ID="platform:el8"
Run the same command, but now inside of a container
srun --container-image=/share/apps-x86/containers/enroot/ubuntu-rolling.sqsh --container-name=ubuntu -p normal-a100-40 -A <slurm_account> -t 00:30:00 cat /etc/os-release
…
PRETTY_NAME="Ubuntu 22.04.4 LTS"
Mount a file from the host and run the command on it, from inside the container
srun --container-image=/share/apps-x86/containers/enroot/ubuntu-rolling.sqsh --container-mounts=/etc/os-release:/etc/os-release --container-name=ubuntu -p all-a100 -A <slurm_account> -t 00:30:00 cat /etc/os-release
…
PLATFORM_ID="platform:el8"
To see more options
srun --help | grep container
sbatch Execute the sbatch script inside a container image:
#!/bin/bash
#SBATCH -p all-a100 -t 30:00
#SBATCH --container-mounts /path/to/code
#SBATCH --container-workdir=/path/to/code
# SBATCH --container-image /path/to/container.sqsh
#SBATCH --container-name <container_name>
srun code input
Singularity
- It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible.
- You can build a container using SingularityCE on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container is a single file, and you don't have to worry about how to install all the software you need on each different operating system.
- Installed version: 3.11.3
Usage
Full SBATCH Singularity example:
#!/bin/bash
#SBATCH -p all-a100 -t 00:30:00
# REFERENCE: https://sylabs.io/guides/3.11/user-guide/gpu.html
#https://www.tensorflow.org/install/docker
#2.10.1 release w/ GPU support: cuda 11.2
#To be downloaded from login nodes (internet access)
#singularity pull tensorflow.sif docker://tensorflow/tensorflow:2.10.1-gpu
#wget https://github.com/tensorflow/benchmarks/archive/039fa9550c9efd25e09db2e73f360b67378436fe.zip
#mv benchmarks-039fa9550c9efd25e09db2e73f360b67378436fe/ benchmarks
CONTAINER_URI='tensorflow.sif'
#--nv option will setup the container's environment to use an NVIDIA GPU and the basic CUDA libraries to run a CUDA enabled application.
# instance tensorflow started
singularity instance start --nv $CONTAINER_URI tensorflow
singularity exec instance://tensorflow python \
$(pwd)/benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--num_gpus=4 --model resnet50 --batch_size 32
singularity exec instance://tensorflow python \
$(pwd)/benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--num_gpus=4 --model inception3 --batch_size 32
#shutdown instance
singularity instance stop -t 30 tensorflow
To get full examples of MPI integration using Singularity please go to our Containerization Tab.