Running on HPC Clusters

Running on HPC Clusters#

Many HPC clusters run containers through singularity, which supports shared systems without requiring root privileges. Use the workflow below to run Isaac Lab training and evaluation jobs on these clusters.

Convert the Isaac Lab Docker image to a Singularity image, upload it, and submit a job to your cluster.

Attention

Cluster setup varies across different institutions. The following instructions have been tested on the ETH Zurich Euler cluster (which uses the SLURM workload manager), and the IIT Genoa Franklin cluster (which uses PBS workload manager).

The instructions may need to be adapted for other clusters. If you have successfully adapted the instructions for another cluster, please consider contributing to the documentation.

Prerequisites#

Install apptainer to export the image. The steps below install it locally; see its documentation for other installation options.

sudo apt update
sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y apptainer

Set up SSH access from your development machine to the cluster to simplify file transfers and avoid repeated password prompts.

Attention

The workflow has been tested with:

apptainer version 1.2.5-1.el7 and docker version 24.0.7
apptainer version 1.3.4 and docker version 27.3.1

In the case of issues, please try to switch to those versions.

Cluster configuration#

Configure the cluster-specific parameters in docker/cluster/.env.cluster. Each parameter is documented inline in the file:

###
# Cluster specific settings
###

# Job scheduler used by cluster.
# Currently supports PBS and SLURM
CLUSTER_JOB_SCHEDULER=SLURM
# Docker cache dir for Isaac Sim (has to end on docker-isaac-sim)
# e.g. /cluster/scratch/$USER/docker-isaac-sim
CLUSTER_ISAAC_SIM_CACHE_DIR=/some/path/on/cluster/docker-isaac-sim
# Isaac Lab directory on the cluster (has to end on isaaclab)
# e.g. /cluster/home/$USER/isaaclab
CLUSTER_ISAACLAB_DIR=/some/path/on/cluster/isaaclab
# Cluster login
CLUSTER_LOGIN=username@cluster_ip
# Cluster scratch directory to store the SIF file
# e.g. /cluster/scratch/$USER
CLUSTER_SIF_PATH=/some/path/on/cluster/
# Remove the temporary isaaclab code copy after the job is done
REMOVE_CODE_COPY_AFTER_JOB=false
# Python executable within Isaac Lab directory to run with the submitted job
CLUSTER_PYTHON_EXECUTABLE=scripts/reinforcement_learning/train.py

A few constraints are worth calling out:

CLUSTER_ISAAC_SIM_CACHE_DIR must end in docker-isaac-sim. It is copied to the compute node and mounted into the singularity container, which speeds up simulation startup.
CLUSTER_ISAACLAB_DIR must end in isaaclab. On submission, the latest local changes are copied to ${CLUSTER_ISAACLAB_DIR}_${datetime}, so several jobs can run different code versions at once. Logs are written back to the permanent CLUSTER_ISAACLAB_DIR.
REMOVE_CODE_COPY_AFTER_JOB deletes only that code copy, never the logs. Enable it to save disk space on the cluster.

When a job is submitted, it will also use variables defined in docker/.env.base, though these should be correct by default.

Exporting the Apptainer image#

Export and upload the image before your first job, then repeat this step whenever the Docker image changes, such as after an Isaac Sim or dependency update.

To export to a singularity image, execute the following command:

./docker/cluster/cluster_interface.sh push [profile]

This command will create a singularity image under docker/exports directory and upload it to the defined location on the cluster. It requires that you have previously built the image with the container.py interface. Be aware that creating the singularity image can take a while. [profile] is an optional argument that specifies the container profile to be used. If no profile is specified, the default profile base will be used.

Note

By default, the singularity image is created without root access by providing the --fakeroot flag to the apptainer build command. In case the image creation fails, you can try to create it with root access by removing the flag in docker/cluster/cluster_interface.sh.

Defining job parameters#

The job parameters need to be defined based on the job scheduler used by your cluster. You only need to update the appropriate script for the scheduler available to you.

SLURM#

The job parameters are defined inside the docker/cluster/submit_job_slurm.sh. A typical SLURM operation requires specifying the number of CPUs and GPUs, the memory, and the time limit. For more information, please check the SLURM documentation.

The default configuration is as follows:

#SBATCH --cpus-per-task=8
#SBATCH --gpus=rtx_3090:1
#SBATCH --time=23:00:00
#SBATCH --mem-per-cpu=4048
#SBATCH --mail-type=END
#SBATCH --mail-user=name@mail
#SBATCH --job-name="training-$(date +"%Y-%m-%dT%H:%M")"

An essential requirement for the cluster is that the compute node has access to the internet at all times. This is required to load assets from the Nucleus server. For some cluster architectures, extra modules must be loaded to allow internet access.

For instance, on ETH Zurich Euler cluster, the eth_proxy module needs to be loaded. This can be done by adding the following line to the submit_job_slurm.sh script:

# in the case you need to load specific modules on the cluster, add them here
# e.g., `module load eth_proxy`

PBS#

The job parameters are defined inside the docker/cluster/submit_job_pbs.sh. A typical PBS operation requires specifying the number of CPUs and GPUs, and the time limit. For more information, please check the PBS Official Site.

The default configuration is as follows:

#PBS -l select=1:ncpus=8:mpiprocs=1:ngpus=1
#PBS -l walltime=01:00:00
#PBS -j oe
#PBS -q gpu
#PBS -N isaaclab
#PBS -m bea -M "user@mail"

Submitting a job#

To submit a job on the cluster, the following command can be used:

./docker/cluster/cluster_interface.sh job [profile] "argument1" "argument2" ...

This command will copy the latest changes in your code to the cluster and submit a job. Please ensure that your Python executable’s output is stored under isaaclab/logs as this directory is synced between the compute node and CLUSTER_ISAACLAB_DIR.

[profile] is an optional argument that specifies which singularity image corresponding to the container profile will be used. If no profile is specified, the default profile base will be used. The profile has be defined directly after the job command. All other arguments are passed to the Python executable. If no profile is defined, all arguments are passed to the Python executable.

The training arguments are passed to the Python executable. As an example, the standard ANYmal rough terrain locomotion training can be executed with the following command:

./docker/cluster/cluster_interface.sh job --rl_library rsl_rl --task IsaacContrib-Velocity-Rough-AnymalC --video

The above will, in addition, also render videos of the training progress and store them under isaaclab/logs directory.

Submitting to OSMO#

NVIDIA OSMO is a cloud-native orchestration platform for scheduling robotics workloads. Unlike the SLURM and PBS workflows above, it runs the Isaac Lab Docker image directly, so no singularity conversion is needed and there is no code copy step – the image is the unit of deployment.

The workflow definition lives in docker/cluster/osmo_multi_gpu_workflow.yaml. It requests a single node and scales training across the GPUs on that node through the train_multigpu command. The following parameters can be overridden at submission time:

Parameter	Description
image	The Isaac Lab container image to run. Defaults to the released `nvcr.io/nvidia/isaac-lab:3.0.0` image. Note that the repository does not publish a `latest` tag, so this must name an explicit version.
rl_library	The RL library used for training. Defaults to `rsl_rl`.
num_gpu	The number of GPUs to train on. When set to `1`, the single-process `train` entry point is used instead of `train_multigpu`.
num_cpu	The number of CPUs requested for the node. These are split evenly across the per-GPU worker processes via `OMP_NUM_THREADS`.
memory	The amount of memory requested for the node, in GiB.
storage	The amount of scratch storage requested for the node, in GiB.
platform	The OSMO platform (node type) to schedule on, e.g. `dgx-h100`.
master_port	The port used by the torchrun rendezvous. Only relevant when `num_gpu` is greater than 1.
args	The arguments forwarded to the training script, e.g. `--task <task-name> --num_envs 4096`. No visualizer is requested by the workflow, so training runs headless.

To submit a job, use the osmo CLI:

osmo workflow submit docker/cluster/osmo_multi_gpu_workflow.yaml \
  --set num_gpu=4 \
  --set args="--task Isaac-Reorient-KukaAllegro --num_envs 4096"