.. _deployment-cluster: Running on HPC clusters ======================= Clusters are a great way to speed up training and evaluation of learning algorithms. While the Isaac Lab Docker image can be used to run jobs on a cluster, many clusters only support singularity images. This is because `singularity`_ is designed for ease-of-use on shared multi-user systems and high performance computing (HPC) environments. It does not require root privileges to run containers and can be used to run user-defined containers. Singularity is compatible with all Docker images. This page describes how to convert the :ref:`Isaac Lab Docker image ` into a singularity image and use it to submit jobs to a cluster. .. attention:: Cluster setup varies across different institutions. The following instructions have been tested on the `ETH Zurich Euler`_ cluster (which uses the SLURM workload manager), and the IIT Genoa Franklin cluster (which uses PBS workload manager). The instructions may need to be adapted for other clusters. If you have successfully adapted the instructions for another cluster, please consider contributing to the documentation. Prerequisites ------------- In order to export the Docker Image to a singularity image, `apptainer`_ is required. A detailed overview of the installation procedure for ``apptainer`` can be found in its `documentation`_. For convenience, we summarize the steps here for a local installation: .. code:: bash sudo apt update sudo apt install -y software-properties-common sudo add-apt-repository -y ppa:apptainer/ppa sudo apt update sudo apt install -y apptainer For simplicity, we recommend that an SSH connection is set up between the local development machine and the cluster. Such a connection will simplify the file transfer and prevent the user cluster password from being requested multiple times. .. attention:: The workflow has been tested with: - ``apptainer version 1.2.5-1.el7`` and ``docker version 24.0.7`` - ``apptainer version 1.3.4`` and ``docker version 27.3.1`` In the case of issues, please try to switch to those versions. Cluster configuration --------------------- Configure the cluster-specific parameters in ``docker/cluster/.env.cluster``. Each parameter is documented inline in the file: .. literalinclude:: ../../../../docker/cluster/.env.cluster :language: bash A few constraints are worth calling out: * ``CLUSTER_ISAAC_SIM_CACHE_DIR`` must end in ``docker-isaac-sim``. It is copied to the compute node and mounted into the singularity container, which speeds up simulation startup. * ``CLUSTER_ISAACLAB_DIR`` must end in ``isaaclab``. On submission, the latest local changes are copied to ``${CLUSTER_ISAACLAB_DIR}_${datetime}``, so several jobs can run different code versions at once. Logs are written back to the permanent ``CLUSTER_ISAACLAB_DIR``. * ``REMOVE_CODE_COPY_AFTER_JOB`` deletes only that code copy, never the logs. Enable it to save disk space on the cluster. When a ``job`` is submitted, it will also use variables defined in ``docker/.env.base``, though these should be correct by default. Exporting the Apptainer image ----------------------------- Next, we need to export the Docker image to a singularity image and upload it to the cluster. This step is only required once when the first job is submitted or when the Docker image is updated. For instance, due to an upgrade of the Isaac Sim version, or additional requirements for your project. To export to a singularity image, execute the following command: .. code:: bash ./docker/cluster/cluster_interface.sh push [profile] This command will create a singularity image under ``docker/exports`` directory and upload it to the defined location on the cluster. It requires that you have previously built the image with the ``container.py`` interface. Be aware that creating the singularity image can take a while. ``[profile]`` is an optional argument that specifies the container profile to be used. If no profile is specified, the default profile ``base`` will be used. .. note:: By default, the singularity image is created without root access by providing the ``--fakeroot`` flag to the ``apptainer build`` command. In case the image creation fails, you can try to create it with root access by removing the flag in ``docker/cluster/cluster_interface.sh``. Defining job parameters ----------------------- The job parameters need to be defined based on the job scheduler used by your cluster. You only need to update the appropriate script for the scheduler available to you. SLURM ~~~~~ The job parameters are defined inside the ``docker/cluster/submit_job_slurm.sh``. A typical SLURM operation requires specifying the number of CPUs and GPUs, the memory, and the time limit. For more information, please check the `SLURM documentation`_. The default configuration is as follows: .. literalinclude:: ../../../../docker/cluster/submit_job_slurm.sh :language: bash :lines: 12-19 :linenos: :lineno-start: 12 An essential requirement for the cluster is that the compute node has access to the internet at all times. This is required to load assets from the Nucleus server. For some cluster architectures, extra modules must be loaded to allow internet access. For instance, on ETH Zurich Euler cluster, the ``eth_proxy`` module needs to be loaded. This can be done by adding the following line to the ``submit_job_slurm.sh`` script: .. literalinclude:: ../../../../docker/cluster/submit_job_slurm.sh :language: bash :lines: 3-5 :linenos: :lineno-start: 3 PBS ~~~ The job parameters are defined inside the ``docker/cluster/submit_job_pbs.sh``. A typical PBS operation requires specifying the number of CPUs and GPUs, and the time limit. For more information, please check the `PBS Official Site`_. The default configuration is as follows: .. literalinclude:: ../../../../docker/cluster/submit_job_pbs.sh :language: bash :lines: 11-17 :linenos: :lineno-start: 11 Submitting a job ---------------- To submit a job on the cluster, the following command can be used: .. code:: bash ./docker/cluster/cluster_interface.sh job [profile] "argument1" "argument2" ... This command will copy the latest changes in your code to the cluster and submit a job. Please ensure that your Python executable's output is stored under ``isaaclab/logs`` as this directory is synced between the compute node and ``CLUSTER_ISAACLAB_DIR``. ``[profile]`` is an optional argument that specifies which singularity image corresponding to the container profile will be used. If no profile is specified, the default profile ``base`` will be used. The profile has be defined directlty after the ``job`` command. All other arguments are passed to the Python executable. If no profile is defined, all arguments are passed to the Python executable. The training arguments are passed to the Python executable. As an example, the standard ANYmal rough terrain locomotion training can be executed with the following command: .. code:: bash ./docker/cluster/cluster_interface.sh job --rl_library rsl_rl --task IsaacContrib-Velocity-Rough-AnymalC --video The above will, in addition, also render videos of the training progress and store them under ``isaaclab/logs`` directory. Submitting to OSMO ------------------ `NVIDIA OSMO`_ is a cloud-native orchestration platform for scheduling robotics workloads. Unlike the SLURM and PBS workflows above, it runs the Isaac Lab Docker image directly, so no singularity conversion is needed and there is no code copy step -- the image is the unit of deployment. The workflow definition lives in ``docker/cluster/osmo_multi_gpu_workflow.yaml``. It requests a single node and scales training across the GPUs on that node through the :ref:`train_multigpu ` command. The following parameters can be overridden at submission time: .. list-table:: :header-rows: 1 :widths: 20 80 * - Parameter - Description * - image - The Isaac Lab container image to run. Defaults to the released ``nvcr.io/nvidia/isaac-lab:3.0.0-rc1`` image. Note that the repository does not publish a ``latest`` tag, so this must name an explicit version. * - rl_library - The RL library used for training. Defaults to ``rsl_rl``. * - num_gpu - The number of GPUs to train on. When set to ``1``, the single-process ``train`` entry point is used instead of ``train_multigpu``. * - num_cpu - The number of CPUs requested for the node. These are split evenly across the per-GPU worker processes via ``OMP_NUM_THREADS``. * - memory - The amount of memory requested for the node, in GiB. * - storage - The amount of scratch storage requested for the node, in GiB. * - platform - The OSMO platform (node type) to schedule on, e.g. ``dgx-h100``. * - master_port - The port used by the torchrun rendezvous. Only relevant when ``num_gpu`` is greater than 1. * - args - The arguments forwarded to the training script, e.g. ``--task --num_envs 4096``. No visualizer is requested by the workflow, so training runs headless. To submit a job, use the ``osmo`` CLI: .. code:: bash osmo workflow submit docker/cluster/osmo_multi_gpu_workflow.yaml \ --set num_gpu=4 \ --set args="--task Isaac-Reorient-KukaAllegro --num_envs 4096" .. _Singularity: https://docs.sylabs.io/guides/2.6/user-guide/index.html .. _ETH Zurich Euler: https://www.gdc-docs.ethz.ch/EulerManual/site/overview/ .. _PBS Official Site: https://openpbs.org/ .. _apptainer: https://apptainer.org/ .. _documentation: https://www.apptainer.org/docs/admin/main/installation.html#install-ubuntu-packages .. _SLURM documentation: https://www.slurm.schedmd.com/sbatch.html .. _NVIDIA OSMO: https://developer.nvidia.com/osmo