Examples: Data Generation and Imitation Learning for Humanoids

These Isaac Lab Mimic examples cover three humanoid tasks:

• Demo 1: Data generation and policy training for a humanoid robot (GR-1 pick and place)

• Demo 2: Visuomotor policy for a humanoid robot (GR-1 nut pouring)

• Demo 3: Data generation and policy training for humanoid robot locomanipulation (Unitree G1)

Important

The Isaac Lab Mimic workflows and Pink IK tasks on this page are supported only on Linux. Pink IK and its dependencies are not available on Windows.

Complete the tutorial in Synthetic Data Generation and Imitation Learning with Isaac Lab Mimic before proceeding with the following demonstrations to understand the data collection, annotation, and generation steps of Isaac Lab Mimic.

Demo 1: Data Generation and Policy Training for a Humanoid Robot

Isaac Lab Mimic supports data generation for robots with multiple end effectors. In the following demonstration, we will show how to generate data to train a Fourier GR-1 humanoid robot to perform a pick and place task.

Optional: Collect and annotate demonstrations

Collect human demonstrations

Note

Data collection for the GR-1 humanoid robot environment requires use of an Apple Vision Pro headset. If you do not have access to an Apple Vision Pro, you may skip this step and continue on to the next step: Generate the dataset. A pre-recorded annotated dataset is provided in the next step.

Tip

The GR1 scene utilizes the wrist poses from the Apple Vision Pro (AVP) as setpoints for a differential IK controller (Pink-IK). The differential IK controller requires the user’s wrist pose to be close to the robot’s initial or current pose for optimal performance. Rapid movements of the user’s wrist may cause it to deviate significantly from the goal state, which could prevent the IK controller from finding the optimal solution. This may result in a mismatch between the user’s wrist and the robot’s wrist. You can increase the gain of all the Pink-IK controller’s FrameTasks to track the AVP wrist poses with lower latency. However, this may lead to more jerky motion. Separately, the finger joints of the robot are retargeted to the user’s finger joints using the dex-retargeting library.

Set up the CloudXR Runtime and Apple Vision Pro for teleoperation by following the steps in Setting up Isaac Capture with CloudXR. CPU simulation is used in the following steps for better XR performance when running a single environment.

Collect a set of human demonstrations. A success demo requires the object to be placed in the bin and for the robot’s right arm to be retracted to the starting position.

The GR-1 example has one left-hand subtask and two right-hand subtasks. Keep the right hand still while the left hand picks up the object and holds it in a stable position. Mimic uses the object pose to interpolate the right-hand trajectory when object positions change during generation.

Left: A good human demonstration with smooth and steady motion. Right: A bad demonstration with jerky and exaggerated motion.

Collect five demonstrations by running the following command:

uv run --extra teleop,isaacsim isaaclab teleop record \
--task IsaacContrib-PickPlace-GR1T2-Abs \
--visualizer kit \
--xr \
--device cpu \
--num_demos 5 \
--dataset_file ./datasets/dataset_gr1.hdf5

Note

We also provide a GR-1 pick and place task with waist degrees-of-freedom enabled IsaacContrib-PickPlace-GR1T2-WaistEnabled-Abs (see Environments for details on the available environments, including the GR1 Waist Enabled variant). The same command above applies but with the task name changed to IsaacContrib-PickPlace-GR1T2-WaistEnabled-Abs.

Tip

If a demo fails during data collection, the environment can be reset using the teleoperation controls panel in the XR teleop client on the Apple Vision Pro or via voice control by saying “reset”. See Teleoperate with Apple Vision Pro for more details.

The robot uses simplified collision meshes for physics calculations that differ from the detailed visual meshes displayed in the simulation. Due to this difference, you may occasionally observe visual artifacts where parts of the robot appear to penetrate other objects or itself, even though proper collision handling is occurring in the physics simulation.

You can replay the collected demonstrations by running the following command:

uv run --extra teleop,isaacsim isaaclab teleop replay \
--task IsaacContrib-PickPlace-GR1T2-Abs \
--visualizer kit \
--device cpu \
--dataset_file ./datasets/dataset_gr1.hdf5

Note

Non-determinism may be observed during replay as physics in IsaacLab are not deterministically reproducible when using env.reset.

Annotate the demonstrations

Unlike the Franka stacking task, the GR-1 pick and place task uses manual annotation to define subtasks.

The pick and place task has one subtask for the left arm (pick) and two subtasks for the right arm (idle, place). Annotations denote the end of a subtask. For the pick and place task, this means there are no annotations for the left arm and one annotation for the right arm (the end of the final subtask is always implicit).

Each demo requires a single annotation between the first and second subtask of the right arm. This annotation (“S” button press) should be done when the right robot arm finishes the “idle” subtask and begins to move towards the target object. An example of a correct annotation is shown below:

Annotate the demonstrations by running the following command:

uv run --extra isaacsim,mimic python scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \
--task Isaac-PickPlace-GR1T2-Abs-Mimic-v0 \
--visualizer kit \
--device cpu \
--input_file ./datasets/dataset_gr1.hdf5 \
--output_file ./datasets/dataset_annotated_gr1.hdf5

Note

The script prints the keyboard commands for manual annotation and the current subtask being annotated:

Annotating episode #0 (demo_0)
   Playing the episode for subtask annotations for eef "right".
   Subtask signals to annotate:
      - Termination:      ['idle_right']

   Press "N" to begin.
   Press "B" to pause.
   Press "S" to annotate subtask signals.
   Press "Q" to skip the episode.

Tip

If the object does not get placed in the bin during annotation, you can press “N” to replay the episode and annotate again. Or you can press “Q” to skip the episode and annotate the next one.

Generate the dataset

If you skipped the prior collection and annotation step, download the pre-recorded annotated dataset dataset_annotated_gr1.hdf5 from here: [Annotated GR1 Dataset]. Place the file under IsaacLab/datasets and run the following command to generate a new dataset with 1000 demonstrations.

uv run --extra isaacsim,mimic python scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--device cpu \
--num_envs 20 \
--generation_num_trials 1000 \
--input_file ./datasets/dataset_annotated_gr1.hdf5 \
--output_file ./datasets/generated_dataset_gr1.hdf5

Train a policy

Use Robomimic to train a policy for the generated dataset.

uv run --extra isaacsim,mimic python scripts/imitation_learning/robomimic/train.py \
--task IsaacContrib-PickPlace-GR1T2-Abs \
--algo bc \
--normalize_training_actions \
--dataset ./datasets/generated_dataset_gr1.hdf5

The training script will normalize the actions in the dataset to the range [-1, 1]. The normalization parameters are saved in the model directory under PATH_TO_MODEL_DIRECTORY/logs/normalization_params.txt. Record the normalization parameters for later use in the visualization step.

Note

By default the trained models and logs will be saved to IsaacLab/logs/robomimic.

Visualize the results

Visualize the results of the trained policy by running the following command, using the normalization parameters recorded in the prior training step:

uv run --extra isaacsim,mimic python scripts/imitation_learning/robomimic/play.py \
--task IsaacContrib-PickPlace-GR1T2-Abs \
--visualizer kit \
--device cpu \
--num_rollouts 50 \
--horizon 400 \
--norm_factor_min <NORM_FACTOR_MIN> \
--norm_factor_max <NORM_FACTOR_MAX> \
--checkpoint /PATH/TO/desired_model_checkpoint.pth

Note

Change the NORM_FACTOR in the above command with the values generated in the training step.

Tip

If you don’t see expected performance results: It is critical to test policies from various checkpoint epochs. Performance can vary significantly between epochs, and the best-performing checkpoint is often not the final one.

The trained policy performing the pick and place task in Isaac Lab.

Note

Expected Success Rates and Timings for Pick and Place GR1T2 Task

• Success rate for data generation depends on the quality of human demonstrations (how well the user performs them) and dataset annotation quality. Both data generation and downstream policy success are sensitive to these factors and can show high variance. See Common Pitfalls when Generating Data for tips to improve your dataset.

• Data generation success for this task is typically 65-80% over 1000 demonstrations, taking 18-40 minutes depending on GPU hardware and success rate (19 minutes on a RTX ADA 6000 @ 80% success rate).

• Behavior Cloning (BC) policy success is typically 75-86% (evaluated on 50 rollouts) when trained on 1000 generated demonstrations for 2000 epochs (default), depending on demonstration quality. Training takes approximately 29 minutes on a RTX ADA 6000.

• Recommendation: Train for 2000 epochs with 1000 generated demonstrations, and evaluate multiple checkpoints saved between the 1000th and 2000th epochs to select the best-performing policy. Testing various epochs is essential for finding optimal performance.

Demo 2: Visuomotor Policy for a Humanoid Robot

Download the Dataset

Download the pre-generated dataset from here and place it under IsaacLab/datasets/generated_dataset_gr1_nut_pouring.hdf5 (Note: The dataset size is approximately 15GB). The dataset contains 1000 demonstrations of a humanoid robot performing a pouring/placing task that was generated using Isaac Lab Mimic for the Isaac-NutPour-GR1T2-Pink-IK-Abs-Mimic-v0 task.

Hint

If desired, data collection, annotation, and generation can be done using the same commands as the prior examples.

The robot first picks up the red beaker and pours the contents into the yellow bowl. Then, it drops the red beaker into the blue bin. Lastly, it places the yellow bowl onto the white scale. See the video in the Visualize the results section below for a visual demonstration of the task.

The success criteria for this task requires the red beaker to be placed in the blue bin, the green nut to be in the yellow bowl, and the yellow bowl to be placed on top of the white scale.

Attention

The following commands are only for your reference and are not required for this demo.

To collect demonstrations:

uv run --extra teleop,isaacsim isaaclab teleop record \
--task IsaacContrib-NutPour-GR1T2-Pink-IK-Abs \
--visualizer kit \
--device cpu \
--xr \
--num_demos 5 \
--dataset_file ./datasets/dataset_gr1_nut_pouring.hdf5

To annotate the demonstrations:

uv run --extra isaacsim,mimic python scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \
--task Isaac-NutPour-GR1T2-Pink-IK-Abs-Mimic-v0 \
--visualizer kit \
--device cpu \
--input_file ./datasets/dataset_gr1_nut_pouring.hdf5 \
--output_file ./datasets/dataset_annotated_gr1_nut_pouring.hdf5

Warning

There are multiple right eef annotations for this task. Annotations for subtasks for the same eef cannot have the same action index. Make sure to annotate the right eef subtasks with different action indices.

To generate the dataset:

uv run --extra isaacsim,mimic python scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--task Isaac-NutPour-GR1T2-Pink-IK-Abs-Mimic-v0 \
--visualizer kit \
--device cpu \
--generation_num_trials 1000 \
--num_envs 5 \
--input_file ./datasets/dataset_annotated_gr1_nut_pouring.hdf5 \
--output_file ./datasets/generated_dataset_gr1_nut_pouring.hdf5

Train a policy

Use Robomimic to train a visuomotor BC agent for the task.

uv run --extra isaacsim,mimic python scripts/imitation_learning/robomimic/train.py \
--task IsaacContrib-NutPour-GR1T2-Pink-IK-Abs \
--algo bc \
--normalize_training_actions \
--dataset ./datasets/generated_dataset_gr1_nut_pouring.hdf5

The training script will normalize the actions in the dataset to the range [-1, 1]. The normalization parameters are saved in the model directory under PATH_TO_MODEL_DIRECTORY/logs/normalization_params.txt. Record the normalization parameters for later use in the visualization step.

Note

By default the trained models and logs will be saved to IsaacLab/logs/robomimic.

You can also post-train a GR00T foundation model to deploy a Vision-Language-Action policy for the task.

Please refer to the IsaacLabEvalTasks repository for more details.

Visualize the results

Visualize the results of the trained policy by running the following command, using the normalization parameters recorded in the prior training step:

uv run --extra isaacsim,mimic python scripts/imitation_learning/robomimic/play.py \
--task IsaacContrib-NutPour-GR1T2-Pink-IK-Abs \
--visualizer kit \
--device cpu \
--num_rollouts 50 \
--horizon 350 \
--norm_factor_min <NORM_FACTOR_MIN> \
--norm_factor_max <NORM_FACTOR_MAX> \
--checkpoint /PATH/TO/desired_model_checkpoint.pth

Note

Change the NORM_FACTOR in the above command with the values generated in the training step.

Tip

If you don’t see expected performance results: Test policies from various checkpoint epochs, not just the final one. Policy performance can vary substantially across training, and intermediate checkpoints often yield better results.

The trained visuomotor policy performing the pouring task in Isaac Lab.

Note

Expected Success Rates and Timings for Visuomotor Nut Pour GR1T2 Task

• Success rate for data generation depends on the quality of human demonstrations (how well the user performs them) and dataset annotation quality. Both data generation and downstream policy success are sensitive to these factors and can show high variance. See Common Pitfalls when Generating Data for tips to improve your dataset.

• Data generation for 1000 demonstrations takes approximately 10 hours on a RTX ADA 6000.

• Behavior Cloning (BC) policy success is typically 50-60% (evaluated on 50 rollouts) when trained on 1000 generated demonstrations for 600 epochs (default). Training takes approximately 15 hours on a RTX ADA 6000.

• Recommendation: Train for 600 epochs with 1000 generated demonstrations, and evaluate multiple checkpoints saved between the 300th and 600th epochs to select the best-performing policy. Testing various epochs is critical for achieving optimal performance.

Demo 3: Data Generation and Policy Training for Humanoid Robot Locomanipulation with Unitree G1

This example combines G1 manipulation and navigation. First generate pick-and-place data, then add a route between two locations. The robot picks up the object at point A, walks to point B, and places it there.

Note

Locomotion policy training

This example uses a locomotion policy trained with AGILE, which builds on Isaac Lab’s manager-based environments. The repository provides pretrained upper- and lower-body policies, along with tools to train locomotion and whole-body control policies.

Generate the manipulation dataset

The same data generation and policy training steps from Demo 1 can be applied to the G1 humanoid robot with locomanipulation capabilities. This demonstration shows how to train a G1 robot to perform pick and place tasks with full-body locomotion and manipulation.

The process follows the same workflow as Demo 1, but uses the IsaacContrib-PickPlace-Locomanipulation-G1-Abs task environment.

Follow the same data collection, annotation, and generation process as demonstrated in Demo 1, but adapted for the G1 locomanipulation task.

Hint

If desired, data collection and annotation can be done using the same commands as the prior examples for validation of the dataset.

The G1 robot with locomanipulation capabilities combines full-body locomotion with manipulation to perform pick and place tasks.

Note that the following commands are only for your reference and dataset validation purposes - they are not required for this demo.

To collect demonstrations:

uv run --extra teleop,isaacsim isaaclab teleop record \
--device cpu \
--xr \
--visualizer kit \
--task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
--dataset_file ./datasets/dataset_g1_locomanip.hdf5 \
--num_demos 5

Note

Depending on how the Apple Vision Pro app was initialized, the hands of the operator might be very far up or far down compared to the hands of the G1 robot. If this is the case, you can click Stop XR in the XR tab in Isaac Lab, and move the AR Anchor prim. Adjust it down to bring the hands of the operator lower, and up to bring them higher. Click Start XR to resume teleoperation session. Make sure to match the hands of the robot before clicking Play in the Apple Vision Pro, otherwise there will be an undesired large force generated initially.

You can replay the collected demonstrations by running:

uv run --extra teleop,isaacsim isaaclab teleop replay \
--device cpu \
--visualizer kit \
--task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
--dataset_file ./datasets/dataset_g1_locomanip.hdf5

To annotate the demonstrations:

uv run --extra isaacsim,mimic python scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \
--device cpu \
--visualizer kit \
--task Isaac-Locomanipulation-G1-Abs-Mimic-v0 \
--input_file ./datasets/dataset_g1_locomanip.hdf5 \
--output_file ./datasets/dataset_annotated_g1_locomanip.hdf5

If you skipped the prior collection and annotation step, download the pre-recorded annotated dataset dataset_annotated_g1_locomanip.hdf5 from here: [Annotated G1 Dataset]. Place the file under IsaacLab/datasets and run the following command to generate a new dataset with 1000 demonstrations.

uv run --extra isaacsim,mimic python scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--device cpu --num_envs 20 --generation_num_trials 1000 \
--input_file ./datasets/dataset_annotated_g1_locomanip.hdf5 --output_file ./datasets/generated_dataset_g1_locomanip.hdf5

Train a manipulation-only policy

At this point you can train a policy that only performs manipulation tasks using the generated dataset:

uv run --extra isaacsim,mimic python scripts/imitation_learning/robomimic/train.py \
--task IsaacContrib-PickPlace-Locomanipulation-G1-Abs --algo bc \
--normalize_training_actions \
--dataset ./datasets/generated_dataset_g1_locomanip.hdf5

Visualize the results

Visualize the trained policy performance:

uv run --extra isaacsim,mimic python scripts/imitation_learning/robomimic/play.py \
--device cpu \
--visualizer kit \
--task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
--num_rollouts 50 \
--horizon 400 \
--norm_factor_min <NORM_FACTOR_MIN> \
--norm_factor_max <NORM_FACTOR_MAX> \
--checkpoint /PATH/TO/desired_model_checkpoint.pth

Note

Change the NORM_FACTOR in the above command with the values generated in the training step.

Tip

If you don’t see expected performance results: Always test policies from various checkpoint epochs. Different epochs can produce significantly different results, so evaluate multiple checkpoints to find the optimal model.

The trained policy performing the pick and place task in Isaac Lab.

Note

Expected Success Rates and Timings for Locomanipulation Pick and Place Task

• Success rate for data generation depends on the quality of human demonstrations (how well the user performs them) and dataset annotation quality. Both data generation and downstream policy success are sensitive to these factors and can show high variance. See Common Pitfalls when Generating Data for tips to improve your dataset.

• Data generation success for this task is typically 65-82% over 1000 demonstrations, taking 18-40 minutes depending on GPU hardware and success rate (18 minutes on a RTX ADA 6000 @ 82% success rate).

• Behavior Cloning (BC) policy success is typically 75-85% (evaluated on 50 rollouts) when trained on 1000 generated demonstrations for 2000 epochs (default), depending on demonstration quality. Training takes approximately 40 minutes on a RTX ADA 6000.

• Recommendation: Train for 2000 epochs with 1000 generated demonstrations, and evaluate multiple checkpoints saved between the 1000th and 2000th epochs to select the best-performing policy. Testing various epochs is essential for finding optimal performance.

Generate the dataset with manipulation and point-to-point navigation

To create a comprehensive locomanipulation dataset that combines both manipulation and navigation capabilities, you can generate a navigation dataset using the manipulation dataset from the previous step as input.

Tip

Skip data generation: A pre-made locomanipulation dataset in LeRobot format is available on Hugging Face at nvidia/g1_locomanip_dataset. Downloading it lets you skip this section and the dataset conversion step, proceeding directly to Finetune the policy below.

Download and unzip the dataset:

hf download nvidia/g1_locomanip_dataset --repo-type dataset --local-dir ./datasets/g1_locomanip_hf
unzip ./datasets/g1_locomanip_hf/*.zip -d ./datasets/

The archive extracts to ./datasets/g1_simple_high_var_lerobot/. Use this path as the --dataset-path in the finetuning step. Policies trained on this dataset require --policy_quat_format wxyz at rollout time.

Default: no obstacles (open scene).

With 8 forklift obstacles and --randomize_placement enabled.

The locomanipulation dataset generation process takes the previously generated manipulation dataset and creates scenarios where the robot must navigate from one location to another while performing manipulation tasks. This creates a more complex dataset that includes both locomotion and manipulation behaviors.

To generate the locomanipulation dataset, use the following command:

uv run python \
    scripts/imitation_learning/locomanipulation_sdg/generate_data.py \
    --device cpu \
    --kit_args="--enable isaacsim.replicator.experimental.mobility_gen" \
    --task="Isaac-G1-SteeringWheel-Locomanipulation" \
    --dataset ./datasets/generated_dataset_g1_locomanip.hdf5 \
    --num_runs 1 \
    --lift_step 60 \
    --navigate_step 130 \
    --output_file ./datasets/generated_dataset_g1_locomanipulation_sdg.hdf5 \
    --randomize_placement \
    --visualizer kit

Note

The input dataset (--dataset) should be the manipulation dataset generated in the previous step. You can specify any output filename using the --output_file_name parameter.

The key parameters for locomanipulation dataset generation are:

• --lift_step 60: Number of steps for the lifting phase of the manipulation task. This should mark the point immediately after the robot has grasped the object.

• --navigate_step 130: Number of steps for the navigation phase between locations. This should mark the point where the robot has lifted the object and is ready to walk.

• --output_file: Name of the output dataset file

Note

You can change the number of obstacles (forklifts and boxes) in the scene by editing the locomanipulation SDG environment configuration. In source/isaaclab_mimic/isaaclab_mimic/locomanipulation_sdg/envs/g1_locomanipulation_sdg_env.py, set the module-level constants NUM_FORKLIFTS and NUM_BOXES to the desired counts. Use --randomize_placement when running generate_data.py to randomize obstacle and fixture positions each run.

This process creates a dataset where the robot performs the manipulation task at different locations, requiring it to navigate between points while maintaining the learned manipulation behaviors. The resulting dataset can be used to train policies that combine both locomotion and manipulation capabilities.

Note

You can visualize the robot trajectory results with the following script command:

uv run python scripts/imitation_learning/locomanipulation_sdg/plot_navigation_trajectory.py --input_file datasets/generated_dataset_g1_locomanipulation_sdg.hdf5 --output_dir /PATH/TO/DESIRED_OUTPUT_DIR

The data generated from this locomanipulation pipeline can also be used to finetune an imitation learning policy using GR00T N1.5. The following steps describe how to install GR00T, convert the dataset to LeRobot format, finetune the policy, and run rollouts in Isaac Lab.

Finetune GR00T N1.5 policy for locomanipulation

Prerequisites: Generate the locomanipulation dataset using the command in the previous section (e.g. generated_dataset_g1_locomanipulation_sdg.hdf5). The conversion step accepts a directory of SDG HDF5 files, so you may group multiple generate_data.py outputs together — but the directory must contain only SDG outputs, not other HDF5 files from earlier steps (e.g. dataset_annotated_g1_locomanip.hdf5 or generated_dataset_g1_locomanip.hdf5).

Install GR00T in a separate environment (uv)

GR00T N1.5 and Isaac Lab require incompatible NumPy and Transformers versions. Keep the simulator in the Isaac Lab environment and run conversion, training, and policy serving in a separate GR00T environment. Rollout communicates with the policy server over localhost.

Start from the IsaacLab repository root. The following commands create a sibling GR00T checkout and save absolute paths for the later steps. Keep these variables in the shell, or set them again when opening a new terminal:

export ISAACLAB_ROOT="$(pwd)"
export GR00T_ROOT="$(dirname "$ISAACLAB_ROOT")/Isaac-GR00T"
export GR00T_PYTHON="$GR00T_ROOT/.venv/bin/python"
export GR00T_SCRIPTS="$ISAACLAB_ROOT/scripts/imitation_learning/locomanipulation_sdg/gr00t"
git clone https://github.com/NVIDIA/Isaac-GR00T "$GR00T_ROOT"
git -C "$GR00T_ROOT" checkout 4af2b622892f7dcb5aae5a3fb70bcb02dc217b96

This revision is the N1.5 version used by this workflow. Copy the G1 configuration and apply the supplied PyTorch SDPA patch once to this fresh checkout:

cp "$GR00T_SCRIPTS/data_config.py" "$GR00T_ROOT/gr00t/experiment/data_config.py"
git -C "$GR00T_ROOT" apply "$GR00T_SCRIPTS/no_flash_attn.patch"

Create the separate Python 3.12 environment. The commands below use the CUDA 13.0 PyTorch wheels on Linux x86_64. Other platforms require matching PyTorch and Torchvision builds for their hardware. The SDPA patch avoids a separate flash-attn build.

uv venv --no-config --python 3.12 "$GR00T_ROOT/.venv"
uv pip install --no-config --python "$GR00T_PYTHON" \
    --index-url https://download.pytorch.org/whl/cu130 torch==2.11.0 torchvision==0.26.0
uv pip install --no-config --python "$GR00T_PYTHON" \
    -r "$GR00T_SCRIPTS/requirements.txt" -e "$GR00T_ROOT"
PYTORCH3D_NO_EXTENSION=1 uv pip install --no-config --python "$GR00T_PYTHON" \
    --no-build-isolation -c "$GR00T_SCRIPTS/requirements.txt" \
    'git+https://github.com/facebookresearch/pytorch3d.git@33824be3cbc87a7dd1db0f6a9a9de9ac81b2d0ba'
uv pip check --no-config --python "$GR00T_PYTHON"

The decord2 distribution retains the decord Python import used by GR00T and provides pre-built wheels for both x86_64 and aarch64 systems, including DGX Spark.

Important

Every GR00T install command specifies both --no-config and --python. This prevents Isaac Lab’s NumPy override from replacing GR00T’s pin even when installing from the Isaac Lab checkout, and avoids modifying the simulator environment. The requirements also pin Diffusers and Hugging Face Hub together to preserve Transformers compatibility. Use the same requirements as constraints when adding packages to this environment.

Run GR00T commands with uv run --no-project --python "$GR00T_PYTHON" as shown below. --no-sync alone does not select the GR00T interpreter or bypass install-time overrides. If an earlier shared installation changed Isaac Lab’s dependencies, restore that environment with uv sync --extra isaacsim --extra mimic before rollout.

The compiled PyTorch3D extension is intentionally disabled here because GR00T N1.5 uses only pytorch3d.transforms. This avoids compiling unused CUDA renderers and supports systems where the extension cannot be linked, including aarch64 Blackwell systems.

Convert dataset to LeRobot format

GR00T N1.5 expects data in LeRobot format. From the IsaacLab repository root, run the conversion script. <input_dir> is a directory containing one or more SDG .hdf5 files produced by generate_data.py (and no other HDF5 files). <output_path> is the LeRobot-format output directory (e.g. ./datasets/datasets_train_200_lerobot). Episodes with very low object displacement are skipped.

uv run --no-project --python "$GR00T_PYTHON" \
    "$GR00T_SCRIPTS/convert_dataset.py" <input_dir> <output_path>

Example — move the SDG output into its own directory first so the converter only sees SDG files:

mkdir -p ./datasets/locomanip_sdg
mv ./datasets/generated_dataset_g1_locomanipulation_sdg.hdf5 ./datasets/locomanip_sdg/
uv run --no-project --python "$GR00T_PYTHON" \
    "$GR00T_SCRIPTS/convert_dataset.py" ./datasets/locomanip_sdg ./datasets/datasets_train_200_lerobot

Finetune the policy

Run finetuning from the Isaac-GR00T repository root. Use the LeRobot-format output path from the previous step as --dataset-path and choose an --output-dir for checkpoints. The --data-config g1_locomanipulation_sdg and --embodiment-tag new_embodiment options are required for the G1 locomanipulation task.

cd "$GR00T_ROOT"
uv run --no-project --python "$GR00T_PYTHON" scripts/gr00t_finetune.py \
    --dataset-path <path_to_lerobot_output> \
    --output-dir <checkpoint_dir> \
    --data-config g1_locomanipulation_sdg \
    --embodiment-tag new_embodiment \
    --num-gpus 1 \
    --max-steps 10000 \
    --save-steps 1000 \
    --video-backend decord \
    --report-to tensorboard

See the GR00T N1.5 repository documentation for additional training options.

Tip

Skip finetuning: A pre-trained GR00T N1.5 checkpoint for this task is available on Hugging Face at nvidia/g1_locomanip_finetune. Downloading it lets you skip the finetuning step and proceed directly to rollout.

Download and unzip the checkpoint:

uv run --no-project --python "$GR00T_PYTHON" hf download nvidia/g1_locomanip_finetune --local-dir ./checkpoints/g1_locomanip_finetune_hf
unzip ./checkpoints/g1_locomanip_finetune_hf/*.zip -d ./checkpoints/

The archive extracts to ./checkpoints/g1_locomanip_finetune_20260129_231610/. Use ./checkpoints/g1_locomanip_finetune_20260129_231610/checkpoint-20000 as the --model_path in the server command below. This checkpoint requires --policy_quat_format wxyz on the rollout client.

Rollout the policy in Isaac Lab

Start the policy server in the GR00T environment, supplying an absolute path to your finetuned checkpoint. Wait for Server is ready and listening before starting rollout:

uv run --no-project --python "$GR00T_PYTHON" \
    "$GR00T_SCRIPTS/serve_policy.py" \
    --model_path <absolute_checkpoint_directory> --embodiment_tag new_embodiment

In a second terminal, from the IsaacLab repository root, run the simulator with the static manipulation dataset (used for scene/demo setup) and the task name. The two --with options supply only the lightweight messaging dependencies; the simulator does not import GR00T, Transformers, or GR00T’s dataset schema through the policy client:

uv run --extra isaacsim --extra mimic --with msgpack==1.2.3 --with pyzmq==27.2.0 \
    python scripts/imitation_learning/locomanipulation_sdg/gr00t/rollout_policy.py \
    --policy_host 127.0.0.1 \
    --dataset ./datasets/generated_dataset_g1_locomanip.hdf5 \
    --demo demo_0 \
    --output_file ./datasets/rollout_output.hdf5 \
    --task Isaac-G1-SteeringWheel-Locomanipulation \
    --device cpu \
    --visualizer kit

Optional arguments include --randomize_placement and --policy_quat_format wxyz (use if your checkpoint was trained with wxyz quaternion format). Use --max_steps 32 for a bounded smoke run. --policy_port defaults to 5555 and --policy_timeout_ms defaults to 15000. A timeout raises an error instead of hanging the simulator. Stop the server with Ctrl-C when finished. It binds to localhost by default; its protocol is intended for a trusted local connection.

The legacy --model_path rollout option remains available for custom compatible environments, but the standard Isaac Lab environment requires the server/client workflow.

Simulation rollout of GR00T N1.5 policy finetuned for locomanipulation.

The policy shown above uses the camera image, hand poses, hand joint positions, object pose, and base goal pose as inputs. The output of the model is the target base velocity, hand poses, and hand joint positions for the next several timesteps.

Integrating 3D Gaussian Splatting into SDG

This section extends locomanipulation SDG by replacing the synthetic background with a 3D Gaussian Splatting (NuRec) scene. The workflow takes a manipulation dataset as input and produces a combined navigation and manipulation dataset as an HDF5 file. The navigation replay happens in a neurally-reconstructed environment, and an ego-centric camera captures the result, producing more realistic training data. For more info on NuRec rendering in Isaac Sim, see Neural Volume Rendering for details.

Note

This section focuses on data generation with a 3DGS background. To train a policy on the generated data, see Finetune GR00T N1.5 policy for locomanipulation.

Note

The locomanipulation SDG pipeline currently runs a single environment. Parallel environment support is not yet available for this workflow.

Setup: downloading example assets

We provide a sample asset, hand_hold-voyager-babyboom, on Hugging Face.

Log in to Hugging Face:

hf auth login --token <your_huggingface_access_token>

Download the required USDZ stage file and occupancy maps:

hf download nvidia/PhysicalAI-Robotics-NuRec \
   hand_hold-voyager-babyboom/particle_sh_optimized.usdz \
   hand_hold-voyager-babyboom/occupancy_map.png \
   hand_hold-voyager-babyboom/occupancy_map.yaml \
   --repo-type dataset \
   --local-dir <PATH_TO_USD_ASSET>

Asset requirements

If you are using custom 3D Gaussian assets, ensure they meet these specifications to be compatible with the SDG pipeline:

• The scene has sufficient free space (e.g. 5m x 5m) for asset placement and robot navigation.

• The ground surface is aligned with the z=0 plane, as the pipeline assumes this elevation for object placement.

• An occupancy map is required for path planning.

Generating the dataset

Before proceeding, ensure you have generated a manipulation dataset or downloaded the sample dataset provided in the Generate the manipulation dataset section.

At runtime, the script adds a ground plane at z=0 to the scene. It then proceeds through four stages:

1. Pick: The robot picks up an object at the start location by replaying the manipulation trajectory. --lift_step marks the end of this stage (immediately after grasp).

2. Navigate: The robot travels to the target location using occupancy-map path planning and its locomotion policy. --navigate_step marks the end of this stage (when the robot is in place to release the object).

3. Place: The robot places the object at the target location, completing the trajectory.

4. Record: Joint states, poses, and the ego-centric video are saved to the HDF5 file specified by --output_file.

Run the generation command:

uv run --extra isaacsim,mimic python scripts/imitation_learning/locomanipulation_sdg/generate_data.py \
    --device cpu \
    --kit_args="--enable isaacsim.replicator.experimental.mobility_gen" \
    --task="Isaac-G1-SteeringWheel-Locomanipulation" \
    --dataset <DATASET_FOLDER>/dataset_annotated_g1_locomanip.hdf5 \
    --num_runs 1 \
    --lift_step 60 \
    --navigate_step 130 \
    --output_file <DATASET_FOLDER>/generated_dataset_g1_locomanipulation_sdg_nurec.hdf5 \
    --background_usd_path <PATH_TO_USD_ASSET>/hand_hold-voyager-babyboom/particle_sh_optimized.usdz \
    --background_occupancy_yaml_file <PATH_TO_USD_ASSET>/hand_hold-voyager-babyboom/occupancy_map.yaml \
    --high_res_video

To inspect the run in the GUI, add --viz kit; adjust the perspective camera, or switch the viewport to the robot camera view for clearer visualization.

The key parameters are:

• --background_usd_path: Path to the 3D Gaussian NuRec USD asset.

• --background_occupancy_yaml_file: Path to the NuRec occupancy map file.

• Placement randomization is enabled automatically when a NuRec scene is used.

• --high_res_video: Capture the ego-centric camera at 512×320 instead of the default 256×160.

When the run completes successfully, an HDF5 dataset is generated containing camera observations. Install FFmpeg-enabled OpenCV, then convert the ego-centric camera view to MP4:

uv pip install opencv-python

uv run --extra mimic python scripts/tools/hdf5_to_mp4.py \
   --input_file <DATASET_FOLDER>/generated_dataset_g1_locomanipulation_sdg_nurec.hdf5 \
   --output_dir <DATASET_FOLDER>/ \
   --input_keys robot_pov_cam \
   --video_width 512 \
   --video_height 320

Set --video_width and --video_height to match the resolution captured during generation: 512×320 with --high_res_video, or 256×160 without it.

The figure above shows recorded ego-centric camera views in the 3D Gaussian background when the robot replays the pick and place trajectory and navigates to the target location.