Humanoid Pick-and-Place with DexMimicGen

This example demonstrates the AutoData workflow using DexMimicGen to generate a synthetic pick-and-place dataset for a bimanual humanoid robot. It covers recording source demonstrations by Apple Vision Pro teleoperation, annotating their per-arm subtask boundaries, generating a large dataset with DexMimicGen, and validating the result.

This tutorial supports two humanoids through the same workflow — the Fourier GR-1 and the Unitree G1 (shown below). Throughout this tutorial, use the tabs to switch every command on the page to the desired robot.

The GR-1 pick-and-place task.

The G1 pick-and-place task.

Task Overview

A humanoid grasps a steering wheel with its left hand, transfers it to its right hand, and places it into a bin. Both arms are driven together (two end-effectors), which is what makes this a DexMimicGen rather than a MimicGen task.

GR-1G1

Property	Value
Algorithm	DexMimicGen (two end-effectors)
Environment name	Isaac-PickPlace-GR1T2-Abs-v0
Embodiment	Upper-body IK with absolute-pose actions and dexterous hands per arm
Task descriptor	gr1_pick_place.yaml
Embodiment config	gr1_ik_abs.yaml
Subtasks	Right arm: idle then grasp (idle_right) → transport & place (end of trajectory). Left arm: grasp and transport (end of trajectory).
Pre-annotated source dataset	datasets/annotated_datasets/dataset_gr1_annotated.hdf5

Why Multi-End-Effector Generation Differs

Follow the Franka cube-stacking workflow for an overview of the single-arm (MimicGen) case. This section highlights what changes for the humanoid (DexMimicGen) case.

• Per-arm subtask sequences. Each end-effector declares its own subtask list in the task descriptor (subtasks.right, subtasks.left), each with its own reference objects, boundary signals, and generation knobs. Segment selection and transformation happen independently per arm; by default all arms reuse the source demonstration selected by the first arm (generation_policy.select_src_per_arm: false), keeping the arms’ motions mutually consistent.

• Coordination constraints. For tasks where the arms must interact (handovers, bimanual lifts) the task descriptor’s constraints section synchronizes specific subtask pairs across arms at runtime. The pick-and-place examples here need none. See Task Descriptors.

• Absolute-pose embodiments. The humanoid embodiment adapters expose one pose slice and one set of hand-joint indices per arm (see Embodiments). The action vector is reassembled from per-arm targets each step.

Workflow

The tutorial covers the pipeline to go from a handful of human demonstrations to a large synthetically generated dataset ready for policy training. You can follow the whole pipeline, or skip directly to Step 3: Generate the Dataset using the pre-annotated dataset that ships with the repository.

Prerequisites

Start the dev container (see Installation):

./docker/run_docker.sh

Attention

Recording humanoid demonstrations requires an Apple Vision Pro and the CloudXR runtime. Step 1: Record Source Demonstrations covers the setup and links the Isaac Lab teleop guides. If you don’t have an Apple Vision Pro, skip to Step 3: Generate the Dataset and use the pre-annotated source dataset.

Workflow Steps

Follow these steps to complete the workflow:

• Step 1: Record Source Demonstrations

• Step 2: Annotate Demonstrations

• Step 3: Generate the Dataset

• Step 4: Validate the Generated Dataset