Attention

Recording your own demonstrations requires an Apple Vision Pro and the CloudXR runtime. Annotating an existing recording only requires the simulator and a Kit window. If you do not have the required hardware, you may skip Step 1: Record Source Demonstrations and Step 2: Annotate Demonstrations by using the pre-annotated source dataset that ships with the repository and jumping directly to Step 3: Generate the Dataset.

Step 1: Record Source Demonstrations#

AutoData consumes source demonstrations recorded as Isaac Lab HDF5 datasets (per-episode actions, initial state, and observations). This step collects a small set of successful teleoperated demonstrations of the humanoid pick-and-place task.

Unlike the single-arm Franka task — which can be teleoperated with a SpaceMouse or keyboard — the dexterous, bimanual humanoids are teleoperated with an Apple Vision Pro through NVIDIA IsaacTeleop and the CloudXR runtime. The headset’s wrist poses drive the robot’s IK controllers, and the finger joints are retargeted onto the robot’s hands. Complete Installation for your chosen runtime before following the steps below. A headset, its client application, and a working CloudXR connection are still required.

Note

For supported IsaacTeleop hardware, see Supported Input Devices, and review the IsaacTeleop system requirements before starting.

Important

A stable network connection meeting the CloudXR network requirements is required before starting the steps below. For connection and tracking failures, see CloudXR and Apple Vision Pro.

Start the CloudXR Runtime#

The CloudXR runtime bridges the Apple Vision Pro and the simulator. Use two terminals in the same chosen runtime: two shells in the same Docker container, or two host terminals at the same repository root for native uv. Start CloudXR in the first terminal and leave it running for the whole recording session.

  1. On the host machine (for Docker, outside the development container), configure the firewall to allow CloudXR traffic:

    # Signaling (use one based on connection mode)
    sudo ufw allow 48010/tcp   # Standard mode
    sudo ufw allow 48322/tcp   # Secure mode
    # Video
    sudo ufw allow 47998/udp
    sudo ufw allow 48005/udp
    sudo ufw allow 48008/udp
    sudo ufw allow 48012/udp
    # Input
    sudo ufw allow 47999/udp
    # Audio
    sudo ufw allow 48000/udp
    sudo ufw allow 48002/udp
    
  2. Prepare the first terminal for your chosen runtime:

    If you are not already in a development container, run this from the repository root in a terminal on the host, outside any container:

    ./docker/run_docker.sh
    

    If the container is already running, this opens another shell in it. Continue inside the container at /workspaces/autodata.

    Run commands from the repository root in a terminal on the host. uv run uses your installed uv environment, so you don’t need to activate it.

  3. In that terminal, create a CloudXR config file for the Apple Vision Pro at the repository root:

    printf '%s\n' \
        'NV_DEVICE_PROFILE=auto-native' \
        'NV_CXR_ENABLE_PUSH_DEVICES=0' \
        'NV_ENABLE_POSE_WAIT=0' > avp.env
    
  4. Start the CloudXR runtime with that config:

    python -m isaacteleop.cloudxr --cloudxr-env-config=avp.env
    
    uv run --locked python -m isaacteleop.cloudxr --cloudxr-env-config=avp.env
    

Attention

The first run prompts you to accept the NVIDIA CloudXR License Agreement. Reply Yes when prompted:

NVIDIA CloudXR EULA must be accepted to run. View: https://github.com/NVIDIA/IsaacTeleop/blob/main/deps/cloudxr/CLOUDXR_LICENSE

Accept NVIDIA CloudXR EULA? [y/N]: Yes

Start Recording#

  1. Open a second terminal in the same runtime as CloudXR. For Docker, attach to the same running container. For native uv, use the same host user and repository root as the first terminal:

    If you are not already in a development container, run this from the repository root in a terminal on the host, outside any container:

    ./docker/run_docker.sh
    

    If the container is already running, this opens another shell in it. Continue inside the container at /workspaces/autodata.

    Run commands from the repository root in a terminal on the host. uv run uses your installed uv environment, so you don’t need to activate it.

  2. Activate the IsaacTeleop CloudXR environment settings written by the runtime:

    source ~/.cloudxr/run/cloudxr.env
    

    Important

    Order matters. source ~/.cloudxr/run/cloudxr.env after the CloudXR runtime from the previous section is already running, and before you start the recording script — the app must inherit the CloudXR environment variables.

  3. Run the recording script. CPU simulation gives smoother XR performance with a single environment:

    Both commands select the AVP CloudXR environment and disable automatic CloudXR launch because the runtime is already running in the first terminal.

    python submodules/IsaacLab-Arena/submodules/IsaacLab/scripts/tools/record_demos.py \
        --task IsaacContrib-PickPlace-GR1T2-Abs \
        --viz kit \
        --device cpu \
        --xr \
        --cloudxr_env avp \
        --no-auto_launch_cloudxr \
        --dataset_file ./datasets/dataset_gr1.hdf5 \
        --num_demos 5
    
    uv run --locked python submodules/IsaacLab-Arena/submodules/IsaacLab/scripts/tools/record_demos.py \
        --task IsaacContrib-PickPlace-GR1T2-Abs \
        --viz kit \
        --device cpu \
        --xr \
        --cloudxr_env avp \
        --no-auto_launch_cloudxr \
        --dataset_file ./datasets/dataset_gr1.hdf5 \
        --num_demos 5
    
  4. In the running application window, press the Start XR button under the XR tab on the right side of the screen.

    Start XR button in the XR tab

Connect the Apple Vision Pro and Record#

For detailed connection instructions, see Connect an XR Device in the Isaac Lab docs.

  1. On the Apple Vision Pro, launch the Isaac XR Teleop app.

  2. Enter your workstation’s IP address and connect.

  3. Before pressing Connect, pinch the bar at the bottom of the CloudXR controls window and move it closer and to your left, so nearby objects don’t occlude it.

  4. Press Connect and wait until you see the simulation in VR.

CloudXR control panel

CloudXR control panel — move this window to your left to avoid occlusion by nearby objects.#

Once connected, complete the pick-and-place task:

  • Your hands control the robot’s hands; your fingers control the robot’s fingers.

  • On task completion the environment resets automatically.

  • Repeat until all --num_demos (5 above) successful demonstrations are recorded; the script then shuts down and saves the dataset.

Performing the Demonstrations#

The humanoid pick-and-place task is set up so the left hand has a single subtask (pickup and transport) while the right hand has two (an idle phase, then the place). During the idle phase the right hand should stay still while the left hand brings the object to a position where the right hand will grasp it. This lets DexMimicGen interpolate the right hand’s trajectory accurately from the object’s pose when scenes are randomized. Avoid moving the right hand while the left hand is picking up the object.

Tip

For best results during the recording session:

  • Move slowly and smoothly, and keep your hands within the tracking volume.

  • Take a direct path toward the goal instead of following axes.

  • Avoid extended pauses — smooth, continuous motion with brief pauses is easier to learn than unexplained stops.

  • Ensure good lighting for hand tracking, and collect at least 5 successful demonstrations.

Expected Output#

Verify the datasets contain the recorded episodes using:

python scripts/validate_dataset.py datasets/dataset_gr1.hdf5
uv run --locked python scripts/validate_dataset.py datasets/dataset_gr1.hdf5

Continue to Step 2: Annotate Demonstrations.