Teleoperation Data Collection#

This workflow covers collecting demonstrations for the G1 loco-manipulation task using Meta Quest 3 supported by Nvidia IsaacTeleop.

Note

For supported IsaacTeleop hardware devices, see Supported Input Devices. Before starting teleoperation, also review the IsaacTeleop system requirements.

Important

A stable network connection meeting the CloudXR network requirements is required before starting the steps below.

Before starting teleoperation, configure the host firewall to allow CloudXR traffic. The required ports depend on the client type:

sudo ufw allow 49100/tcp   # Signaling
sudo ufw allow 47998/udp   # Media stream
sudo ufw allow 48322/tcp   # Proxy (HTTPS mode only)

Step 1: Start Arena Teleop#

  1. Start the Arena Docker container:

    ./docker/run_docker.sh

  2. Run Isaac Lab’s teleop script with Arena’s environment registration callback. The script launches the CloudXR runtime automatically:

    python submodules/IsaacLab/scripts/environments/teleoperation/teleop_se3_agent.py \
  --viz kit \
  --device cpu \
  --xr \
  --external_callback isaaclab_arena.environments.isaaclab_interop.environment_registration_callback \
  --task galileo_g1_locomanip_pick_and_place \
  --arena_teleop_device openxr

  3. In the running application, start the session from the XR tab in the application window.

    Arena teleop with XR running (stereoscopic view and OpenXR settings)

    Arena teleop session with XR running. Stereoscopic view (left) and OpenXR settings in the XR tab (right).#

Step 2: Connect from Meta Quest 3#

For detail instructions please refer to Connect an XR Device:

  1. Open the browser on your headset and navigate to https://nvidia.github.io/IsaacTeleop/client.

  2. Enter the IP address of your Isaac Lab host machine in the Server IP field.

  3. Click the Click https://<ip>:48322/ to accept cert link that appears on the page. Accept the certificate in the new page that opens, then navigate back to the CloudXR.js client page.

  4. Click Connect to begin teleoperation.

  5. Teleoperation Controls:

    • Left joystick: Move the body forward/backward/left/right.

    • Right joystick: Squat (down), rotate torso (left/right).

    • Controllers: Move end-effector (EE) targets for the arms.

Note

If the simulation runs at too low FPS and makes the teleoperation feel laggy, you can try to reduce the XR resolution from the XR tab / Advanced Settings / Render Resolution.

XR resolution panel

Reducing render resolution from 1 (default) to 0.2.#

Once you have verified the teleoperation pipeline, exit VR from the Quest 3 headset, and stop the Arena teleop app.

Step 3: Record with Quest 3#

  1. Run the recording script from the Arena container:

    export DATASET_DIR=/datasets/isaaclab_arena/locomanipulation_tutorial
mkdir -p $DATASET_DIR

# Record demonstrations with OpenXR teleop
python submodules/IsaacLab/scripts/tools/record_demos.py \
  --viz kit \
  --device cpu \
  --xr \
  --dataset_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_recorded.hdf5 \
  --num_demos 10 \
  --num_success_steps 2 \
  --external_callback isaaclab_arena.environments.isaaclab_interop.environment_registration_callback \
  --task galileo_g1_locomanip_pick_and_place \
  --arena_teleop_device openxr

    Warning

    If you exit Sim with Ctrl-C, you need to manually clean up the spawned CloudXR process with:

    pkill -KILL -f '[i]saacteleop.cloudxr.runtime'

    Otherwise the next record_demos.py run will crash with an error looking like XR_ERROR_INSTANCE_LOST in xrPollEvent: Call to "xrt_session_poll_events" failed.

  2. In the running application, start the session from the XR tab in the application window.

  3. Follow Step 2 to connect the Quest 3 headset again.

  4. Complete the task for each demo. Reset between demos. The script saves successful runs to the HDF5 file above.

Hint

Suggested sequence for the task:

  1. Align your body with the robot.

  2. Walk forward (left joystick forward).

  3. Grab the box (controllers).

  4. Walk backward (left joystick back).

  5. Turn toward the bin (right joystick).

  6. Walk forward to the bin.

  7. Squat (right joystick down).

  8. Place the box in the bin (controllers).

../../../_images/g1_galileo_arena_box_pnp_locomanip.gif

Warning

Known issue: the squat height does not reset correctly between demos. As a workaround, after each completed demo:

  1. Use the right joystick (up) to stand the robot back up.

  2. Use the control panel to Reset, then Play to start the next demo.

Step 4: Replay Recorded Demos (Optional)#

To replay the recorded demos:

# Replay from the recorded HDF5 dataset
python submodules/IsaacLab/scripts/tools/replay_demos.py \
  --viz kit \
  --device cpu \
  --dataset_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_recorded.hdf5 \
  --external_callback isaaclab_arena.environments.isaaclab_interop.environment_registration_callback \
  --task galileo_g1_locomanip_pick_and_place