Teleoperation Data Collection#

This workflow covers collecting demonstrations for the G1 loco-manipulation task using Meta Quest 3 supported by Nvidia IsaacTeleop.

Step 1: Start the CloudXR Runtime#

  1. On the host machine, configure the firewall to allow CloudXR traffic. The required ports depend on the client type.

    sudo ufw allow 49100/tcp   # Signaling
sudo ufw allow 47998/udp   # Media stream
sudo ufw allow 48322/tcp   # Proxy (HTTPS mode only)

  2. Start the CloudXR runtime from the Arena Docker container:

    ./docker/run_docker.sh

    python -m isaacteleop.cloudxr

Attention

The first run will prompt users to accept the NVIDIA CloudXR License Agreement. To accept the EULA, reply Yes when prompted with the below message:

NVIDIA CloudXR EULA must be accepted to run. View: https://github.com/NVIDIA/IsaacTeleop/blob/main/deps/cloudxr/CLOUDXR_LICENSE

Accept NVIDIA CloudXR EULA? [y/N]: Yes

Step 2: Start Arena Teleop#

  1. In another terminal, start the Arena Docker container:

    ./docker/run_docker.sh

  2. Run the following command to activate IsaacTeleop CloudXR environment settings:

    source ~/.cloudxr/run/cloudxr.env

    Important

    Order matters. In the terminal where you will run Arena, source ~/.cloudxr/run/cloudxr.env after the CloudXR runtime from Step 1 is already running, and before you start the Arena app. The Arena app must inherit the IsaacTeleop CloudXR environment variables.

  3. Run the teleop script:

    python isaaclab_arena/scripts/imitation_learning/teleop.py \
  --viz kit \
  --device cpu \
  galileo_g1_locomanip_pick_and_place \
  --teleop_device openxr

  4. In the running application, start the session from the XR tab in the application window.

    Arena teleop with XR running (stereoscopic view and OpenXR settings)

    Arena teleop session with XR running. Stereoscopic view (left) and OpenXR settings in the XR tab (right).#

Step 3: Connect from Meta Quest 3#

For detail instructions please refer to Connect an XR Device:

A strong wireless connection is essential for a high-quality streaming experience. Refer to the CloudXR Network Setup guide for router configuration.

  1. Open the browser on your headset and navigate to https://nvidia.github.io/IsaacTeleop/client.

  2. Enter the IP address of your Isaac Lab host machine in the Server IP field.

  3. Click the Click https://<ip>:48322/ to accept cert link that appears on the page. Accept the certificate in the new page that opens, then navigate back to the CloudXR.js client page.

  4. Click Connect to begin teleoperation.

  5. Teleoperation Controls:

    • Left joystick: Move the body forward/backward/left/right.

    • Right joystick: Squat (down), rotate torso (left/right).

    • Controllers: Move end-effector (EE) targets for the arms.

Note

If the simulation runs at too low FPS and makes the teleoperation feel laggy, you can try to reduce the XR resolution from the XR tab / Advanced Settings / Render Resolution.

XR resolution panel

Reducing render resolution from 1 (default) to 0.2.#

Once you have verified the teleoperation pipeline, exit VR from the Quest 3 headset, and stop the Arena teleop app.

Step 4: Record with Quest 3#

Note

Run the following command to activate IsaacTeleop CloudXR environment settings again if you are starting the recording app from a different terminal.

source ~/.cloudxr/run/cloudxr.env

  1. Run the recording script from the Arena container:

    export DATASET_DIR=/datasets/isaaclab_arena/locomanipulation_tutorial
mkdir -p $DATASET_DIR

# Record demonstrations with OpenXR teleop
python isaaclab_arena/scripts/imitation_learning/record_demos.py \
  --viz kit \
  --device cpu \
  --dataset_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_recorded.hdf5 \
  --num_demos 10 \
  --num_success_steps 2 \
  galileo_g1_locomanip_pick_and_place \
  --teleop_device openxr

  2. In the running application, start the session from the XR tab in the application window.

  3. Follow Step 3 to connect the Quest 3 headset again.

  4. Complete the task for each demo. Reset between demos. The script saves successful runs to the HDF5 file above.

Hint

Suggested sequence for the task:

  1. Align your body with the robot.

  2. Walk forward (left joystick forward).

  3. Grab the box (controllers).

  4. Walk backward (left joystick back).

  5. Turn toward the bin (right joystick).

  6. Walk forward to the bin.

  7. Squat (right joystick down).

  8. Place the box in the bin (controllers).

../../../_images/g1_galileo_arena_box_pnp_locomanip.gif

Warning

Known issue: the squat height does not reset correctly between demos. As a workaround, after each completed demo:

  1. Use the right joystick (up) to stand the robot back up.

  2. Use the control panel to Reset, then Play to start the next demo.

Step 5: Replay Recorded Demos (Optional)#

To replay the recorded demos:

# Replay from the recorded HDF5 dataset
python isaaclab_arena/scripts/imitation_learning/replay_demos.py \
  --viz kit \
  --device cpu \
  --dataset_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_recorded.hdf5 \
  galileo_g1_locomanip_pick_and_place