Teleoperation Data Collection#

This workflow covers collecting demonstrations for the G1 loco-manipulation task using Meta Quest 3 supported by Nvidia IsaacTeleop.

Step 1: Start the CloudXR Runtime#

On the host machine, configure the firewall to allow CloudXR traffic. The required ports depend on the client type.

sudo ufw allow 49100/tcp   # Signaling
sudo ufw allow 47998/udp   # Media stream
sudo ufw allow 48322/tcp   # Proxy (HTTPS mode only)

Start the CloudXR runtime from the Arena Docker container:

./docker/run_docker.sh

python -m isaacteleop.cloudxr

Step 2: Start Arena Teleop#

In another terminal, start the Arena Docker container and launch the teleop session to verify the pipeline:

./docker/run_docker.sh

source ~/.cloudxr/run/cloudxr.env
python isaaclab_arena/scripts/imitation_learning/teleop.py \
  --visualizer kit \
  --device cpu \
  galileo_g1_locomanip_pick_and_place \
  --teleop_device openxr

Start the session from the XR tab in the application window.

Arena teleop with XR running (stereoscopic view and OpenXR settings)

Arena teleop session with XR running. Stereoscopic view (left) and OpenXR settings in the XR tab (right).#

Step 3: Connect from Meta Quest 3#

For detail instructions please refer to Connect an XR Device:

A strong wireless connection is essential for a high-quality streaming experience. Refer to the CloudXR Network Setup guide for router configuration.

  1. Open the browser on your headset and navigate to https://nvidia.github.io/IsaacTeleop/client.

  2. Enter the IP address of your Isaac Lab host machine in the Server IP field.

  3. Click the Click https://<ip>:48322/ to accept cert link that appears on the page. Accept the certificate in the new page that opens, then navigate back to the CloudXR.js client page.

  4. Click Connect to begin teleoperation.

  5. Teleoperation Controls:

  • Left joystick: Move the body forward/backward/left/right.

  • Right joystick: Squat (down), rotate torso (left/right).

  • Controllers: Move end-effector (EE) targets for the arms.

Note

If the simulation runs at too low FPS and makes the teleoperation feel laggy, you can try to reduce the XR resolution from the XR tab / Advanced Settings / Render Resolution.

XR resolution panel

Reducing render resolution from 1 (default) to 0.2.#

Step 4: Record with Quest 3#

  1. Recording: When ready to collect data, run the recording script from the Arena container:

    export DATASET_DIR=/datasets/isaaclab_arena/locomanipulation_tutorial
mkdir -p $DATASET_DIR

# Record demonstrations with OpenXR teleop
python isaaclab_arena/scripts/imitation_learning/record_demos.py \
  --visualizer kit \
  --device cpu \
  --dataset_file $DATASET_DIR/arena_g1_locomanipulation_dataset_recorded.hdf5 \
  --num_demos 10 \
  --num_success_steps 2 \
  galileo_g1_locomanip_pick_and_place \
  --teleop_device openxr

  2. Complete the task for each demo. Reset between demos. The script saves successful runs to the HDF5 file above.

Hint

Suggested sequence for the task:

  1. Align your body with the robot.

  2. Walk forward (left joystick forward).

  3. Grab the box (controllers).

  4. Walk backward (left joystick back).

  5. Turn toward the bin (right joystick).

  6. Walk forward to the bin.

  7. Squat (right joystick down).

  8. Place the box in the bin (controllers).

../../../_images/g1_galileo_arena_box_pnp_locomanip.gif

Step 5: Replay Recorded Demos (Optional)#

To replay the recorded demos:

# Replay from the recorded HDF5 dataset
python isaaclab_arena/scripts/imitation_learning/replay_demos.py \
  --visualizer kit \
  --device cpu \
  --dataset_file $DATASET_DIR/arena_g1_locomanipulation_dataset_recorded.hdf5 \
  galileo_g1_locomanip_pick_and_place