Isaac Teleop

Isaac Teleop#

Isaac Teleop is the unified framework for high-fidelity egocentric and robot data collection. It provides a standardized device interface, a flexible graph-based retargeting pipeline, and works seamlessly across simulated and real-world robots.

Isaac Teleop replaces the previous native XR teleop stack (isaaclab.devices.openxr) in Isaac Lab. For migration details see Migrating to Isaac Lab 3.0.

Tip

Just want to get running? Follow the Setting up Isaac Teleop with CloudXR how-to guide for installation and first-run steps, then come back here for deeper topics.

Supported Devices#

Isaac Teleop supports multiple XR headsets and tracking peripherals. Each device provides different input modes, which determine which retargeters and control schemes are available.

Device	Input Modes	Client / Connection	Notes
Apple Vision Pro	Hand tracking (26 joints), spatial controllers	Native visionOS app (Isaac XR Teleop Sample Client)	Build from source; see Build and Install the Client App
Meta Quest 3	Motion controllers (triggers, thumbsticks, squeeze), hand tracking	CloudXR.js WebXR client (browser)	CloudXR client; see connection guide
Pico 4 Ultra	Motion controllers, hand tracking	CloudXR.js WebXR client (browser)	Requires Pico OS 15.4.4U+; must use HTTPS mode
Manus Gloves	High-fidelity finger tracking (Manus SDK)	Isaac Teleop plugin (bundled)	Migrated from the now-deprecated `isaac-teleop-device-plugins` repo. Combine with an external wrist-tracking source for wrist positioning. See Manus Gloves.

Choose a Control Scheme#

The right combination of input device and retargeters depends on your task. Use this table as a starting point, then see the detailed pipeline examples below.

Task Type	Recommended Input	Retargeters	Action Dim	Reference Config
Manipulation (e.g. Franka)	Motion controllers	`Se3AbsRetargeter` + `GripperRetargeter`	8	`stack_ik_abs_env_cfg.py`
Bimanual dex + locomotion (e.g. G1 TriHand)	Motion controllers	Bimanual `Se3AbsRetargeter` + `TriHandMotionControllerRetargeter` + `LocomotionRootCmdRetargeter`	32	`locomanipulation_g1_env_cfg.py`
Bimanual dex, fixed base (e.g. G1)	Motion controllers	Bimanual `Se3AbsRetargeter` + `TriHandMotionControllerRetargeter`	28	`fixed_base_upper_body_ik_g1_env_cfg.py`
Complex dex hand (e.g. GR1T2, G1 Inspire)	Hand tracking / Manus gloves	Bimanual `Se3AbsRetargeter` + `DexBiManualRetargeter`	36+	`pickplace_gr1t2_env_cfg.py`

Why motion controllers for manipulation? Controllers provide precise spatial control via a grip pose and a physical trigger for gripper actuation, making them ideal for pick-and-place tasks.

Why hand tracking for complex dex hands? Hand tracking captures the full 26-joint hand pose required for high-fidelity dexterous retargeting. This is essential when individual finger control matters.

How It Works#

The IsaacTeleopDevice is the main integration point between Isaac Teleop and Isaac Lab. It composes three collaborators:

XrAnchorManager – creates and synchronizes an XR anchor prim in the simulation, and computes the world_T_anchor transform matrix that maps XR tracking data into the simulation coordinate frame.
TeleopSessionLifecycle – builds the retargeting pipeline, acquires OpenXR handles from Isaac Sim’s XR bridge (or, in standalone mode, creates its own OpenXR session through the CloudXR runtime – see Run Without a Headset (Standalone I/O)), creates the TeleopSession, and steps it each frame to produce an action tensor.
CommandHandler – lightweight callback registry for START / STOP / RESET commands. Scripts can register callbacks via add_callback(), but the primary control path uses poll_control_events() (see Teleop Control States (Start / Stop / Reset)).

Visualize XR Tracking#

Isaac Teleop can draw the raw XR tracking poses in the Isaac Lab world frame. Use this visualization to confirm that tracking data is available and aligned with the simulated robot:

red spheres show the 26 joints of each tracked hand when the retargeting pipeline contains a HandsSource;
RGB coordinate axes show the OpenXR aim pose of each tracked controller. The X, Y, and Z axes are red, green, and blue, respectively, and the controller’s -Z axis points forward in its natural pointing direction.

Enable the visualization when launching a teleoperation session:

uv (Recommended)

uv run --extra teleop isaaclab teleop run \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit \
    --xr \
    --enable_debug_visualization

isaaclab.sh / isaaclab.bat

./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit \
    --xr \
    --enable_debug_visualization

The --enable_debug_visualization flag is also available in scripts/tools/record_demos.py and scripts/environments/teleoperation/teleop_replay_agent.py. The option is applied when the Isaac Teleop device is created and cannot be toggled during a running device session. The markers are diagnostic only and do not change the actions produced by the retargeting pipeline.

Note

Tracking markers are only visible when the Kit visualizer is active. A marker is created after the corresponding hand or controller first produces valid tracking data.

Teleop Control States (Start / Stop / Reset)#

Isaac Lab supports remote teleop control commands – start, stop, and reset – sent from the XR headset to the simulation. These are used to begin and end demonstration recording, pause the robot, or reset the environment without touching the simulation host. The same commands can be driven locally without a headset via request_start() / request_stop() and reset() (see Run Without a Headset (Standalone I/O)).

How it works#

By default, every IsaacTeleopCfg enables a control message channel using the well-known UUID uuid5(NAMESPACE_DNS, "teleop_command"). The channel is created as a teleop_control_pipeline inside TeleopCore’s TeleopSession, which means:

A MessageChannelSource opens an OpenXR opaque data channel (XR_NV_opaque_data_channel) with the agreed-upon UUID.

The CloudXR JS client (or any other client) discovers the channel by UUID and sends UTF-8 JSON commands:

{"type": "teleop_command", "message": {"command": "start teleop"}}
{"type": "teleop_command", "message": {"command": "stop teleop"}}
{"type": "teleop_command", "message": {"command": "reset teleop"}}

A TeleopMessageProcessor parses these payloads and produces boolean pulse signals (run_toggle, kill, reset).
DefaultTeleopStateManager consumes the boolean signals, runs its state machine (edge detection, fail-safe), and produces teleop_state (one-hot) and reset_event (bool pulse) outputs.
TeleopCore decodes these outputs into ExecutionEvents and injects them into every retargeter’s ComputeContext, so stateful retargeters can react to state changes (e.g. reinitializing cross-step state on reset).

Polling control events in your script#

Use poll_control_events() to read the latest control state each frame:

from isaaclab_teleop import poll_control_events

with IsaacTeleopDevice(cfg) as device:
    running = False
    while sim_app.is_running():
        action = device.advance()

        ctrl = poll_control_events(device)
        if ctrl.is_active is not None:
            running = ctrl.is_active      # True after "start", False after "stop"
        if ctrl.should_reset:
            env.reset()                    # "reset" command received this frame

        if action is not None and running:
            env.step(action.repeat(num_envs, 1))
        else:
            env.sim.render()

ControlEvents has two fields:

is_active – True after a “start” command, False after “stop”, None when no command has been received yet (callers should leave their own flag unchanged).
should_reset – True for exactly one frame after a “reset” command.

Disabling the control channel#

If you do not need headset-driven start/stop/reset (e.g. keyboard-only workflows), set control_channel_uuid=None in your config:

IsaacTeleopCfg(
    pipeline_builder=_build_my_pipeline,
    control_channel_uuid=None,   # no opaque data channel created
)

Using a custom channel UUID#

To use a different channel UUID (e.g. for a separate control protocol), pass any 16-byte bytes value:

import uuid

MY_UUID = uuid.uuid5(uuid.NAMESPACE_DNS, "my_custom_control").bytes

IsaacTeleopCfg(
    pipeline_builder=_build_my_pipeline,
    control_channel_uuid=MY_UUID,
)

The CloudXR JS client must be updated to discover this UUID when sending commands.

Run Without a Headset (Standalone I/O)#

By default the teleop scripts drive Isaac Teleop through Isaac Sim’s Kit XR bridge, which renders the scene to a connected XR headset. When you only need Isaac Teleop as an input/output transport – for example a joint-space leader arm streaming encoder angles, or any non-XR device – you can run it standalone, with no Kit XR rendering.

The teleop_se3_agent.py and record_demos.py scripts select the mode with the --xr flag:

With --xr – the full Kit XR path (headset rendering, anchor, hand / controller tracking).
Without --xr – Isaac Teleop creates and owns its own OpenXR session through the CloudXR runtime (create_isaac_teleop_device(..., use_kit_xr_bridge=False)); teleop input/output works headless, with no Kit XR rendering.

Starting teleop without a headset#

Without a headset there is no client to send the start command, so the scripts drive the state machine locally:

On startup they call request_start(), which transitions the teleop state machine to RUNNING (see Teleop Control States (Start / Stop / Reset)) – so a headless session begins running immediately, with no headset UI.
When a Kit viewport window is present, they also bind keys for interactive control:

Key

Action

B

Start / resume teleoperation.

P

Pause teleoperation (robot holds position).

R

Reset the environment.

Both paths flow through the same state machine, so request_start() / request_stop() and remote headset commands are interchangeable, and poll_control_events() stays authoritative.

Note

A clientless CloudXR runtime advertises no HMD, so the standalone OpenXR session needs a device profile that reports a system without a connected client. The scripts default --cloudxr_env to CLOUDXR_STANDALONE_ENV when --xr is omitted (and to cloudxrjs when it is passed); see CloudXR Environment Profiles.

Example: SO-101 leader-arm joint teleoperation#

IsaacContrib-Stack-Cube-SO101-Joint-Teleop-v0 mirrors the joint angles streamed by a physical SO-101 leader arm directly onto the simulated follower – no XR headset, inverse kinematics, or anchor. Its pipeline is JointStateSource -> JointStateRetargeter (mode="joint") -> TensorReorderer.

For the end-to-end walkthrough – building the plugin, hardware setup, calibration, launching, and recording a dataset – see Data Collection in Sim in the Isaac Teleop documentation. The prerequisites below are the minimum needed to get this example running.

Prerequisites#

SO-101 hardware: A physical SO-101 leader arm connected to the workstation over USB.
The so101_leader plugin, built from Isaac Teleop source: the leader’s joint state is streamed by a standalone C++ plugin that you run alongside the sim.

Important

so101_leader_plugin is not shipped with Isaac Lab, is not part of the isaacteleop pip package, and is not in any release archive. It exists only after building the Isaac Teleop repository from source. If install/plugins/so101_leader/so101_leader_plugin does not exist in your Isaac Teleop checkout, this step has not been completed.

Install the build prerequisites first – a missing clang-format-14 is the most common cause of a failed build, because the format check is enforced by default on Linux:
```
sudo apt-get update
sudo apt-get install -y build-essential cmake libx11-dev clang-format-14 ccache patchelf
```
Then clone, configure, build, and install. To target a specific Python version, pass -DISAAC_TELEOP_PYTHON_VERSION=3.12 (or 3.11, 3.13) on the configure line; each version needs its own build directory if building multiple at once:
```
git clone https://github.com/NVIDIA/IsaacTeleop.git
cd IsaacTeleop

cmake -B build                       # configure (default: Python 3.11)
cmake --build build --parallel       # build
cmake --install build                # install into ./install
```
The plugin is installed to <IsaacTeleop>/install/plugins/so101_leader/so101_leader_plugin. Every later command in this section runs from the Isaac Teleop checkout root; substitute your own path for /path/to/IsaacTeleop. Verify the build by running the plugin with no arguments – that selects the synthetic backend, so it starts without any hardware attached:
```
cd /path/to/IsaacTeleop
./install/plugins/so101_leader/so101_leader_plugin
```
See Build from Source for the full prerequisite list and all build options, and the build-troubleshooting table in Data Collection in Sim for common failures (including the clang-format not found but ENABLE_CLANG_FORMAT_CHECK is ON CMake error).
Calibration: The SO-101 arm must be calibrated before use. The plugin talks to the FEETECH servos directly – it has no lerobot or FEETECH SDK dependency – so calibrate with its own calibrate subcommand, which needs no OpenXR runtime:
```
cd /path/to/IsaacTeleop
./install/plugins/so101_leader/so101_leader_plugin calibrate /dev/ttyACM0 so101_leader.calib
```
It runs two interactive steps (hold the arm at mid-range, then sweep every joint through its full range) and writes the calibration file, which the plugin reads at startup to map raw encoder values to joint angles. An existing LeRobot calibration .json can be passed instead. Running without calibration produces incorrect joint mappings and the follower arm will not track the leader.

Run the simulation#

The SO-101 workflow supports two monitoring modes – choose based on whether you have an XR headset available:

Without a headset (local viewport only)

Omit --xr. The sim runs in standalone mode and the teleop state machine starts automatically on launch – no headset connection is needed (see Run Without a Headset (Standalone I/O)). Pass --visualizer kit to watch the follower in the local Kit viewport; headless is the default.

uv (Recommended)

uv run --extra teleop isaaclab teleop run \
    --task IsaacContrib-Stack-Cube-SO101-Joint-Teleop-v0 \
    --num_envs 1 \
    --visualizer kit

isaaclab.sh / isaaclab.bat

./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
    --task IsaacContrib-Stack-Cube-SO101-Joint-Teleop-v0 \
    --num_envs 1 \
    --visualizer kit

With a headset (immersive XR view)

Add --xr to stream the simulation to a Quest, Pico, or Apple Vision Pro headset while the SO-101 leader arm still drives the follower joints. The retargeting pipeline is unchanged; --xr only controls whether the scene is rendered to the headset. Follow the connection steps in Connect an XR Device to pair the headset after launching.

uv (Recommended)

uv run --extra teleop isaaclab teleop run \
    --task IsaacContrib-Stack-Cube-SO101-Joint-Teleop-v0 \
    --num_envs 1 \
    --visualizer kit --xr

isaaclab.sh / isaaclab.bat

./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
    --task IsaacContrib-Stack-Cube-SO101-Joint-Teleop-v0 \
    --num_envs 1 \
    --visualizer kit --xr

Start the plugin#

Isaac Lab does not spawn the plugin for you. Launch the sim first – it brings up the CloudXR runtime the plugin connects through – then, in a separate terminal, source the environment file the runtime writes on startup (this points the OpenXR loader at CloudXR) and start the plugin on the leader’s serial port:

cd /path/to/IsaacTeleop
source ~/.cloudxr/run/cloudxr.env
./install/plugins/so101_leader/so101_leader_plugin /dev/ttyACM0 so101_leader so101_leader.calib

Arguments are positional: [device_path] [collection_id] [calibration_file]. The collection_id must stay so101_leader – that is the tensor collection this task’s JointStateSource subscribes to.

Running the plugin with no arguments at all streams a synthetic trajectory on the default so101_leader collection, which is a good way to confirm the sim side is wired up before connecting hardware. Because the arguments are positional, there is no way to skip only the device path while still passing the later two:

./install/plugins/so101_leader/so101_leader_plugin   # synthetic, no hardware needed

See the SO-101 plugin README and Data Collection in Sim for all configuration options.

Start teleoperation#

Without ``–xr``: the script calls request_start() automatically on startup, so the follower begins mirroring the leader immediately with no additional input required.

With ``–xr``: a start command can be sent from the connected headset UI (Quest / Pico: the CloudXR.js Connect button; Apple Vision Pro: the Play button in the Isaac XR Teleop Sample Client). The SO-101 then begins driving the follower as soon as the headset sends the start command.

Regardless of mode, when the Kit viewport is open you can also control the session with keyboard shortcuts:

Key	Action
`B`	Start / resume teleoperation.
`P`	Pause teleoperation (follower holds position).
`R`	Reset the environment.

Move the physical SO-101 leader arm and the simulated follower will mirror its joint angles in real time. To record demonstrations from this task, run scripts/tools/record_demos.py with the same --task and the plugin running in its second terminal – see Data Collection in Sim for the full recording workflow and runtime troubleshooting.

Retargeting Framework#

Isaac Teleop uses a graph-based retargeting pipeline. Data flows from source nodes through retargeters and is combined into a single action tensor.

Source Nodes#

HandsSource – provides hand tracking data (left/right, 26 joints each).
ControllersSource – provides motion controller data (grip pose, trigger, thumbstick, etc.).

Available Retargeters#

Retargeters are provided by the isaacteleop package from the Isaac Teleop repository. The retargeters listed below are those used by the built-in Isaac Lab environments. Isaac Teleop may offer additional retargeters not listed here – refer to the Isaac Teleop repository for the full set.

The built-in Isaac Lab environments use these retargeters as follows:

Environment	Retargeters Used
Franka manipulation (stack, pick-place)	`Se3AbsRetargeter`, `GripperRetargeter`, `TensorReorderer`
G1 Inspire dexterous pick-place	`Se3AbsRetargeter`, `DexHandRetargeter`, `TensorReorderer`
GR1-T2 dexterous pick-place	`Se3AbsRetargeter`, `DexHandRetargeter`, `TensorReorderer`
G1 upper-body (fixed base)	`Se3AbsRetargeter`, `TriHandMotionControllerRetargeter`, `TensorReorderer`
G1 loco-manipulation	`Se3AbsRetargeter`, `TriHandMotionControllerRetargeter`, `LocomotionRootCmdRetargeter`, `TensorReorderer`

Teleoperation Environment Reference#

The tables below list every built-in Isaac Lab environment that supports teleoperation, organized by input method. For closed-loop policy evaluation, the play script automatically applies each environment config’s play_mode overrides; pass --train_env_cfg to play the training configuration as-is.

Isaac Teleop (XR Headset) Environments#

These environments use the Isaac Teleop XR pipeline with motion controllers or hand tracking.

Task ID	Input Mode	Hands	Operator Interaction
`IsaacContrib-Stack-Cube-Franka-IK-Abs`	Controllers	Right	Arm: right controller grip pose drives end-effector. Gripper: right trigger.
`IsaacContrib-Stack-Cube-SO101-IK-Abs-v0`	Controllers	Right	Arm: right controller grip pose drives the end-effector via absolute IK (clutch-rebased; IK tracks position, orientation soft-weighted). Gripper: right trigger (analog).
`IsaacContrib-PickPlace-GR1T2-Abs`	Hand tracking	Both	Arms: left/right hand wrist pose drives each end-effector. Hands: full 26-joint hand tracking retargeted to 11 DOF per Fourier hand via `DexHandRetargeter`.
`IsaacContrib-PickPlace-GR1T2-WaistEnabled-Abs`	Hand tracking	Both	Same as `IsaacContrib-PickPlace-GR1T2-Abs` with waist DOFs enabled.
`IsaacContrib-NutPour-GR1T2-Pink-IK-Abs`	Hand tracking	Both	Same retargeting pipeline as `IsaacContrib-PickPlace-GR1T2-Abs` (different task scene).
`IsaacContrib-ExhaustPipe-GR1T2-Pink-IK-Abs`	Hand tracking	Both	Same retargeting pipeline as `IsaacContrib-PickPlace-GR1T2-Abs` (different task scene).
`IsaacContrib-PickPlace-G1-InspireFTP-Abs`	Hand tracking	Both	Arms: left/right hand wrist pose drives each end-effector. Hands: full 26-joint hand tracking retargeted to 12 DOF per Inspire hand via `DexHandRetargeter`.
`IsaacContrib-PickPlace-FixedBaseUpperBodyIK-G1-Abs`	Controllers	Both	Arms: left/right controller grip pose drives each end-effector. Hands: trigger closes index, squeeze closes middle, both together close thumb (7 DOF TriHand per hand).
`IsaacContrib-PickPlace-Locomanipulation-G1-Abs`	Controllers	Both	Arms: same as fixed-base G1 above. Hands: same TriHand mapping. Locomotion: left thumbstick = linear velocity (x/y), right thumbstick X = rotational velocity, right thumbstick Y = hip height.

Tip

Controllers provide a grip pose plus physical buttons (trigger, squeeze, thumbstick), ideal for tasks that need a gripper or simple hand mapping. Hand tracking captures 26 wrist and finger joints per hand, required for dexterous retargeting to complex robot hands.

Keyboard and SpaceMouse Environments#

Note

Keyboard and SpaceMouse teleoperation uses the legacy native Isaac Lab teleop stack (isaaclab.devices), not Isaac Teleop. These environments do not require an XR headset.

The device button layouts below apply to all environments in this section. Per-environment differences (gripper enabled/disabled, sensitivity) are noted in the environment table that follows.

Keyboard

Function	Keys	Description
Position X	`W` / `S`	Move end-effector forward / backward.
Position Y	`A` / `D`	Move end-effector left / right.
Position Z	`Q` / `E`	Move end-effector up / down.
Roll	`Z` / `X`	Rotate about X axis.
Pitch	`T` / `G`	Rotate about Y axis.
Yaw	`C` / `V`	Rotate about Z axis.
Gripper toggle	`K`	Open / close gripper or suction (disabled in Reach envs).
Reset	`L`	Clear accumulated delta pose and gripper state.

SpaceMouse

Function	Control	Description
Translation	6-DOF knob	Push/pull/slide the knob to move the end-effector in X/Y/Z.
Rotation	6-DOF knob	Tilt/twist the knob to rotate the end-effector in roll/pitch/yaw.
Gripper toggle	Left button	Open / close gripper or suction (disabled in Reach envs).
Reset	Right button	Clear accumulated delta pose and gripper state.

Gamepad (Reach environments only)

Function	Control	Description
Position X / Y	Left stick	Move end-effector forward/backward and left/right.
Position Z	Right stick (up/down)	Move end-effector up / down.
Roll / Pitch	D-Pad	Left/right for roll, up/down for pitch.
Yaw	Right stick (left/right)	Rotate about Z axis.
Gripper toggle	X button	Open / close gripper (disabled in Reach envs).

Task ID	Devices	Operator Interaction
`IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow`	Keyboard, SpaceMouse	Arm: end-effector pose via RMPFlow. Gripper: `K` on keyboard, left button on SpaceMouse.
`IsaacContrib-Stack-Cube-Galbot-Right-Arm-Suction-RmpFlow` Note: With the RMPFlow controller, avoid colliding with the cubes during teleoperation: contact forces cause the controller to overtune and the arm to drift. Move the end-effector close to and just above the cube, stop, then close the suction cup.	Keyboard, SpaceMouse	Arm: end-effector pose via RMPFlow. Suction: `K` on keyboard, left button on SpaceMouse.
`IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor`	Keyboard, SpaceMouse	Same as left-arm gripper above with camera observations.
`IsaacContrib-Place-Mug-Agibot-Left-Arm-RmpFlow`	Keyboard, SpaceMouse	Arm: left-arm end-effector pose via RMPFlow. Gripper: `K` on keyboard, left button on SpaceMouse.
`IsaacContrib-Place-Toy2Box-Agibot-Right-Arm-RmpFlow`	Keyboard, SpaceMouse	Arm: right-arm end-effector pose via RMPFlow. Gripper: `K` on keyboard, left button on SpaceMouse.
`IsaacContrib-Stack-Cube-UR10-Long-Suction-IK-Rel`	Keyboard, SpaceMouse	Arm: relative IK end-effector control. Suction: `K` on keyboard, left button on SpaceMouse.
`IsaacContrib-Stack-Cube-UR10-Short-Suction-IK-Rel`	Keyboard, SpaceMouse	Same as long-suction UR10 above with a shorter suction cup.
`Isaac-Reach-Franka` with `physics=isaacsim_physx presets=diffik`	Keyboard, Gamepad, SpaceMouse	Arm: relative IK end-effector control. Gripper disabled.

Note

Humanoid arms (e.g. Galbot, Agibot) have joint limits that inverse kinematics must respect. The differential IK controller ignores these limits, so RMPFlow is preferred for teleoperating them as it enforces joint limits while solving IK. Consequently, the arm may occasionally stop responding when the commanded target pose is unreachable within those limits – this is expected, not a bug.

Leader-Arm Environments#

These environments are driven by a physical leader arm that streams joint angles over the Isaac Teleop tensor transport; the angles are mirrored directly onto the simulated follower with no XR headset or inverse kinematics. See Example: SO-101 leader-arm joint teleoperation above for the run command and pipeline.

Task ID	Device	Operator Interaction
`IsaacContrib-Stack-Cube-SO101-Joint-Teleop-v0`	SO-101 leader arm	Arm + gripper: the leader arm’s six joint angles (five arm DOF + gripper) are mirrored onto the follower via `JointStateRetargeter` (`mode="joint"`).

Switch Between Controllers and Hand Tracking#

The retargeting pipeline determines whether an environment uses motion controllers or hand tracking. Switching input modes requires changing the pipeline_builder function in your environment config. No other environment-level changes are needed as long as the action space (TensorReorderer output order) stays the same.

Controller to hand tracking

The key changes are:

Create a HandsSource and apply the world-to-anchor transform to it (instead of ControllersSource).
Point the Se3RetargeterConfig.input_device at the appropriate HandsSource key.
Set use_wrist_rotation=True and use_wrist_position=True so that the SE3 retargeter reads from the hand wrist joint rather than the controller grip pose.
The GripperRetargeter already supports both inputs – it uses the controller trigger when connected to a ControllersSource or thumb-index pinch when connected to a HandsSource.

Here is the Franka stack environment’s controller-based pipeline alongside a hand-tracking variant for comparison.

Original (controller-based):

# SE3: tracks right controller grip pose
se3_cfg = Se3RetargeterConfig(
    input_device=ControllersSource.RIGHT,
    use_wrist_rotation=False,
    use_wrist_position=False,
    target_offset_roll=90.0,
)
se3 = Se3AbsRetargeter(se3_cfg, name="ee_pose")
connected_se3 = se3.connect({
    ControllersSource.RIGHT: transformed_controllers.output(
        ControllersSource.RIGHT
    ),
})

Modified (hand-tracking-based):

se3_cfg = Se3RetargeterConfig(
    input_device=HandsSource.RIGHT,
    use_wrist_rotation=True,
    use_wrist_position=True,
    target_offset_roll=0.0,
)
se3 = Se3AbsRetargeter(se3_cfg, name="ee_pose")

transformed_hands = hands.transformed(transform_input.output(ValueInput.VALUE))
connected_se3 = se3.connect({
    HandsSource.RIGHT: transformed_hands.output(HandsSource.RIGHT),
})

The GripperRetargeter needs no changes – it accepts both controller and hand inputs and uses whichever source is connected.

Hand tracking to controller

Reverse the steps above: set input_device to a ControllersSource key, transform the controllers instead of the hands, and set use_wrist_rotation=False and use_wrist_position=False. Adjust target_offset_roll/pitch/yaw to account for the controller grip frame orientation (typically 90 degrees roll for Franka-style grippers).

Note

When switching between input modes, you may need to tune the target_offset_roll, target_offset_pitch, and target_offset_yaw values. Controller grip frames and hand wrist frames have different default orientations relative to the robot end-effector.

Build a Retargeting Pipeline#

A pipeline builder is a callable that constructs the retargeting graph and returns an OutputCombiner with a single "action" key. Here is a complete example for a Franka manipulator (from stack_ik_abs_env_cfg.py):

def _build_franka_stack_pipeline():
    from isaacteleop.retargeting_engine.deviceio_source_nodes import ControllersSource, HandsSource
    from isaacteleop.retargeting_engine.interface import OutputCombiner, ValueInput
    from isaacteleop.retargeters import (
        GripperRetargeter, GripperRetargeterConfig,
        Se3AbsRetargeter, Se3RetargeterConfig,
        TensorReorderer,
    )
    from isaacteleop.retargeting_engine.tensor_types import TransformMatrix

    # 1. Create input sources
    controllers = ControllersSource(name="controllers")
    hands = HandsSource(name="hands")

    # 2. Apply coordinate-frame transform (world_T_anchor provided by IsaacTeleopDevice)
    transform_input = ValueInput("world_T_anchor", TransformMatrix())
    transformed_controllers = controllers.transformed(
        transform_input.output(ValueInput.VALUE)
    )

    # 3. Create and connect retargeters
    se3_cfg = Se3RetargeterConfig(
        input_device=ControllersSource.RIGHT,
        target_offset_roll=90.0,
    )
    se3 = Se3AbsRetargeter(se3_cfg, name="ee_pose")
    connected_se3 = se3.connect({
        ControllersSource.RIGHT: transformed_controllers.output(ControllersSource.RIGHT),
    })

    gripper_cfg = GripperRetargeterConfig(hand_side="right")
    gripper = GripperRetargeter(gripper_cfg, name="gripper")
    connected_gripper = gripper.connect({
        ControllersSource.RIGHT: transformed_controllers.output(ControllersSource.RIGHT),
        HandsSource.RIGHT: hands.output(HandsSource.RIGHT),
    })

    # 4. Flatten into a single action tensor with TensorReorderer
    ee_elements = ["pos_x", "pos_y", "pos_z", "quat_x", "quat_y", "quat_z", "quat_w"]
    reorderer = TensorReorderer(
        input_config={
            "ee_pose": ee_elements,
            "gripper_command": ["gripper_value"],
        },
        output_order=ee_elements + ["gripper_value"],
        name="action_reorderer",
        input_types={"ee_pose": "array", "gripper_command": "scalar"},
    )
    connected_reorderer = reorderer.connect({
        "ee_pose": connected_se3.output("ee_pose"),
        "gripper_command": connected_gripper.output("gripper_command"),
    })

    # 5. Return OutputCombiner with "action" key
    return OutputCombiner({"action": connected_reorderer.output("output")})

Tip

The output_order of the TensorReorderer must match the action space of your environment. Mismatches will cause silent control errors.

Configure Your Environment#

Register the pipeline in your environment configuration using IsaacTeleopCfg:

from isaaclab_teleop import IsaacTeleopCfg, XrCfg

@configclass
class MyTeleopEnvCfg(ManagerBasedRLEnvCfg):

    xr: XrCfg = XrCfg(anchor_pos=(0.5, 0.0, 0.5))

    def __post_init__(self):
        super().__post_init__()

        self.isaac_teleop = IsaacTeleopCfg(
            pipeline_builder=_build_my_pipeline,
            sim_device=self.sim.device,
            xr_cfg=self.xr,
        )

Key IsaacTeleopCfg fields:

pipeline_builder – callable that returns an OutputCombiner with an "action" output.
retargeters_to_tune – optional callable returning retargeters to expose in the live tuning UI.
xr_cfg – XrCfg for anchor configuration (see below).
xr_camera_feeds – ordered selection and per-feed panel settings for existing task cameras. The list is empty by default, so tasks opt in to PiP explicitly.
xr_camera_feed_layout – viewer reference, fixed world pose, and manual, horizontal, vertical, or grid panel packing.
plugins – list of Isaac Teleop plugin configurations (e.g. Manus).
sim_device – torch device string (default "cuda:0").
retargeting_execution – IsaacTeleop retargeting execution settings. Defaults to RetargetingExecutionConfig(mode="pipelined") with DeadlinePacingConfig(safety_margin_s=0.025) so retargeting can run on the IsaacTeleop worker instead of blocking the simulation loop. The 25 ms safety margin staggers IsaacTeleop’s Python work behind Isaac Lab’s step Python, giving native work such as rendering time to overlap instead of having both Python stacks contend for the GIL at the start of the step.

Warning

pipeline_builder and retargeters_to_tune must be callables (functions or lambdas), not pre-built objects. The @configclass decorator deep-copies mutable attributes, which would break pre-built pipeline graphs.

CloudXR Environment Profiles#

Isaac Lab ships three .env profiles that configure the CloudXR runtime – two for XR devices and one for headless standalone use. These are bundled inside the isaaclab_teleop package and can be referenced via constants:

Constant	File	`NV_DEVICE_PROFILE`	`NV_CXR_ENABLE_PUSH_DEVICES`	`NV_ENABLE_POSE_WAIT`
`CLOUDXR_JS_ENV`	`cloudxrjs-cloudxr.env`	`auto-webrtc`	`0`	`0`
`CLOUDXR_AVP_ENV`	`avp-cloudxr.env`	`auto-native`	`0`	`0`
`CLOUDXR_STANDALONE_ENV`	`cloudxr-standalone.env`	`quest3`	`0`	`0`

All three profiles set NV_CXR_ENABLE_PUSH_DEVICES=0, which is correct for headset optical hand tracking (the most common setup). For external push-device peripherals such as Manus gloves, set this to 1 in a custom profile (see below). They also set NV_ENABLE_POSE_WAIT=0 so CloudXR does not throttle the application when frame times spike. This favors lower latency over CloudXR’s pose-wait smoothing.

CLOUDXR_STANDALONE_ENV is the default when running without --xr (see Run Without a Headset (Standalone I/O)). It pins an emulated quest3 device profile so a CloudXR runtime with no client still advertises an OpenXR system, working around XR_ERROR_FORM_FACTOR_UNAVAILABLE.

Override at launch time#

The --cloudxr_env flag on teleop_se3_agent.py and record_demos.py selects which .env profile to use. When unset it defaults to cloudxrjs (Quest/Pico) with --xr and standalone without --xr. Use the avp shorthand for Apple Vision Pro, standalone for the headless profile, or pass a full file path for a custom profile:

uv (Recommended)

# Use the AVP profile
uv run --extra teleop isaaclab teleop run \
    --task IsaacContrib-PickPlace-GR1T2-WaistEnabled-Abs \
    --visualizer kit --xr \
    --cloudxr_env avp

isaaclab.sh / isaaclab.bat

# Use the AVP profile
./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
    --task IsaacContrib-PickPlace-GR1T2-WaistEnabled-Abs \
    --visualizer kit --xr \
    --cloudxr_env avp

Create a custom profile#

Copy a shipped profile and edit it:

# Start from the Quest/Pico profile
cp $(uv run --extra teleop python -c \
    "from isaaclab_teleop import CLOUDXR_JS_ENV; print(CLOUDXR_JS_ENV)") ~/my-cloudxr.env

Edit ~/my-cloudxr.env to change any values (e.g. NV_CXR_ENABLE_PUSH_DEVICES=1 for Manus gloves), then pass it via --cloudxr_env ~/my-cloudxr.env.

Disable auto-launch#

If you prefer to run the CloudXR runtime manually in a separate terminal (python -m isaacteleop.cloudxr), you can disable auto-launch in several ways:

CLI flag: --no-auto_launch_cloudxr on the teleop script.
Disable CloudXR entirely: --cloudxr_env none.
Environment variable: ISAACLAB_CXR_SKIP_AUTOLAUNCH=1 overrides the CLI flag at runtime.

uv (Recommended)

# Disable via CLI flag
uv run --extra teleop isaaclab teleop run \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit --xr \
    --no-auto_launch_cloudxr

# Or disable via environment variable
ISAACLAB_CXR_SKIP_AUTOLAUNCH=1 uv run --extra teleop isaaclab teleop run \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit --xr

isaaclab.sh / isaaclab.bat

# Disable via CLI flag
./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit --xr \
    --no-auto_launch_cloudxr

# Or disable via environment variable
ISAACLAB_CXR_SKIP_AUTOLAUNCH=1 ./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit --xr

Configure the XR Anchor#

The XrCfg controls how the simulation is positioned and oriented in the XR device’s view.

anchor_pos / anchor_rot

Static anchor placement. The simulation point at these coordinates appears at the XR device’s local origin (floor level). Set to a point on the floor beneath the robot to position it in front of the user.

anchor_prim_path

Attach the anchor to a USD prim for dynamic positioning. Use this for locomotion tasks where the robot moves and the XR camera should follow.

anchor_rotation_mode

Controls how anchor rotation behaves:

Mode	Description
`FIXED`	Sets rotation once from `anchor_rot`. Best for static manipulation setups.
`FOLLOW_PRIM`	Rotation continuously tracks the attached prim. Best for locomotion where the user should face the robot’s heading direction.
`FOLLOW_PRIM_SMOOTHED`	Same as `FOLLOW_PRIM` with slerp interpolation. Controlled by `anchor_rotation_smoothing_time` (seconds, default 1.0). Reduces motion sickness from abrupt rotation changes. Typical range: 0.3–1.5 s.
`CUSTOM`	User-provided callable `anchor_rotation_custom_func(headpose, primpose) -> quaternion` for fully custom logic.

fixed_anchor_height

When True (default), keeps the anchor height at its initial value. Prevents vertical bobbing during locomotion.

near_plane

Closest render distance for the XR device (default 0.15 m).

Note

On Apple Vision Pro, the local coordinate frame can be reset to a point on the floor beneath the user by holding the digital crown.

Tip

Camera sensors add GPU cost. Strip them with --disable_external_cameras when the workflow needs neither image observations nor XR camera feedback. That flag also skips configured PiP feeds.

XR Camera Feedback#

An ordered list of XrCameraFeedCfg objects selects existing task scene cameras. The manager publishes each new RGBA frame after rendering, while XrCameraFeedLayoutCfg places the panels manually or in horizontal, vertical, and grid layouts. IsaacContrib-PickPlace-GR1T2-Abs and IsaacContrib-PickPlace-Locomanipulation-G1-Abs are the primary reference examples.

teleop_se3_agent.py and record_demos.py show every enabled feed when an IsaacTeleop-enabled environment runs with --xr. PiP is absent unless the task explicitly selects an existing CameraCfg through xr_camera_feeds. In the reference examples, the selected robot_pov_cam is also a policy image observation, so the normal demonstration recorder stores the same view shown to the operator. Both reference cameras are parented to a physical robot body link, so the recorded view follows robot motion. The NutPour and ExhaustPipe GR1T2 teleoperation tasks also present their existing recorded robot_pov_cam:

uv run --extra teleop isaaclab teleop run \
    --task IsaacContrib-PickPlace-GR1T2-Abs \
    --xr --device cpu

uv run --extra teleop isaaclab teleop run \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --xr --device cpu

XR camera PiP currently supports exactly one environment. When a task has enabled PiP feeds, startup rejects --num_envs values other than 1; IsaacTeleop XR behavior without PiP is unchanged.

The reference feeds request render-product-local DLSS Ray Reconstruction and quality execution mode through XrCameraFeedCfg. The private PiP adapter authors those two attributes only on the selected camera render product. On Isaac Sim 6.1 and newer, Ray Reconstruction also requires the process-global responsive-denoising setting, which the session enables before environment construction. On earlier versions, selected PiP feeds fall back to classic DLSS because responsive denoising is unavailable.

Camera selection#

Tasks declare their default selection through IsaacTeleopCfg.xr_camera_feeds:

from isaaclab_teleop import IsaacTeleopCfg, XrCameraFeedCfg

self.isaac_teleop = IsaacTeleopCfg(
    pipeline_builder=_build_my_pipeline,
    xr_camera_feeds=[
        XrCameraFeedCfg(
            camera_name="left_wrist_camera",
            enable_dlss_ray_reconstruction=True,
            dlss_exec_mode="quality",
        ),
        XrCameraFeedCfg(camera_name="overview_camera", enabled=False),
    ],
)

For a recorded training view, define the named CameraCfg in the task scene and a matching mdp.image term in observations.policy. The normal recorder then stores that observation, while the PiP declaration above only selects it for presentation. Enabled task-declared entries control camera selection, panel count, display order, and automatic-layout order without mutating the reusable task configuration.

Placement#

The default placement is "viewer_start". The presenter waits for the first valid headset pose, captures its eye position and upright yaw, then places the layout 0.8 m ahead at eye height. That pose remains fixed in the world while the user moves. Panels hide when the XR display disconnects and capture a new starting pose after reconnect; resetting the environment does not recenter them.

Three placement references are available:

Placement	Behavior
`viewer_start`	Capture the first valid eye position and upright yaw, then keep the layout world-static. This is the default.
`head_locked`	Follow the current headset position and orientation with the configured offset and distance.
`world`	Use `world_position_m` and `world_orientation_xyzw` as a fixed pose in the Isaac Lab USD stage world. `distance_m` is unused.

from isaaclab_teleop import XrCameraFeedLayoutCfg

# Default: fixed in the world at the user's starting eye pose.
self.isaac_teleop.xr_camera_feed_layout = XrCameraFeedLayoutCfg()

# Follow the headset and pack feeds left to right.
self.isaac_teleop.xr_camera_feed_layout = XrCameraFeedLayoutCfg(
    placement="head_locked",
    mode="horizontal",
    distance_m=0.8,
)

# Place a grid in an Isaac Lab Z-up world. This example assumes an eye at
# (0, 0, 1.6) looking along world +Y.
self.isaac_teleop.xr_camera_feed_layout = XrCameraFeedLayoutCfg(
    placement="world",
    mode="grid",
    world_position_m=(0.0, 0.8, 1.6),
    world_orientation_xyzw=(0.7071067812, 0.0, 0.0, 0.7071067812),
    max_columns=2,
)

manual mode preserves each feed’s offset_m and distance_m. horizontal, vertical, and grid modes pack enabled feeds around center_offset_m using panel_gap_m; grid mode also honors max_columns. Offsets are measured in the selected layout plane. With viewer_start and head_locked, distance_m places that plane in front of the viewer reference. In manual mode, feeds with identical offsets overlap; select an automatic mode or assign distinct feed offsets when showing multiple panels.

With world, the explicit pose is measured in the Isaac Lab USD stage world in meters, not OpenXR physical space. Isaac Lab stages are Z-up. world_orientation_xyzw maps panel-local coordinates into that world: local +X is image right, local +Y is image up, and local +Z points from the readable face toward the viewer. Feed and automatic-layout offsets are applied in the panel’s local XY plane.

Disable cameras and PiP#

Set enabled to False to suppress one feed, or set xr_camera_feeds=[] to suppress all PiP. The list is empty by default. These choices do not remove camera sensors independently owned by the task.

--disable_external_cameras is the master camera-rendering switch for teleop_se3_agent.py and record_demos.py. It strips camera sensors and suppresses every configured PiP feed.

Kit Scene UI presentation#

PiP presentation uses Kit Scene UI and SpatialSource placement. Kit imports are deferred until an enabled feed is requested. If the Scene UI modules cannot be imported, the scripts log a warning and continue without PiP; task-owned cameras and recording observations remain unchanged. This keeps the camera selection configuration usable when a future kitless entry point no longer provides Scene UI. Render-product tuning is also best-effort: backends without a compatible render product keep the Camera-buffer path, and schema-authoring failures warn while PiP continues. Other configuration, camera-buffer, and panel-initialization errors still fail during startup.

Haptic Feedback#

Isaac Teleop can render haptic feedback on the operator’s device from a sim-side signal, closing the loop on grasp feel during teleoperation and demonstration recording. Two backends ship today:

Controller vibration – a motion controller rumbles in proportion to the total contact force on the corresponding robot hand.
Haptic glove – each finger of a haptic glove vibrates in proportion to how tightly that finger grips the manipulated object.

Feedback is an output path that mirrors the input retargeting pipeline in reverse. Each step, a HapticFeedbackDriver reads a per-hand signal vector from the scene and pushes it to the IsaacTeleopDevice, which renders it through an Isaac Teleop HapticSink. The signal-to-device mapping runs inside the retargeting graph, so no device-specific code lives in your environment.

The design keeps three concerns pluggable, so the same seam serves both devices:

Signal source – how the per-hand vector is read from the environment.
Output port – the HapticFeedbackReceiver protocol, whose payload is a vector (one scalar for a rumble motor, one value per finger for a glove).
Device backend – selected by the concrete HapticFeedbackCfg subclass, which wires the retargeter and device behind the HapticSink.

To enable feedback, add per-hand ContactSensor s to the scene and attach a HapticFeedbackCfg subclass as a sibling of isaac_teleop. The teleop scripts (teleop_se3_agent.py and record_demos.py) detect the haptic_feedback attribute automatically – no extra flags are needed. If the active device cannot render haptics (e.g. keyboard) or the env has no haptic_feedback, the feature is silently skipped.

Controller vibration#

Use ControllerHapticFeedbackCfg. It sums each hand’s contact force to a single amplitude and maps it via amplitude = clamp(gain * (force - deadband), 0, saturation) (fields gain [1/N], deadband [N], saturation, plus frequency_hz / duration_s for the pulse).

from isaaclab.sensors import ContactSensorCfg
from isaaclab_teleop import ControllerHapticFeedbackCfg

@configclass
class MySceneCfg(InteractiveSceneCfg):
    left_hand_contact = ContactSensorCfg(
        prim_path="{ENV_REGEX_NS}/Robot/left_hand_.*_link", update_period=0.0, history_length=3
    )
    right_hand_contact = ContactSensorCfg(
        prim_path="{ENV_REGEX_NS}/Robot/right_hand_.*_link", update_period=0.0, history_length=3
    )

# in the env cfg __post_init__:
self.scene.robot.spawn.activate_contact_sensors = True
self.haptic_feedback = ControllerHapticFeedbackCfg(
    left_sensor_name="left_hand_contact",
    right_sensor_name="right_hand_contact",
)

Enabled on the two G1 loco-manipulation teleop environments (IsaacContrib-PickPlace-Locomanipulation-G1-Abs, IsaacContrib-PickPlace-FixedBaseUpperBodyIK-G1-Abs). Controller vibration is delivered over OpenXR (xrApplyHapticFeedback), so no CloudXR .env profile change is required.

Haptic glove (per-finger grip)#

Use GloveHapticFeedbackCfg. Each hand’s contact sensor is filtered against the grasped object (filter_prim_paths_expr), so force_matrix_w reports each finger’s force on that object and nothing else. The driver orders the per-finger magnitudes Thumb..Pinky (matched from the sensor’s body names via finger_order) into a FingerPowerVector rendered by a cross-process glove device on collection_id.

from isaaclab.sensors import ContactSensorCfg
from isaaclab_teleop import GloveHapticFeedbackCfg

@configclass
class MySceneCfg(InteractiveSceneCfg):
    object = RigidObjectCfg(prim_path="{ENV_REGEX_NS}/Object", ...)
    left_hand_contact = ContactSensorCfg(
        prim_path="{ENV_REGEX_NS}/Robot/.*L_(thumb_distal|index_intermediate|"
        "middle_intermediate|ring_intermediate|pinky_intermediate)_link",
        filter_prim_paths_expr=["{ENV_REGEX_NS}/Object"],
        update_period=0.0, history_length=3,
    )
    right_hand_contact = ContactSensorCfg(
        prim_path="{ENV_REGEX_NS}/Robot/.*R_(...)_link",
        filter_prim_paths_expr=["{ENV_REGEX_NS}/Object"],
        update_period=0.0, history_length=3,
    )

# in the env cfg __post_init__:
self.haptic_feedback = GloveHapticFeedbackCfg(
    left_sensor_name="left_hand_contact",
    right_sensor_name="right_hand_contact",
)

Enabled on the two GR1T2 pick-place teleop environments (IsaacContrib-PickPlace-GR1T2-Abs, IsaacContrib-PickPlace-GR1T2-WaistEnabled-Abs).

Note

The glove is a cross-process device: GloveHapticFeedbackCfg pushes per-finger powers on collection_id (default "manus_glove_haptic") to a vendor plugin. Run that plugin and set NV_CXR_ENABLE_PUSH_DEVICES=1 in a custom CloudXR profile (see CloudXR Environment Profiles). Contact-sensor prim_path regexes must match your robot’s finger body names – verify with print(robot.data.body_names) if unsure.

Record Demonstrations for Imitation Learning#

Isaac Teleop integrates with Isaac Lab’s record_demos.py script for recording teleoperated demonstrations, exposed as the isaaclab teleop record command.

When your environment configuration has an isaac_teleop attribute, the script automatically uses create_isaac_teleop_device() – no --teleop_device flag is needed:

uv (Recommended)

uv run --extra teleop isaaclab teleop record \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit \
    --xr

isaaclab.sh / isaaclab.bat

./isaaclab.sh -p scripts/tools/record_demos.py \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit \
    --xr

Some environments use the legacy teleop_devices configuration instead of isaac_teleop (e.g. the Galbot RmpFlow relative-mode tasks). For these, pass --teleop_device to select the input device:

uv (Recommended)

uv run --extra teleop isaaclab teleop record \
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow \
    --visualizer kit \
    --teleop_device keyboard

isaaclab.sh / isaaclab.bat

./isaaclab.sh -p scripts/tools/record_demos.py \
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow \
    --visualizer kit \
    --teleop_device keyboard

The workflow is:

Configure your environment with IsaacTeleopCfg (see Configure Your Environment) or teleop_devices for legacy devices (keyboard, spacemouse).
Run record_demos.py with the task name.
For XR tasks: start AR, connect your XR device, and teleoperate. For legacy tasks: use the configured input device directly.
Demonstrations are recorded to HDF5 files.
Use the recorded data with Isaac Lab Mimic or other imitation learning frameworks.

For the broader imitation learning pipeline (replay, augmentation, policy training), see Synthetic Data Generation and Imitation Learning with Isaac Lab Mimic.

Add a New Robot#

To add teleoperation support for a new robot in Isaac Lab:

Choose a control scheme. Refer to the Choose a Control Scheme table to determine which retargeters match your robot’s capabilities.
Build the pipeline. If existing retargeters are sufficient (e.g. Se3AbsRetargeter + GripperRetargeter for a new manipulator), write a pipeline builder function following the pattern in Build a Retargeting Pipeline. Configure the TensorReorderer output order to match your environment’s action space.
For dexterous hands: create a robot hand URDF and YAML config for DexHandRetargeter. Ensure fingertip links are positioned at the actual fingertips, not mid-finger.
For a custom retargeter: see Add a New Retargeter below.
Configure the XR anchor for your robot (static for manipulation, dynamic for locomotion). See Configure the XR Anchor.
Register in env config via IsaacTeleopCfg (see Configure Your Environment).

Add a New Retargeter#

If the built-in retargeters do not cover your use case, you can implement a custom one in the Isaac Teleop repository:

Inherit from BaseRetargeter and implement input_spec(), output_spec(), and compute().
Optionally add a ParameterState for parameters that should be live-tunable via the retargeter tuning UI.
Connect to existing source nodes (HandsSource, ControllersSource) or create a new IDeviceIOSource subclass for custom input devices.

See the Isaac Teleop repository and Contributing Guide for details.

Add a New Device#

There are two levels of device integration:

Isaac Teleop plugin (C++ level)

For new hardware that requires a custom driver or SDK. Plugins push data via OpenXR tensor collections. Existing plugins include Manus gloves, OAK-D camera, controller synthetic hands, and foot pedals. After creating the plugin, update the retargeting pipeline config to consume data from the new plugin’s source node.

See the Plugins directory for examples.

Pipeline configuration only

For devices already supported by Isaac Teleop (or whose data is available as hand / controller tracking). Simply update your pipeline_builder to use the appropriate source nodes and retargeters for the device’s data format.

Optimize XR Performance#

Switch the viewport to the RTX - Minimal renderer

The RTX - Minimal renderer trades image fidelity for substantially lower per-frame GPU cost. It is the recommended choice when the simulation cannot sustain the XR device’s display rate in real time – for example on lower-spec GPUs, in scenes with many lights or complex materials, or when you have already configured sim.dt and sim.render_interval and still see dropped frames.

Known issue

Starting an XR session while RTX - Minimal is already the active renderer is a known issue: robot control inputs may never be applied and teleoperation stays non-functional, with no error reported. Switching to RTX - Minimal after the XR session has started and teleoperation is confirmed working is not affected.

As a workaround, start under the default renderer before switching to RTX - Minimal:

Launch Isaac Lab with the default renderer and start the XR session.
Confirm that teleoperation is working (the robot responds to your hand motions).
Switch the viewport renderer to RTX - Minimal as described below.

To enable it, click the renderer dropdown at the top-left of the Isaac Lab viewport and select RTX - Minimal:

Viewport renderer dropdown with RTX - Minimal selected — Selecting the **RTX - Minimal** renderer from the viewport dropdown.#

For best results, open Render Settings from the top-right of the Isaac Lab UI, switch to the Minimal tab, and set Minimal Shading Mode to Diffuse/Glossy/Emission:

Render Settings panel showing the Minimal Shading Mode options — The **Render Settings** panel with the **Minimal Shading Mode** dropdown open (recommended: **Diffuse/Glossy/Emission**).#

Note

The RTX Minimal renderer currently only supports DistantLight prims for scene illumination – DomeLight prims are ignored. If your environment uses a DomeLight, swap (or supplement) it with a DistantLight so the scene is lit when running under RTX Minimal:

import isaaclab.sim as sim_utils
from isaaclab.assets import AssetBaseCfg

light = AssetBaseCfg(
    prim_path="/World/light",
    spawn=sim_utils.DistantLightCfg(color=(0.75, 0.75, 0.75), intensity=3000.0),
)

Depending on your environment, the default DistantLight orientation may cast shadows that overlap the robot and reduce visibility during teleoperation. If you encounter this, adjust the light’s orientation via init_state on AssetBaseCfg to position the light source at an angle that gives clear visibility:

import isaaclab.sim as sim_utils
from isaaclab.assets import AssetBaseCfg

light = AssetBaseCfg(
    prim_path="/World/light",
    spawn=sim_utils.DistantLightCfg(color=(0.75, 0.75, 0.75), intensity=3000.0),
    init_state=AssetBaseCfg.InitialStateCfg(
        rot=(0.0, 0.0, 0.0, 1.0),  # quaternion (x, y, z, w); adjust to reduce shadow overlap
    ),
)

Experiment with different orientations in your scene to find an angle that avoids shadow overlap on the robot. A slight tilt away from the camera viewpoint is a good starting point.

Known Issues#

XR_ERROR_VALIDATION_FAILURE: xrWaitFrame(frameState->type == 0) when stopping AR Mode

Can be safely ignored. Caused by a race condition in the exit handler.
XR_ERROR_INSTANCE_LOST in xrPollEvent

Occurs if the CloudXR runtime exits before Isaac Lab. Restart the runtime to resume.
[omni.usd] TF_PYTHON_EXCEPTION when starting/stopping AR Mode

Can be safely ignored. Caused by a race condition in the enter/exit handler.
Invalid version string in _ParseVersionString

Caused by shader assets authored with older USD versions. Typically safe to ignore.
XR device connects but no video is displayed (viewport responds to tracking)

The GPU index may differ between host and container. Set NV_GPU_INDEX to 0, 1, or 2 in the runtime to match the host GPU.

API Reference#

See the isaaclab_teleop for full class and function documentation:

Key	Action
`B`	Start / resume teleoperation.
`P`	Pause teleoperation (robot holds position).
`R`	Reset the environment.

Isaac Teleop

Contents

Isaac Teleop#

Supported Devices#

Choose a Control Scheme#

How It Works#

Visualize XR Tracking#

Teleop Control States (Start / Stop / Reset)#

How it works#

Polling control events in your script#

Disabling the control channel#

Using a custom channel UUID#

Run Without a Headset (Standalone I/O)#

Starting teleop without a headset#

Example: SO-101 leader-arm joint teleoperation#

Prerequisites#

Run the simulation#

Start the plugin#

Start teleoperation#

Retargeting Framework#

Source Nodes#

Available Retargeters#

Teleoperation Environment Reference#

Isaac Teleop (XR Headset) Environments#

Keyboard and SpaceMouse Environments#

Leader-Arm Environments#

Switch Between Controllers and Hand Tracking#

Build a Retargeting Pipeline#

Configure Your Environment#

CloudXR Environment Profiles#

Override at launch time#

Create a custom profile#

Disable auto-launch#

Configure the XR Anchor#

XR Camera Feedback#

Camera selection#

Placement#

Disable cameras and PiP#

Kit Scene UI presentation#

Haptic Feedback#

Controller vibration#

Haptic glove (per-finger grip)#

Record Demonstrations for Imitation Learning#

Add a New Robot#

Add a New Retargeter#

Add a New Device#

Optimize XR Performance#

Known Issues#

API Reference#