isaaclab_teleop

isaaclab_teleop#

Package providing IsaacTeleop-based teleoperation for Isaac Lab.

Classes

`IsaacTeleopCfg`	Configuration for IsaacTeleop-based teleoperation.
`IsaacTeleopDevice`	A IsaacTeleop-based teleoperation device for Isaac Lab.
`XrCfg`	Configuration for viewing and interacting with the environment through an XR device.
`XrAnchorRotationMode`	Enumeration for XR anchor rotation modes.
`XrAnchorSynchronizer`	Keeps the XR anchor prim aligned with a reference prim according to XR config.

Functions

`create_isaac_teleop_device`(cfg[, ...])	Create an `IsaacTeleopDevice` with required Omniverse extension setup.
`remove_camera_configs`(env_cfg)	Removes cameras from environments when using XR devices.

Configuration#

class isaaclab_teleop.IsaacTeleopCfg[source]#

Configuration for IsaacTeleop-based teleoperation.

This configuration class defines the parameters needed to create a IsaacTeleop teleoperation session integrated with Isaac Lab environments.

The pipeline_builder is a callable that constructs the IsaacTeleop retargeting pipeline. It should return an OutputCombiner with a single “action” output that contains the flattened action tensor (typically via TensorReorderer).

If the pipeline builder also produces retargeters that should be exposed in the tuning UI, the env cfg should call the builder, unpack the results, and populate both pipeline_builder and retargeters_to_tune explicitly. Both fields must be callables (lambdas / functions) so they survive the deepcopy performed by @configclass on mutable attributes.

Example

def build_pipeline():
    controllers = ControllersSource(name="controllers")
    se3 = Se3AbsRetargeter(cfg, name="ee_pose")
    # ... connect and flatten with TensorReorderer ...
    pipeline = OutputCombiner({"action": reorderer.output("output")})
    return pipeline, [se3]  # return retargeters separately


pipeline, retargeters = build_pipeline()
teleop_cfg = IsaacTeleopCfg(
    xr_cfg=XrCfg(anchor_pos=(0.5, 0.0, 0.5)),
    pipeline_builder=lambda: pipeline,
    retargeters_to_tune=lambda: retargeters,
)

Attributes:

`xr_cfg`	XR anchor configuration for positioning the user in the simulation.
`pipeline_builder`	Callable that builds the IsaacTeleop retargeting pipeline.
`plugins`	List of IsaacTeleop plugin configurations.
`sim_device`	Torch device string for placing output action tensors.
`teleoperation_active_default`	Whether teleoperation should be active by default when the session starts.
`retargeters_to_tune`	Optional callable returning retargeters to expose in the tuning UI.
`app_name`	Application name for the IsaacTeleop session.

Methods:

__init__([xr_cfg, pipeline_builder, ...])

xr_cfg: XrCfg#

XR anchor configuration for positioning the user in the simulation.

This includes anchor position, rotation, and optional dynamic anchoring to follow a prim (e.g., robot base) during locomotion tasks.

pipeline_builder: Callable[[], OutputCombiner]#

Callable that builds the IsaacTeleop retargeting pipeline.

The function should return an OutputCombiner with an “action” output containing the flattened action tensor matching the Isaac Lab action space. Use TensorReorderer to flatten multiple retargeter outputs into a single array.

To expose retargeters for the tuning UI, populate retargeters_to_tune directly when constructing this config rather than encoding them into the builder’s return value.

plugins: list[PluginConfig]#

List of IsaacTeleop plugin configurations.

Plugins can provide additional functionality like synthetic hand tracking from controller inputs.

__init__(xr_cfg: XrCfg = <factory>, pipeline_builder: Callable[[], OutputCombiner] = <factory>, plugins: list[PluginConfig] = <factory>, sim_device: str = <factory>, teleoperation_active_default: bool = <factory>, retargeters_to_tune: Callable[[], list[BaseRetargeter]] | None = <factory>, app_name: str = <factory>) → None#

sim_device: str#: Torch device string for placing output action tensors.

teleoperation_active_default: bool#

Whether teleoperation should be active by default when the session starts.

When False (the default), the teleop session remains inactive until a "START" command is received from xr_core via the message bus.

retargeters_to_tune: Callable[[], list[BaseRetargeter]] | None#

Optional callable returning retargeters to expose in the tuning UI.

Must be a callable (e.g. lambda: [retargeter1, retargeter2]) rather than a plain list because @configclass deep-copies mutable attributes and retargeter objects often contain non-picklable C++/SWIG handles. Wrapping in a callable makes the value opaque to deepcopy.

When set and the tuning UI is enabled, the returned retargeters will be displayed in the MultiRetargeterTuningUIImGui window, allowing real-time adjustment of their tunable parameters. Only retargeters that have a ParameterState (i.e. tunable parameters) will appear.

If None, the tuning UI will not be opened.

app_name: str#: Application name for the IsaacTeleop session.

class isaaclab_teleop.XrCfg[source]#

Configuration for viewing and interacting with the environment through an XR device.

Attributes:

`anchor_pos`	Specifies the position (in m) of the simulation when viewed in an XR device.
`anchor_rot`	Specifies the rotation (as a quaternion xyzw) of the simulation when viewed in an XR device.
`anchor_prim_path`	Specifies the prim path to attach the XR anchor to for dynamic positioning.
`anchor_rotation_mode`	Specifies how the XR anchor rotation should behave when attached to a prim.
`anchor_rotation_smoothing_time`	Wall-clock time constant (seconds) for rotation smoothing in FOLLOW_PRIM_SMOOTHED mode.
`anchor_rotation_custom_func`	Specifies the function to calculate the rotation of the XR anchor when anchor_rotation_mode is CUSTOM.
`near_plane`	Specifies the near plane distance for the XR device.
`fixed_anchor_height`	Specifies if the anchor height should be fixed.

Methods:

__init__([anchor_pos, anchor_rot, ...])

anchor_pos: tuple[float, float, float]#

Specifies the position (in m) of the simulation when viewed in an XR device.

Specifically: this position will appear at the origin of the XR device’s local coordinate frame.

anchor_rot: tuple[float, float, float, float]#

Specifies the rotation (as a quaternion xyzw) of the simulation when viewed in an XR device.

Specifically: this rotation will determine how the simulation is rotated with respect to the origin of the XR device’s local coordinate frame.

This quantity is only effective if xr_anchor_pos is set.

anchor_prim_path: str | None#

Specifies the prim path to attach the XR anchor to for dynamic positioning.

When set, the XR anchor will be attached to the specified prim (e.g., robot root prim), allowing the XR camera to move with the prim. This is particularly useful for locomotion robot teleoperation where the robot moves and the XR camera should follow it.

If None, the anchor will use the static anchor_pos and anchor_rot values.

__init__(anchor_pos: tuple[float, float, float] = <factory>, anchor_rot: tuple[float, float, float, float] = <factory>, anchor_prim_path: str | None = <factory>, anchor_rotation_mode: ~isaaclab_teleop.xr_cfg.XrAnchorRotationMode = <factory>, anchor_rotation_smoothing_time: float = <factory>, anchor_rotation_custom_func: ~collections.abc.Callable[[numpy.ndarray, numpy.ndarray], numpy.ndarray] = <factory>, near_plane: float = <factory>, fixed_anchor_height: bool = <factory>) → None#

anchor_rotation_mode: XrAnchorRotationMode#

Specifies how the XR anchor rotation should behave when attached to a prim.

The available modes are: - XrAnchorRotationMode.FIXED: Sets rotation once to anchor_rot value - XrAnchorRotationMode.FOLLOW_PRIM: Rotation follows prim’s rotation - XrAnchorRotationMode.FOLLOW_PRIM_SMOOTHED: Rotation smoothly follows prim’s rotation using slerp - XrAnchorRotationMode.CUSTOM: user provided function to calculate the rotation

anchor_rotation_smoothing_time: float#

Wall-clock time constant (seconds) for rotation smoothing in FOLLOW_PRIM_SMOOTHED mode.

This time constant is applied using wall-clock delta time between frames (not physics dt). Smaller values (e.g., 0.1) result in faster/snappier response but less smoothing. Larger values (e.g., 0.75–2.0) result in slower/smoother response but more lag. Typical useful range: 0.3 – 1.5 seconds depending on runtime frame-rate and comfort.

anchor_rotation_custom_func: Callable[[numpy.ndarray, numpy.ndarray], numpy.ndarray]#

Specifies the function to calculate the rotation of the XR anchor when anchor_rotation_mode is CUSTOM.

Parameters:

headpose – Previous head pose as numpy array [x, y, z, w, x, y, z] (position + quaternion)
pose – Anchor prim pose as numpy array [x, y, z, w, x, y, z] (position + quaternion)

Returns:

Quaternion as numpy array [w, x, y, z]

Return type:

np.ndarray

near_plane: float#

Specifies the near plane distance for the XR device.

This value determines the closest distance at which objects will be rendered in the XR device.

fixed_anchor_height: bool#

Specifies if the anchor height should be fixed.

If True, the anchor height will be fixed to the initial height of the anchor prim.

class isaaclab_teleop.XrAnchorRotationMode[source]#

Enumeration for XR anchor rotation modes.

Attributes:

`FIXED`	sets rotation once and doesn't change it.
`FOLLOW_PRIM`	rotation follows prim's rotation.
`FOLLOW_PRIM_SMOOTHED`	rotation smoothly follows prim's rotation using slerp.
`CUSTOM`	user provided function to calculate the rotation.

FIXED = 'fixed'#

sets rotation once and doesn’t change it.

Type:: Fixed rotation mode

FOLLOW_PRIM = 'follow_prim'#

rotation follows prim’s rotation.

Type:: Follow prim rotation mode

FOLLOW_PRIM_SMOOTHED = 'follow_prim_smoothed'#

rotation smoothly follows prim’s rotation using slerp.

Type:: Follow prim rotation mode with smooth interpolation

CUSTOM = 'custom_rotation'#

user provided function to calculate the rotation.

Type:: Custom rotation mode

Device#

class isaaclab_teleop.IsaacTeleopDevice[source]#

Bases: object

A IsaacTeleop-based teleoperation device for Isaac Lab.

This device provides an interface between IsaacTeleop’s retargeting pipeline and Isaac Lab environments. It composes three focused collaborators:

XrAnchorManager – XR anchor prim setup, synchronization, and coordinate-frame transform computation.
TeleopSessionLifecycle – pipeline building, OpenXR handle acquisition, session creation/destruction, and action-tensor extraction.
CommandHandler – callback registration and XR message-bus command dispatch.

Together they manage:

XR anchor configuration and synchronization
IsaacTeleop session lifecycle
Action tensor generation from the retargeting pipeline

The device uses IsaacTeleop’s TensorReorderer to flatten pipeline outputs into a single action tensor matching the environment’s action space.

Teleop commands:: The device supports callbacks for START, STOP, and RESET commands that can be triggered via XR controller buttons or the message bus.

Example

cfg = IsaacTeleopCfg(
    pipeline_builder=my_pipeline_builder,
    sim_device="cuda:0",
)

with IsaacTeleopDevice(cfg) as device:
    while running:
        action = device.advance()
        env.step(action.repeat(num_envs, 1))

Methods:

`__init__`(cfg)	Initialize the IsaacTeleop device.
`reset`()	Reset the device state.
`add_callback`(key, func)	Add a callback function for teleop commands.
`advance`()	Process current device state and return control commands.

__init__(cfg: IsaacTeleopCfg)[source]#

Initialize the IsaacTeleop device.

Parameters:: cfg – Configuration object for IsaacTeleop settings.

reset() → None[source]#

Reset the device state.

Resets the XR anchor synchronizer if present.

add_callback(key: str, func: Callable) → None[source]#

Add a callback function for teleop commands.

Parameters:

key – The command type to bind to. Valid values are “START”, “STOP”, “RESET”, and “R”.
func – The function to call when the command is received. Should take no arguments.

advance() → torch.Tensor | None[source]#

Process current device state and return control commands.

If the IsaacTeleop session has not been started yet (because the OpenXR handles were not available at __enter__ time), this method will attempt to start it on each call. Once the user clicks “Start AR” and the handles become available, the session is created transparently.

Returns:: A flattened action torch.Tensor ready for the Isaac Lab environment, or None if the session has not started yet (e.g. still waiting for the user to start AR).
Raises:: RuntimeError – If called outside of a context manager.

isaaclab_teleop.create_isaac_teleop_device(cfg: IsaacTeleopCfg, sim_device: str | None = None, callbacks: dict[str, Callable] | None = None) → IsaacTeleopDevice[source]#

Create an IsaacTeleopDevice with required Omniverse extension setup.

This helper centralises the boilerplate that every script must execute before constructing an IsaacTeleopDevice:

Disable default OpenXR input bindings (prevents conflicts).
Enable the isaacsim.kit.xr.teleop.bridge extension.
Optionally override IsaacTeleopCfg.sim_device so action tensors land on the same device the caller uses for the simulation.

Note

When sim_device is provided, cfg.sim_device is mutated in place before the device is constructed.

Parameters:

cfg – IsaacTeleop configuration.
sim_device – If provided, overrides cfg.sim_device so action tensors are placed on the requested torch device (e.g. "cuda:0").
callbacks – Optional mapping of command keys (e.g. "START", "STOP", "RESET") to callables registered on the device.

Returns:

A fully configured IsaacTeleopDevice ready for use in a with block.

XR Anchor#

class isaaclab_teleop.XrAnchorSynchronizer[source]#

Keeps the XR anchor prim aligned with a reference prim according to XR config.

Methods:

`__init__`(xr_core, xr_cfg, xr_anchor_headset_path)
`get_world_transform`()	Return the anchor world transform.
`sync_headset_to_anchor`()	Sync XR anchor pose in USD for both dynamic and static anchoring.

__init__(xr_core: Any, xr_cfg: Any, xr_anchor_headset_path: str)[source]#

get_world_transform() → tuple[numpy.ndarray, numpy.ndarray] | None[source]#

Return the anchor world transform.

Returns the cached world transform that was computed by the most recent call to sync_headset_to_anchor(). Using the cached value avoids a Fabric/USD layer mismatch: when the XR anchor prim is a child of a physics-driven prim (e.g. the robot pelvis), reading GetFabricHierarchyWorldMatrixAttr would compose the Fabric-side parent transform (updated by physics) with a local xform that was decomposed against the USD-side parent (which can lag behind), producing an incorrect world matrix that drifts as the robot moves.

Returns:: A (position, quat_xyzw) tuple of numpy float64 arrays, or None if sync_headset_to_anchor() has not run yet.

sync_headset_to_anchor()[source]#

Sync XR anchor pose in USD for both dynamic and static anchoring.

For dynamic anchoring (anchor_prim_path is set), the reference prim’s world position is read from Fabric and anchor_pos is added as an offset. For static anchoring (no prim path), anchor_pos is used directly as the world position.

In both cases the function calls set_world_transform_matrix on the XR core so that the rendering anchor and the pipeline’s world_T_anchor matrix are guaranteed to agree, and caches the world transform for get_world_transform().

isaaclab_teleop.remove_camera_configs(env_cfg: Any) → Any[source]#

Removes cameras from environments when using XR devices.

Having additional cameras cause operation performance issues. This function scans the environment configuration for camera objects and removes them, along with any associated observation terms that reference these cameras.

Parameters:: env_cfg – The environment configuration to modify.
Returns:: The modified environment configuration with cameras removed.

isaaclab_teleop

Contents

isaaclab_teleop#

Configuration#

Device#

XR Anchor#