isaaclab.devices#
Sub-package providing interfaces to different teleoperation devices.
Currently, the following categories of devices are supported:
OpenXR: Uses hand tracking of index/thumb tip avg to drive the target pose. Gripping is done with pinching.
Haply: Haptic device (Inverse3 + VerseGrip) with position, orientation tracking and force feedback.
All device interfaces inherit from the DeviceBase class, which provides a
common interface for all devices. The device interface reads the input data when
the DeviceBase.advance() method is called. It also provides the function DeviceBase.add_callback()
to add user-defined callback functions to be called when a particular input is pressed from
the peripheral device.
Keyboard, gamepad and SpaceMouse input is implemented by isaaclab_teleop through Isaac Capture.
Their deprecated configs remain as data-only declarations translated before device creation.
Classes
An interface class for teleoperation devices. |
|
Base interface for input data retargeting. |
|
|
A Haply device controller for sending SE(3) commands with force feedback. |
|
An OpenXR-powered device for teleoperation and interaction. |
|
Manus gloves and Vive trackers for teleoperation and interaction. |
|
Retargeter specifically for gripper control based on hand tracking data. |
|
Retargets OpenXR hand tracking data to end-effector commands using absolute positioning. |
|
Retargets OpenXR hand tracking data to end-effector commands using relative positioning. |
|
Retargets OpenXR hand tracking data to GR1T2 hand end-effector commands. |
Modules
|
Device Base#
- class isaaclab.devices.DeviceBase[source]#
An interface class for teleoperation devices.
Derived classes have two implementation options:
Override _get_raw_data() and use the base advance() implementation: This approach is suitable for devices that want to leverage the built-in retargeting logic but only need to customize the raw data acquisition.
Override advance() completely: This approach gives full control over the command generation process, and _get_raw_data() can be ignored entirely.
Methods:
__init__([retargeters])Initialize the teleoperation interface.
reset()Reset the internals.
add_callback(key, func)Add additional functions to bind keyboard.
advance()Process current device state and return control commands.
Classes:
Standard tracking targets shared across devices.
Rows in the motion-controller 2x7 array.
Indices in the motion-controller input row.
- __init__(retargeters: list[RetargeterBase] | None = None)[source]#
Initialize the teleoperation interface.
- Parameters:
retargeters¶ – List of components that transform device data into robot commands. If None or empty list, the device will output its native data format.
- abstractmethod add_callback(key: Any, func: Callable)[source]#
Add additional functions to bind keyboard.
- advance() torch.Tensor[source]#
Process current device state and return control commands.
This method retrieves raw data from the device and optionally applies retargeting to convert it to robot commands.
Derived classes can either: 1. Override _get_raw_data() and use this base implementation, or 2. Override this method completely for custom command processing
- Returns:
When no retargeters are configured, returns raw device data in its native format. When retargeters are configured, returns a torch.Tensor containing the concatenated outputs from all retargeters.
Retargeter Base#
- class isaaclab.devices.RetargeterBase[source]#
Base interface for input data retargeting.
Deprecated since version Use: the Isaac Capture retargeting engine via
isaaclab_teleopinstead.This abstract class defines the interface for components that transform raw device data into robot control commands. Implementations can handle various types of transformations including: - Hand joint data to end-effector poses - Input device commands to robot movements - Sensor data to control signals
Methods:
__init__(cfg)Initialize the retargeter.
retarget(data)Retarget input data to desired output format.
Return the list of required data features for this retargeter.
Classes:
Features a retargeter may require from a device's raw data feed.
- __init__(cfg: RetargeterCfg)[source]#
Initialize the retargeter.
- Parameters:
cfg¶ – Configuration for the retargeter
- abstractmethod retarget(data: Any) Any[source]#
Retarget input data to desired output format.
- Parameters:
data¶ – Raw input data to be transformed
- Returns:
Retargeted data in implementation-specific format
- get_requirements() list[Requirement][source]#
Return the list of required data features for this retargeter.
Defaults to requesting all available features for backward compatibility. Implementations should override to narrow to only what they need.
Haply#
- class isaaclab.devices.HaplyDevice[source]
Bases:
DeviceBaseA Haply device controller for sending SE(3) commands with force feedback.
This class provides an interface to Haply robotic devices (Inverse3 + VerseGrip) for teleoperation. It communicates via WebSocket and supports:
Position tracking from Inverse3 device
Orientation and button inputs from VerseGrip device
Directional force feedback to Inverse3
Real-time data streaming at configurable rates
The device provides raw data:
Position: 3D position (x, y, z) in meters from Inverse3
Orientation: Quaternion (x, y, z, w) from VerseGrip
Buttons: Three buttons (a, b, c) from VerseGrip with state (pressed/not pressed)
Note: All button logic (e.g., gripper control, reset, mode switching) should be implemented in the application layer using the raw button states from advance().
Note
Requires the Haply SDK to be running and accessible via WebSocket. Install dependencies: pip install websockets
Methods:
__init__(cfg[, retargeters])Initialize the Haply device interface.
reset()Reset the device internal state.
add_callback(key, func)Add additional functions to bind to button events.
advance()Provides the result from Haply device state.
push_force(forces, position)Push force vector to Haply Inverse3 device.
Classes:
MotionControllerDataRowIndexRows in the motion-controller 2x7 array.
MotionControllerInputIndexIndices in the motion-controller input row.
TrackingTargetStandard tracking targets shared across devices.
- __init__(cfg: HaplyDeviceCfg, retargeters: list[RetargeterBase] | None = None)[source]
Initialize the Haply device interface.
- Parameters:
- Raises:
ImportError – If websockets module is not installed.
RuntimeError – If connection to Haply device fails.
- reset()[source]
Reset the device internal state.
- advance() torch.Tensor[source]
Provides the result from Haply device state.
- Returns:
- A tensor containing the raw device data:
- 10 elements: [x, y, z, qx, qy, qz, qw, button_a, button_b, button_c]
where (x, y, z) is position, (qx, qy, qz, qw) is quaternion orientation, and buttons are 1.0 (pressed) or 0.0 (not pressed)
- Return type:
- push_force(forces: torch.Tensor, position: torch.Tensor) None[source]
Push force vector to Haply Inverse3 device.
Overrides DeviceBase.push_force() to provide force feedback for Haply Inverse3. Forces are clipped to [-limit_force, limit_force] range for safety.
OpenXR#
- class isaaclab.devices.OpenXRDevice[source]
Bases:
DeviceBaseAn OpenXR-powered device for teleoperation and interaction.
This device tracks hand joints using OpenXR and makes them available as:
A dictionary of tracking data (when used without retargeters)
Retargeted commands for robot control (when retargeters are provided)
Raw data format (_get_raw_data output):
A dictionary with keys matching TrackingTarget enum values (HAND_LEFT, HAND_RIGHT, HEAD)
Each hand tracking entry contains a dictionary of joint poses
Each joint pose is a 7D vector (x, y, z, qw, qx, qy, qz) in meters and quaternion units
Joint names are defined in HAND_JOINT_NAMES from isaaclab_teleop.deprecated.openxr.common
Supported joints include palm, wrist, and joints for thumb, index, middle, ring and little fingers
Teleop commands: The device responds to several teleop commands that can be subscribed to via add_callback():
“START”: Resume hand tracking data flow
“STOP”: Pause hand tracking data flow
“RESET”: Reset the tracking and signal simulation reset
The device tracks the left hand, right hand, head position, or any combination of these based on the TrackingTarget enum values. When retargeters are provided, the raw tracking data is transformed into robot control commands suitable for teleoperation.
Methods:
__init__(cfg[, retargeters])Initialize the OpenXR device.
reset()Reset the internals.
add_callback(key, func)Add additional functions to bind to client messages.
advance()Process current device state and return control commands.
Classes:
MotionControllerDataRowIndexRows in the motion-controller 2x7 array.
MotionControllerInputIndexIndices in the motion-controller input row.
TrackingTargetStandard tracking targets shared across devices.
- __init__(cfg: OpenXRDeviceCfg, retargeters: list[RetargeterBase] | None = None)[source]
Initialize the OpenXR device.
- reset()[source]
Reset the internals.
- advance() torch.Tensor[source]
Process current device state and return control commands.
This method retrieves raw data from the device and optionally applies retargeting to convert it to robot commands.
Derived classes can either: 1. Override _get_raw_data() and use this base implementation, or 2. Override this method completely for custom command processing
- Returns:
When no retargeters are configured, returns raw device data in its native format. When retargeters are configured, returns a torch.Tensor containing the concatenated outputs from all retargeters.
Manus + Vive#
- class isaaclab.devices.ManusVive[source]
Bases:
DeviceBaseManus gloves and Vive trackers for teleoperation and interaction.
This device tracks hand joints using Manus gloves and Vive trackers and makes them available as:
A dictionary of tracking data (when used without retargeters)
Retargeted commands for robot control (when retargeters are provided)
The user needs to install the Manus SDK and add {path_to_manus_sdk}/manus_sdk/lib to LD_LIBRARY_PATH. Data are acquired by ManusViveIntegration from isaaclab_teleop.deprecated.openxr.manus_vive_utils, including
Vive tracker poses in scene frame, calibrated from AVP wrist poses.
Hand joints calculated from Vive wrist joints and Manus hand joints (relative to wrist).
Vive trackers are automatically mapped to the left and right wrist joints.
Raw data format (_get_raw_data output): consistent with
OpenXRDevice. Joint names are defined in HAND_JOINT_MAP from isaaclab_teleop.deprecated.openxr.manus_vive_utils.Teleop commands: consistent with
OpenXRDevice.The device tracks the left hand, right hand, head position, or any combination of these based on the TrackingTarget enum values. When retargeters are provided, the raw tracking data is transformed into robot control commands suitable for teleoperation.
Methods:
__init__(cfg[, retargeters])Initialize the Manus+Vive device.
reset()Reset cached joint and head poses.
add_callback(key, func)Register a callback for a given key.
advance()Process current device state and return control commands.
Classes:
MotionControllerDataRowIndexRows in the motion-controller 2x7 array.
MotionControllerInputIndexIndices in the motion-controller input row.
TrackingTargetStandard tracking targets shared across devices.
- __init__(cfg: ManusViveCfg, retargeters: list[RetargeterBase] | None = None)[source]
Initialize the Manus+Vive device.
- reset()[source]
Reset cached joint and head poses.
- advance() torch.Tensor[source]
Process current device state and return control commands.
This method retrieves raw data from the device and optionally applies retargeting to convert it to robot commands.
Derived classes can either: 1. Override _get_raw_data() and use this base implementation, or 2. Override this method completely for custom command processing
- Returns:
When no retargeters are configured, returns raw device data in its native format. When retargeters are configured, returns a torch.Tensor containing the concatenated outputs from all retargeters.
Retargeters#
- class isaaclab.devices.openxr.retargeters.GripperRetargeter[source]
Bases:
RetargeterBaseRetargeter specifically for gripper control based on hand tracking data.
This retargeter analyzes the distance between thumb and index finger tips to determine whether the gripper should be open or closed. It includes hysteresis to prevent rapid toggling between states when the finger distance is near the threshold.
Features: - Tracks thumb and index finger distance - Implements hysteresis for stable gripper control - Outputs boolean command (True = close gripper, False = open gripper)
Methods:
__init__(cfg)Initialize the retargeter.
retarget(data)Convert hand joint poses to gripper command.
get_requirements()Return the list of required data features for this retargeter.
Classes:
RequirementFeatures a retargeter may require from a device's raw data feed.
- __init__(cfg: GripperRetargeterCfg)[source]
Initialize the retargeter.
- Parameters:
cfg¶ – Configuration for the retargeter
- retarget(data: dict) torch.Tensor[source]
Convert hand joint poses to gripper command.
- Parameters:
data¶ – Dictionary mapping tracking targets to joint data dictionaries. The joint names are defined in isaaclab.devices.openxr.common.HAND_JOINT_NAMES
- Returns:
Tensor containing a single bool value where True = close gripper, False = open gripper
- Return type:
- get_requirements() list[Requirement][source]
Return the list of required data features for this retargeter.
Defaults to requesting all available features for backward compatibility. Implementations should override to narrow to only what they need.
- class isaaclab.devices.openxr.retargeters.Se3AbsRetargeter[source]
Bases:
RetargeterBaseRetargets OpenXR hand tracking data to end-effector commands using absolute positioning.
This retargeter maps hand joint poses directly to robot end-effector positions and orientations, rather than using relative movements. It can either: - Use the wrist position and orientation - Use the midpoint between thumb and index finger (pinch position)
Features: - Optional constraint to zero out X/Y rotations (keeping only Z-axis rotation) - Optional visualization of the target end-effector pose
Methods:
__init__(cfg)Initialize the retargeter.
retarget(data)Convert hand joint poses to robot end-effector command.
get_requirements()Return the list of required data features for this retargeter.
Classes:
RequirementFeatures a retargeter may require from a device's raw data feed.
- __init__(cfg: Se3AbsRetargeterCfg)[source]
Initialize the retargeter.
- Parameters:
bound_hand¶ – The hand to track (DeviceBase.TrackingTarget.HAND_LEFT or DeviceBase.TrackingTarget.HAND_RIGHT)
zero_out_xy_rotation¶ – If True, zero out rotation around x and y axes
use_wrist_rotation¶ – If True, use wrist rotation instead of finger average
use_wrist_position¶ – If True, use wrist position instead of pinch position
enable_visualization¶ – If True, visualize the target pose in the scene
device¶ – The device to place the returned tensor on (‘cpu’ or ‘cuda’)
- retarget(data: dict) torch.Tensor[source]
Convert hand joint poses to robot end-effector command.
- Parameters:
data¶ – Dictionary mapping tracking targets to joint data dictionaries. The joint names are defined in isaaclab.devices.openxr.common.HAND_JOINT_NAMES
- Returns:
- 7D tensor containing position (xyz) and orientation (quaternion)
for the robot end-effector
- Return type:
- get_requirements() list[Requirement][source]
Return the list of required data features for this retargeter.
Defaults to requesting all available features for backward compatibility. Implementations should override to narrow to only what they need.
- class isaaclab.devices.openxr.retargeters.Se3RelRetargeter[source]
Bases:
RetargeterBaseRetargets OpenXR hand tracking data to end-effector commands using relative positioning.
This retargeter calculates delta poses between consecutive hand joint poses to generate incremental robot movements. It can either: - Use the wrist position and orientation - Use the midpoint between thumb and index finger (pinch position)
Features: - Optional constraint to zero out X/Y rotations (keeping only Z-axis rotation) - Motion smoothing with adjustable parameters - Optional visualization of the target end-effector pose
Methods:
__init__(cfg)Initialize the relative motion retargeter.
retarget(data)Convert hand joint poses to robot end-effector command.
get_requirements()Return the list of required data features for this retargeter.
Classes:
RequirementFeatures a retargeter may require from a device's raw data feed.
- __init__(cfg: Se3RelRetargeterCfg)[source]
Initialize the relative motion retargeter.
- Parameters:
bound_hand¶ – The hand to track (DeviceBase.TrackingTarget.HAND_LEFT or DeviceBase.TrackingTarget.HAND_RIGHT)
zero_out_xy_rotation¶ – If True, ignore rotations around x and y axes, allowing only z-axis rotation
use_wrist_rotation¶ – If True, use wrist rotation for control instead of averaging finger orientations
use_wrist_position¶ – If True, use wrist position instead of pinch position (midpoint between fingers)
delta_pos_scale_factor¶ – Amplification factor for position changes (higher = larger robot movements)
delta_rot_scale_factor¶ – Amplification factor for rotation changes (higher = larger robot rotations)
alpha_pos¶ – Position smoothing parameter (0-1); higher values track more closely to input, lower values smooth more
alpha_rot¶ – Rotation smoothing parameter (0-1); higher values track more closely to input, lower values smooth more
enable_visualization¶ – If True, show a visual marker representing the target end-effector pose
device¶ – The device to place the returned tensor on (‘cpu’ or ‘cuda’)
- retarget(data: dict) torch.Tensor[source]
Convert hand joint poses to robot end-effector command.
- Parameters:
data¶ – Dictionary mapping tracking targets to joint data dictionaries. The joint names are defined in isaaclab.devices.openxr.common.HAND_JOINT_NAMES
- Returns:
- 6D tensor containing position (xyz) and rotation vector (rx,ry,rz)
for the robot end-effector
- Return type:
- get_requirements() list[Requirement][source]
Return the list of required data features for this retargeter.
Defaults to requesting all available features for backward compatibility. Implementations should override to narrow to only what they need.
- class isaaclab.devices.openxr.retargeters.GR1T2Retargeter[source]
Bases:
RetargeterBaseRetargets OpenXR hand tracking data to GR1T2 hand end-effector commands.
This retargeter maps hand tracking data from OpenXR to joint commands for the GR1T2 robot’s hands. It handles both left and right hands, converting poses of the hands in OpenXR format joint angles for the GR1T2 robot’s hands.
Methods:
__init__(cfg)Initialize the GR1T2 hand retargeter.
retarget(data)Convert hand joint poses to robot end-effector commands.
get_requirements()Return the list of required data features for this retargeter.
Classes:
RequirementFeatures a retargeter may require from a device's raw data feed.
- __init__(cfg: GR1T2RetargeterCfg)[source]
Initialize the GR1T2 hand retargeter.
- retarget(data: dict) torch.Tensor[source]
Convert hand joint poses to robot end-effector commands.
- Parameters:
data¶ – Dictionary mapping tracking targets to joint data dictionaries.
- Returns:
Left wrist pose Right wrist pose in USD frame Retargeted hand joint angles
- Return type:
tuple containing
- get_requirements() list[Requirement][source]
Return the list of required data features for this retargeter.
Defaults to requesting all available features for backward compatibility. Implementations should override to narrow to only what they need.
Additional Public Classes#
The following classes are part of the public isaaclab.devices API.
Configuration for teleoperation devices. |
|
Configuration for all supported teleoperation devices. |
|
Configuration for Haply device. |
|
Base configuration for hand tracking retargeters. |
|
Configuration translated to Isaac Capture before device creation. |
|
Configuration translated to Isaac Capture before device creation. |
|
Configuration translated to Isaac Capture before device creation. |
|
Configuration translated to Isaac Capture before device creation. |
|
Configuration translated to Isaac Capture before device creation. |
|
Configuration translated to Isaac Capture before device creation. |
- class isaaclab.devices.DeviceCfg[source]#
Bases:
objectConfiguration for teleoperation devices.
Methods:
- classmethod __new__(*args, **kwargs)#
- class isaaclab.devices.DevicesCfg[source]#
Bases:
objectConfiguration for all supported teleoperation devices.
Methods:
- classmethod __new__(*args, **kwargs)#
- class isaaclab.devices.HaplyDeviceCfg[source]#
Bases:
DeviceCfgConfiguration for Haply device.
Methods:
- classmethod __new__(*args, **kwargs)#
- __init__(teleoperation_active_default: bool = <factory>, sim_device: str = <factory>, retargeters: list[RetargeterCfg] = <factory>, class_type: type[DeviceBase] = <class 'isaaclab.devices.haply.se3_haply.HaplyDevice'>, websocket_uri: str = <factory>, pos_sensitivity: float = <factory>, data_rate: float = <factory>, limit_force: float = <factory>) None#
- class isaaclab.devices.RetargeterCfg[source]#
Bases:
objectBase configuration for hand tracking retargeters.
Deprecated since version Use: the Isaac Capture retargeting engine via
isaaclab_teleopinstead.Methods:
- classmethod __new__(*args, **kwargs)#
- class isaaclab.devices.Se2GamepadCfg[source]#
Bases:
DeviceCfgConfiguration translated to Isaac Capture before device creation.
Deprecated since version 3.2: Use
isaaclab_teleop.se2_gamepad_teleop_cfg()for new configurations.Methods:
- classmethod __new__(*args, **kwargs)#
- __init__(teleoperation_active_default: bool = <factory>, sim_device: str = <factory>, retargeters: list[RetargeterCfg] = <factory>, class_type: type[DeviceBase] | None = <factory>, v_x_sensitivity: float = <factory>, v_y_sensitivity: float = <factory>, omega_z_sensitivity: float = <factory>, dead_zone: float = <factory>) None#
- class isaaclab.devices.Se2KeyboardCfg[source]#
Bases:
DeviceCfgConfiguration translated to Isaac Capture before device creation.
Deprecated since version 3.2: Use
isaaclab_teleop.se2_keyboard_teleop_cfg()for new configurations.Methods:
- classmethod __new__(*args, **kwargs)#
- class isaaclab.devices.Se2SpaceMouseCfg[source]#
Bases:
DeviceCfgConfiguration translated to Isaac Capture before device creation.
Deprecated since version 3.2: Use
isaaclab_teleop.se2_spacemouse_teleop_cfg()for new configurations.Methods:
- classmethod __new__(*args, **kwargs)#
- class isaaclab.devices.Se3GamepadCfg[source]#
Bases:
DeviceCfgConfiguration translated to Isaac Capture before device creation.
Deprecated since version 3.2: Use
isaaclab_teleop.se3_gamepad_teleop_cfg()for new configurations.Methods:
- classmethod __new__(*args, **kwargs)#
- __init__(teleoperation_active_default: bool = <factory>, sim_device: str = <factory>, retargeters: list[RetargeterCfg] = <factory>, class_type: type[DeviceBase] | None = <factory>, gripper_term: bool = <factory>, dead_zone: float = <factory>, pos_sensitivity: float = <factory>, rot_sensitivity: float = <factory>) None#
- class isaaclab.devices.Se3KeyboardCfg[source]#
Bases:
DeviceCfgConfiguration translated to Isaac Capture before device creation.
Deprecated since version 3.2: Use
isaaclab_teleop.se3_keyboard_teleop_cfg()for new configurations.Methods:
- classmethod __new__(*args, **kwargs)#
- class isaaclab.devices.Se3SpaceMouseCfg[source]#
Bases:
DeviceCfgConfiguration translated to Isaac Capture before device creation.
Deprecated since version 3.2: Use
isaaclab_teleop.se3_spacemouse_teleop_cfg()for new configurations.Methods:
- classmethod __new__(*args, **kwargs)#