isaaclab.sensors

isaaclab.sensors#

Sub-package containing various sensor classes implementations.

This subpackage contains the sensor classes that are compatible with Isaac Sim. We include both USD-based and custom sensors:

USD-prim sensors: Available in Omniverse and require creating a USD prim for them. For instance, RTX ray tracing camera and lidar sensors.
USD-schema sensors: Available in Omniverse and require creating a USD schema on an existing prim. For instance, contact sensors and frame transformers.
Custom sensors: Implemented in Python and do not require creating any USD prim or schema. For instance, warp-based ray-casters.

Due to the above categorization, the prim paths passed to the sensor’s configuration class are interpreted differently based on the sensor type. The following table summarizes the interpretation of the prim paths for different sensor types:

Sensor Type	Example Prim Path	Pre-check
Camera	/World/robot/base/camera	Leaf is available, and it will spawn a USD camera
Contact Sensor	/World/robot/feet_*	Leaf is available and checks if the schema exists
Ray Caster	/World/robot/base/raycast	`spawn` creates an Xform leaf; else the leaf must exist
Frame Transformer	/World/robot/base	Leaf exists and is a physics body (Articulation / Rigid Body)
Imu	/World/robot/base	Leaf exists and is a physics body (Rigid Body)
Pva	/World/robot/base	Leaf exists and is a physics body (Rigid Body)
Joint Wrench Sensor	/World/robot	Leaf exists and is an articulation

Submodules

patterns

Sub-module for ray-casting patterns used by the ray-caster.

Classes

`SensorBase`	The base class for implementing a sensor.
`SensorBaseCfg`	Configuration parameters for a sensor.
`Camera`	The camera sensor for acquiring visual data.
`CameraData`	Data container for the camera sensor.
`CameraCfg`	Configuration for a camera sensor.
`ContactSensor`	Factory for creating contact sensor instances.
`ContactSensorData`	Factory for creating contact sensor data instances.
`ContactSensorCfg`	Configuration for the contact sensor.
`FrameTransformer`	Factory for creating frame transformer instances.
`FrameTransformerData`	Factory for creating frame transformer data instances.
`FrameTransformerCfg`	Configuration for the frame transformer sensor.
`RayCaster`	Backend-dispatching ray-caster sensor.
`RayCasterData`	Data container for the ray-cast sensor.
`RayCasterCfg`	Configuration for the ray-cast sensor.
`RayCasterCamera`	Backend-dispatching ray-caster camera sensor.
`RayCasterCameraCfg`	Configuration for the ray-cast sensor.
`MultiMeshRayCaster`	Backend-dispatching multi-mesh ray-caster sensor.
`MultiMeshRayCasterData`	Data container for the multi-mesh ray-cast sensor.
`MultiMeshRayCasterCfg`	Configuration for the multi-mesh ray-cast sensor.
`MultiMeshRayCasterCamera`	Backend-dispatching multi-mesh ray-caster camera sensor.
`MultiMeshRayCasterCameraCfg`	Configuration for the multi-mesh ray-cast camera sensor.
`Imu`	Factory for creating IMU sensor instances.
`ImuCfg`	Configuration for an Inertial Measurement Unit (IMU) sensor.
`Pva`	Factory for creating PVA sensor instances.
`PvaData`	Factory for creating PVA data instances.
`PvaCfg`	Configuration for a Pose Velocity Acceleration (PVA) sensor.
`JointWrenchSensor`	Factory for creating joint-wrench sensor instances.
`JointWrenchSensorData`	Factory for creating joint-wrench sensor data instances.
`JointWrenchSensorCfg`	Configuration for a joint reaction wrench sensor.

Sensor Base#

class isaaclab.sensors.SensorBase[source]#

The base class for implementing a sensor.

The implementation is based on lazy evaluation. The sensor data is only updated when the user tries accessing the data through the data property or sets force_compute=True in the update() method. This is done to avoid unnecessary computation when the sensor data is not used.

The sensor is updated at the specified update period. If the update period is zero, then the sensor is updated at every simulation step.

Methods:

`__init__`(cfg)	Initialize the sensor class.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.
`reset`([env_ids, env_mask])	Resets the sensor internals.

Attributes:

`is_initialized`	Whether the sensor is initialized.
`num_instances`	Number of instances of the sensor.
`device`	Memory device for computation.
`data`	Data from the sensor.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.

__init__(cfg: SensorBaseCfg)[source]#

Initialize the sensor class.

Parameters:: cfg – The configuration parameters for the sensor.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

property device: str#: Memory device for computation.

abstract property data: Any#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

set_debug_vis(debug_vis: bool) → bool[source]#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None) → None[source]#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

class isaaclab.sensors.SensorBaseCfg[source]#

Configuration parameters for a sensor.

Attributes:

`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

USD Camera#

class isaaclab.sensors.Camera[source]#

Bases: SensorBase

The camera sensor for acquiring visual data.

This class wraps over the UsdGeom Camera for providing a consistent API for acquiring visual data. It ensures that the camera follows the ROS convention for the coordinate system.

Summarizing from the replicator extension, the following sensor types are supported:

"rgb": A 3-channel rendered color image.
"rgba": A 4-channel rendered color image with alpha channel.
"albedo": A 4-channel fast diffuse-albedo only path for color image. Note that this path will achieve the best performance when used alone or with depth only.
"distance_to_camera": An image containing the distance to camera optical center.
"distance_to_image_plane": An image containing distances of 3D points from camera plane along camera’s z-axis.
"depth": The same as "distance_to_image_plane".
"simple_shading_constant_diffuse": Simple shading (constant diffuse) RGB approximation.
"simple_shading_diffuse_mdl": Simple shading (diffuse MDL) RGB approximation.
"simple_shading_full_mdl": Simple shading (full MDL) RGB approximation.
"normals": An image containing the local surface normal vectors at each pixel.
"motion_vectors": An image containing the motion vector data at each pixel.
"semantic_segmentation": The semantic segmentation data.
"instance_segmentation_fast": The instance segmentation data.
"instance_id_segmentation_fast": The instance id segmentation data.

Note

Currently the following sensor types are not supported in a “view” format:

"instance_segmentation": The instance segmentation data. Please use the fast counterparts instead.
"instance_id_segmentation": The instance id segmentation data. Please use the fast counterparts instead.
"bounding_box_2d_tight": The tight 2D bounding box data (only contains non-occluded regions).
"bounding_box_2d_tight_fast": The tight 2D bounding box data (only contains non-occluded regions).
"bounding_box_2d_loose": The loose 2D bounding box data (contains occluded regions).
"bounding_box_2d_loose_fast": The loose 2D bounding box data (contains occluded regions).
"bounding_box_3d": The 3D view space bounding box data.
"bounding_box_3d_fast": The 3D view space bounding box data.

Attributes:

`cfg`	The configuration parameters.
`UNSUPPORTED_TYPES`	The set of sensor types that are not supported by the camera class.
`num_instances`	Number of instances of the sensor.
`data`	Data from the sensor.
`frame`	Frame number when the measurement took place.
`image_shape`	A tuple containing (height, width) of the camera sensor.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.

Methods:

`__init__`(cfg)	Initializes the camera sensor.
`set_intrinsic_matrices`(matrices[, ...])	Set parameters of the USD camera from its intrinsic matrix.
`set_world_poses`([positions, orientations, ...])	Set the pose of the camera w.r.t.
`set_world_poses_from_view`(eyes, targets[, ...])	Set the poses of the camera from the eye position and look-at target position.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

cfg: CameraCfg#: The configuration parameters.

UNSUPPORTED_TYPES: set[str] = {'bounding_box_2d_loose', 'bounding_box_2d_loose_fast', 'bounding_box_2d_tight', 'bounding_box_2d_tight_fast', 'bounding_box_3d', 'bounding_box_3d_fast', 'instance_id_segmentation', 'instance_segmentation'}#: The set of sensor types that are not supported by the camera class.

__init__(cfg: CameraCfg)[source]#

Initializes the camera sensor.

Parameters:

cfg – The configuration parameters.

Raises:

RuntimeError – If no camera prim is found at the given path.
ValueError – If the provided data types are not supported by the camera.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

property data: CameraData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property frame: ProxyArray#: Frame number when the measurement took place.

property image_shape: tuple[int, int]#: A tuple containing (height, width) of the camera sensor.

set_intrinsic_matrices(matrices: torch.Tensor | wp.array, focal_length: float | None = None, env_ids: Sequence[int] | None = None)[source]#

Set parameters of the USD camera from its intrinsic matrix.

The intrinsic matrix is used to set the following parameters to the USD camera:

focal_length: The focal length of the camera.
horizontal_aperture: The horizontal aperture of the camera.
vertical_aperture: The vertical aperture of the camera.
horizontal_aperture_offset: The horizontal offset of the camera.
vertical_aperture_offset: The vertical offset of the camera.

Warning

Due to limitations of Omniverse camera, we need to assume that the camera is a spherical lens, i.e. has square pixels, and the optical center is centered at the camera eye. If this assumption is not true in the input intrinsic matrix, then the camera will not set up correctly.

Parameters:

matrices – The intrinsic matrices for the camera. Shape is (N, 3, 3).
focal_length – Perspective focal length (in cm) used to calculate pixel size. Defaults to None. If None, focal_length will be calculated 1 / width.
env_ids – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

set_world_poses(positions: torch.Tensor | None = None, orientations: torch.Tensor | None = None, env_ids: Sequence[int] | None = None, convention: Literal['opengl', 'ros', 'world'] = 'ros')[source]#

Set the pose of the camera w.r.t. the world frame using specified convention.

Since different fields use different conventions for camera orientations, the method allows users to set the camera poses in the specified convention. Possible conventions are:

"opengl" - forward axis: -Z - up axis +Y - Offset is applied in the OpenGL (Usd.Camera) convention
"ros" - forward axis: +Z - up axis -Y - Offset is applied in the ROS convention
"world" - forward axis: +X - up axis +Z - Offset is applied in the World Frame convention

See isaaclab.sensors.camera.utils.convert_camera_frame_orientation_convention() for more details on the conventions.

Parameters:

positions – The cartesian coordinates (in meters). Shape is (N, 3). Defaults to None, in which case the camera position in not changed.
orientations – The quaternion orientation in (x, y, z, w). Shape is (N, 4). Defaults to None, in which case the camera orientation in not changed.
env_ids – A sensor ids to manipulate. Defaults to None, which means all sensor indices.
convention – The convention in which the poses are fed. Defaults to “ros”.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.

set_world_poses_from_view(eyes: torch.Tensor, targets: torch.Tensor, env_ids: Sequence[int] | None = None)[source]#

Set the poses of the camera from the eye position and look-at target position.

Parameters:

eyes – The positions of the camera’s eye. Shape is (N, 3).
targets – The target locations to look at. Shape is (N, 3).
env_ids – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.
NotImplementedError – If the stage up-axis is not “Y” or “Z”.
ValueError – If every eye position equals its target (look-at direction undefined for the whole batch). When only some rows are degenerate, those rows are skipped and the remaining poses are still applied; a warning is logged.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)[source]#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

property device: str#: Memory device for computation.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

class isaaclab.sensors.CameraData[source]#

Data container for the camera sensor.

Public properties return ProxyArray wrappers. Use .torch for a cached zero-copy torch.Tensor view or .warp for the underlying warp.array.

Attributes:

`image_shape`	A tuple containing (height, width) of the camera sensor.
`info`	The retrieved sensor info with sensor types as key.
`pos_w`	Position of the sensor origin in world frame [m], following ROS convention.
`quat_w_world`	Quaternion orientation `(x, y, z, w)` of the sensor origin in world frame, following the world coordinate frame convention.
`intrinsic_matrices`	The intrinsic matrices for the camera.
`output`	The retrieved sensor data with sensor types as key.
`quat_w_ros`	Quaternion orientation `(x, y, z, w)` of the sensor origin in the world frame, following ROS convention.
`quat_w_opengl`	Quaternion orientation `(x, y, z, w)` of the sensor origin in the world frame, following Opengl / USD Camera convention.

Methods:

`create_buffers`(num_views, device)	Allocate warp arrays for pose and intrinsics and create their `ProxyArray` wrappers.
`allocate`(data_types, height, width, ...)	Build a `CameraData` with output buffers pre-allocated as warp arrays.

image_shape: tuple[int, int] | None#: A tuple containing (height, width) of the camera sensor.

info: dict[str, Any] | None#

The retrieved sensor info with sensor types as key.

This contains extra information provided by the sensor such as semantic segmentation label mapping, prim paths. For semantic-based data, this corresponds to the "info" key in the output of the sensor. For other sensor types, the info is empty.

property pos_w: ProxyArray#

Position of the sensor origin in world frame [m], following ROS convention.

Shape is (N,), dtype wp.vec3f. In torch this resolves to (N, 3), where N is the number of sensors. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

property quat_w_world: ProxyArray#

Quaternion orientation (x, y, z, w) of the sensor origin in world frame, following the world coordinate frame convention.

Note

World frame convention follows the camera aligned with forward axis +X and up axis +Z.

Shape is (N,), dtype wp.quatf. In torch this resolves to (N, 4), where N is the number of sensors. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

property intrinsic_matrices: ProxyArray#

The intrinsic matrices for the camera.

Shape is (N,), dtype wp.mat33f. In torch this resolves to (N, 3, 3), where N is the number of sensors. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

property output: dict[str, ProxyArray] | None#

The retrieved sensor data with sensor types as key.

Each value is a ProxyArray of shape (N, H, W, C) where N is the number of views, H/W are image dimensions, and C is the number of channels. Use .torch for a torch.Tensor view or .warp for the underlying wp.array.

The format of the data is available in the Replicator Documentation. For semantic-based data, this corresponds to the "data" key in the output of the sensor.

create_buffers(num_views: int, device: str) → None[source]#

Allocate warp arrays for pose and intrinsics and create their ProxyArray wrappers.

Called by Camera after allocate() to populate the pose and intrinsics buffers.

Parameters:

num_views – Number of camera views (batch dimension).
device – Device for tensor storage (e.g. "cuda:0").

classmethod allocate(data_types: list[str], height: int, width: int, num_views: int, device: str, supported_specs: dict[RenderBufferKind, RenderBufferSpec]) → CameraData[source]#

Build a CameraData with output buffers pre-allocated as warp arrays.

Allocates one (num_views, height, width, channels) warp array per kind in the intersection of data_types and supported_specs, using the channels and dtype from each RenderBufferSpec. Each buffer is wrapped in a ProxyArray; call .torch on the result to obtain a zero-copy torch.Tensor view.

Parameters:

data_types – Requested output names (typically CameraCfg.data_types). Every name must be a member of RenderBufferKind.
height – Image height in pixels.
width – Image width in pixels.
num_views – Number of camera views (batch dimension).
device – Device on which to allocate the buffers.
supported_specs – Per-buffer layout the active renderer can produce, keyed by RenderBufferKind. Names absent from this mapping are not allocated.

Returns:

A new CameraData with image_shape, output, and info populated; pose/intrinsic buffers must be created separately via create_buffers().

Raises:

ValueError – If data_types contains names that are not members of RenderBufferKind.

property quat_w_ros: ProxyArray#

Quaternion orientation (x, y, z, w) of the sensor origin in the world frame, following ROS convention.

Note

ROS convention follows the camera aligned with forward axis +Z and up axis -Y.

Shape is (N,), dtype wp.quatf. In torch this resolves to (N, 4), where N is the number of sensors. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

property quat_w_opengl: ProxyArray#

Quaternion orientation (x, y, z, w) of the sensor origin in the world frame, following Opengl / USD Camera convention.

Note

OpenGL convention follows the camera aligned with forward axis -Z and up axis +Y.

Shape is (N,), dtype wp.quatf. In torch this resolves to (N, 4), where N is the number of sensors. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

class isaaclab.sensors.CameraCfg[source]#

Bases: SensorBaseCfg

Configuration for a camera sensor.

Attributes:

`offset`	The offset pose of the sensor's frame from the sensor's parent frame.
`spawn`	Spawn configuration for the asset.
`depth_clipping_behavior`	Clipping behavior for the camera for values exceed the maximum value.
`data_types`	List of sensor names/types to enable for the camera.
`width`	Width of the image in pixels.
`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`height`	Height of the image in pixels.
`update_latest_camera_pose`	Whether to update the latest camera pose when fetching the camera's data.
`semantic_filter`	A string or a list specifying a semantic filter predicate.
`colorize_semantic_segmentation`	Whether to colorize the semantic segmentation images.
`colorize_instance_id_segmentation`	Whether to colorize the instance ID segmentation images.
`colorize_instance_segmentation`	Whether to colorize the instance ID segmentation images.
`semantic_segmentation_mapping`	Dictionary mapping semantics to specific colours
`renderer_cfg`	Renderer configuration for camera sensor.

offset: OffsetCfg#: The offset pose of the sensor’s frame from the sensor’s parent frame. Defaults to identity.

Note

The parent frame is the frame the sensor attaches to. For example, the parent frame of a camera at path /World/envs/env_0/Robot/Camera is /World/envs/env_0/Robot.

spawn: PinholeCameraCfg | FisheyeCameraCfg | None#

Spawn configuration for the asset.

If None, then the prim is not spawned by the asset. Instead, it is assumed that the asset is already present in the scene.

depth_clipping_behavior: Literal['max', 'zero', 'none']#

Clipping behavior for the camera for values exceed the maximum value. Defaults to “none”.

"max": Values are clipped to the maximum value.
"zero": Values are clipped to zero.
"none: No clipping is applied. Values will be returned as inf.

Deprecated since version 4.6.22: This field is RTX-specific. Set depth_clipping_behavior on renderer_cfg instead.

data_types: list[str]#

List of sensor names/types to enable for the camera. Defaults to [“rgb”].

Please refer to the Camera class for a list of available data types.

width: int#: Width of the image in pixels.

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

height: int#: Height of the image in pixels.

update_latest_camera_pose: bool#

Whether to update the latest camera pose when fetching the camera’s data. Defaults to False.

If True, the latest camera pose is updated in the camera’s data which will slow down performance due to the use of FrameView. If False, the pose of the camera during initialization is returned.

semantic_filter: str | list[str]#

A string or a list specifying a semantic filter predicate. Defaults to "*:*".

If a string, it should be a disjunctive normal form of (semantic type, labels). For examples:

"typeA : labelA & !labelB | labelC , typeB: labelA ; typeC: labelE": All prims with semantic type “typeA” and label “labelA” but not “labelB” or with label “labelC”. Also, all prims with semantic type “typeB” and label “labelA”, or with semantic type “typeC” and label “labelE”.
"typeA : * ; * : labelA": All prims with semantic type “typeA” or with label “labelA”

If a list of strings, each string should be a semantic type. The segmentation for prims with semantics of the specified types will be retrieved. For example, if the list is [“class”], only the segmentation for prims with semantics of type “class” will be retrieved.

Contact Sensor#

class isaaclab.sensors.ContactSensor[source]#

Bases: FactoryBase, BaseContactSensor

Factory for creating contact sensor instances.

Attributes:

`data`	Data from the sensor.
`body_names`	Ordered names of shapes or bodies with contact sensors attached.
`contact_view`	View for the contact forces captured.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`num_bodies`	Deprecated property.
`num_instances`	Number of instances of the sensor.
`num_sensors`	Number of sensors per environment.
`cfg`	The configuration parameters.

Methods:

`__new__`(cls, args, *kwargs)	Create a new instance of a contact sensor based on the backend.
`__init__`(cfg)	Initializes the contact sensor object.
`compute_first_air`(dt[, abs_tol])	Checks if bodies that have broken contact within the last `dt` seconds.
`compute_first_contact`(dt[, abs_tol])	Checks if bodies that have established contact within the last `dt` seconds.
`find_bodies`(name_keys[, preserve_order])	Deprecated method.
`find_sensors`(name_keys[, preserve_order])	Find sensors in the contact sensor based on the name keys.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.
`reset`([env_ids, env_mask])	Resets the sensor.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

abstract property data: BaseContactSensorData#: Data from the sensor.

static __new__(cls, *args, **kwargs) → BaseContactSensor | PhysXContactSensor | NewtonContactSensor[source]#: Create a new instance of a contact sensor based on the backend.

__init__(cfg: ContactSensorCfg)#

Initializes the contact sensor object.

Parameters:: cfg – The configuration parameters.

abstract property body_names: list[str] | None#: Ordered names of shapes or bodies with contact sensors attached.

abstractmethod compute_first_air(dt: float, abs_tol: float = 1e-08) → ProxyArray#

Checks if bodies that have broken contact within the last dt seconds.

This function checks if the bodies have broken contact within the last dt seconds by comparing the current air time with the given time period. If the air time is less than the given time period, then the bodies are considered to not be in contact.

Note

It assumes that dt is a factor of the sensor update time-step. In other words, \(dt / dt_sensor = n\), where \(n\) is a natural number. This is always true if the sensor is updated by the physics or the environment stepping time-step and the sensor is read by the environment stepping time-step.

Parameters:

dt – The time period since the contract is broken.
abs_tol – The absolute tolerance for the comparison.

Returns:

A boolean tensor indicating the bodies that have broken contact within the last dt seconds. Shape is (N, B), where N is the number of sensors and B is the number of bodies in each sensor.

Raises:

RuntimeError – If the sensor is not configured to track contact time.

abstractmethod compute_first_contact(dt: float, abs_tol: float = 1e-08) → ProxyArray#

Checks if bodies that have established contact within the last dt seconds.

This function checks if the bodies have established contact within the last dt seconds by comparing the current contact time with the given time period. If the contact time is less than the given time period, then the bodies are considered to be in contact.

Note

The function assumes that dt is a factor of the sensor update time-step. In other words \(dt / dt_sensor = n\), where \(n\) is a natural number. This is always true if the sensor is updated by the physics or the environment stepping time-step and the sensor is read by the environment stepping time-step.

Parameters:

dt – The time period since the contact was established.
abs_tol – The absolute tolerance for the comparison.

Returns:

A boolean tensor indicating the bodies that have established contact within the last dt seconds. Shape is (N, B), where N is the number of sensors and B is the number of bodies in each sensor.

Raises:

RuntimeError – If the sensor is not configured to track contact time.

abstract property contact_view: None#: View for the contact forces captured.

Note

None if there is no view associated with the sensor.

property device: str#: Memory device for computation.

find_bodies(name_keys: str | Sequence[str], preserve_order: bool = False) → tuple[list[int], list[str]]#: Deprecated method. Please use find_sensors instead.

find_sensors(name_keys: str | Sequence[str], preserve_order: bool = False) → tuple[list[int], list[str]]#

Find sensors in the contact sensor based on the name keys.

Parameters:

name_keys – A regular expression or a list of regular expressions to match the body names.
preserve_order – Whether to preserve the order of the name keys in the output. Defaults to False.

Returns:

A tuple of lists containing the sensor indices and names.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

property num_bodies: int#: Deprecated property. Please use num_sensors instead.

abstract property num_instances: int | None#: Number of instances of the sensor.

abstract property num_sensors: int#: Number of sensors per environment.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

abstractmethod reset(env_ids: Sequence[int] | None = None, env_mask: wp.array(dtype=wp.bool) | None = None)#

Resets the sensor.

Parameters:

env_ids – The indices of the environments to reset. Defaults to None: all the environments are reset.
env_mask – The masks of the environments to reset. Defaults to None: all the environments are reset.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

cfg: ContactSensorCfg#: The configuration parameters.

class isaaclab.sensors.ContactSensorData[source]#

Factory for creating contact sensor data instances.

Methods:

`__new__`(cls, args, *kwargs)	Create a new instance of a contact sensor data based on the backend.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.

Attributes:

`contact_pos_w`	Average position of contact points.
`current_air_time`	Time spent in air since last detach.
`current_contact_time`	Time spent in contact since last contact.
`force_matrix_w`	Normal contact forces filtered between sensor and filtered bodies.
`force_matrix_w_history`	History of filtered contact forces.
`friction_forces_w`	Sum of friction forces.
`last_air_time`	Time spent in air before last contact.
`last_contact_time`	Time spent in contact before last detach.
`net_forces_w`	The net normal contact forces in world frame.
`net_forces_w_history`	History of net normal contact forces.
`pos_w`	Position of the sensor origin in world frame.
`pose_w`	Pose of the sensor origin in world frame.
`quat_w`	Orientation of the sensor origin in world frame.

static __new__(cls, *args, **kwargs) → BaseContactSensorData | PhysXContactSensorData | NewtonContactSensorData[source]#: Create a new instance of a contact sensor data based on the backend.

abstract property contact_pos_w: ProxyArray | None#

Average position of contact points.

Shape is (num_instances, num_sensors, num_filter_shapes), dtype = wp.vec3f. In torch this resolves to (num_instances, num_sensors, num_filter_shapes, 3).

None if ContactSensorCfg.track_contact_points is False.

abstract property current_air_time: ProxyArray | None#

Time spent in air since last detach.

Shape is (num_instances, num_sensors), dtype = wp.float32.

None if ContactSensorCfg.track_air_time is False.

abstract property current_contact_time: ProxyArray | None#

Time spent in contact since last contact.

Shape is (num_instances, num_sensors), dtype = wp.float32.

None if ContactSensorCfg.track_air_time is False.

abstract property force_matrix_w: ProxyArray | None#

Normal contact forces filtered between sensor and filtered bodies.

Shape is (num_instances, num_sensors, num_filter_shapes), dtype = wp.vec3f. In torch this resolves to (num_instances, num_sensors, num_filter_shapes, 3).

None if ContactSensorCfg.filter_prim_paths_expr is empty.

abstract property force_matrix_w_history: ProxyArray | None#

History of filtered contact forces.

Shape is (num_instances, history_length, num_sensors, num_filter_shapes), dtype = wp.vec3f. In torch this resolves to (num_instances, history_length, num_sensors, num_filter_shapes, 3).

None if ContactSensorCfg.filter_prim_paths_expr is empty.

abstract property friction_forces_w: ProxyArray | None#

Sum of friction forces.

Shape is (num_instances, num_sensors, num_filter_shapes), dtype = wp.vec3f. In torch this resolves to (num_instances, num_sensors, num_filter_shapes, 3).

None if ContactSensorCfg.track_friction_forces is False.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

abstract property last_air_time: ProxyArray | None#

Time spent in air before last contact.

Shape is (num_instances, num_sensors), dtype = wp.float32.

None if ContactSensorCfg.track_air_time is False.

abstract property last_contact_time: ProxyArray | None#

Time spent in contact before last detach.

Shape is (num_instances, num_sensors), dtype = wp.float32.

None if ContactSensorCfg.track_air_time is False.

abstract property net_forces_w: ProxyArray | None#

The net normal contact forces in world frame.

Shape is (num_instances, num_sensors), dtype = wp.vec3f. In torch this resolves to (num_instances, num_sensors, 3).

abstract property net_forces_w_history: ProxyArray | None#

History of net normal contact forces.

Shape is (num_instances, history_length, num_sensors), dtype = wp.vec3f. In torch this resolves to (num_instances, history_length, num_sensors, 3).

abstract property pos_w: ProxyArray | None#

Position of the sensor origin in world frame.

Shape is (num_instances, num_sensors), dtype = wp.vec3f. In torch this resolves to (num_instances, num_sensors, 3).

None if ContactSensorCfg.track_pose is False.

abstract property pose_w: ProxyArray | None#

Pose of the sensor origin in world frame.

None if ContactSensorCfg.track_pose is False.

abstract property quat_w: ProxyArray | None#

Orientation of the sensor origin in world frame.

Shape is (num_instances, num_sensors), dtype = wp.quatf. In torch this resolves to (num_instances, num_sensors, 4). The orientation is provided in (x, y, z, w) format.

None if ContactSensorCfg.track_pose is False.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

class isaaclab.sensors.ContactSensorCfg[source]#

Bases: SensorBaseCfg

Configuration for the contact sensor.

Sensing bodies are selected via SensorBaseCfg.prim_path. Filter bodies for per-partner force reporting are selected via filter_prim_paths_expr.

Only body-level sensing and filtering are supported. For shape-level granularity, see NewtonContactSensorCfg in isaaclab_newton.

Attributes:

`track_pose`	Whether to track the pose of the sensor's origin.
`track_contact_points`	Whether to track the contact point locations.
`track_friction_forces`	Whether to track the friction forces at the contact points.
`max_contact_data_count_per_prim`	The maximum number of contacts across all batches of the sensor to keep track of.
`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`track_air_time`	Whether to track the air/contact time of the bodies (time between contacts).
`force_threshold`	The threshold on the norm of the contact force that determines whether two bodies are in collision or not.
`history_length`	Number of past frames to store in the sensor buffers.
`filter_prim_paths_expr`	List of body prim path expressions to filter contacts against.
`visualizer_cfg`	The configuration object for the visualization markers.

track_pose: bool#: Whether to track the pose of the sensor’s origin. Defaults to False.

track_contact_points: bool#: Whether to track the contact point locations. Defaults to False.

track_friction_forces: bool#: Whether to track the friction forces at the contact points. Defaults to False.

max_contact_data_count_per_prim: int | None#

The maximum number of contacts across all batches of the sensor to keep track of. Default is 4, where supported.

This parameter sets the total maximum counts of the simulation across all bodies and environments. The total number of contacts allowed is max_contact_data_count_per_prim*num_envs*num_sensor_bodies.

Note

If the environment is very contact rich it is suggested to increase this parameter to avoid out of bounds memory errors and loss of contact data leading to inaccurate measurements.

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

track_air_time: bool#: Whether to track the air/contact time of the bodies (time between contacts). Defaults to False.

force_threshold: float | None#

The threshold on the norm of the contact force that determines whether two bodies are in collision or not. Defaults to None, in which case the sensor backend chooses an appropriate value.

This value is only used for tracking the mode duration (the time in contact or in air), if track_air_time is True.

history_length: int#: Number of past frames to store in the sensor buffers. Defaults to 0, which means that only the current data is stored (no history).

filter_prim_paths_expr: list[str]#

List of body prim path expressions to filter contacts against. Defaults to empty, meaning contacts with all bodies are aggregated into the net force.

If provided, a per-partner force matrix (ContactSensorData.force_matrix_w) is reported in addition to the net force. Each expression is matched against body prim paths in the scene.

For shape-level filtering, see NewtonContactSensorCfg in isaaclab_newton.

Note

Expressions can contain the environment namespace regex {ENV_REGEX_NS}, which is replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Object becomes /World/envs/env_.*/Object.

Attention

Filtered contact reporting only works when SensorBaseCfg.prim_path matches a single primitive per environment. For many-to-many filtering, see NewtonContactSensorCfg in isaaclab_newton.

visualizer_cfg: VisualizationMarkersCfg#: The configuration object for the visualization markers. Defaults to CONTACT_SENSOR_MARKER_CFG.

Note

This attribute is only used when debug visualization is enabled.

Frame Transformer#

class isaaclab.sensors.FrameTransformer[source]#

Bases: FactoryBase, BaseFrameTransformer

Factory for creating frame transformer instances.

Attributes:

`data`	Data from the sensor.
`body_names`	Returns the names of the target bodies being tracked.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`num_bodies`	Returns the number of target bodies being tracked.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

Methods:

`__new__`(cls, args, *kwargs)	Create a new instance of a frame transformer based on the backend.
`__init__`(cfg)	Initializes the frame transformer object.
`find_bodies`(name_keys[, preserve_order])	Find bodies in the articulation based on the name keys.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

abstract property data: BaseFrameTransformerData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

static __new__(cls, *args, **kwargs) → BaseFrameTransformer | NewtonFrameTransformer | PhysXFrameTransformer[source]#: Create a new instance of a frame transformer based on the backend.

__init__(cfg: FrameTransformerCfg)#

Initializes the frame transformer object.

Parameters:: cfg – The configuration parameters.

abstract property body_names: list[str]#: Returns the names of the target bodies being tracked.

Deprecated since version Use: data.target_frame_names instead. This property will be removed in a future release.

property device: str#: Memory device for computation.

find_bodies(name_keys: str | Sequence[str], preserve_order: bool = False) → tuple[list[int], list[str]]#

Find bodies in the articulation based on the name keys.

Parameters:

name_keys – A regular expression or a list of regular expressions to match the body names.
preserve_order – Whether to preserve the order of the name keys in the output. Defaults to False.

Returns:

A tuple of lists containing the body indices and names.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

abstract property num_bodies: int#: Returns the number of target bodies being tracked.

Deprecated since version Use: len(data.target_frame_names) instead. This property will be removed in a future release.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None) → None#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

cfg: FrameTransformerCfg#: The configuration parameters.

class isaaclab.sensors.FrameTransformerData[source]#

Factory for creating frame transformer data instances.

Methods:

`__new__`(cls, args, *kwargs)	Create a new instance of a frame transformer data based on the backend.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.

Attributes:

`source_pos_w`	Position of the source frame after offset in world frame.
`source_pose_w`	Pose of the source frame after offset in world frame.
`source_quat_w`	Orientation of the source frame after offset in world frame.
`target_frame_names`	Target frame names (order matches data ordering).
`target_pos_source`	Position of the target frame(s) relative to source frame.
`target_pos_w`	Position of the target frame(s) after offset in world frame.
`target_pose_source`	Pose of the target frame(s) relative to source frame.
`target_pose_w`	Pose of the target frame(s) after offset in world frame.
`target_quat_source`	Orientation of the target frame(s) relative to source frame.
`target_quat_w`	Orientation of the target frame(s) after offset in world frame.

static __new__(cls, *args, **kwargs) → BaseFrameTransformerData | NewtonFrameTransformerData | PhysXFrameTransformerData[source]#: Create a new instance of a frame transformer data based on the backend.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

abstract property source_pos_w: ProxyArray#

Position of the source frame after offset in world frame.

Shape is (num_instances,), dtype = wp.vec3f. In torch this resolves to (num_instances, 3).

abstract property source_pose_w: ProxyArray | None#

Pose of the source frame after offset in world frame.

Shape is (num_instances,), dtype = wp.transformf. In torch this resolves to (num_instances, 7). The pose is provided in (x, y, z, qx, qy, qz, qw) format.

abstract property source_quat_w: ProxyArray#

Orientation of the source frame after offset in world frame.

Shape is (num_instances,), dtype = wp.quatf. In torch this resolves to (num_instances, 4). The orientation is provided in (x, y, z, w) format.

abstract property target_frame_names: list[str]#

Target frame names (order matches data ordering).

Resolved from FrameTransformerCfg.FrameCfg.name.

abstract property target_pos_source: ProxyArray#

Position of the target frame(s) relative to source frame.

Shape is (num_instances, num_target_frames), dtype = wp.vec3f. In torch this resolves to (num_instances, num_target_frames, 3).

abstract property target_pos_w: ProxyArray#

Position of the target frame(s) after offset in world frame.

Shape is (num_instances, num_target_frames), dtype = wp.vec3f. In torch this resolves to (num_instances, num_target_frames, 3).

abstract property target_pose_source: ProxyArray | None#

Pose of the target frame(s) relative to source frame.

Shape is (num_instances, num_target_frames), dtype = wp.transformf. In torch this resolves to (num_instances, num_target_frames, 7). The pose is provided in (x, y, z, qx, qy, qz, qw) format.

abstract property target_pose_w: ProxyArray | None#

Pose of the target frame(s) after offset in world frame.

Shape is (num_instances, num_target_frames), dtype = wp.transformf. In torch this resolves to (num_instances, num_target_frames, 7). The pose is provided in (x, y, z, qx, qy, qz, qw) format.

abstract property target_quat_source: ProxyArray#

Orientation of the target frame(s) relative to source frame.

Shape is (num_instances, num_target_frames), dtype = wp.quatf. In torch this resolves to (num_instances, num_target_frames, 4). The orientation is provided in (x, y, z, w) format.

abstract property target_quat_w: ProxyArray#

Orientation of the target frame(s) after offset in world frame.

Shape is (num_instances, num_target_frames), dtype = wp.quatf. In torch this resolves to (num_instances, num_target_frames, 4). The orientation is provided in (x, y, z, w) format.

class isaaclab.sensors.FrameTransformerCfg[source]#

Bases: SensorBaseCfg

Configuration for the frame transformer sensor.

Classes:

FrameCfg

Information specific to a coordinate frame.

Attributes:

`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`prim_path`	The prim path of the body to transform from (source frame).
`source_frame_offset`	The pose offset from the source prim frame.
`target_frames`	A list of the target frames.
`visualizer_cfg`	The configuration object for the visualization markers.

class FrameCfg[source]#

Bases: object

Information specific to a coordinate frame.

Attributes:

`prim_path`	The prim path corresponding to a rigid body.
`name`	User-defined name for the new coordinate frame.
`offset`	The pose offset from the parent prim frame.

prim_path: str#

The prim path corresponding to a rigid body.

This can be a regex pattern to match multiple prims. For example, “/Robot/.*” will match all prims under “/Robot”.

This means that if the source FrameTransformerCfg.prim_path is “/Robot/base”, and the target FrameTransformerCfg.FrameCfg.prim_path is “/Robot/.*”, then the frame transformer will track the poses of all the prims under “/Robot”, including “/Robot/base” (even though this will result in an identity pose w.r.t. the source frame).

name: str | None#

User-defined name for the new coordinate frame. Defaults to None.

If None, then the name is extracted from the leaf of the prim path.

offset: OffsetCfg#: The pose offset from the parent prim frame.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

prim_path: str#: The prim path of the body to transform from (source frame).

source_frame_offset: OffsetCfg#: The pose offset from the source prim frame.

target_frames: list[FrameCfg]#

A list of the target frames.

This allows a single FrameTransformer to handle multiple target prims. For example, in a quadruped, we can use a single FrameTransformer to track each foot’s position and orientation in the body frame using four frame offsets.

visualizer_cfg: VisualizationMarkersCfg#: The configuration object for the visualization markers. Defaults to FRAME_MARKER_CFG.

Note

This attribute is only used when debug visualization is enabled.

class isaaclab.sensors.OffsetCfg[source]#

The offset pose of one frame relative to another frame.

Attributes:

`pos`	Translation w.r.t.
`rot`	Quaternion rotation (x, y, z, w) w.r.t.

pos: tuple[float, float, float]#: Translation w.r.t. the parent frame. Defaults to (0.0, 0.0, 0.0).

rot: tuple[float, float, float, float]#: Quaternion rotation (x, y, z, w) w.r.t. the parent frame. Defaults to (0.0, 0.0, 0.0, 1.0).

Ray-Cast Sensor#

class isaaclab.sensors.RayCaster[source]#

Bases: FactoryBase, BaseRayCaster

Backend-dispatching ray-caster sensor.

Methods:

`__init__`(cfg)	Initializes the ray-caster object.
`__new__`(cls, args, *kwargs)	Create a new instance of an implementation based on the backend.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

Attributes:

`data`	Data from the sensor.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`meshes`	A dictionary to store warp meshes for raycasting, shared across all instances.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

__init__(cfg: RayCasterCfg)#

Initializes the ray-caster object.

Parameters:: cfg – The configuration parameters.

static __new__(cls, *args, **kwargs)#: Create a new instance of an implementation based on the backend.

property data: RayCasterData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property device: str#: Memory device for computation.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

meshes: ClassVar[dict[tuple[str, str], wp.Mesh]] = {}#

A dictionary to store warp meshes for raycasting, shared across all instances.

The keys are (prim_path, device) tuples and values are the corresponding warp Mesh objects. Meshes are created lazily for the sensor’s active device, not eagerly for every device. Including the device in the key prevents a mesh created on one device (e.g. CPU) from being reused by a kernel running on a different device (e.g. CUDA) when multiple simulation contexts or tests use different devices in the same Python process.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

cfg: RayCasterCfg#: The configuration parameters.

class isaaclab.sensors.RayCasterData[source]#

Data container for the ray-cast sensor.

Public properties return ProxyArray wrappers. Use .torch for a cached zero-copy torch.Tensor view or .warp for the underlying warp.array.

Attributes:

`pos_w`	Position of the sensor origin in world frame [m].
`quat_w`	Orientation of the sensor origin in quaternion (x, y, z, w) in world frame.
`ray_hits_w`	The ray hit positions in the world frame [m].

Methods:

create_buffers(num_envs, num_rays, device)

Create internal warp buffers and their ProxyArray wrappers.

property pos_w: ProxyArray#

Position of the sensor origin in world frame [m].

Shape is (N,), dtype wp.vec3f. In torch this resolves to (N, 3), where N is the number of sensors. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

property quat_w: ProxyArray#

Orientation of the sensor origin in quaternion (x, y, z, w) in world frame.

Shape is (N,), dtype wp.quatf. In torch this resolves to (N, 4), where N is the number of sensors. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

property ray_hits_w: ProxyArray#

The ray hit positions in the world frame [m].

Shape is (N, B), dtype wp.vec3f. In torch this resolves to (N, B, 3), where N is the number of sensors and B is the number of rays per sensor. Contains inf for missed hits. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

create_buffers(num_envs: int, num_rays: int, device: str) → None[source]#

Create internal warp buffers and their ProxyArray wrappers.

Parameters:

num_envs – Number of environments / sensors.
num_rays – Number of rays per sensor.
device – Device for tensor storage.

class isaaclab.sensors.RayCasterCfg[source]#

Bases: SensorBaseCfg

Configuration for the ray-cast sensor.

Classes:

OffsetCfg

The offset pose of the sensor's frame from the sensor's parent frame.

Attributes:

`spawn`	Spawn configuration for the sensor Xform prim.
`mesh_prim_paths`	The list of mesh primitive paths to ray cast against.
`offset`	The offset pose of the sensor's frame from the sensor's parent frame.
`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`ray_alignment`	Specify in what frame the rays are projected onto the ground.
`pattern_cfg`	The pattern that defines the local ray starting positions and directions.
`max_distance`	Maximum distance (in meters) from the sensor to ray cast to.
`drift_range`	The range of drift (in meters) to add to the ray starting positions (xyz) in world frame.
`ray_cast_drift_range`	The range of drift (in meters) to add to the projected ray points in local projection frame.
`visualizer_cfg`	The configuration object for the visualization markers.

class OffsetCfg[source]#

Bases: object

The offset pose of the sensor’s frame from the sensor’s parent frame.

Attributes:

`pos`	Translation w.r.t.
`rot`	Quaternion rotation (x, y, z, w) w.r.t.

pos: tuple[float, float, float]#: Translation w.r.t. the parent frame. Defaults to (0.0, 0.0, 0.0).

rot: tuple[float, float, float, float]#: Quaternion rotation (x, y, z, w) w.r.t. the parent frame. Defaults to (0.0, 0.0, 0.0, 1.0).

spawn: SensorFrameCfg | None#

Spawn configuration for the sensor Xform prim.

A plain USD Xform is created at prim_path before initialization, matching the pattern used by CameraCfg (which spawns a Camera prim). The prim_path can be either:

A new child path under a parent link (e.g. {ENV_REGEX_NS}/Robot/base).
A physics body path (e.g. {ENV_REGEX_NS}/Robot/base). In this case, the sensor will automatically create a child Xform at {prim_path}.

If None, the prim at prim_path must already exist on the USD stage and must not be a physics body.

mesh_prim_paths: list[str]#: The list of mesh primitive paths to ray cast against.

Note

Currently, only a single static mesh is supported. We are working on supporting multiple static meshes and dynamic meshes.

offset: OffsetCfg#: The offset pose of the sensor’s frame from the sensor’s parent frame. Defaults to identity.

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

ray_alignment: Literal['base', 'yaw', 'world']#

Specify in what frame the rays are projected onto the ground. Default is “base”.

The options are:

base if the rays’ starting positions and directions track the full root position and orientation.
yaw if the rays’ starting positions track root position and the yaw component of the orientation, while ray directions remain fixed in world frame. This is useful for ray-casting height maps where the scan footprint should follow the body heading without tilting when the body pitches or rolls.
world if rays’ starting positions and directions are always fixed. This is useful in combination with a mapping package on the robot and querying ray-casts in a global frame.

pattern_cfg: PatternBaseCfg#: The pattern that defines the local ray starting positions and directions.

max_distance: float#: Maximum distance (in meters) from the sensor to ray cast to. Defaults to 1e6.

drift_range: tuple[float, float]#

The range of drift (in meters) to add to the ray starting positions (xyz) in world frame. Defaults to (0.0, 0.0).

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

ray_cast_drift_range: dict[str, tuple[float, float]]#

The range of drift (in meters) to add to the projected ray points in local projection frame. Defaults to a dictionary with zero drift for each x, y and z axis.

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

visualizer_cfg: VisualizationMarkersCfg#: The configuration object for the visualization markers. Defaults to RAY_CASTER_MARKER_CFG.

Note

This attribute is only used when debug visualization is enabled.

Ray-Cast Camera#

class isaaclab.sensors.RayCasterCamera[source]#

Bases: FactoryBase, BaseRayCasterCamera

Backend-dispatching ray-caster camera sensor.

Attributes:

`UNSUPPORTED_TYPES`	A set of sensor types that are not supported by the ray-caster camera.
`data`	Data from the sensor.
`device`	Memory device for computation.
`frame`	Frame number when the measurement took place.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`image_shape`	A tuple containing (height, width) of the camera sensor.
`is_initialized`	Whether the sensor is initialized.
`meshes`	A dictionary to store warp meshes for raycasting, shared across all instances.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

Methods:

`__init__`(cfg)	Initializes the camera object.
`__new__`(cls, args, *kwargs)	Create a new instance of an implementation based on the backend.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.
`set_intrinsic_matrices`(matrices[, ...])	Set the intrinsic matrix of the camera.
`set_world_poses`([positions, orientations, ...])	Set the pose of the camera w.r.t.
`set_world_poses_from_view`(eyes, targets[, ...])	Set the poses of the camera from the eye position and look-at target position.

UNSUPPORTED_TYPES: ClassVar[set[str]] = {'bounding_box_2d_loose', 'bounding_box_2d_loose_fast', 'bounding_box_2d_tight', 'bounding_box_2d_tight_fast', 'bounding_box_3d', 'bounding_box_3d_fast', 'instance_id_segmentation', 'instance_id_segmentation_fast', 'instance_segmentation', 'instance_segmentation_fast', 'motion_vectors', 'rgb', 'semantic_segmentation', 'skeleton_data'}#: A set of sensor types that are not supported by the ray-caster camera.

__init__(cfg: RayCasterCameraCfg)#

Initializes the camera object.

Parameters:: cfg – The configuration parameters.
Raises:: ValueError – If the provided data types are not supported by the ray-caster camera.

static __new__(cls, *args, **kwargs)#: Create a new instance of an implementation based on the backend.

property data: CameraData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property device: str#: Memory device for computation.

property frame: torch.tensor#: Frame number when the measurement took place.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property image_shape: tuple[int, int]#: A tuple containing (height, width) of the camera sensor.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

meshes: ClassVar[dict[tuple[str, str], wp.Mesh]] = {}#

A dictionary to store warp meshes for raycasting, shared across all instances.

The keys are (prim_path, device) tuples and values are the corresponding warp Mesh objects. Meshes are created lazily for the sensor’s active device, not eagerly for every device. Including the device in the key prevents a mesh created on one device (e.g. CPU) from being reused by a kernel running on a different device (e.g. CUDA) when multiple simulation contexts or tests use different devices in the same Python process.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

set_intrinsic_matrices(matrices: torch.Tensor, focal_length: float = 1.0, env_ids: Sequence[int] | None = None)#

Set the intrinsic matrix of the camera.

Parameters:

matrices – The intrinsic matrices for the camera. Shape is (N, 3, 3).
focal_length – Focal length to use when computing aperture values (in cm). Defaults to 1.0.
env_ids – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

set_world_poses(positions: torch.Tensor | None = None, orientations: torch.Tensor | None = None, env_ids: Sequence[int] | None = None, convention: Literal['opengl', 'ros', 'world'] = 'ros')#

Set the pose of the camera w.r.t. the world frame using specified convention.

Since different fields use different conventions for camera orientations, the method allows users to set the camera poses in the specified convention. Possible conventions are:

"opengl" - forward axis: -Z - up axis +Y - Offset is applied in the OpenGL (Usd.Camera) convention
"ros" - forward axis: +Z - up axis -Y - Offset is applied in the ROS convention
"world" - forward axis: +X - up axis +Z - Offset is applied in the World Frame convention

See isaaclab.utils.math.convert_camera_frame_orientation_convention() for more details on the conventions.

Parameters:

positions – The cartesian coordinates (in meters). Shape is (N, 3). Defaults to None, in which case the camera position in not changed.
orientations – The quaternion orientation in (x, y, z, w). Shape is (N, 4). Defaults to None, in which case the camera orientation in not changed.
env_ids – A sensor ids to manipulate. Defaults to None, which means all sensor indices.
convention – The convention in which the poses are fed. Defaults to “ros”.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.

set_world_poses_from_view(eyes: torch.Tensor, targets: torch.Tensor, env_ids: Sequence[int] | None = None)#

Set the poses of the camera from the eye position and look-at target position.

Parameters:

eyes – The positions of the camera’s eye. Shape is (N, 3).
targets – The target locations to look at. Shape is (N, 3).
env_ids – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.
NotImplementedError – If the stage up-axis is not “Y” or “Z”.

cfg: RayCasterCameraCfg#: The configuration parameters.

class isaaclab.sensors.RayCasterCameraCfg[source]#

Bases: RayCasterCfg

Configuration for the ray-cast sensor.

Attributes:

`offset`	The offset pose of the sensor's frame from the sensor's parent frame.
`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`spawn`	Spawn configuration for the sensor Xform prim.
`mesh_prim_paths`	The list of mesh primitive paths to ray cast against.
`ray_alignment`	Specify in what frame the rays are projected onto the ground.
`max_distance`	Maximum distance (in meters) from the sensor to ray cast to.
`drift_range`	The range of drift (in meters) to add to the ray starting positions (xyz) in world frame.
`ray_cast_drift_range`	The range of drift (in meters) to add to the projected ray points in local projection frame.
`visualizer_cfg`	The configuration object for the visualization markers.
`data_types`	List of sensor names/types to enable for the camera.
`depth_clipping_behavior`	Clipping behavior for the camera for values exceed the maximum value.
`pattern_cfg`	The pattern that defines the local ray starting positions and directions in a pinhole camera pattern.

offset: OffsetCfg#: The offset pose of the sensor’s frame from the sensor’s parent frame. Defaults to identity.

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

spawn: SensorFrameCfg | None#

Spawn configuration for the sensor Xform prim.

A plain USD Xform is created at prim_path before initialization, matching the pattern used by CameraCfg (which spawns a Camera prim). The prim_path can be either:

A new child path under a parent link (e.g. {ENV_REGEX_NS}/Robot/base).
A physics body path (e.g. {ENV_REGEX_NS}/Robot/base). In this case, the sensor will automatically create a child Xform at {prim_path}.

If None, the prim at prim_path must already exist on the USD stage and must not be a physics body.

mesh_prim_paths: list[str]#: The list of mesh primitive paths to ray cast against.

Note

Currently, only a single static mesh is supported. We are working on supporting multiple static meshes and dynamic meshes.

ray_alignment: Literal['base', 'yaw', 'world']#

Specify in what frame the rays are projected onto the ground. Default is “base”.

The options are:

base if the rays’ starting positions and directions track the full root position and orientation.
yaw if the rays’ starting positions track root position and the yaw component of the orientation, while ray directions remain fixed in world frame. This is useful for ray-casting height maps where the scan footprint should follow the body heading without tilting when the body pitches or rolls.
world if rays’ starting positions and directions are always fixed. This is useful in combination with a mapping package on the robot and querying ray-casts in a global frame.

max_distance: float#: Maximum distance (in meters) from the sensor to ray cast to. Defaults to 1e6.

drift_range: tuple[float, float]#

The range of drift (in meters) to add to the ray starting positions (xyz) in world frame. Defaults to (0.0, 0.0).

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

ray_cast_drift_range: dict[str, tuple[float, float]]#

The range of drift (in meters) to add to the projected ray points in local projection frame. Defaults to a dictionary with zero drift for each x, y and z axis.

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

visualizer_cfg: VisualizationMarkersCfg#: The configuration object for the visualization markers. Defaults to RAY_CASTER_MARKER_CFG.

Note

This attribute is only used when debug visualization is enabled.

data_types: list[str]#: List of sensor names/types to enable for the camera. Defaults to [“distance_to_image_plane”].

depth_clipping_behavior: Literal['max', 'zero', 'none']#

Clipping behavior for the camera for values exceed the maximum value. Defaults to “none”.

"max": Values are clipped to the maximum value.
"zero": Values are clipped to zero.
"none": No clipping is applied. Values will be returned as inf for missed rays in both distance_to_camera and distance_to_image_plane data types.

pattern_cfg: PinholeCameraPatternCfg#: The pattern that defines the local ray starting positions and directions in a pinhole camera pattern.

Multi-Mesh Ray-Cast Sensor#

class isaaclab.sensors.MultiMeshRayCaster[source]#

Bases: FactoryBase, BaseMultiMeshRayCaster

Backend-dispatching multi-mesh ray-caster sensor.

Methods:

`__init__`(cfg)	Initializes the ray-caster object.
`__new__`(cls, args, *kwargs)	Create a new instance of an implementation based on the backend.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

Attributes:

`data`	Data from the sensor.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`meshes`	A dictionary to store warp meshes for raycasting, shared across all instances.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

__init__(cfg: MultiMeshRayCasterCfg)#

Initializes the ray-caster object.

Parameters:: cfg – The configuration parameters.

static __new__(cls, *args, **kwargs)#: Create a new instance of an implementation based on the backend.

property data: MultiMeshRayCasterData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property device: str#: Memory device for computation.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

meshes: ClassVar[dict[tuple[str, str], wp.Mesh]] = {}#

A dictionary to store warp meshes for raycasting, shared across all instances.

The keys are (prim_path, device) tuples and values are the corresponding warp Mesh objects. Meshes are created lazily for the sensor’s active device, not eagerly for every device. Including the device in the key prevents a mesh created on one device (e.g. CPU) from being reused by a kernel running on a different device (e.g. CUDA) when multiple simulation contexts or tests use different devices in the same Python process.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

cfg: MultiMeshRayCasterCfg#: The configuration parameters.

class isaaclab.sensors.MultiMeshRayCasterData[source]#

Data container for the multi-mesh ray-cast sensor.

Methods:

create_buffers(num_envs, num_rays, device)

Create internal warp buffers and their ProxyArray wrappers.

Attributes:

`pos_w`	Position of the sensor origin in world frame [m].
`quat_w`	Orientation of the sensor origin in quaternion (x, y, z, w) in world frame.
`ray_hits_w`	The ray hit positions in the world frame [m].
`ray_mesh_ids`	The mesh ids of the ray hits.

create_buffers(num_envs: int, num_rays: int, device: str) → None#

Create internal warp buffers and their ProxyArray wrappers.

Parameters:

num_envs – Number of environments / sensors.
num_rays – Number of rays per sensor.
device – Device for tensor storage.

property pos_w: ProxyArray#

Position of the sensor origin in world frame [m].

Shape is (N,), dtype wp.vec3f. In torch this resolves to (N, 3), where N is the number of sensors. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

property quat_w: ProxyArray#

Orientation of the sensor origin in quaternion (x, y, z, w) in world frame.

Shape is (N,), dtype wp.quatf. In torch this resolves to (N, 4), where N is the number of sensors. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

property ray_hits_w: ProxyArray#

The ray hit positions in the world frame [m].

Shape is (N, B), dtype wp.vec3f. In torch this resolves to (N, B, 3), where N is the number of sensors and B is the number of rays per sensor. Contains inf for missed hits. Use .warp for the underlying wp.array or .torch for a cached zero-copy torch.Tensor view.

ray_mesh_ids: ProxyArray = None#

The mesh ids of the ray hits.

Shape is (N, B, 1), where N is the number of sensors, B is the number of rays in the scan pattern per sensor.

class isaaclab.sensors.MultiMeshRayCasterCfg[source]#

Bases: RayCasterCfg

Configuration for the multi-mesh ray-cast sensor.

Classes:

RaycastTargetCfg

Configuration for different ray-cast targets.

Attributes:

`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`spawn`	Spawn configuration for the sensor Xform prim.
`offset`	The offset pose of the sensor's frame from the sensor's parent frame.
`ray_alignment`	Specify in what frame the rays are projected onto the ground.
`pattern_cfg`	The pattern that defines the local ray starting positions and directions.
`max_distance`	Maximum distance (in meters) from the sensor to ray cast to.
`drift_range`	The range of drift (in meters) to add to the ray starting positions (xyz) in world frame.
`ray_cast_drift_range`	The range of drift (in meters) to add to the projected ray points in local projection frame.
`visualizer_cfg`	The configuration object for the visualization markers.
`mesh_prim_paths`	The list of mesh primitive paths to ray cast against.
`update_mesh_ids`	Whether to update the mesh ids of the ray hits in the `data` container.
`reference_meshes`	Whether to reference duplicated meshes instead of loading each one separately into memory.

class RaycastTargetCfg[source]#

Bases: object

Configuration for different ray-cast targets.

Attributes:

`prim_expr`	The regex to specify the target prim to ray cast against.
`is_shared`	Whether the target prim is assumed to be the same mesh across all environments.
`merge_prim_meshes`	Whether to merge the parsed meshes for a prim that contains multiple meshes.
`track_mesh_transforms`	Whether the mesh transformations should be tracked.

prim_expr: str#: The regex to specify the target prim to ray cast against.

is_shared: bool#

Whether the target prim is assumed to be the same mesh across all environments. Defaults to False.

If True, only the first mesh is read and then reused for all environments, rather than re-parsed. This provides a startup performance boost when there are many environments that all use the same asset.

Note

If MultiMeshRayCasterCfg.reference_meshes is False, this flag has no effect.

merge_prim_meshes: bool#

Whether to merge the parsed meshes for a prim that contains multiple meshes. Defaults to True.

This will create a new mesh that combines all meshes in the parsed prim. The raycast hits mesh IDs will then refer to the single merged mesh.

track_mesh_transforms: bool#: Whether the mesh transformations should be tracked. Defaults to True.

Note

Not tracking the mesh transformations is recommended when the meshes are static to increase performance.

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

spawn: SensorFrameCfg | None#

Spawn configuration for the sensor Xform prim.

A plain USD Xform is created at prim_path before initialization, matching the pattern used by CameraCfg (which spawns a Camera prim). The prim_path can be either:

A new child path under a parent link (e.g. {ENV_REGEX_NS}/Robot/base).
A physics body path (e.g. {ENV_REGEX_NS}/Robot/base). In this case, the sensor will automatically create a child Xform at {prim_path}.

If None, the prim at prim_path must already exist on the USD stage and must not be a physics body.

offset: OffsetCfg#: The offset pose of the sensor’s frame from the sensor’s parent frame. Defaults to identity.

ray_alignment: Literal['base', 'yaw', 'world']#

Specify in what frame the rays are projected onto the ground. Default is “base”.

The options are:

base if the rays’ starting positions and directions track the full root position and orientation.
yaw if the rays’ starting positions track root position and the yaw component of the orientation, while ray directions remain fixed in world frame. This is useful for ray-casting height maps where the scan footprint should follow the body heading without tilting when the body pitches or rolls.
world if rays’ starting positions and directions are always fixed. This is useful in combination with a mapping package on the robot and querying ray-casts in a global frame.

pattern_cfg: PatternBaseCfg#: The pattern that defines the local ray starting positions and directions.

max_distance: float#: Maximum distance (in meters) from the sensor to ray cast to. Defaults to 1e6.

drift_range: tuple[float, float]#

The range of drift (in meters) to add to the ray starting positions (xyz) in world frame. Defaults to (0.0, 0.0).

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

ray_cast_drift_range: dict[str, tuple[float, float]]#

The range of drift (in meters) to add to the projected ray points in local projection frame. Defaults to a dictionary with zero drift for each x, y and z axis.

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

visualizer_cfg: VisualizationMarkersCfg#: The configuration object for the visualization markers. Defaults to RAY_CASTER_MARKER_CFG.

Note

This attribute is only used when debug visualization is enabled.

mesh_prim_paths: list[str | RaycastTargetCfg]#

The list of mesh primitive paths to ray cast against.

If an entry is a string, it is internally converted to RaycastTargetCfg with track_mesh_transforms disabled. These settings ensure backwards compatibility with the default raycaster.

update_mesh_ids: bool#: Whether to update the mesh ids of the ray hits in the data container.

reference_meshes: bool#

Whether to reference duplicated meshes instead of loading each one separately into memory. Defaults to True.

When enabled, the raycaster parses all meshes in all environments, but reuses references for duplicates instead of storing multiple copies. This reduces memory footprint.

Multi-Mesh Ray-Cast Camera#

class isaaclab.sensors.MultiMeshRayCasterCamera[source]#

Bases: FactoryBase, BaseMultiMeshRayCasterCamera

Backend-dispatching multi-mesh ray-caster camera sensor.

Attributes:

`UNSUPPORTED_TYPES`	A set of sensor types that are not supported by the ray-caster camera.
`data`	Data from the sensor.
`device`	Memory device for computation.
`frame`	Frame number when the measurement took place.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`image_shape`	A tuple containing (height, width) of the camera sensor.
`is_initialized`	Whether the sensor is initialized.
`meshes`	A dictionary to store warp meshes for raycasting, shared across all instances.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

Methods:

`__init__`(cfg)	Initializes the camera object.
`__new__`(cls, args, *kwargs)	Create a new instance of an implementation based on the backend.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.
`set_intrinsic_matrices`(matrices[, ...])	Set the intrinsic matrix of the camera.
`set_world_poses`([positions, orientations, ...])	Set the pose of the camera w.r.t.
`set_world_poses_from_view`(eyes, targets[, ...])	Set the poses of the camera from the eye position and look-at target position.

UNSUPPORTED_TYPES: ClassVar[set[str]] = {'bounding_box_2d_loose', 'bounding_box_2d_loose_fast', 'bounding_box_2d_tight', 'bounding_box_2d_tight_fast', 'bounding_box_3d', 'bounding_box_3d_fast', 'instance_id_segmentation', 'instance_id_segmentation_fast', 'instance_segmentation', 'instance_segmentation_fast', 'motion_vectors', 'rgb', 'semantic_segmentation', 'skeleton_data'}#: A set of sensor types that are not supported by the ray-caster camera.

__init__(cfg: MultiMeshRayCasterCameraCfg)#

Initializes the camera object.

Parameters:: cfg – The configuration parameters.
Raises:: ValueError – If the provided data types are not supported by the ray-caster camera.

static __new__(cls, *args, **kwargs)#: Create a new instance of an implementation based on the backend.

property data: CameraData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property device: str#: Memory device for computation.

property frame: torch.tensor#: Frame number when the measurement took place.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property image_shape: tuple[int, int]#: A tuple containing (height, width) of the camera sensor.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

meshes: ClassVar[dict[tuple[str, str], wp.Mesh]] = {}#

A dictionary to store warp meshes for raycasting, shared across all instances.

The keys are (prim_path, device) tuples and values are the corresponding warp Mesh objects. Meshes are created lazily for the sensor’s active device, not eagerly for every device. Including the device in the key prevents a mesh created on one device (e.g. CPU) from being reused by a kernel running on a different device (e.g. CUDA) when multiple simulation contexts or tests use different devices in the same Python process.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

set_intrinsic_matrices(matrices: torch.Tensor, focal_length: float = 1.0, env_ids: Sequence[int] | None = None)#

Set the intrinsic matrix of the camera.

Parameters:

matrices – The intrinsic matrices for the camera. Shape is (N, 3, 3).
focal_length – Focal length to use when computing aperture values (in cm). Defaults to 1.0.
env_ids – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

set_world_poses(positions: torch.Tensor | None = None, orientations: torch.Tensor | None = None, env_ids: Sequence[int] | None = None, convention: Literal['opengl', 'ros', 'world'] = 'ros')#

Set the pose of the camera w.r.t. the world frame using specified convention.

Since different fields use different conventions for camera orientations, the method allows users to set the camera poses in the specified convention. Possible conventions are:

"opengl" - forward axis: -Z - up axis +Y - Offset is applied in the OpenGL (Usd.Camera) convention
"ros" - forward axis: +Z - up axis -Y - Offset is applied in the ROS convention
"world" - forward axis: +X - up axis +Z - Offset is applied in the World Frame convention

See isaaclab.utils.math.convert_camera_frame_orientation_convention() for more details on the conventions.

Parameters:

positions – The cartesian coordinates (in meters). Shape is (N, 3). Defaults to None, in which case the camera position in not changed.
orientations – The quaternion orientation in (x, y, z, w). Shape is (N, 4). Defaults to None, in which case the camera orientation in not changed.
env_ids – A sensor ids to manipulate. Defaults to None, which means all sensor indices.
convention – The convention in which the poses are fed. Defaults to “ros”.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.

set_world_poses_from_view(eyes: torch.Tensor, targets: torch.Tensor, env_ids: Sequence[int] | None = None)#

Set the poses of the camera from the eye position and look-at target position.

Parameters:

eyes – The positions of the camera’s eye. Shape is (N, 3).
targets – The target locations to look at. Shape is (N, 3).
env_ids – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.
NotImplementedError – If the stage up-axis is not “Y” or “Z”.

cfg: MultiMeshRayCasterCameraCfg#: The configuration parameters.

class isaaclab.sensors.MultiMeshRayCasterCameraCfg[source]#

Bases: RayCasterCameraCfg, MultiMeshRayCasterCfg

Configuration for the multi-mesh ray-cast camera sensor.

Attributes:

`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`spawn`	Spawn configuration for the sensor Xform prim.
`mesh_prim_paths`	The list of mesh primitive paths to ray cast against.
`offset`	The offset pose of the sensor's frame from the sensor's parent frame.
`ray_alignment`	Specify in what frame the rays are projected onto the ground.
`pattern_cfg`	The pattern that defines the local ray starting positions and directions in a pinhole camera pattern.
`max_distance`	Maximum distance (in meters) from the sensor to ray cast to.
`drift_range`	The range of drift (in meters) to add to the ray starting positions (xyz) in world frame.
`ray_cast_drift_range`	The range of drift (in meters) to add to the projected ray points in local projection frame.
`visualizer_cfg`	The configuration object for the visualization markers.
`update_mesh_ids`	Whether to update the mesh ids of the ray hits in the `data` container.
`reference_meshes`	Whether to reference duplicated meshes instead of loading each one separately into memory.
`data_types`	List of sensor names/types to enable for the camera.
`depth_clipping_behavior`	Clipping behavior for the camera for values exceed the maximum value.

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

spawn: SensorFrameCfg | None#

Spawn configuration for the sensor Xform prim.

A plain USD Xform is created at prim_path before initialization, matching the pattern used by CameraCfg (which spawns a Camera prim). The prim_path can be either:

A new child path under a parent link (e.g. {ENV_REGEX_NS}/Robot/base).
A physics body path (e.g. {ENV_REGEX_NS}/Robot/base). In this case, the sensor will automatically create a child Xform at {prim_path}.

If None, the prim at prim_path must already exist on the USD stage and must not be a physics body.

mesh_prim_paths: list[str]#

The list of mesh primitive paths to ray cast against.

If an entry is a string, it is internally converted to RaycastTargetCfg with track_mesh_transforms disabled. These settings ensure backwards compatibility with the default raycaster.

offset: OffsetCfg#: The offset pose of the sensor’s frame from the sensor’s parent frame. Defaults to identity.

ray_alignment: Literal['base', 'yaw', 'world']#

Specify in what frame the rays are projected onto the ground. Default is “base”.

The options are:

base if the rays’ starting positions and directions track the full root position and orientation.
yaw if the rays’ starting positions track root position and the yaw component of the orientation, while ray directions remain fixed in world frame. This is useful for ray-casting height maps where the scan footprint should follow the body heading without tilting when the body pitches or rolls.
world if rays’ starting positions and directions are always fixed. This is useful in combination with a mapping package on the robot and querying ray-casts in a global frame.

pattern_cfg: PinholeCameraPatternCfg#: The pattern that defines the local ray starting positions and directions in a pinhole camera pattern.

max_distance: float#: Maximum distance (in meters) from the sensor to ray cast to. Defaults to 1e6.

drift_range: tuple[float, float]#

The range of drift (in meters) to add to the ray starting positions (xyz) in world frame. Defaults to (0.0, 0.0).

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

ray_cast_drift_range: dict[str, tuple[float, float]]#

The range of drift (in meters) to add to the projected ray points in local projection frame. Defaults to a dictionary with zero drift for each x, y and z axis.

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

visualizer_cfg: VisualizationMarkersCfg#: The configuration object for the visualization markers. Defaults to RAY_CASTER_MARKER_CFG.

Note

This attribute is only used when debug visualization is enabled.

update_mesh_ids: bool#: Whether to update the mesh ids of the ray hits in the data container.

reference_meshes: bool#

Whether to reference duplicated meshes instead of loading each one separately into memory. Defaults to True.

When enabled, the raycaster parses all meshes in all environments, but reuses references for duplicates instead of storing multiple copies. This reduces memory footprint.

data_types: list[str]#: List of sensor names/types to enable for the camera. Defaults to [“distance_to_image_plane”].

depth_clipping_behavior: Literal['max', 'zero', 'none']#

Clipping behavior for the camera for values exceed the maximum value. Defaults to “none”.

"max": Values are clipped to the maximum value.
"zero": Values are clipped to zero.
"none": No clipping is applied. Values will be returned as inf for missed rays in both distance_to_camera and distance_to_image_plane data types.

Inertia Measurement Unit#

class isaaclab.sensors.Imu[source]#

Bases: FactoryBase, BaseImu

Factory for creating IMU sensor instances.

Attributes:

`data`	Data from the sensor.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

Methods:

`__new__`(cls, args, *kwargs)	Create a new instance of an IMU sensor based on the backend.
`__init__`(cfg)	Initializes the IMU sensor.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

abstract property data: BaseImuData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

static __new__(cls, *args, **kwargs) → BaseImu | PhysXImu | NewtonImu[source]#: Create a new instance of an IMU sensor based on the backend.

__init__(cfg: ImuCfg)#

Initializes the IMU sensor.

Parameters:: cfg – The configuration parameters.

property device: str#: Memory device for computation.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None) → None#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

cfg: ImuCfg#: The configuration parameters.

class isaaclab.sensors.ImuCfg[source]#

Bases: SensorBaseCfg

Configuration for an Inertial Measurement Unit (IMU) sensor.

This configures a sensor that provides the two physical quantities measured by a real IMU: angular velocity (gyroscope) and linear acceleration (accelerometer). For a richer sensor that also provides pose, velocity, and angular acceleration, see PvaCfg.

Classes:

OffsetCfg

The offset pose of the sensor's frame from the sensor's parent frame.

Attributes:

`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`offset`	The offset pose of the sensor's frame from the sensor's parent frame.

class OffsetCfg[source]#

Bases: object

The offset pose of the sensor’s frame from the sensor’s parent frame.

Attributes:

`pos`	Translation w.r.t.
`rot`	Quaternion rotation (x, y, z, w) w.r.t.

pos: tuple[float, float, float]#: Translation w.r.t. the parent frame [m]. Defaults to (0.0, 0.0, 0.0).

rot: tuple[float, float, float, float]#: Quaternion rotation (x, y, z, w) w.r.t. the parent frame. Defaults to (0.0, 0.0, 0.0, 1.0).

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

offset: OffsetCfg#: The offset pose of the sensor’s frame from the sensor’s parent frame. Defaults to identity.

Pose Velocity Acceleration Sensor#

class isaaclab.sensors.Pva[source]#

Bases: FactoryBase, BasePva

Factory for creating PVA sensor instances.

Attributes:

`data`	Data from the sensor.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

Methods:

`__new__`(cls, args, *kwargs)	Create a new instance of a PVA sensor based on the backend.
`__init__`(cfg)	Initializes the PVA sensor.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

abstract property data: BasePvaData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

static __new__(cls, *args, **kwargs) → BasePva | NewtonPva | PhysXPva[source]#: Create a new instance of a PVA sensor based on the backend.

__init__(cfg: PvaCfg)#

Initializes the PVA sensor.

Parameters:: cfg – The configuration parameters.

property device: str#: Memory device for computation.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None) → None#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

cfg: PvaCfg#: The configuration parameters.

class isaaclab.sensors.PvaData[source]#

Factory for creating PVA data instances.

Methods:

`__new__`(cls, args, *kwargs)	Create a new instance of PVA data based on the backend.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.

Attributes:

`ang_acc_b`	PVA frame angular acceleration relative to the world expressed in PVA frame [rad/s^2].
`ang_vel_b`	PVA frame angular velocity relative to the world expressed in PVA frame [rad/s].
`lin_acc_b`	Linear acceleration (coordinate) in the PVA frame [m/s^2].
`lin_vel_b`	PVA frame linear velocity relative to the world expressed in PVA frame [m/s].
`pos_w`	Position of the sensor origin in world frame [m].
`pose_w`	Pose of the sensor origin in world frame [m, unitless].
`projected_gravity_b`	Gravity direction unit vector projected on the PVA frame.
`quat_w`	Orientation of the sensor origin in world frame.

static __new__(cls, *args, **kwargs) → BasePvaData | NewtonPvaData | PhysXPvaData[source]#: Create a new instance of PVA data based on the backend.

abstract property ang_acc_b: ProxyArray#

PVA frame angular acceleration relative to the world expressed in PVA frame [rad/s^2].

Shape is (num_instances,), dtype = wp.vec3f. In torch this resolves to (num_instances, 3).

abstract property ang_vel_b: ProxyArray#

PVA frame angular velocity relative to the world expressed in PVA frame [rad/s].

Shape is (num_instances,), dtype = wp.vec3f. In torch this resolves to (num_instances, 3).

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

abstract property lin_acc_b: ProxyArray#

Linear acceleration (coordinate) in the PVA frame [m/s^2].

Equal to -g in freefall, zero at rest.

Shape is (num_instances,), dtype = wp.vec3f. In torch this resolves to (num_instances, 3).

abstract property lin_vel_b: ProxyArray#

PVA frame linear velocity relative to the world expressed in PVA frame [m/s].

Shape is (num_instances,), dtype = wp.vec3f. In torch this resolves to (num_instances, 3).

abstract property pos_w: ProxyArray#

Position of the sensor origin in world frame [m].

Shape is (num_instances,), dtype = wp.vec3f. In torch this resolves to (num_instances, 3).

abstract property pose_w: ProxyArray | None#

Pose of the sensor origin in world frame [m, unitless].

Shape is (num_instances,), dtype = wp.transformf. In torch this resolves to (num_instances, 7). The pose is provided in (x, y, z, qx, qy, qz, qw) format.

abstract property projected_gravity_b: ProxyArray#

Gravity direction unit vector projected on the PVA frame.

Shape is (num_instances,), dtype = wp.vec3f. In torch this resolves to (num_instances, 3).

abstract property quat_w: ProxyArray#

Orientation of the sensor origin in world frame.

Shape is (num_instances,), dtype = wp.quatf. In torch this resolves to (num_instances, 4). The orientation is provided in (x, y, z, w) format.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

class isaaclab.sensors.PvaCfg[source]#

Bases: SensorBaseCfg

Configuration for a Pose Velocity Acceleration (PVA) sensor.

Classes:

OffsetCfg

The offset pose of the sensor's frame from the sensor's parent frame.

Attributes:

`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`offset`	The offset pose of the sensor's frame from the sensor's parent frame.
`visualizer_cfg`	The configuration object for the visualization markers.

class OffsetCfg[source]#

Bases: object

The offset pose of the sensor’s frame from the sensor’s parent frame.

Attributes:

`pos`	Translation w.r.t.
`rot`	Quaternion rotation (x, y, z, w) w.r.t.

pos: tuple[float, float, float]#: Translation w.r.t. the parent frame [m]. Defaults to (0.0, 0.0, 0.0).

rot: tuple[float, float, float, float]#: Quaternion rotation (x, y, z, w) w.r.t. the parent frame. Defaults to (0.0, 0.0, 0.0, 1.0).

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

offset: OffsetCfg#: The offset pose of the sensor’s frame from the sensor’s parent frame. Defaults to identity.

visualizer_cfg: VisualizationMarkersCfg#

The configuration object for the visualization markers. Defaults to RED_ARROW_X_MARKER_CFG.

This attribute is only used when debug visualization is enabled.

Joint Wrench Sensor#

class isaaclab.sensors.JointWrenchSensor[source]#

Bases: FactoryBase, BaseJointWrenchSensor

Factory for creating joint-wrench sensor instances.

Attributes:

`data`	The sensor data container, populated after simulation initialization.
`body_names`	Ordered names of the bodies whose incoming joint wrench is reported.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`num_bodies`	Number of bodies whose incoming joint wrench is reported.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

Methods:

`__new__`(cls, args, *kwargs)	Create a new instance of a joint-wrench sensor based on the backend.
`__init__`(cfg)	Initialize the joint wrench sensor.
`find_bodies`(name_keys[, preserve_order])	Find reported bodies based on name keys.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

abstract property data: BaseJointWrenchSensorData#: The sensor data container, populated after simulation initialization.

static __new__(cls, *args, **kwargs) → BaseJointWrenchSensor | PhysXJointWrenchSensor | NewtonJointWrenchSensor[source]#: Create a new instance of a joint-wrench sensor based on the backend.

__init__(cfg: JointWrenchSensorCfg)#

Initialize the joint wrench sensor.

Parameters:: cfg – The configuration parameters.

abstract property body_names: list[str]#: Ordered names of the bodies whose incoming joint wrench is reported.

property device: str#: Memory device for computation.

find_bodies(name_keys: str | Sequence[str], preserve_order: bool = False) → tuple[list[int], list[str]]#

Find reported bodies based on name keys.

Parameters:

name_keys – A regular expression or list of regular expressions to match the body names.
preserve_order – Whether to preserve the order of the name keys in the output. Defaults to False.

Returns:

The matching body indices and names.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

property num_bodies: int#: Number of bodies whose incoming joint wrench is reported.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None) → None#

Resets the sensor internals.

Parameters:

env_ids – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

cfg: JointWrenchSensorCfg#: The configuration parameters.

class isaaclab.sensors.JointWrenchSensorData[source]#

Factory for creating joint-wrench sensor data instances.

Methods:

`__new__`(cls, args, *kwargs)	Create a new instance of joint-wrench sensor data based on the backend.
`get_registry_keys`()	Returns a list of registered backend names.
`register`(name, sub_class)	Register a new implementation class.

Attributes:

`force`	Linear component of the joint reaction wrench [N].
`torque`	Angular component of the joint reaction wrench [N·m].

static __new__(cls, *args, **kwargs) → BaseJointWrenchSensorData | PhysXJointWrenchSensorData | NewtonJointWrenchSensorData[source]#: Create a new instance of joint-wrench sensor data based on the backend.

abstract property force: ProxyArray | None#

Linear component of the joint reaction wrench [N].

Expressed in the frame selected by convention. Shape is (num_envs, num_bodies), dtype wp.vec3f. In torch this resolves to (num_envs, num_bodies, 3). None before the simulation is initialized.

classmethod get_registry_keys() → list[str]#: Returns a list of registered backend names.

classmethod register(name: str, sub_class) → None#: Register a new implementation class.

abstract property torque: ProxyArray | None#

Angular component of the joint reaction wrench [N·m].

Expressed in the frame selected by convention. Shape is (num_envs, num_bodies), dtype wp.vec3f. In torch this resolves to (num_envs, num_bodies, 3). None before the simulation is initialized.

class isaaclab.sensors.JointWrenchSensorCfg[source]#

Bases: SensorBaseCfg

Configuration for a joint reaction wrench sensor.

Attributes:

`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`convention`	Coordinate convention for the reported wrench.

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

convention: Literal['incoming_joint_frame']#

Coordinate convention for the reported wrench. Defaults to "incoming_joint_frame".

"incoming_joint_frame" — child-side joint frame, child-side joint anchor as reference point. Matches what a real 6-axis F/T sensor mounted at the joint would measure. Backends convert their native solver outputs to this convention.

isaaclab.sensors

Contents

isaaclab.sensors#

Sensor Base#

USD Camera#

Contact Sensor#

Frame Transformer#

Ray-Cast Sensor#

Ray-Cast Camera#

Multi-Mesh Ray-Cast Sensor#

Multi-Mesh Ray-Cast Camera#

Inertia Measurement Unit#

Pose Velocity Acceleration Sensor#

Joint Wrench Sensor#