isaaclab_newton.sensors

isaaclab_newton.sensors#

Sub-package containing Newton-specific sensor implementations.

Classes

`NewtonRaycastSensor`	Ray-cast sensor that queries the whole Newton scene through the model's shape BVH.
`NewtonRaycastSensorCfg`	Configuration for the Newton BVH ray-cast sensor.
`NewtonRaycastSensorData`	Data container for the Newton BVH ray-cast sensor.
`LegacyRayCaster`	Legacy Newton ray caster that queries one configured Warp mesh.
`LegacyRayCasterCamera`	Legacy Newton ray-caster camera backed by configured Warp meshes.
`LegacyMultiMeshRayCaster`	Legacy Newton ray caster for configured static and dynamic Warp meshes.
`LegacyMultiMeshRayCasterCamera`	Legacy Newton ray-caster camera for configured Warp meshes.

Ray Caster#

class isaaclab_newton.sensors.NewtonRaycastSensor[source]#

Bases: _NewtonRayCasterPoseMixin, BaseRayCaster

Ray-cast sensor that queries the whole Newton scene through the model’s shape BVH.

Rays are cast with newton.intersect_ray() against every collision shape in the sensor’s own world plus the global world (e.g. terrain), so dynamic bodies are hit without configuring target meshes. The full update (sensor pose, ray transform, BVH query, hit resolve) is registered as a task with NewtonManager, which owns the shared BVH refit and sensor execution graph.

Attributes:

`cfg`	The configuration parameters.
`data`	Data from the sensor.
`ray_starts_w`	World-frame ray start positions as of the last update [m].
`ray_directions_w`	World-frame ray directions (unit vectors) as of the last update.
`count`	Number of resolved Newton sites tracked as sensor frames.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`meshes`	A dictionary to store warp meshes for raycasting, shared across all instances.
`num_instances`	Number of instances of the sensor.

Methods:

`__init__`(cfg)	Register the sensor site before Newton model finalization.
`get_world_poses`([indices])	Return world poses for legacy camera helpers.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

cfg: NewtonRaycastSensorCfg#: The configuration parameters.

__init__(cfg: NewtonRaycastSensorCfg)[source]#: Register the sensor site before Newton model finalization.

property data: NewtonRaycastSensorData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property ray_starts_w: ProxyArray#

World-frame ray start positions as of the last update [m].

Shape is (N, B), dtype wp.vec3f. In torch this resolves to (N, B, 3).

property ray_directions_w: ProxyArray#

World-frame ray directions (unit vectors) as of the last update.

Shape is (N, B), dtype wp.vec3f. In torch this resolves to (N, B, 3).

property count: int#: Number of resolved Newton sites tracked as sensor frames.

property device: str#: Memory device for computation.

get_world_poses(indices=None) → tuple[ProxyArray, ProxyArray]#: Return world poses for legacy camera helpers.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

meshes: ClassVar[dict[tuple[str, str], wp.Mesh]] = {}#

A dictionary to store warp meshes for raycasting, shared across all instances.

The keys are (prim_path, device) tuples and values are the corresponding warp Mesh objects. Meshes are created lazily for the sensor’s active device, not eagerly for every device. Including the device in the key prevents a mesh created on one device (e.g. CPU) from being reused by a kernel running on a different device (e.g. CUDA) when multiple simulation contexts or tests use different devices in the same Python process.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)#

Resets the sensor internals.

Parameters:

env_ids¶ – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask¶ – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis¶ – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

class isaaclab_newton.sensors.NewtonRaycastSensorCfg[source]#

Bases: RayCasterCfg

Configuration for the Newton BVH ray-cast sensor.

Unlike RayCasterCfg, no target meshes are configured: rays are cast against every collision shape in the Newton scene through the model’s shape BVH, including dynamic bodies.

Attributes:

`class_type`	The associated sensor class.
`cloning_contexts`	Cloning contexts for this sensor.
`prim_path`	Prim path (or expression) to the sensor.
`update_period`	Update period of the sensor buffers (in seconds).
`debug_vis`	Whether to visualize the sensor.
`spawn`	Spawn configuration for the sensor Xform prim.
`mesh_prim_paths`	Unused.
`offset`	The offset pose of the sensor's frame from the sensor's parent frame.
`ray_alignment`	Specify in what frame the rays are projected onto the ground.
`pattern_cfg`	The pattern that defines the local ray starting positions and directions.
`max_distance`	Maximum distance (in meters) from the sensor to ray cast to.
`global_world_only`	Cast rays against the global world only.
`drift_range`	The range of drift (in meters) to add to the ray starting positions (xyz) in world frame.
`ray_cast_drift_range`	The range of drift (in meters) to add to the projected ray points in local projection frame.
`visualizer_cfg`	The configuration object for the visualization markers.

class_type: type[NewtonRaycastSensor] | str#

The associated sensor class.

The class should inherit from isaaclab.sensors.sensor_base.SensorBase.

cloning_contexts: tuple[str | type, ...] | None#

Cloning contexts for this sensor. Defaults to USD-only cloning.

Sensors carry no physics of their own; see cloning_contexts.

prim_path: str#: Prim path (or expression) to the sensor.

Note

The expression can contain the environment namespace regex {ENV_REGEX_NS} which will be replaced with the environment namespace.

Example: {ENV_REGEX_NS}/Robot/sensor will be replaced with /World/envs/env_.*/Robot/sensor.

update_period: float#: Update period of the sensor buffers (in seconds). Defaults to 0.0 (update every step).

debug_vis: bool#: Whether to visualize the sensor. Defaults to False.

spawn: SensorFrameCfg | None#

Spawn configuration for the sensor Xform prim.

A plain USD Xform is created at prim_path before initialization, matching the pattern used by CameraCfg (which spawns a Camera prim). The prim_path can be either:

A new child path under a parent link (e.g. {ENV_REGEX_NS}/Robot/base).
A physics body path (e.g. {ENV_REGEX_NS}/Robot/base). In this case, the sensor will automatically create a child Xform at {prim_path}.

If None, the prim at prim_path must already exist on the USD stage and must not be a physics body.

mesh_prim_paths: list[str]#: Unused. Rays hit every shape in the Newton scene BVH.

offset: OffsetCfg#: The offset pose of the sensor’s frame from the sensor’s parent frame. Defaults to identity.

ray_alignment: Literal['base', 'yaw', 'world']#

Specify in what frame the rays are projected onto the ground. Default is “base”.

The options are:

base if the rays’ starting positions and directions track the full root position and orientation.
yaw if the rays’ starting positions track root position and the yaw component of the orientation, while ray directions remain fixed in world frame. This is useful for ray-casting height maps where the scan footprint should follow the body heading without tilting when the body pitches or rolls.
world if rays’ starting positions and directions are always fixed. This is useful in combination with a mapping package on the robot and querying ray-casts in a global frame.

pattern_cfg: PatternBaseCfg#: The pattern that defines the local ray starting positions and directions.

max_distance: float#: Maximum distance (in meters) from the sensor to ray cast to. Defaults to 1e6.

global_world_only: bool#

Cast rays against the global world only. Defaults to False.

Newton backend only (ignored by the PhysX and OvPhysX backends, which ray cast against the meshes in mesh_prim_paths). The global world holds shapes shared by all environments (e.g. terrain). When False, rays additionally hit the shapes of the sensor’s own environment — including the sensor’s carrier body when it lies in the ray path; use offset to start the rays outside the carrier’s geometry in that case.

drift_range: tuple[float, float]#

The range of drift (in meters) to add to the ray starting positions (xyz) in world frame. Defaults to (0.0, 0.0).

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

ray_cast_drift_range: dict[str, tuple[float, float]]#

The range of drift (in meters) to add to the projected ray points in local projection frame. Defaults to a dictionary with zero drift for each x, y and z axis.

For floating base robots, this is useful for simulating drift in the robot’s pose estimation.

visualizer_cfg: VisualizationMarkersCfg#: The configuration object for the visualization markers. Defaults to RAY_CASTER_MARKER_CFG.

Note

This attribute is only used when debug visualization is enabled.

class isaaclab_newton.sensors.NewtonRaycastSensorData[source]#

Bases: RayCasterData

Data container for the Newton BVH ray-cast sensor.

Extends RayCasterData with hit distances and surface normals, which the Newton BVH query provides at no extra cost.

Attributes:

`ray_distances`	Ray hit distances from the ray origins [m].
`ray_normals_w`	Surface normals at the ray hit points in world frame.

Methods:

create_buffers(num_envs, num_rays, device)

Create internal warp buffers and their ProxyArray wrappers.

property ray_distances: ProxyArray#

Ray hit distances from the ray origins [m].

Shape is (N, B), dtype wp.float32, where N is the number of sensors and B is the number of rays per sensor. Contains inf for missed hits.

property ray_normals_w: ProxyArray#

Surface normals at the ray hit points in world frame.

Shape is (N, B), dtype wp.vec3f. In torch this resolves to (N, B, 3). Contains inf for missed hits.

create_buffers(num_envs: int, num_rays: int, device: str) → None[source]#

Create internal warp buffers and their ProxyArray wrappers.

Parameters:

num_envs¶ – Number of environments / sensors.
num_rays¶ – Number of rays per sensor.
device¶ – Device for tensor storage.

Legacy Ray Casters#

class isaaclab_newton.sensors.LegacyRayCaster[source]#

Bases: _LegacyNewtonRayCasterMixin, BaseRayCaster

Legacy Newton ray caster that queries one configured Warp mesh.

Methods:

`__init__`(cfg)	Register the sensor site before Newton model finalization.
`get_world_poses`([indices])	Return world poses for legacy camera helpers.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

Attributes:

`count`	Number of resolved Newton sites tracked as sensor frames.
`data`	Data from the sensor.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`meshes`	A dictionary to store warp meshes for raycasting, shared across all instances.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

__init__(cfg)#: Register the sensor site before Newton model finalization.

property count: int#: Number of resolved Newton sites tracked as sensor frames.

property data: RayCasterData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property device: str#: Memory device for computation.

get_world_poses(indices=None) → tuple[ProxyArray, ProxyArray]#: Return world poses for legacy camera helpers.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

meshes: ClassVar[dict[tuple[str, str], wp.Mesh]] = {}#

A dictionary to store warp meshes for raycasting, shared across all instances.

The keys are (prim_path, device) tuples and values are the corresponding warp Mesh objects. Meshes are created lazily for the sensor’s active device, not eagerly for every device. Including the device in the key prevents a mesh created on one device (e.g. CPU) from being reused by a kernel running on a different device (e.g. CUDA) when multiple simulation contexts or tests use different devices in the same Python process.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)#

Resets the sensor internals.

Parameters:

env_ids¶ – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask¶ – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis¶ – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

cfg: RayCasterCfg#: The configuration parameters.

class isaaclab_newton.sensors.LegacyRayCasterCamera[source]#

Bases: _LegacyNewtonRayCasterMixin, BaseRayCasterCamera

Legacy Newton ray-caster camera backed by configured Warp meshes.

Attributes:

`UNSUPPORTED_TYPES`	A set of sensor types that are not supported by the ray-caster camera.
`count`	Number of resolved Newton sites tracked as sensor frames.
`data`	Data from the sensor.
`device`	Memory device for computation.
`frame`	Frame number when the measurement took place.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`image_shape`	A tuple containing (height, width) of the camera sensor.
`is_initialized`	Whether the sensor is initialized.
`meshes`	A dictionary to store warp meshes for raycasting, shared across all instances.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

Methods:

`__init__`(cfg)	Register the sensor site before Newton model finalization.
`get_world_poses`([indices])	Return world poses for legacy camera helpers.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.
`set_intrinsic_matrices`(matrices[, ...])	Set the intrinsic matrix of the camera.
`set_world_poses`([positions, orientations, ...])	Set the pose of the camera w.r.t.
`set_world_poses_from_view`(eyes, targets[, ...])	Set the poses of the camera from the eye position and look-at target position.

UNSUPPORTED_TYPES: ClassVar[set[str]] = {'bounding_box_2d_loose', 'bounding_box_2d_loose_fast', 'bounding_box_2d_tight', 'bounding_box_2d_tight_fast', 'bounding_box_3d', 'bounding_box_3d_fast', 'instance_id_segmentation', 'instance_id_segmentation_fast', 'instance_segmentation', 'motion_vectors', 'rgb', 'semantic_segmentation', 'skeleton_data'}#: A set of sensor types that are not supported by the ray-caster camera.

__init__(cfg)#: Register the sensor site before Newton model finalization.

property count: int#: Number of resolved Newton sites tracked as sensor frames.

property data: CameraData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property device: str#: Memory device for computation.

property frame: torch.tensor#: Frame number when the measurement took place.

get_world_poses(indices=None) → tuple[ProxyArray, ProxyArray]#: Return world poses for legacy camera helpers.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property image_shape: tuple[int, int]#: A tuple containing (height, width) of the camera sensor.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

meshes: ClassVar[dict[tuple[str, str], wp.Mesh]] = {}#

A dictionary to store warp meshes for raycasting, shared across all instances.

The keys are (prim_path, device) tuples and values are the corresponding warp Mesh objects. Meshes are created lazily for the sensor’s active device, not eagerly for every device. Including the device in the key prevents a mesh created on one device (e.g. CPU) from being reused by a kernel running on a different device (e.g. CUDA) when multiple simulation contexts or tests use different devices in the same Python process.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)#

Resets the sensor internals.

Parameters:

env_ids¶ – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask¶ – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis¶ – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

set_intrinsic_matrices(matrices: torch.Tensor, focal_length: float = 1.0, env_ids: Sequence[int] | None = None)#

Set the intrinsic matrix of the camera.

Parameters:

matrices¶ – The intrinsic matrices for the camera. Shape is (N, 3, 3).
focal_length¶ – Focal length to use when computing aperture values (in cm). Defaults to 1.0.
env_ids¶ – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

set_world_poses(positions: torch.Tensor | None = None, orientations: torch.Tensor | None = None, env_ids: Sequence[int] | None = None, convention: Literal['opengl', 'ros', 'world'] = 'ros')#

Set the pose of the camera w.r.t. the world frame using specified convention.

Since different fields use different conventions for camera orientations, the method allows users to set the camera poses in the specified convention. Possible conventions are:

"opengl" - forward axis: -Z - up axis +Y - Offset is applied in the OpenGL (Usd.Camera) convention
"ros" - forward axis: +Z - up axis -Y - Offset is applied in the ROS convention
"world" - forward axis: +X - up axis +Z - Offset is applied in the World Frame convention

See isaaclab.utils.math.convert_camera_frame_orientation_convention() for more details on the conventions.

Parameters:

positions¶ – The cartesian coordinates (in meters). Shape is (N, 3). Defaults to None, in which case the camera position in not changed.
orientations¶ – The quaternion orientation in (x, y, z, w). Shape is (N, 4). Defaults to None, in which case the camera orientation in not changed.
env_ids¶ – A sensor ids to manipulate. Defaults to None, which means all sensor indices.
convention¶ – The convention in which the poses are fed. Defaults to “ros”.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.

set_world_poses_from_view(eyes: torch.Tensor, targets: torch.Tensor, env_ids: Sequence[int] | None = None)#

Set the poses of the camera from the eye position and look-at target position.

Parameters:

eyes¶ – The positions of the camera’s eye. Shape is (N, 3).
targets¶ – The target locations to look at. Shape is (N, 3).
env_ids¶ – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.
NotImplementedError – If the stage up-axis is not “Y” or “Z”.

cfg: RayCasterCameraCfg#: The configuration parameters.

class isaaclab_newton.sensors.LegacyMultiMeshRayCaster[source]#

Bases: _LegacyNewtonRayCasterMixin, BaseMultiMeshRayCaster

Legacy Newton ray caster for configured static and dynamic Warp meshes.

Methods:

`__init__`(cfg)	Register the sensor site before Newton model finalization.
`get_world_poses`([indices])	Return world poses for legacy camera helpers.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.

Attributes:

`count`	Number of resolved Newton sites tracked as sensor frames.
`data`	Data from the sensor.
`device`	Memory device for computation.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`is_initialized`	Whether the sensor is initialized.
`meshes`	A dictionary to store warp meshes for raycasting, shared across all instances.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

__init__(cfg)#: Register the sensor site before Newton model finalization.

property count: int#: Number of resolved Newton sites tracked as sensor frames.

property data: MultiMeshRayCasterData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property device: str#: Memory device for computation.

get_world_poses(indices=None) → tuple[ProxyArray, ProxyArray]#: Return world poses for legacy camera helpers.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

meshes: ClassVar[dict[tuple[str, str], wp.Mesh]] = {}#

A dictionary to store warp meshes for raycasting, shared across all instances.

The keys are (prim_path, device) tuples and values are the corresponding warp Mesh objects. Meshes are created lazily for the sensor’s active device, not eagerly for every device. Including the device in the key prevents a mesh created on one device (e.g. CPU) from being reused by a kernel running on a different device (e.g. CUDA) when multiple simulation contexts or tests use different devices in the same Python process.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)#

Resets the sensor internals.

Parameters:

env_ids¶ – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask¶ – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis¶ – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

cfg: MultiMeshRayCasterCfg#: The configuration parameters.

class isaaclab_newton.sensors.LegacyMultiMeshRayCasterCamera[source]#

Bases: _LegacyNewtonRayCasterMixin, BaseMultiMeshRayCasterCamera

Legacy Newton ray-caster camera for configured Warp meshes.

Attributes:

`UNSUPPORTED_TYPES`	A set of sensor types that are not supported by the ray-caster camera.
`count`	Number of resolved Newton sites tracked as sensor frames.
`data`	Data from the sensor.
`device`	Memory device for computation.
`frame`	Frame number when the measurement took place.
`has_debug_vis_implementation`	Whether the sensor has a debug visualization implemented.
`image_shape`	A tuple containing (height, width) of the camera sensor.
`is_initialized`	Whether the sensor is initialized.
`meshes`	A dictionary to store warp meshes for raycasting, shared across all instances.
`num_instances`	Number of instances of the sensor.
`cfg`	The configuration parameters.

Methods:

`__init__`(cfg)	Register the sensor site before Newton model finalization.
`get_world_poses`([indices])	Return world poses for legacy camera helpers.
`reset`([env_ids, env_mask])	Resets the sensor internals.
`set_debug_vis`(debug_vis)	Sets whether to visualize the sensor data.
`set_intrinsic_matrices`(matrices[, ...])	Set the intrinsic matrix of the camera.
`set_world_poses`([positions, orientations, ...])	Set the pose of the camera w.r.t.
`set_world_poses_from_view`(eyes, targets[, ...])	Set the poses of the camera from the eye position and look-at target position.

UNSUPPORTED_TYPES: ClassVar[set[str]] = {'bounding_box_2d_loose', 'bounding_box_2d_loose_fast', 'bounding_box_2d_tight', 'bounding_box_2d_tight_fast', 'bounding_box_3d', 'bounding_box_3d_fast', 'instance_id_segmentation', 'instance_id_segmentation_fast', 'instance_segmentation', 'motion_vectors', 'rgb', 'semantic_segmentation', 'skeleton_data'}#: A set of sensor types that are not supported by the ray-caster camera.

__init__(cfg)#: Register the sensor site before Newton model finalization.

property count: int#: Number of resolved Newton sites tracked as sensor frames.

property data: CameraData#

Data from the sensor.

This property is only updated when the user tries to access the data. This is done to avoid unnecessary computation when the sensor data is not used.

For updating the sensor when this property is accessed, you can use the following code snippet in your sensor implementation:

# update sensors if needed
self._update_outdated_buffers()
# return the data (where `_data` is the data for the sensor)
return self._data

property device: str#: Memory device for computation.

property frame: torch.tensor#: Frame number when the measurement took place.

get_world_poses(indices=None) → tuple[ProxyArray, ProxyArray]#: Return world poses for legacy camera helpers.

property has_debug_vis_implementation: bool#: Whether the sensor has a debug visualization implemented.

property image_shape: tuple[int, int]#: A tuple containing (height, width) of the camera sensor.

property is_initialized: bool#

Whether the sensor is initialized.

Returns True if the sensor is initialized, False otherwise.

meshes: ClassVar[dict[tuple[str, str], wp.Mesh]] = {}#

A dictionary to store warp meshes for raycasting, shared across all instances.

The keys are (prim_path, device) tuples and values are the corresponding warp Mesh objects. Meshes are created lazily for the sensor’s active device, not eagerly for every device. Including the device in the key prevents a mesh created on one device (e.g. CPU) from being reused by a kernel running on a different device (e.g. CUDA) when multiple simulation contexts or tests use different devices in the same Python process.

property num_instances: int#

Number of instances of the sensor.

This is equal to the number of sensors per environment multiplied by the number of environments.

reset(env_ids: Sequence[int] | None = None, env_mask: wp.array | None = None)#

Resets the sensor internals.

Parameters:

env_ids¶ – The environment indices to reset. Defaults to None, in which case all environments are reset.
env_mask¶ – A boolean warp array indicating which environments to reset. If provided, takes priority over env_ids. Defaults to None.

set_debug_vis(debug_vis: bool) → bool#

Sets whether to visualize the sensor data.

Parameters:: debug_vis¶ – Whether to visualize the sensor data.
Returns:: Whether the debug visualization was successfully set. False if the sensor does not support debug visualization.

set_intrinsic_matrices(matrices: torch.Tensor, focal_length: float = 1.0, env_ids: Sequence[int] | None = None)#

Set the intrinsic matrix of the camera.

Parameters:

matrices¶ – The intrinsic matrices for the camera. Shape is (N, 3, 3).
focal_length¶ – Focal length to use when computing aperture values (in cm). Defaults to 1.0.
env_ids¶ – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

set_world_poses(positions: torch.Tensor | None = None, orientations: torch.Tensor | None = None, env_ids: Sequence[int] | None = None, convention: Literal['opengl', 'ros', 'world'] = 'ros')#

Set the pose of the camera w.r.t. the world frame using specified convention.

Since different fields use different conventions for camera orientations, the method allows users to set the camera poses in the specified convention. Possible conventions are:

"opengl" - forward axis: -Z - up axis +Y - Offset is applied in the OpenGL (Usd.Camera) convention
"ros" - forward axis: +Z - up axis -Y - Offset is applied in the ROS convention
"world" - forward axis: +X - up axis +Z - Offset is applied in the World Frame convention

See isaaclab.utils.math.convert_camera_frame_orientation_convention() for more details on the conventions.

Parameters:

positions¶ – The cartesian coordinates (in meters). Shape is (N, 3). Defaults to None, in which case the camera position in not changed.
orientations¶ – The quaternion orientation in (x, y, z, w). Shape is (N, 4). Defaults to None, in which case the camera orientation in not changed.
env_ids¶ – A sensor ids to manipulate. Defaults to None, which means all sensor indices.
convention¶ – The convention in which the poses are fed. Defaults to “ros”.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.

set_world_poses_from_view(eyes: torch.Tensor, targets: torch.Tensor, env_ids: Sequence[int] | None = None)#

Set the poses of the camera from the eye position and look-at target position.

Parameters:

eyes¶ – The positions of the camera’s eye. Shape is (N, 3).
targets¶ – The target locations to look at. Shape is (N, 3).
env_ids¶ – A sensor ids to manipulate. Defaults to None, which means all sensor indices.

Raises:

RuntimeError – If the camera prim is not set. Need to call initialize() method first.
NotImplementedError – If the stage up-axis is not “Y” or “Z”.

cfg: MultiMeshRayCasterCameraCfg#: The configuration parameters.

isaaclab_newton.sensors

Contents

isaaclab_newton.sensors#

Ray Caster#

Legacy Ray Casters#