isaaclab.sensors.ray_caster

Sub-module for Warp-based ray-cast sensor.

The sub-module contains two implementations of the ray-cast sensor:

• isaaclab.sensors.ray_caster.RayCaster: A basic ray-cast sensor that can be used to ray-cast against a single mesh.

• isaaclab.sensors.ray_caster.MultiMeshRayCaster: A multi-mesh ray-cast sensor that can be used to ray-cast against multiple meshes. For these meshes, it tracks their transformations and updates the warp meshes accordingly.

Corresponding camera implementations are also provided for each of the sensor implementations. Internally, they perform the same ray-casting operations as the sensor implementations, but return the results as images.

Classes

The following classes are part of the public isaaclab.sensors.ray_caster API.

BaseMultiMeshRayCaster	A multi-mesh ray-casting sensor.
BaseMultiMeshRayCasterCamera	A multi-mesh ray-casting camera sensor.
BaseRayCaster	A ray-casting sensor.
BaseRayCasterCamera	A ray-casting camera sensor.

class isaaclab.sensors.ray_caster.BaseMultiMeshRayCaster

Bases: BaseRayCaster

A multi-mesh ray-casting sensor.

The ray-caster uses a set of rays to detect collisions with meshes in the scene. The rays are defined in the sensor’s local coordinate frame. The sensor can be configured to ray-cast against a set of meshes with a given ray pattern.

The meshes are parsed from the list of primitive paths provided in the configuration. These are then converted to warp meshes and stored in the meshes dictionary. The ray-caster then ray-casts against these warp meshes using the ray pattern provided in the configuration.

Compared to the default RayCaster, the MultiMeshRayCaster provides additional functionality and flexibility as an extension of the default RayCaster with the following enhancements:

• Raycasting against multiple target types : Supports primitive shapes (spheres, cubes, etc.) as well as arbitrary meshes.

• Dynamic mesh tracking : Keeps track of specified meshes, enabling raycasting against moving parts (e.g., robot links, articulated bodies, or dynamic obstacles).

• Memory-efficient caching : Avoids redundant memory usage by reusing mesh data across environments.

Warning

Known limitation (multi-mesh closest-hit resolution): When two meshes produce a hit at the exact same distance for a given ray, the atomic_min + equality-check pattern in the raycasting kernel is not fully thread-safe. The hit position is always correct, but auxiliary outputs (normals, face IDs, mesh IDs) may originate from different meshes for the affected ray. This requires an exact floating-point tie and is rare in practice. See warp#1058 for upstream progress on a thread-safe atomic_min return value.

Example usage to raycast against the visual meshes of a robot (e.g. ANYmal):

ray_caster_cfg = MultiMeshRayCasterCfg(
    prim_path="{ENV_REGEX_NS}/Robot",
    mesh_prim_paths=[
        "/World/Ground",
        MultiMeshRayCasterCfg.RaycastTargetCfg(prim_expr="{ENV_REGEX_NS}/Robot/LF_[^/]*/visuals"),
        MultiMeshRayCasterCfg.RaycastTargetCfg(prim_expr="{ENV_REGEX_NS}/Robot/RF_[^/]*/visuals"),
        MultiMeshRayCasterCfg.RaycastTargetCfg(prim_expr="{ENV_REGEX_NS}/Robot/LH_[^/]*/visuals"),
        MultiMeshRayCasterCfg.RaycastTargetCfg(prim_expr="{ENV_REGEX_NS}/Robot/RH_[^/]*/visuals"),
        MultiMeshRayCasterCfg.RaycastTargetCfg(prim_expr="{ENV_REGEX_NS}/Robot/base/visuals"),
    ],
    ray_alignment="world",
    pattern_cfg=patterns.GridPatternCfg(resolution=0.02, size=(2.5, 2.5), direction=(0, 0, -1)),
)

Methods:

__init__(cfg)	Initializes the ray-caster object.
__new__(*args, **kwargs)

__init__(cfg: MultiMeshRayCasterCfg)

Initializes the ray-caster object.

Parameters:

cfg – The configuration parameters.

classmethod __new__(*args, **kwargs)

class isaaclab.sensors.ray_caster.BaseMultiMeshRayCasterCamera

Bases: BaseRayCasterCamera, BaseMultiMeshRayCaster

A multi-mesh ray-casting camera sensor.

The ray-caster camera uses a set of rays to get the distances to meshes in the scene. The rays are defined in the sensor’s local coordinate frame. The sensor has the same interface as the isaaclab.sensors.Camera that implements the camera class through USD camera prims. However, this class provides a faster image generation. The sensor converts meshes from the list of primitive paths provided in the configuration to Warp meshes. The camera then ray-casts against these Warp meshes only.

Currently, only the following annotators are supported:

• "distance_to_camera": An image containing the distance to camera optical center.

• "distance_to_image_plane": An image containing distances of 3D points from camera plane along camera’s z-axis.

• "normals": An image containing the local surface normal vectors at each pixel.

Methods:

__init__(cfg)	Initializes the camera object.
__new__(*args, **kwargs)

__init__(cfg: MultiMeshRayCasterCameraCfg)

Initializes the camera object.

Parameters:

cfg – The configuration parameters.

Raises:

ValueError – If the provided data types are not supported by the ray-caster camera.

classmethod __new__(*args, **kwargs)

class isaaclab.sensors.ray_caster.BaseRayCaster

Bases: SensorBase

A ray-casting sensor.

The ray-caster uses a set of rays to detect collisions with meshes in the scene. The rays are defined in the sensor’s local coordinate frame. The sensor can be configured to ray-cast against a set of meshes with a given ray pattern.

The meshes are parsed from the list of primitive paths provided in the configuration. These are then converted to warp meshes and stored in the meshes dictionary. The ray-caster then ray-casts against these warp meshes using the ray pattern provided in the configuration.

Note

Currently, only static meshes are supported. Extending the warp mesh to support dynamic meshes is a work in progress.

Methods:

__init__(cfg)	Initializes the ray-caster object.
__new__(*args, **kwargs)

__init__(cfg: RayCasterCfg)

Initializes the ray-caster object.

Parameters:

cfg – The configuration parameters.

classmethod __new__(*args, **kwargs)

class isaaclab.sensors.ray_caster.BaseRayCasterCamera

Bases: BaseRayCaster

A ray-casting camera sensor.

The ray-caster camera uses a set of rays to get the distances to meshes in the scene. The rays are defined in the sensor’s local coordinate frame. The sensor has the same interface as the isaaclab.sensors.Camera that implements the camera class through USD camera prims. However, this class provides a faster image generation. The sensor converts meshes from the list of primitive paths provided in the configuration to Warp meshes. The camera then ray-casts against these Warp meshes only.

Currently, only the following annotators are supported:

• "distance_to_camera": An image containing the distance to camera optical center.

• "distance_to_image_plane": An image containing distances of 3D points from camera plane along camera’s z-axis.

• "normals": An image containing the local surface normal vectors at each pixel.

Note

Currently, only static meshes are supported. Extending the warp mesh to support dynamic meshes is a work in progress.

Methods:

__init__(cfg)	Initializes the camera object.
__new__(*args, **kwargs)

__init__(cfg: RayCasterCameraCfg)

Initializes the camera object.

Parameters:

cfg – The configuration parameters.

Raises:

ValueError – If the provided data types are not supported by the ray-caster camera.

classmethod __new__(*args, **kwargs)