Ray Caster#
The Ray Caster sensor (and the ray caster camera) are similar to RTX based rendering in that they both involve casting rays. The difference here is that the rays cast by the Ray Caster sensor return strictly collision information along the cast, and the direction of each individual ray can be specified. They do not bounce, nor are they affected by things like materials or opacity. For each ray specified by the sensor, a line is traced along the path of the ray and the location of first collision with the specified mesh is returned. This is the method used by some of our quadruped examples to measure the local height field.
The ray-casting implementation depends on the physics backend. With PhysX and
OV PhysX, configured meshes are loaded into Warp
when the sensor initializes and must remain static. With Newton, the sensor
queries the live scene BVH and therefore sees every collision shape in its
environment, global terrain, and dynamic bodies. Newton ignores
mesh_prim_paths; use
isaaclab_newton.sensors.NewtonRaycastSensorCfg to make that behavior
explicit and to access Newton-specific options.
Using a ray caster sensor requires a pattern and a parent xform to be attached to. The pattern defines how the rays are cast, while the prim properties defines the orientation and position of the sensor (additional offsets can be specified for more exact placement). Isaac Lab supports a number of ray casting pattern configurations, including a generic LIDAR and grid pattern.
@configclass
class RaycasterSensorSceneCfg(InteractiveSceneCfg):
"""Design the scene with sensors on the robot."""
# ground plane
ground = AssetBaseCfg(
prim_path="/World/Ground",
spawn=sim_utils.UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Environments/Terrains/rough_plane.usd",
scale=(1, 1, 1),
),
)
# lights
dome_light = AssetBaseCfg(
prim_path="/World/Light", spawn=sim_utils.DomeLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75))
)
# robot
robot = replace(ANYMAL_C_CFG, prim_path="{ENV_REGEX_NS}/Robot")
ray_caster = RayCasterCfg(
prim_path="{ENV_REGEX_NS}/Robot/base",
update_period=1 / 60,
offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0.5)),
mesh_prim_paths=["/World/Ground"],
ray_alignment="yaw",
pattern_cfg=patterns.LidarPatternCfg(
channels=100, vertical_fov_range=[-90, 90], horizontal_fov_range=[-90, 90], horizontal_res=1.0
),
debug_vis=bool(args_cli.visualizer),
)
Notice that the units on the pattern config is in degrees! Also, we enable visualization here to explicitly show the pattern in the rendering, but this is not required and should be disabled for performance tuning.
Querying the sensor for data can be done at simulation run time like any other sensor.
def run_simulator(sim: sim_utils.SimulationContext, scene: InteractiveScene):
.
.
.
# Simulate physics
while sim.is_running():
.
.
.
# print information from the sensors
print("-------------------------------")
print(scene["ray_caster"])
print("Ray cast hit results: ", scene["ray_caster"].data.ray_hits_w)
-------------------------------
Ray-caster @ '/World/envs/env_.*/Robot/base/lidar_cage':
view type : <class 'isaacsim.core.experimental.prims.xform_prim.XformPrim'>
update period (s) : 0.016666666666666666
number of meshes : 1
number of sensors : 1
number of rays/sensor: 18000
total number of rays : 18000
Ray cast hit results: tensor([[[-0.3698, 0.0357, 0.0000],
[-0.3698, 0.0357, 0.0000],
[-0.3698, 0.0357, 0.0000],
...,
[ inf, inf, inf],
[ inf, inf, inf],
[ inf, inf, inf]]], device='cuda:0')
-------------------------------
Here we can see the data returned by the sensor itself. Notice first that there are 3 closed brackets at the beginning and the end: this is because the data returned is batched by the number of sensors. The ray cast pattern itself has also been flattened, and so the dimensions of the array are [N, B, 3] where N is the number of sensors, B is the number of cast rays in the pattern, and 3 is the dimension of the casting space. Finally, notice that the first several values in this casting pattern are the same: this is because the lidar pattern is spherical and we have specified our FOV to be hemispherical, which includes the poles. In this configuration, the “flattening pattern” becomes apparent: the first 180 entries will be the same because it’s the bottom pole of this hemisphere, and there will be 180 of them because our horizontal FOV is 180 degrees with a resolution of 1 degree.
You can use this script to experiment with pattern configurations and build an intuition about how the data is stored by altering the triggered variable on line 81.
Code for raycaster_sensor.py
1# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
2# All rights reserved.
3#
4# SPDX-License-Identifier: BSD-3-Clause
5
6"""Cast a lidar-style ray pattern against rough terrain."""
7
8import argparse
9from typing import TYPE_CHECKING
10
11import torch
12
13import isaaclab.sim as sim_utils
14from isaaclab.app import add_launcher_args, launch_simulation
15from isaaclab.assets import AssetBaseCfg
16from isaaclab.physics import PhysicsCfg
17from isaaclab.scene import InteractiveSceneCfg
18from isaaclab.sensors.ray_caster import RayCasterCfg, patterns
19from isaaclab.utils import configclass, instantiate, replace
20from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
21
22from isaaclab_assets.robots.anymal import ANYMAL_C_CFG
23
24if TYPE_CHECKING:
25 from isaaclab.scene import InteractiveScene
26
27parser = argparse.ArgumentParser(description="Example on using the raycaster sensor.")
28parser.add_argument("--num_envs", type=int, default=1, help="Number of environments to spawn.")
29parser.add_argument("--log_interval", type=int, default=100, help="Steps between compact sensor summaries.")
30parser.add_argument("--max_steps", type=int, default=-1, help="Stop after this many steps; negative runs forever.")
31parser.add_argument(
32 "--physics",
33 default="isaacsim_physx",
34 choices=["isaacsim_physx"],
35 help="Physics backend.",
36)
37add_launcher_args(parser)
38parser.set_defaults(visualizer=["kit"])
39args_cli = parser.parse_args()
40if args_cli.log_interval < 1:
41 parser.error("--log_interval must be at least 1.")
42if args_cli.max_steps == 0 or args_cli.max_steps < -1:
43 parser.error("--max_steps must be positive or -1.")
44
45
46@configclass
47class RaycasterSensorSceneCfg(InteractiveSceneCfg):
48 """Design the scene with sensors on the robot."""
49
50 # ground plane
51 ground = AssetBaseCfg(
52 prim_path="/World/Ground",
53 spawn=sim_utils.UsdFileCfg(
54 usd_path=f"{ISAAC_NUCLEUS_DIR}/Environments/Terrains/rough_plane.usd",
55 scale=(1, 1, 1),
56 ),
57 )
58
59 # lights
60 dome_light = AssetBaseCfg(
61 prim_path="/World/Light", spawn=sim_utils.DomeLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75))
62 )
63
64 # robot
65 robot = replace(ANYMAL_C_CFG, prim_path="{ENV_REGEX_NS}/Robot")
66
67 ray_caster = RayCasterCfg(
68 prim_path="{ENV_REGEX_NS}/Robot/base",
69 update_period=1 / 60,
70 offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0.5)),
71 mesh_prim_paths=["/World/Ground"],
72 ray_alignment="yaw",
73 pattern_cfg=patterns.LidarPatternCfg(
74 channels=100, vertical_fov_range=[-90, 90], horizontal_fov_range=[-90, 90], horizontal_res=1.0
75 ),
76 debug_vis=bool(args_cli.visualizer),
77 )
78
79
80def run_simulator(sim: sim_utils.SimulationContext, scene: "InteractiveScene") -> None:
81 """Run the simulator."""
82 # Define simulation stepping
83 sim_dt = sim.get_physics_dt()
84 count = 0
85
86 while sim.is_running() and (args_cli.max_steps < 0 or count < args_cli.max_steps):
87 if count % 500 == 0:
88 # reset the scene entities
89 # root state
90 # we offset the root state by the origin since the states are written in simulation world frame
91 # if this is not done, then the robots will be spawned at the (0, 0, 0) of the simulation world
92 root_pose = scene["robot"].data.default_root_pose.torch.clone()
93 root_pose[:, :3] += scene.env_origins
94 scene["robot"].write_root_pose_to_sim_index(root_pose=root_pose)
95 root_vel = scene["robot"].data.default_root_vel.torch.clone()
96 scene["robot"].write_root_velocity_to_sim_index(root_velocity=root_vel)
97 # set joint positions with some noise
98 joint_pos, joint_vel = (
99 scene["robot"].data.default_joint_pos.torch.clone(),
100 scene["robot"].data.default_joint_vel.torch.clone(),
101 )
102 joint_pos += torch.rand_like(joint_pos) * 0.1
103 scene["robot"].write_joint_position_to_sim_index(position=joint_pos)
104 scene["robot"].write_joint_velocity_to_sim_index(velocity=joint_vel)
105 # clear internal buffers
106 scene.reset()
107 print("[INFO]: Resetting robot state...")
108 targets = scene["robot"].data.default_joint_pos.torch
109 scene["robot"].set_joint_position_target_index(target=targets)
110 scene.write_data_to_sim()
111 sim.step()
112 count += 1
113 scene.update(sim_dt)
114
115 if count % args_cli.log_interval == 0:
116 hits = scene["ray_caster"].data.ray_hits_w.torch
117 valid = torch.isfinite(hits).all(dim=-1)
118 print(f"[INFO] step={count} ray hit rate={valid.float().mean().item():.1%}")
119
120
121def main() -> None:
122 """Run the ray-caster example."""
123 with launch_simulation(cfg=PhysicsCfg(), launcher_args=args_cli) as physics_cfg:
124 sim_cfg = sim_utils.SimulationCfg(dt=0.005, device=args_cli.device, physics=physics_cfg)
125 sim = sim_utils.SimulationContext(sim_cfg)
126 sim.set_camera_view(eye=[3.5, 3.5, 3.5], target=[0.0, 0.0, 0.0])
127 scene_cfg = RaycasterSensorSceneCfg(num_envs=args_cli.num_envs, env_spacing=2.0)
128 scene: InteractiveScene = instantiate(scene_cfg)
129 sim.reset()
130 print("[INFO]: Setup complete...")
131 run_simulator(sim, scene)
132
133
134if __name__ == "__main__":
135 main()