Find How Many/What Cameras You Should Train With#

Currently in Isaac Lab, there are several camera types; USD Cameras (standard), Tiled Cameras, and Ray Caster cameras. These camera types differ in functionality and performance. The benchmark_cameras.py script can be used to understand the difference in cameras types, as well to characterize their relative performance at different parameters such as camera quantity, image dimensions, and data types.

This utility is provided so that one easily can find the camera type/parameters that are the most performant while meeting the requirements of the user’s scenario. This utility also helps estimate the maximum number of cameras one can realistically run, assuming that one wants to maximize the number of environments while minimizing step time.

This utility can inject cameras into an existing task from the gym registry, which can be useful for benchmarking cameras in a specific scenario. Also, if you install pynvml, you can let this utility automatically find the maximum numbers of cameras that can run in your task environment up to a certain specified system resource utilization threshold (without training; taking zero actions at each timestep).

This guide accompanies the benchmark_cameras.py script in the source/standalone/benchmarks directory.

Code for benchmark_cameras.py
  1# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
  2# All rights reserved.
  3#
  4# SPDX-License-Identifier: BSD-3-Clause
  5
  6"""
  7This script might help you determine how many cameras your system can realistically run
  8at different desired settings.
  9
 10You can supply different task environments to inject cameras into, or just test a sample scene.
 11Additionally, you can automatically find the maximum amount of cameras you can run a task with
 12through the auto-tune functionality.
 13
 14.. code-block:: bash
 15
 16    # Usage with GUI
 17    ./isaaclab.sh -p source/standalone/benchmarks/benchmark_cameras.py -h
 18
 19    # Usage with headless
 20    ./isaaclab.sh -p source/standalone/benchmarks/benchmark_cameras.py -h --headless
 21
 22"""
 23
 24"""Launch Isaac Sim Simulator first."""
 25
 26import argparse
 27from collections.abc import Callable
 28
 29from omni.isaac.lab.app import AppLauncher
 30
 31# parse the arguments
 32args_cli = argparse.Namespace()
 33
 34parser = argparse.ArgumentParser(description="This script can help you benchmark how many cameras you could run.")
 35
 36"""
 37The following arguments only need to be supplied for when one wishes
 38to try injecting cameras into their environment, and automatically determining
 39the maximum camera count.
 40"""
 41parser.add_argument(
 42    "--task",
 43    type=str,
 44    default=None,
 45    required=False,
 46    help="Supply this argument to spawn cameras within an known manager-based task environment.",
 47)
 48
 49parser.add_argument(
 50    "--autotune",
 51    default=False,
 52    action="store_true",
 53    help=(
 54        "Autotuning is only supported for provided task environments."
 55        " Supply this argument to increase the number of environments until a desired threshold is reached."
 56        "Install pynvml in your environment; ./isaaclab.sh -m pip install pynvml"
 57    ),
 58)
 59
 60parser.add_argument(
 61    "--task_num_cameras_per_env",
 62    type=int,
 63    default=1,
 64    help="The number of cameras per environment to use when using a known task.",
 65)
 66
 67parser.add_argument(
 68    "--use_fabric", action="store_true", default=False, help="Enable fabric and use USD I/O operations."
 69)
 70
 71parser.add_argument(
 72    "--autotune_max_percentage_util",
 73    nargs="+",
 74    type=float,
 75    default=[100.0, 80.0, 80.0, 80.0],
 76    required=False,
 77    help=(
 78        "The system utilization percentage thresholds to reach before an autotune is finished. "
 79        "If any one of these limits are hit, the autotune stops."
 80        "Thresholds are, in order, maximum CPU percentage utilization,"
 81        "maximum RAM percentage utilization, maximum GPU compute percent utilization, "
 82        "amd maximum GPU memory utilization."
 83    ),
 84)
 85
 86parser.add_argument(
 87    "--autotune_max_camera_count", type=int, default=4096, help="The maximum amount of cameras allowed in an autotune."
 88)
 89
 90parser.add_argument(
 91    "--autotune_camera_count_interval",
 92    type=int,
 93    default=25,
 94    help=(
 95        "The number of cameras to try to add to the environment if the current camera count"
 96        " falls within permitted system resource utilization limits."
 97    ),
 98)
 99
100"""
101The following arguments are shared for when injecting cameras into a task environment,
102as well as when creating cameras independent of a task environment.
103"""
104
105parser.add_argument(
106    "--num_tiled_cameras",
107    type=int,
108    default=0,
109    required=False,
110    help="Number of tiled cameras to create. For autotuning, this is how many cameras to start with.",
111)
112
113parser.add_argument(
114    "--num_standard_cameras",
115    type=int,
116    default=0,
117    required=False,
118    help="Number of standard cameras to create. For autotuning, this is how many cameras to start with.",
119)
120
121parser.add_argument(
122    "--num_ray_caster_cameras",
123    type=int,
124    default=0,
125    required=False,
126    help="Number of ray caster cameras to create. For autotuning, this is how many cameras to start with.",
127)
128
129parser.add_argument(
130    "--tiled_camera_data_types",
131    nargs="+",
132    type=str,
133    default=["rgb", "depth"],
134    help="The data types rendered by the tiled camera",
135)
136
137parser.add_argument(
138    "--standard_camera_data_types",
139    nargs="+",
140    type=str,
141    default=["rgb", "distance_to_image_plane", "distance_to_camera"],
142    help="The data types rendered by the standard camera",
143)
144
145parser.add_argument(
146    "--ray_caster_camera_data_types",
147    nargs="+",
148    type=str,
149    default=["distance_to_image_plane"],
150    help="The data types rendered by the ray caster camera.",
151)
152
153parser.add_argument(
154    "--ray_caster_visible_mesh_prim_paths",
155    nargs="+",
156    type=str,
157    default=["/World/ground"],
158    help="WARNING: Ray Caster can currently only cast against a single, static, object",
159)
160
161parser.add_argument(
162    "--convert_depth_to_camera_to_image_plane",
163    action="store_true",
164    default=True,
165    help=(
166        "Enable undistorting from perspective view (distance to camera data_type)"
167        "to orthogonal view (distance to plane data_type) for depth."
168        "This is currently needed to create undisorted depth images/point cloud."
169    ),
170)
171
172parser.add_argument(
173    "--keep_raw_depth",
174    dest="convert_depth_to_camera_to_image_plane",
175    action="store_false",
176    help=(
177        "Disable undistorting from perspective view (distance to camera)"
178        "to orthogonal view (distance to plane data_type) for depth."
179    ),
180)
181
182parser.add_argument(
183    "--height",
184    type=int,
185    default=120,
186    required=False,
187    help="Height in pixels of cameras",
188)
189
190parser.add_argument(
191    "--width",
192    type=int,
193    default=140,
194    required=False,
195    help="Width in pixels of cameras",
196)
197
198parser.add_argument(
199    "--warm_start_length",
200    type=int,
201    default=3,
202    required=False,
203    help=(
204        "Number of steps to run the sim before starting benchmark."
205        "Needed to avoid blank images at the start of the simulation."
206    ),
207)
208
209parser.add_argument(
210    "--experiment_length",
211    type=int,
212    default=15,
213    required=False,
214    help="Number of steps to average over",
215)
216
217# This argument is only used when a task is not provided.
218parser.add_argument(
219    "--num_objects",
220    type=int,
221    default=10,
222    required=False,
223    help="Number of objects to spawn into the scene when not using a known task.",
224)
225
226
227AppLauncher.add_app_launcher_args(parser)
228args_cli = parser.parse_args()
229args_cli.enable_cameras = True
230
231if args_cli.autotune:
232    import pynvml
233
234if len(args_cli.ray_caster_visible_mesh_prim_paths) > 1:
235    print("[WARNING]: Ray Casting is only currently supported for a single, static object")
236# launch omniverse app
237app_launcher = AppLauncher(args_cli)
238simulation_app = app_launcher.app
239
240"""Rest everything follows."""
241
242import gymnasium as gym
243import numpy as np
244import random
245import time
246import torch
247
248import omni.isaac.core.utils.prims as prim_utils
249import psutil
250from omni.isaac.core.utils.stage import create_new_stage
251
252import omni.isaac.lab.sim as sim_utils
253from omni.isaac.lab.assets import RigidObject, RigidObjectCfg
254from omni.isaac.lab.scene.interactive_scene import InteractiveScene
255from omni.isaac.lab.sensors import (
256    Camera,
257    CameraCfg,
258    RayCasterCamera,
259    RayCasterCameraCfg,
260    TiledCamera,
261    TiledCameraCfg,
262    patterns,
263)
264from omni.isaac.lab.utils.math import orthogonalize_perspective_depth, unproject_depth
265
266from omni.isaac.lab_tasks.utils import load_cfg_from_registry
267
268"""
269Camera Creation
270"""
271
272
273def create_camera_base(
274    camera_cfg: type[CameraCfg | TiledCameraCfg],
275    num_cams: int,
276    data_types: list[str],
277    height: int,
278    width: int,
279    prim_path: str | None = None,
280    instantiate: bool = True,
281) -> Camera | TiledCamera | CameraCfg | TiledCameraCfg | None:
282    """Generalized function to create a camera or tiled camera sensor."""
283    # Determine prim prefix based on the camera class
284    name = camera_cfg.class_type.__name__
285
286    if instantiate:
287        # Create the necessary prims
288        for idx in range(num_cams):
289            prim_utils.create_prim(f"/World/{name}_{idx:02d}", "Xform")
290    if prim_path is None:
291        prim_path = f"/World/{name}_.*/{name}"
292    # If valid camera settings are provided, create the camera
293    if num_cams > 0 and len(data_types) > 0 and height > 0 and width > 0:
294        cfg = camera_cfg(
295            prim_path=prim_path,
296            update_period=0,
297            height=height,
298            width=width,
299            data_types=data_types,
300            spawn=sim_utils.PinholeCameraCfg(
301                focal_length=24, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 1e4)
302            ),
303        )
304        if instantiate:
305            return camera_cfg.class_type(cfg=cfg)
306        else:
307            return cfg
308    else:
309        return None
310
311
312def create_tiled_cameras(
313    num_cams: int = 2, data_types: list[str] | None = None, height: int = 100, width: int = 120
314) -> TiledCamera | None:
315    if data_types is None:
316        data_types = ["rgb", "depth"]
317    """Defines the tiled camera sensor to add to the scene."""
318    return create_camera_base(
319        camera_cfg=TiledCameraCfg,
320        num_cams=num_cams,
321        data_types=data_types,
322        height=height,
323        width=width,
324    )
325
326
327def create_cameras(
328    num_cams: int = 2, data_types: list[str] | None = None, height: int = 100, width: int = 120
329) -> Camera | None:
330    """Defines the Standard cameras."""
331    if data_types is None:
332        data_types = ["rgb", "depth"]
333    return create_camera_base(
334        camera_cfg=CameraCfg, num_cams=num_cams, data_types=data_types, height=height, width=width
335    )
336
337
338def create_ray_caster_cameras(
339    num_cams: int = 2,
340    data_types: list[str] = ["distance_to_image_plane"],
341    mesh_prim_paths: list[str] = ["/World/ground"],
342    height: int = 100,
343    width: int = 120,
344    prim_path: str = "/World/RayCasterCamera_.*/RayCaster",
345    instantiate: bool = True,
346) -> RayCasterCamera | RayCasterCameraCfg | None:
347    """Create the raycaster cameras; different configuration than Standard/Tiled camera"""
348    for idx in range(num_cams):
349        prim_utils.create_prim(f"/World/RayCasterCamera_{idx:02d}/RayCaster", "Xform")
350
351    if num_cams > 0 and len(data_types) > 0 and height > 0 and width > 0:
352        cam_cfg = RayCasterCameraCfg(
353            prim_path=prim_path,
354            mesh_prim_paths=mesh_prim_paths,
355            update_period=0,
356            offset=RayCasterCameraCfg.OffsetCfg(pos=(0.0, 0.0, 0.0), rot=(1.0, 0.0, 0.0, 0.0)),
357            data_types=data_types,
358            debug_vis=False,
359            pattern_cfg=patterns.PinholeCameraPatternCfg(
360                focal_length=24.0,
361                horizontal_aperture=20.955,
362                height=480,
363                width=640,
364            ),
365        )
366        if instantiate:
367            return RayCasterCamera(cfg=cam_cfg)
368        else:
369            return cam_cfg
370
371    else:
372        return None
373
374
375def create_tiled_camera_cfg(prim_path: str) -> TiledCameraCfg:
376    """Grab a simple tiled camera config for injecting into task environments."""
377    return create_camera_base(
378        TiledCameraCfg,
379        num_cams=args_cli.num_tiled_cameras,
380        data_types=args_cli.tiled_camera_data_types,
381        width=args_cli.width,
382        height=args_cli.height,
383        prim_path="{ENV_REGEX_NS}/" + prim_path,
384        instantiate=False,
385    )
386
387
388def create_standard_camera_cfg(prim_path: str) -> CameraCfg:
389    """Grab a simple standard camera config for injecting into task environments."""
390    return create_camera_base(
391        CameraCfg,
392        num_cams=args_cli.num_standard_cameras,
393        data_types=args_cli.standard_camera_data_types,
394        width=args_cli.width,
395        height=args_cli.height,
396        prim_path="{ENV_REGEX_NS}/" + prim_path,
397        instantiate=False,
398    )
399
400
401def create_ray_caster_camera_cfg(prim_path: str) -> RayCasterCameraCfg:
402    """Grab a simple ray caster config for injecting into task environments."""
403    return create_ray_caster_cameras(
404        num_cams=args_cli.num_ray_caster_cameras,
405        data_types=args_cli.ray_caster_camera_data_types,
406        width=args_cli.width,
407        height=args_cli.height,
408        prim_path="{ENV_REGEX_NS}/" + prim_path,
409    )
410
411
412"""
413Scene Creation
414"""
415
416
417def design_scene(
418    num_tiled_cams: int = 2,
419    num_standard_cams: int = 0,
420    num_ray_caster_cams: int = 0,
421    tiled_camera_data_types: list[str] | None = None,
422    standard_camera_data_types: list[str] | None = None,
423    ray_caster_camera_data_types: list[str] | None = None,
424    height: int = 100,
425    width: int = 200,
426    num_objects: int = 20,
427    mesh_prim_paths: list[str] = ["/World/ground"],
428) -> dict:
429    """Design the scene."""
430    if tiled_camera_data_types is None:
431        tiled_camera_data_types = ["rgb"]
432    if standard_camera_data_types is None:
433        standard_camera_data_types = ["rgb"]
434    if ray_caster_camera_data_types is None:
435        ray_caster_camera_data_types = ["distance_to_image_plane"]
436
437    # Populate scene
438    # -- Ground-plane
439    cfg = sim_utils.GroundPlaneCfg()
440    cfg.func("/World/ground", cfg)
441    # -- Lights
442    cfg = sim_utils.DistantLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75))
443    cfg.func("/World/Light", cfg)
444
445    # Create a dictionary for the scene entities
446    scene_entities = {}
447
448    # Xform to hold objects
449    prim_utils.create_prim("/World/Objects", "Xform")
450    # Random objects
451    for i in range(num_objects):
452        # sample random position
453        position = np.random.rand(3) - np.asarray([0.05, 0.05, -1.0])
454        position *= np.asarray([1.5, 1.5, 0.5])
455        # sample random color
456        color = (random.random(), random.random(), random.random())
457        # choose random prim type
458        prim_type = random.choice(["Cube", "Cone", "Cylinder"])
459        common_properties = {
460            "rigid_props": sim_utils.RigidBodyPropertiesCfg(),
461            "mass_props": sim_utils.MassPropertiesCfg(mass=5.0),
462            "collision_props": sim_utils.CollisionPropertiesCfg(),
463            "visual_material": sim_utils.PreviewSurfaceCfg(diffuse_color=color, metallic=0.5),
464            "semantic_tags": [("class", prim_type)],
465        }
466        if prim_type == "Cube":
467            shape_cfg = sim_utils.CuboidCfg(size=(0.25, 0.25, 0.25), **common_properties)
468        elif prim_type == "Cone":
469            shape_cfg = sim_utils.ConeCfg(radius=0.1, height=0.25, **common_properties)
470        elif prim_type == "Cylinder":
471            shape_cfg = sim_utils.CylinderCfg(radius=0.25, height=0.25, **common_properties)
472        # Rigid Object
473        obj_cfg = RigidObjectCfg(
474            prim_path=f"/World/Objects/Obj_{i:02d}",
475            spawn=shape_cfg,
476            init_state=RigidObjectCfg.InitialStateCfg(pos=position),
477        )
478        scene_entities[f"rigid_object{i}"] = RigidObject(cfg=obj_cfg)
479
480    # Sensors
481    standard_camera = create_cameras(
482        num_cams=num_standard_cams, data_types=standard_camera_data_types, height=height, width=width
483    )
484    tiled_camera = create_tiled_cameras(
485        num_cams=num_tiled_cams, data_types=tiled_camera_data_types, height=height, width=width
486    )
487    ray_caster_camera = create_ray_caster_cameras(
488        num_cams=num_ray_caster_cams,
489        data_types=ray_caster_camera_data_types,
490        mesh_prim_paths=mesh_prim_paths,
491        height=height,
492        width=width,
493    )
494    # return the scene information
495    if tiled_camera is not None:
496        scene_entities["tiled_camera"] = tiled_camera
497    if standard_camera is not None:
498        scene_entities["standard_camera"] = standard_camera
499    if ray_caster_camera is not None:
500        scene_entities["ray_caster_camera"] = ray_caster_camera
501    return scene_entities
502
503
504def inject_cameras_into_task(
505    task: str,
506    num_cams: int,
507    camera_name_prefix: str,
508    camera_creation_callable: Callable,
509    num_cameras_per_env: int = 1,
510) -> gym.Env:
511    """Loads the task, sticks cameras into the config, and creates the environment."""
512    cfg = load_cfg_from_registry(task, "env_cfg_entry_point")
513    cfg.sim.device = args_cli.device
514    cfg.sim.use_fabric = args_cli.use_fabric
515    scene_cfg = cfg.scene
516
517    num_envs = int(num_cams / num_cameras_per_env)
518    scene_cfg.num_envs = num_envs
519
520    for idx in range(num_cameras_per_env):
521        suffix = "" if idx == 0 else str(idx)
522        name = camera_name_prefix + suffix
523        setattr(scene_cfg, name, camera_creation_callable(name))
524    cfg.scene = scene_cfg
525    env = gym.make(task, cfg=cfg)
526    return env
527
528
529"""
530System diagnosis
531"""
532
533
534def get_utilization_percentages(reset: bool = False, max_values: list[float] = [0.0, 0.0, 0.0, 0.0]) -> list[float]:
535    """Get the maximum CPU, RAM, GPU utilization (processing), and
536    GPU memory usage percentages since the last time reset was true."""
537    if reset:
538        max_values[:] = [0, 0, 0, 0]  # Reset the max values
539
540    # CPU utilization
541    cpu_usage = psutil.cpu_percent(interval=0.1)
542    max_values[0] = max(max_values[0], cpu_usage)
543
544    # RAM utilization
545    memory_info = psutil.virtual_memory()
546    ram_usage = memory_info.percent
547    max_values[1] = max(max_values[1], ram_usage)
548
549    # GPU utilization using pynvml
550    if torch.cuda.is_available():
551
552        if args_cli.autotune:
553            pynvml.nvmlInit()  # Initialize NVML
554            for i in range(torch.cuda.device_count()):
555                handle = pynvml.nvmlDeviceGetHandleByIndex(i)
556
557                # GPU Utilization
558                gpu_utilization = pynvml.nvmlDeviceGetUtilizationRates(handle)
559                gpu_processing_utilization_percent = gpu_utilization.gpu  # GPU core utilization
560                max_values[2] = max(max_values[2], gpu_processing_utilization_percent)
561
562                # GPU Memory Usage
563                memory_info = pynvml.nvmlDeviceGetMemoryInfo(handle)
564                gpu_memory_total = memory_info.total
565                gpu_memory_used = memory_info.used
566                gpu_memory_utilization_percent = (gpu_memory_used / gpu_memory_total) * 100
567                max_values[3] = max(max_values[3], gpu_memory_utilization_percent)
568
569            pynvml.nvmlShutdown()  # Shutdown NVML after usage
570    else:
571        gpu_processing_utilization_percent = None
572        gpu_memory_utilization_percent = None
573    return max_values
574
575
576"""
577Experiment
578"""
579
580
581def run_simulator(
582    sim: sim_utils.SimulationContext | None,
583    scene_entities: dict | InteractiveScene,
584    warm_start_length: int = 10,
585    experiment_length: int = 100,
586    tiled_camera_data_types: list[str] | None = None,
587    standard_camera_data_types: list[str] | None = None,
588    ray_caster_camera_data_types: list[str] | None = None,
589    depth_predicate: Callable = lambda x: "to" in x or x == "depth",
590    perspective_depth_predicate: Callable = lambda x: x == "distance_to_camera",
591    convert_depth_to_camera_to_image_plane: bool = True,
592    max_cameras_per_env: int = 1,
593    env: gym.Env | None = None,
594) -> dict:
595    """Run the simulator with all cameras, and return timing analytics. Visualize if desired."""
596
597    if tiled_camera_data_types is None:
598        tiled_camera_data_types = ["rgb"]
599    if standard_camera_data_types is None:
600        standard_camera_data_types = ["rgb"]
601    if ray_caster_camera_data_types is None:
602        ray_caster_camera_data_types = ["distance_to_image_plane"]
603
604    # Initialize camera lists
605    tiled_cameras = []
606    standard_cameras = []
607    ray_caster_cameras = []
608
609    # Dynamically extract cameras from the scene entities up to max_cameras_per_env
610    for i in range(max_cameras_per_env):
611        # Extract tiled cameras
612        tiled_camera_key = f"tiled_camera{i}" if i > 0 else "tiled_camera"
613        standard_camera_key = f"standard_camera{i}" if i > 0 else "standard_camera"
614        ray_caster_camera_key = f"ray_caster_camera{i}" if i > 0 else "ray_caster_camera"
615
616        try:  # if instead you checked ... if key is in scene_entities... # errors out always even if key present
617            tiled_cameras.append(scene_entities[tiled_camera_key])
618            standard_cameras.append(scene_entities[standard_camera_key])
619            ray_caster_cameras.append(scene_entities[ray_caster_camera_key])
620        except KeyError:
621            break
622
623    # Initialize camera counts
624    camera_lists = [tiled_cameras, standard_cameras, ray_caster_cameras]
625    camera_data_types = [tiled_camera_data_types, standard_camera_data_types, ray_caster_camera_data_types]
626    labels = ["tiled", "standard", "ray_caster"]
627
628    if sim is not None:
629        # Set camera world poses
630        for camera_list in camera_lists:
631            for camera in camera_list:
632                num_cameras = camera.data.intrinsic_matrices.size(0)
633                positions = torch.tensor([[2.5, 2.5, 2.5]], device=sim.device).repeat(num_cameras, 1)
634                targets = torch.tensor([[0.0, 0.0, 0.0]], device=sim.device).repeat(num_cameras, 1)
635                camera.set_world_poses_from_view(positions, targets)
636
637    # Initialize timing variables
638    timestep = 0
639    total_time = 0.0
640    valid_timesteps = 0
641    sim_step_time = 0.0
642
643    while simulation_app.is_running() and timestep < experiment_length:
644        print(f"On timestep {timestep} of {experiment_length}, with warm start of {warm_start_length}")
645        get_utilization_percentages()
646
647        # Measure the total simulation step time
648        step_start_time = time.time()
649
650        if sim is not None:
651            sim.step()
652
653        if env is not None:
654            with torch.inference_mode():
655                # compute zero actions
656                actions = torch.zeros(env.action_space.shape, device=env.unwrapped.device)
657                # apply actions
658                env.step(actions)
659
660        # Update cameras and process vision data within the simulation step
661        clouds = {}
662        images = {}
663        depth_images = {}
664
665        # Loop through all camera lists and their data_types
666        for camera_list, data_types, label in zip(camera_lists, camera_data_types, labels):
667            for cam_idx, camera in enumerate(camera_list):
668
669                if env is None:  # No env, need to step cams manually
670                    # Only update the camera if it hasn't been updated as part of scene_entities.update ...
671                    camera.update(dt=sim.get_physics_dt())
672
673                for data_type in data_types:
674                    data_label = f"{label}_{cam_idx}_{data_type}"
675
676                    if depth_predicate(data_type):  # is a depth image, want to create cloud
677                        depth = camera.data.output[data_type]
678                        depth_images[data_label + "_raw"] = depth
679                        if perspective_depth_predicate(data_type) and convert_depth_to_camera_to_image_plane:
680                            depth = orthogonalize_perspective_depth(
681                                camera.data.output[data_type], camera.data.intrinsic_matrices
682                            )
683                            depth_images[data_label + "_undistorted"] = depth
684
685                        pointcloud = unproject_depth(depth=depth, intrinsics=camera.data.intrinsic_matrices)
686                        clouds[data_label] = pointcloud
687                    else:  # rgb image, just save it
688                        image = camera.data.output[data_type]
689                        images[data_label] = image
690
691        # End timing for the step
692        step_end_time = time.time()
693        sim_step_time += step_end_time - step_start_time
694
695        if timestep > warm_start_length:
696            get_utilization_percentages(reset=True)
697            total_time += step_end_time - step_start_time
698            valid_timesteps += 1
699
700        timestep += 1
701
702    # Calculate average timings
703    if valid_timesteps > 0:
704        avg_timestep_duration = total_time / valid_timesteps
705        avg_sim_step_duration = sim_step_time / experiment_length
706    else:
707        avg_timestep_duration = 0.0
708        avg_sim_step_duration = 0.0
709
710    # Package timing analytics in a dictionary
711    timing_analytics = {
712        "average_timestep_duration": avg_timestep_duration,
713        "average_sim_step_duration": avg_sim_step_duration,
714        "total_simulation_time": sim_step_time,
715        "total_experiment_duration": sim_step_time,
716    }
717
718    system_utilization_analytics = get_utilization_percentages()
719
720    print("--- Benchmark Results ---")
721    print(f"Average timestep duration: {avg_timestep_duration:.6f} seconds")
722    print(f"Average simulation step duration: {avg_sim_step_duration:.6f} seconds")
723    print(f"Total simulation time: {sim_step_time:.6f} seconds")
724    print("\nSystem Utilization Statistics:")
725    print(
726        f"| CPU:{system_utilization_analytics[0]}% | "
727        f"RAM:{system_utilization_analytics[1]}% | "
728        f"GPU Compute:{system_utilization_analytics[2]}% | "
729        f" GPU Memory: {system_utilization_analytics[3]:.2f}% |"
730    )
731
732    return {"timing_analytics": timing_analytics, "system_utilization_analytics": system_utilization_analytics}
733
734
735def main():
736    """Main function."""
737    # Load simulation context
738    if args_cli.num_tiled_cameras + args_cli.num_standard_cameras + args_cli.num_ray_caster_cameras <= 0:
739        raise ValueError("You must select at least one camera.")
740    if (
741        (args_cli.num_tiled_cameras > 0 and args_cli.num_standard_cameras > 0)
742        or (args_cli.num_ray_caster_cameras > 0 and args_cli.num_standard_cameras > 0)
743        or (args_cli.num_ray_caster_cameras > 0 and args_cli.num_tiled_cameras > 0)
744    ):
745        print("[WARNING]: You have elected to use more than one camera type.")
746        print("[WARNING]: For a benchmark to be meaningful, use ONLY ONE camera type at a time.")
747        print(
748            "[WARNING]: For example, if num_tiled_cameras=100, for a meaningful benchmark,"
749            "num_standard_cameras should be 0, and num_ray_caster_cameras should be 0"
750        )
751        raise ValueError("Benchmark one camera at a time.")
752
753    print("[INFO]: Designing the scene")
754    if args_cli.task is None:
755        print("[INFO]: No task environment provided, creating random scene.")
756        sim_cfg = sim_utils.SimulationCfg(device=args_cli.device)
757        sim = sim_utils.SimulationContext(sim_cfg)
758        # Set main camera
759        sim.set_camera_view([2.5, 2.5, 2.5], [0.0, 0.0, 0.0])
760        scene_entities = design_scene(
761            num_tiled_cams=args_cli.num_tiled_cameras,
762            num_standard_cams=args_cli.num_standard_cameras,
763            num_ray_caster_cams=args_cli.num_ray_caster_cameras,
764            tiled_camera_data_types=args_cli.tiled_camera_data_types,
765            standard_camera_data_types=args_cli.standard_camera_data_types,
766            ray_caster_camera_data_types=args_cli.ray_caster_camera_data_types,
767            height=args_cli.height,
768            width=args_cli.width,
769            num_objects=args_cli.num_objects,
770            mesh_prim_paths=args_cli.ray_caster_visible_mesh_prim_paths,
771        )
772        # Play simulator
773        sim.reset()
774        # Now we are ready!
775        print("[INFO]: Setup complete...")
776        # Run simulator
777        run_simulator(
778            sim=sim,
779            scene_entities=scene_entities,
780            warm_start_length=args_cli.warm_start_length,
781            experiment_length=args_cli.experiment_length,
782            tiled_camera_data_types=args_cli.tiled_camera_data_types,
783            standard_camera_data_types=args_cli.standard_camera_data_types,
784            ray_caster_camera_data_types=args_cli.ray_caster_camera_data_types,
785            convert_depth_to_camera_to_image_plane=args_cli.convert_depth_to_camera_to_image_plane,
786        )
787    else:
788        print("[INFO]: Using known task environment, injecting cameras.")
789        autotune_iter = 0
790        max_sys_util_thresh = [0.0, 0.0, 0.0]
791        max_num_cams = max(args_cli.num_tiled_cameras, args_cli.num_standard_cameras, args_cli.num_ray_caster_cameras)
792        cur_num_cams = max_num_cams
793        cur_sys_util = max_sys_util_thresh
794        interval = args_cli.autotune_camera_count_interval
795
796        if args_cli.autotune:
797            max_sys_util_thresh = args_cli.autotune_max_percentage_util
798            max_num_cams = args_cli.autotune_max_camera_count
799            print("[INFO]: Auto tuning until any of the following threshold are met")
800            print(f"|CPU: {max_sys_util_thresh[0]}% | RAM {max_sys_util_thresh[1]}% | GPU: {max_sys_util_thresh[2]}% |")
801            print(f"[INFO]: Maximum number of cameras allowed: {max_num_cams}")
802        # Determine which camera is being tested...
803        tiled_camera_cfg = create_tiled_camera_cfg("tiled_camera")
804        standard_camera_cfg = create_standard_camera_cfg("standard_camera")
805        ray_caster_camera_cfg = create_ray_caster_camera_cfg("ray_caster_camera")
806        camera_name_prefix = ""
807        camera_creation_callable = None
808        num_cams = 0
809        if tiled_camera_cfg is not None:
810            camera_name_prefix = "tiled_camera"
811            camera_creation_callable = create_tiled_camera_cfg
812            num_cams = args_cli.num_tiled_cameras
813        elif standard_camera_cfg is not None:
814            camera_name_prefix = "standard_camera"
815            camera_creation_callable = create_standard_camera_cfg
816            num_cams = args_cli.num_standard_cameras
817        elif ray_caster_camera_cfg is not None:
818            camera_name_prefix = "ray_caster_camera"
819            camera_creation_callable = create_ray_caster_camera_cfg
820            num_cams = args_cli.num_ray_caster_cameras
821
822        while (
823            all(cur <= max_thresh for cur, max_thresh in zip(cur_sys_util, max_sys_util_thresh))
824            and cur_num_cams <= max_num_cams
825        ):
826            cur_num_cams = num_cams + interval * autotune_iter
827            autotune_iter += 1
828
829            env = inject_cameras_into_task(
830                task=args_cli.task,
831                num_cams=cur_num_cams,
832                camera_name_prefix=camera_name_prefix,
833                camera_creation_callable=camera_creation_callable,
834                num_cameras_per_env=args_cli.task_num_cameras_per_env,
835            )
836            env.reset()
837            print(f"Testing with {cur_num_cams} {camera_name_prefix}")
838            analysis = run_simulator(
839                sim=None,
840                scene_entities=env.unwrapped.scene,
841                warm_start_length=args_cli.warm_start_length,
842                experiment_length=args_cli.experiment_length,
843                tiled_camera_data_types=args_cli.tiled_camera_data_types,
844                standard_camera_data_types=args_cli.standard_camera_data_types,
845                ray_caster_camera_data_types=args_cli.ray_caster_camera_data_types,
846                convert_depth_to_camera_to_image_plane=args_cli.convert_depth_to_camera_to_image_plane,
847                max_cameras_per_env=args_cli.task_num_cameras_per_env,
848                env=env,
849            )
850
851            cur_sys_util = analysis["system_utilization_analytics"]
852            print("Triggering reset...")
853            env.close()
854            create_new_stage()
855        print("[INFO]: DONE! Feel free to CTRL + C Me ")
856        print(f"[INFO]: If you've made it this far, you can likely simulate {cur_num_cams} {camera_name_prefix}")
857        print("Keep in mind, this is without any training running on the GPU.")
858        print("Set lower utilization thresholds to account for training.")
859
860        if not args_cli.autotune:
861            print("[WARNING]: GPU Util Statistics only correct while autotuning, ignore above.")
862
863
864if __name__ == "__main__":
865    # run the main function
866    main()
867    # close sim app
868    simulation_app.close()

Possible Parameters#

First, run

./isaaclab.sh -p source/standalone/benchmarks/benchmark_cameras.py -h

to see all possible parameters you can vary with this utility.

See the command line parameters related to autotune for more information about automatically determining maximum camera count.

Compare Performance in Task Environments and Automatically Determine Task Max Camera Count#

Currently, tiled cameras are the most performant camera that can handle multiple dynamic objects.

For example, to see how your system could handle 100 tiled cameras in the cartpole environment, with 2 cameras per environment (so 50 environments total) only in RGB mode, run

./isaaclab.sh -p source/standalone/benchmarks/benchmark_cameras.py \
--task Isaac-Cartpole-v0 --num_tiled_cameras 100 \
--task_num_cameras_per_env 2 \
--tiled_camera_data_types rgb

If you have pynvml installed, (./isaaclab.sh -p -m pip install pynvml), you can also find the maximum number of cameras that you could run in the specified environment up to a certain performance threshold (specified by max CPU utilization percent, max RAM utilization percent, max GPU compute percent, and max GPU memory percent). For example, to find the maximum number of cameras you can run with cartpole, you could run:

./isaaclab.sh -p source/standalone/benchmarks/benchmark_cameras.py \
--task Isaac-Cartpole-v0 --num_tiled_cameras 100 \
--task_num_cameras_per_env 2 \
--tiled_camera_data_types rgb --autotune \
--autotune_max_percentage_util 100 80 50 50

Autotune may lead to the program crashing, which means that it tried to run too many cameras at once. However, the max percentage utilization parameter is meant to prevent this from happening.

The output of the benchmark doesn’t include the overhead of training the network, so consider decreasing the maximum utilization percentages to account for this overhead. The final output camera count is for all cameras, so to get the total number of environments, divide the output camera count by the number of cameras per environment.

Compare Camera Type and Performance (Without a Specified Task)#

This tool can also asses performance without a task environment. For example, to view 100 random objects with 2 standard cameras, one could run

./isaaclab.sh -p source/standalone/benchmarks/benchmark_cameras.py \
--height 100 --width 100 --num_standard_cameras 2 \
--standard_camera_data_types instance_segmentation_fast normals --num_objects 100 \
--experiment_length 100

If your system cannot handle this due to performance reasons, then the process will be killed. It’s recommended to monitor CPU/RAM utilization and GPU utilization while running this script, to get an idea of how many resources rendering the desired camera requires. In Ubuntu, you can use tools like htop and nvtop to live monitor resources while running this script, and in Windows, you can use the Task Manager.

If your system has a hard time handling the desired cameras, you can try the following

  • Switch to headless mode (supply --headless)

  • Ensure you are using the GPU pipeline not CPU!

  • If you aren’t using Tiled Cameras, switch to Tiled Cameras

  • Decrease camera resolution

  • Decrease how many data_types there are for each camera.

  • Decrease the number of cameras

  • Decrease the number of objects in the scene

If your system is able to handle the amount of cameras, then the time statistics will be printed to the terminal. After the simulations stops it can be closed with CTRL+C.