Streaming a camera view in a visualizer

Streaming a camera view in a visualizer#

Display ground-truth camera frames from many environments in one live visualizer panel. For how the view works and which visualizers support it, see Streaming Camera View.

Quick Start#

Run the run_tiled_camera_visualizer.py script in IsaacLab/scripts/tutorials/07_visualizers:

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py ^
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit
Code for run_tiled_camera_visualizer.py
  1# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
  2# All rights reserved.
  3#
  4# SPDX-License-Identifier: BSD-3-Clause
  5
  6"""
  7This script demonstrates the visualizer tiled camera panel.
  8
  9.. code-block:: bash
 10
 11    # Kit visualizer tiled camera panel
 12    uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
 13 --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit
 14
 15    # Newton visualizer tiled camera panel
 16    uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
 17        --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl
 18
 19"""
 20
 21from __future__ import annotations
 22
 23import argparse
 24import contextlib
 25import sys
 26
 27import gymnasium as gym
 28import torch
 29
 30import isaaclab_tasks  # noqa: F401
 31
 32with contextlib.suppress(ImportError):
 33    import isaaclab_tasks_experimental  # noqa: F401
 34from isaaclab.app import add_launcher_args, launch_simulation
 35
 36from isaaclab_tasks.utils import resolve_task_config, setup_preset_cli
 37
 38KIT_DEFAULT_TASK = "Isaac-Velocity-Rough-AnymalD"
 39NEWTON_DEFAULT_TASK = "IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor"
 40SUPPORTED_TILED_VISUALIZERS = {"kit", "newton", "newton_gl", "newton_rtx"}
 41UNSUPPORTED_TILED_VISUALIZERS = {"rerun", "viser"}
 42
 43
 44def _resolve_env_regex_path(prim_path: str) -> str:
 45    """Resolve scene config env namespace macros to the cloned-env regex."""
 46    return prim_path.format(ENV_REGEX_NS="/World/envs/env_.*")
 47
 48
 49def _requested_visualizers(args_cli: argparse.Namespace) -> list[str]:
 50    """Return requested visualizers, defaulting to Kit for this tutorial."""
 51    visualizers = args_cli.visualizer or ["kit"]
 52    visualizers = [str(visualizer).lower() for visualizer in visualizers]
 53
 54    if "none" in visualizers:
 55        raise ValueError("This demo requires a tiled-camera visualizer. Use '--viz kit' or '--viz newton_gl'.")
 56    unsupported = sorted(set(visualizers) & UNSUPPORTED_TILED_VISUALIZERS)
 57    if unsupported:
 58        raise ValueError(
 59            "The visualizer tiled camera panel is only implemented for Kit and Newton. "
 60            f"Unsupported selection: {unsupported}."
 61        )
 62    unknown = sorted(set(visualizers) - SUPPORTED_TILED_VISUALIZERS)
 63    if unknown:
 64        raise ValueError(f"Unknown visualizer selection for this demo: {unknown}.")
 65    return visualizers
 66
 67
 68def _make_kit_visualizer_cfg(env_cfg):
 69    """Create the Kit streaming-camera visualizer for the selected task."""
 70    from isaaclab_visualizers.kit import KitVisualizerCfg
 71
 72    visualizer_cfg = KitVisualizerCfg()
 73    visualizer_cfg.streaming_view = True
 74    visualizer_cfg.streaming_envs = 36
 75
 76    ego_cam_cfg = getattr(env_cfg.scene, "ego_cam", None)
 77    if ego_cam_cfg is not None:
 78        visualizer_cfg.streaming_sensor_prim_path = _resolve_env_regex_path(ego_cam_cfg.prim_path)
 79        return visualizer_cfg
 80
 81    visualizer_cfg.streaming_sensor_prim_path = None
 82    visualizer_cfg.streaming_cam_eye = (3.0, 3.0, 3.0)
 83    visualizer_cfg.streaming_cam_target_prim_path = "/World/envs/*/Robot/base"
 84    # Here is an alternative eye position for a top down view
 85    # visualizer_cfg.streaming_cam_eye = (0.0, 0.0, 5.0)
 86    return visualizer_cfg
 87
 88
 89def _make_newton_visualizer_cfg(env_cfg):
 90    """Create the Newton streaming-camera visualizer for the selected task."""
 91    from isaaclab_visualizers.newton import NewtonGLVisualizerCfg
 92
 93    visualizer_cfg = NewtonGLVisualizerCfg()
 94    visualizer_cfg.streaming_view = True
 95    visualizer_cfg.streaming_envs = 12
 96
 97    ego_cam_cfg = getattr(env_cfg.scene, "ego_cam", None)
 98    if ego_cam_cfg is not None:
 99        visualizer_cfg.streaming_sensor_prim_path = _resolve_env_regex_path(ego_cam_cfg.prim_path)
100        return visualizer_cfg
101
102    # Here are other robot mounted camera options for this environment
103    # visualizer_cfg.streaming_sensor_prim_path = "/World/envs/env_.*/Robot/left_arm_camera_sim_view_frame/left_camera"
104    # visualizer_cfg.streaming_sensor_prim_path = (
105    #     "/World/envs/env_.*/Robot/right_arm_camera_sim_view_frame/right_camera"
106    # )
107    visualizer_cfg.streaming_sensor_prim_path = None
108    visualizer_cfg.streaming_cam_eye = (3.0, 3.0, 3.0)
109    visualizer_cfg.streaming_cam_target_prim_path = "/World/envs/*/Robot/base"
110    return visualizer_cfg
111
112
113def _configure_visualizers(env_cfg, args_cli: argparse.Namespace) -> None:
114    """Attach tiled camera visualizer configs to the environment simulation config."""
115    visualizers = _requested_visualizers(args_cli)
116    args_cli.visualizer = visualizers
117    env_cfg.sim.visualizer_cfgs = [
118        _make_kit_visualizer_cfg(env_cfg) if visualizer == "kit" else _make_newton_visualizer_cfg(env_cfg)
119        for visualizer in visualizers
120    ]
121
122
123def _resolve_task(args_cli: argparse.Namespace) -> str:
124    """Resolve the task for the selected visualizer."""
125    if args_cli.task is not None:
126        return args_cli.task
127    if "newton" in _requested_visualizers(args_cli):
128        return NEWTON_DEFAULT_TASK
129    return KIT_DEFAULT_TASK
130
131
132# add argparse arguments
133parser = argparse.ArgumentParser(description="Showcase the Kit/Newton visualizer tiled camera panel.")
134parser.add_argument("--num_envs", type=int, default=None, help="Number of environments to simulate.")
135parser.add_argument("--task", type=str, default=None, help="Name of the task.")
136# append simulation launcher cli args
137add_launcher_args(parser)
138args_cli, hydra_args = setup_preset_cli(parser)
139args_cli.task = _resolve_task(args_cli)
140sys.argv = [sys.argv[0]] + hydra_args
141
142
143def main():
144    """Run a random-action environment with a tiled camera visualizer."""
145    # parse configuration via Hydra (supports preset selection, e.g. presets=newton_mjwarp)
146    env_cfg, _ = resolve_task_config(args_cli.task, "")
147    _configure_visualizers(env_cfg, args_cli)
148
149    with launch_simulation(env_cfg, args_cli):
150        # override with CLI arguments
151        env_cfg.scene.num_envs = args_cli.num_envs if args_cli.num_envs is not None else env_cfg.scene.num_envs
152        env_cfg.sim.device = args_cli.device if args_cli.device is not None else env_cfg.sim.device
153
154        # create environment
155        env = gym.make(args_cli.task, cfg=env_cfg)
156
157        # print info (this is vectorized environment)
158        print(f"[INFO]: Gym observation space: {env.observation_space}")
159        print(f"[INFO]: Gym action space: {env.action_space}")
160        env.reset()
161
162        # keep stepping until all visualizer windows have been closed
163        sim = env.unwrapped.sim
164        if not sim.visualizers:
165            print("[WARN]: No visualizers found. Exiting.")
166            env.close()
167            return
168
169        while True:
170            if sim.visualizers and not any(v.is_running() and not v.is_closed for v in sim.visualizers):
171                break
172            with torch.inference_mode():
173                actions = 2 * torch.rand(env.action_space.shape, device=env.unwrapped.device) - 1
174                env.step(actions)
175
176        env.close()
177
178
179if __name__ == "__main__":
180    main()

See Examples below for the two ways to run the script. For the VisualizerCfg fields that customize streaming, see Configuration and Troubleshooting.

Examples#

Running run_tiled_camera_visualizer.py demonstrates two ways to use the streaming camera view:

  • auto-created cameras pointed at and following moving AnymalD robots, shown in the Kit visualizer

  • streaming from existing wrist-mounted robot cameras, shown in the Newton visualizer

Example 1: Following AnymalD Robots#

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py ^
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

The script’s KitVisualizerCfg creates cameras that point at and follow each robot’s base prim, offset by streaming_cam_eye (here (3.0, 3.0, 3.0); try (0, 0, 5) for a top-down view). Of the 256 environments, 36 are randomly sampled for the camera view.

Kit visualizer: interactive viewport

Kit visualizer: streaming camera view

Example 2: Streaming from Robot-Mounted Cameras#

uv run --extra teleop python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl

uv run --extra teleop python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py ^
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl

The Galbot cube-stacking environment ships with wrist-mounted cameras giving an egocentric view of the gripper, table, and cubes. The script’s NewtonGLVisualizerCfg streams from the existing sensor at /World/envs/env_.*/Robot/head_camera_sim_view_frame/head_camera; edit streaming_sensor_prim_path to show a different camera. Of the 25 environments, 12 camera feeds are shown by default.

Newton visualizer: interactive viewport

Newton visualizer: streaming camera view

Configuration#

To customize streaming camera behavior, edit the highlighted VisualizerCfg fields in run_tiled_camera_visualizer.py:

  • For auto-created cameras, streaming_cam_target_prim_path chooses the followed prim and streaming_cam_eye sets the camera offset from that prim. Defaults to None, which causes the visualizer to adopt the first scene camera it discovers at init; no explicit path is needed when a TiledCamera sensor is already in the scene.

  • For existing scene cameras, streaming_sensor_prim_path must match an Isaac Lab Camera sensor prim path in the selected task.

  • streaming_envs controls how many environment tiles are shown. Pass an int to randomly sample that many environments, or a list[int] to pin specific environment indices.

  • streaming_gt_types selects which ground-truth types are shown, e.g. ["rgb", "depth", "segmentation", "normals"].

  • streaming_depth_min / streaming_depth_max set the depth colormap range in metres.

Troubleshooting#

  • If a generated view fails with a missing prim error, verify that streaming_cam_target_prim_path resolves in each selected environment; common template forms are /World/envs/*/... and /World/envs/env_.*/.... In most cases you can leave it as None and let the visualizer adopt an existing scene camera automatically.

  • If an existing-camera view reports that no Isaac Lab camera owns the prim, check that streaming_sensor_prim_path matches a Camera sensor in the task.

  • If the depth panel shows a flat color, adjust streaming_depth_min and streaming_depth_max to bracket the expected depth range in your scene.

  • If the view is too expensive, reduce streaming_envs, --num_envs, or the camera resolution.

Warning

Newton MJWarp with replicate_physics=True and auto-created cameras

With replicate_physics=True, only env_0 has a USD prim after physics initialization. Cameras for the remaining environments (env_1 through env_{N-1}) are dropped, causing initialization to fail:

RuntimeError: Number of camera prims in the view (1) does not match
the number of environments (N).

Workaround: set streaming_sensor_prim_path to a scene camera that was declared in the scene config before physics init (for example, a TiledCamera on a vision-based task).

See also#