Visualizer Streaming Camera View

Visualizer Streaming Camera View#

For general visualizer documentation, see Visualization.

The visualizer streaming camera view is a live monitoring and debugging tool. It combines ground-truth camera frames from multiple environments (RGB, depth, segmentation, or surface normals) into a single panel that updates every step, either following robots automatically or streaming from existing scene camera sensors.

Note

The streaming camera view is supported in the Kit, Newton GL, Rerun, and Viser visualizers. The Newton RTX visualizer accepts the configuration but does not display the panel (experimental).

Quick Start#

This guide is accompanied by the run_tiled_camera_visualizer.py script in IsaacLab/scripts/tutorials/07_visualizers:

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit
Code for run_tiled_camera_visualizer.py
  1# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
  2# All rights reserved.
  3#
  4# SPDX-License-Identifier: BSD-3-Clause
  5
  6"""
  7This script demonstrates the visualizer tiled camera panel.
  8
  9.. code-block:: bash
 10
 11    # Kit visualizer tiled camera panel
 12    uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
 13 --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit
 14
 15    # Newton visualizer tiled camera panel
 16    uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
 17        --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton
 18
 19"""
 20
 21from __future__ import annotations
 22
 23import argparse
 24import contextlib
 25import sys
 26
 27import gymnasium as gym
 28import torch
 29
 30import isaaclab_tasks  # noqa: F401
 31
 32with contextlib.suppress(ImportError):
 33    import isaaclab_tasks_experimental  # noqa: F401
 34from isaaclab.app import add_launcher_args, launch_simulation
 35
 36from isaaclab_tasks.utils import resolve_task_config, setup_preset_cli
 37
 38KIT_DEFAULT_TASK = "Isaac-Velocity-Rough-AnymalD"
 39NEWTON_DEFAULT_TASK = "IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor"
 40SUPPORTED_TILED_VISUALIZERS = {"kit", "newton", "newton_gl", "newton_rtx"}
 41UNSUPPORTED_TILED_VISUALIZERS = {"rerun", "viser"}
 42
 43
 44def _resolve_env_regex_path(prim_path: str) -> str:
 45    """Resolve scene config env namespace macros to the cloned-env regex."""
 46    return prim_path.format(ENV_REGEX_NS="/World/envs/env_.*")
 47
 48
 49def _requested_visualizers(args_cli: argparse.Namespace) -> list[str]:
 50    """Return requested visualizers, defaulting to Kit for this tutorial."""
 51    visualizers = args_cli.visualizer or ["kit"]
 52    visualizers = [str(visualizer).lower() for visualizer in visualizers]
 53
 54    if "none" in visualizers:
 55        raise ValueError("This demo requires a tiled-camera visualizer. Use '--viz kit' or '--viz newton_gl'.")
 56    unsupported = sorted(set(visualizers) & UNSUPPORTED_TILED_VISUALIZERS)
 57    if unsupported:
 58        raise ValueError(
 59            "The visualizer tiled camera panel is only implemented for Kit and Newton. "
 60            f"Unsupported selection: {unsupported}."
 61        )
 62    unknown = sorted(set(visualizers) - SUPPORTED_TILED_VISUALIZERS)
 63    if unknown:
 64        raise ValueError(f"Unknown visualizer selection for this demo: {unknown}.")
 65    return visualizers
 66
 67
 68def _make_kit_visualizer_cfg(env_cfg):
 69    """Create the Kit streaming-camera visualizer for the selected task."""
 70    from isaaclab_visualizers.kit import KitVisualizerCfg
 71
 72    visualizer_cfg = KitVisualizerCfg()
 73    visualizer_cfg.streaming_view = True
 74    visualizer_cfg.streaming_envs = 36
 75
 76    ego_cam_cfg = getattr(env_cfg.scene, "ego_cam", None)
 77    if ego_cam_cfg is not None:
 78        visualizer_cfg.streaming_sensor_prim_path = _resolve_env_regex_path(ego_cam_cfg.prim_path)
 79        return visualizer_cfg
 80
 81    visualizer_cfg.streaming_sensor_prim_path = None
 82    visualizer_cfg.streaming_cam_eye = (3.0, 3.0, 3.0)
 83    visualizer_cfg.streaming_cam_target_prim_path = "/World/envs/*/Robot/base"
 84    # Here is an alternative eye position for a top down view
 85    # visualizer_cfg.streaming_cam_eye = (0.0, 0.0, 5.0)
 86    return visualizer_cfg
 87
 88
 89def _make_newton_visualizer_cfg(env_cfg):
 90    """Create the Newton streaming-camera visualizer for the selected task."""
 91    from isaaclab_visualizers.newton import NewtonGLVisualizerCfg
 92
 93    visualizer_cfg = NewtonGLVisualizerCfg()
 94    visualizer_cfg.streaming_view = True
 95    visualizer_cfg.streaming_envs = 12
 96
 97    ego_cam_cfg = getattr(env_cfg.scene, "ego_cam", None)
 98    if ego_cam_cfg is not None:
 99        visualizer_cfg.streaming_sensor_prim_path = _resolve_env_regex_path(ego_cam_cfg.prim_path)
100        return visualizer_cfg
101
102    # Here are other robot mounted camera options for this environment
103    # visualizer_cfg.streaming_sensor_prim_path = "/World/envs/env_.*/Robot/left_arm_camera_sim_view_frame/left_camera"
104    # visualizer_cfg.streaming_sensor_prim_path = (
105    #     "/World/envs/env_.*/Robot/right_arm_camera_sim_view_frame/right_camera"
106    # )
107    visualizer_cfg.streaming_sensor_prim_path = None
108    visualizer_cfg.streaming_cam_eye = (3.0, 3.0, 3.0)
109    visualizer_cfg.streaming_cam_target_prim_path = "/World/envs/*/Robot/base"
110    return visualizer_cfg
111
112
113def _configure_visualizers(env_cfg, args_cli: argparse.Namespace) -> None:
114    """Attach tiled camera visualizer configs to the environment simulation config."""
115    visualizers = _requested_visualizers(args_cli)
116    args_cli.visualizer = visualizers
117    env_cfg.sim.visualizer_cfgs = [
118        _make_kit_visualizer_cfg(env_cfg) if visualizer == "kit" else _make_newton_visualizer_cfg(env_cfg)
119        for visualizer in visualizers
120    ]
121
122
123def _resolve_task(args_cli: argparse.Namespace) -> str:
124    """Resolve the task for the selected visualizer."""
125    if args_cli.task is not None:
126        return args_cli.task
127    if "newton" in _requested_visualizers(args_cli):
128        return NEWTON_DEFAULT_TASK
129    return KIT_DEFAULT_TASK
130
131
132# add argparse arguments
133parser = argparse.ArgumentParser(description="Showcase the Kit/Newton visualizer tiled camera panel.")
134parser.add_argument("--num_envs", type=int, default=None, help="Number of environments to simulate.")
135parser.add_argument("--task", type=str, default=None, help="Name of the task.")
136# append AppLauncher cli args
137add_launcher_args(parser)
138args_cli, hydra_args = setup_preset_cli(parser)
139args_cli.task = _resolve_task(args_cli)
140sys.argv = [sys.argv[0]] + hydra_args
141
142
143def main():
144    """Run a random-action environment with a tiled camera visualizer."""
145    # parse configuration via Hydra (supports preset selection, e.g. presets=newton_mjwarp)
146    env_cfg, _ = resolve_task_config(args_cli.task, "")
147    _configure_visualizers(env_cfg, args_cli)
148
149    with launch_simulation(env_cfg, args_cli):
150        # override with CLI arguments
151        env_cfg.scene.num_envs = args_cli.num_envs if args_cli.num_envs is not None else env_cfg.scene.num_envs
152        env_cfg.sim.device = args_cli.device if args_cli.device is not None else env_cfg.sim.device
153
154        # create environment
155        env = gym.make(args_cli.task, cfg=env_cfg)
156
157        # print info (this is vectorized environment)
158        print(f"[INFO]: Gym observation space: {env.observation_space}")
159        print(f"[INFO]: Gym action space: {env.action_space}")
160        env.reset()
161
162        # keep stepping until all visualizer windows have been closed
163        sim = env.unwrapped.sim
164        if not sim.visualizers:
165            print("[WARN]: No visualizers found. Exiting.")
166            env.close()
167            return
168
169        while True:
170            if sim.visualizers and not any(v.is_running() and not v.is_closed for v in sim.visualizers):
171                break
172            with torch.inference_mode():
173                actions = 2 * torch.rand(env.action_space.shape, device=env.unwrapped.device) - 1
174                env.step(actions)
175
176        env.close()
177
178
179if __name__ == "__main__":
180    main()

See Examples below for the two ways the script can be run, and Usage for the VisualizerCfg fields that customize streaming behavior.

Overview#

Kit launches the streaming view as a separate Streaming View viewport, selectable from the Viewport tabs; it can also be placed side by side with the default interactive viewport for dual monitoring.

Newton GL shows a Streaming View section in the HUD sidebar with a Hide / Open toggle to show or hide the panel, and a source dropdown to select between different camera sensors.

Examples#

Running run_tiled_camera_visualizer.py demonstrates two ways to use the streaming camera view:

  • auto-created cameras pointed at and following moving AnymalD robots, shown in the Kit visualizer

  • streaming from existing wrist-mounted robot cameras, shown in the Newton visualizer

Example 1: Following AnymalD Robots#

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

The script’s KitVisualizerCfg creates cameras that point at and follow each robot’s base prim, offset by streaming_cam_eye (here (3.0, 3.0, 3.0); try (0, 0, 5) for a top-down view). Of the 256 environments, 36 are randomly sampled for the camera view.

Kit visualizer: interactive viewport

Kit visualizer: streaming camera view

Example 2: Streaming from Robot-Mounted Cameras#

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl

The Galbot cube-stacking environment ships with wrist-mounted cameras giving an egocentric view of the gripper, table, and cubes. The script’s NewtonGLVisualizerCfg streams from the existing sensor at /World/envs/env_.*/Robot/head_camera_sim_view_frame/head_camera; edit streaming_sensor_prim_path to show a different camera. Of the 25 environments, 12 camera feeds are shown by default.

Newton visualizer: interactive viewport

Newton visualizer: streaming camera view

Usage#

Configuration notes#

To customize streaming camera behavior, edit the highlighted VisualizerCfg fields in run_tiled_camera_visualizer.py:

  • For auto-created cameras, streaming_cam_target_prim_path chooses the followed prim and streaming_cam_eye sets the camera offset from that prim. Defaults to None, which causes the visualizer to adopt the first scene camera it discovers at init; no explicit path is needed when a TiledCamera sensor is already in the scene.

  • For existing scene cameras, streaming_sensor_prim_path must match an Isaac Lab Camera sensor prim path in the selected task.

  • streaming_envs controls how many environment tiles are shown. Pass an int to randomly sample that many environments, or a list[int] to pin specific environment indices.

  • streaming_gt_types selects which ground-truth types are shown, e.g. ["rgb", "depth", "segmentation", "normals"].

  • streaming_depth_min / streaming_depth_max set the depth colormap range in metres.

Troubleshooting#

  • If a generated view fails with a missing prim error, verify that streaming_cam_target_prim_path resolves in each selected environment; common template forms are /World/envs/*/... and /World/envs/env_.*/.... In most cases you can leave it as None and let the visualizer adopt an existing scene camera automatically.

  • If an existing-camera view reports that no Isaac Lab camera owns the prim, check that streaming_sensor_prim_path matches a Camera sensor in the task.

  • If the depth panel shows a flat color, adjust streaming_depth_min and streaming_depth_max to bracket the expected depth range in your scene.

  • If the view is too expensive, reduce streaming_envs, --num_envs, or the camera resolution.

Warning

Newton MJWarp with replicate_physics=True and auto-created cameras

With replicate_physics=True, only env_0 has a USD prim after physics initialization. Cameras for the remaining environments (env_1 through env_{N-1}) are dropped, causing initialization to fail:

RuntimeError: Number of camera prims in the view (1) does not match
the number of environments (N).

Workaround: set streaming_sensor_prim_path to a scene camera that was declared in the scene config before physics init (for example, a TiledCamera on a vision-based task).

See also#