Streaming a camera view in a visualizer

Streaming a camera view in a visualizer#

Display ground-truth camera frames from many environments in one live visualizer panel. For how the view works and which visualizers support it, see Streaming Camera View.

Sharing a view with recording#

ImageViewCfg selects pixels independently of their destination. A scene sensor name binds an existing camera; its renderer and capture cadence remain owned by the scene. WindowCfg controls presentation independently through VisualizerCfg.window. VisualizerCfg.view and VideoRecorderCfg.source can consume the same selection:

from isaaclab.envs import VideoRecorderCfg
from isaaclab.visualizers import ImageViewCfg, WindowCfg
from isaaclab_visualizers.newton import NewtonGLVisualizerCfg

view = ImageViewCfg(source="front_camera", envs=(0, 2), channels=("rgb", "depth"))
env_cfg.sim.visualizer_cfgs = [NewtonGLVisualizerCfg(view=view, window=WindowCfg(size=(1280, 720)))]
env_cfg.video_recorders = [VideoRecorderCfg(source=view, output_dir="videos")]

Use --viz newton_gl (or --viz newton_rtx for the RTX example) to display the window. Omit --viz for recording alone. The scene must declare front_camera with both requested outputs. The view preserves environment order and native tile dimensions, placing channels beside each other. channels selects existing outputs; depth_range only maps depth values to display colors. Neither changes the source camera’s outputs, measurements, or clipping planes. CameraCfg.width and height set sensor resolution. WindowCfg.size sets the initial window dimensions; the window scales the composed image for display. Resizing a sensor’s window leaves both sensor and composed-image storage unchanged. The recorder encodes the native composed image. Only encoding downloads pixels; display uses the device image. Invalid recording sources or channel selections raise errors when the recorder binds its source after simulation reset; they do not silently disable recording. Subsequent frames use the bound capture function without looking up the producer again.

Reusing the same ImageViewCfg shares one runtime frame within a simulation, including through copies of the enclosing configuration. Use isaaclab.utils.replace(view, ...) for an independent selection. Reading the shared view twice in one physics step reuses its frame and sensor acquisition. Changing the view’s selection rebuilds composition state; captured composition graphs must then be recreated. This does not resize the sensor’s render product.

For Newton RTX, a perspective declaration is expanded into an ordinary scene camera before launch:

from isaaclab.visualizers import PerspectiveCameraCfg
from isaaclab_visualizers.newton import NewtonRTXVisualizerCfg

view = ImageViewCfg(source=PerspectiveCameraCfg(eye=(4.0, -4.0, 3.0)))
env_cfg.sim.visualizer_cfgs = [
    NewtonRTXVisualizerCfg(view=view, window=WindowCfg(size=(960, 540)))
]
env_cfg.video_recorders = [VideoRecorderCfg(source=view)]

Newton RTX fixes perspective render resolution at the initial WindowCfg.size; resizing the window scales its image. Newton GL perspective rendering follows its window framebuffer dimensions. The recorder reads the completed native render without requesting a second render for that step. Perspective views currently provide one RGB image, so keep envs=(0,) and channels=("rgb",). A recorded perspective view requires exactly one configured rendering visualizer. In headless runs, the launcher retains that producer without opening a window; recording a sensor view needs no visualizer.

VisualizerCfg.view is supported by Newton GL and RTX. Existing cameras and streaming_* configurations and CLI recording source strings remain supported; they are translated at initialization. Use view instead of combining it with legacy camera selection fields.

Quick Start#

Run the run_tiled_camera_visualizer.py script in IsaacLab/scripts/tutorials/07_visualizers:

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py ^
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit
Code for run_tiled_camera_visualizer.py
  1# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
  2# All rights reserved.
  3#
  4# SPDX-License-Identifier: BSD-3-Clause
  5
  6"""
  7This script demonstrates the visualizer tiled camera panel.
  8
  9.. code-block:: bash
 10
 11    # Kit visualizer tiled camera panel
 12    uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
 13 --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit
 14
 15    # Newton visualizer tiled camera panel
 16    uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
 17        --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl
 18
 19"""
 20
 21from __future__ import annotations
 22
 23import argparse
 24import contextlib
 25import sys
 26
 27import gymnasium as gym
 28import torch
 29from isaaclab_newton.renderers import NewtonWarpRendererCfg
 30from isaaclab_physx.renderers import IsaacRtxRendererCfg
 31from isaaclab_visualizers.kit import KitVisualizerCfg
 32from isaaclab_visualizers.newton import NewtonGLVisualizerCfg
 33
 34import isaaclab_tasks  # noqa: F401
 35
 36with contextlib.suppress(ImportError):
 37    import isaaclab_tasks_experimental  # noqa: F401
 38from isaaclab.app import add_launcher_args, launch_simulation
 39from isaaclab.sensors import CameraCfg
 40from isaaclab.sim import PinholeCameraCfg
 41from isaaclab.utils.math import create_rotation_matrix_from_view, quat_from_matrix
 42from isaaclab.visualizers import SceneCameraCfg
 43
 44from isaaclab_tasks.utils import resolve_task_config, setup_preset_cli
 45
 46KIT_DEFAULT_TASK = "Isaac-Velocity-Rough-AnymalD"
 47NEWTON_DEFAULT_TASK = "IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor"
 48
 49
 50# add argparse arguments
 51parser = argparse.ArgumentParser(description="Showcase the Kit/Newton visualizer tiled camera panel.")
 52parser.add_argument("--num_envs", type=int, default=None, help="Number of environments to simulate.")
 53parser.add_argument("--task", type=str, default=None, help="Name of the task.")
 54# append simulation launcher cli args
 55add_launcher_args(parser)
 56parser.set_defaults(visualizer=["kit"])
 57args_cli, hydra_args = setup_preset_cli(parser)
 58if args_cli.task is None:
 59    visualizers = next(arg.split("=", 1)[1].split(",") for arg in hydra_args if arg.startswith("visualizer="))
 60    args_cli.task = NEWTON_DEFAULT_TASK if {"newton", "newton_gl"}.intersection(visualizers) else KIT_DEFAULT_TASK
 61sys.argv = [sys.argv[0]] + hydra_args
 62
 63
 64def main():
 65    """Run a random-action environment with a tiled camera visualizer."""
 66    # parse configuration via Hydra (supports preset selection, e.g. presets=newton_mjwarp)
 67    env_cfg, _ = resolve_task_config(args_cli.task, "")
 68    # Declare the shared scene camera before launching either viewer.
 69    visualizers = env_cfg.sim.visualizer_cfgs
 70    visualizers = visualizers if isinstance(visualizers, list) else [visualizers] if visualizers else []
 71    if not visualizers or any(not isinstance(cfg, (KitVisualizerCfg, NewtonGLVisualizerCfg)) for cfg in visualizers):
 72        raise ValueError("This tutorial supports --viz kit or --viz newton_gl.")
 73    camera_cfg = next((cfg for cfg in vars(env_cfg.scene).values() if isinstance(cfg, CameraCfg)), None)
 74    if camera_cfg is None:
 75        eye, target = torch.tensor([[3.0, 3.0, 3.0]]), torch.zeros((1, 3))
 76        rotation = quat_from_matrix(create_rotation_matrix_from_view(eye, target, device="cpu"))[0]
 77        # Attaching the sensor to the base gives it the robot's pose through the normal camera view.
 78        camera_cfg = CameraCfg(
 79            prim_path="{ENV_REGEX_NS}/Robot/base/StreamingCamera",
 80            width=320,
 81            height=240,
 82            data_types=["rgb"],
 83            spawn=PinholeCameraCfg(focal_length=24.0, clipping_range=(0.1, 1.0e5)),
 84            offset=CameraCfg.OffsetCfg(pos=(3.0, 3.0, 3.0), rot=tuple(rotation.tolist()), convention="opengl"),
 85            renderer_cfg=(
 86                IsaacRtxRendererCfg()
 87                if any(isinstance(cfg, KitVisualizerCfg) for cfg in visualizers)
 88                else NewtonWarpRendererCfg()
 89            ),
 90        )
 91        env_cfg.scene.streaming_camera = camera_cfg
 92    for cfg in visualizers:
 93        cfg.streaming_view = True
 94        cfg.streaming_envs = 36 if isinstance(cfg, KitVisualizerCfg) else 12
 95        cfg.cameras = [SceneCameraCfg(prim_path=camera_cfg.prim_path)]
 96
 97    with launch_simulation(env_cfg, args_cli):
 98        # override with CLI arguments
 99        env_cfg.scene.num_envs = args_cli.num_envs if args_cli.num_envs is not None else env_cfg.scene.num_envs
100
101        # create environment
102        env = gym.make(args_cli.task, cfg=env_cfg)
103
104        # print info (this is vectorized environment)
105        print(f"[INFO]: Gym observation space: {env.observation_space}")
106        print(f"[INFO]: Gym action space: {env.action_space}")
107        env.reset()
108
109        # keep stepping until all visualizer windows have been closed
110        while env.unwrapped.sim.is_running():
111            with torch.inference_mode():
112                actions = 2 * torch.rand(env.action_space.shape, device=env.unwrapped.device) - 1
113                env.step(actions)
114
115        env.close()
116
117
118if __name__ == "__main__":
119    main()

See Examples below for the two ways to run the script. For the VisualizerCfg fields that customize streaming, see Configuration, Display sources, and Troubleshooting.

Choose and control displayed views#

Kit launches the streaming view as a separate Streaming View viewport, selectable from the Viewport tabs; it can also be placed side by side with the default interactive viewport for dual monitoring.

Newton GL and Newton RTX have a Camera View selector in the sidebar. Selecting a scene camera displays its tiled output in the main viewport instead of rendering a perspective view behind it. Selecting a perspective camera restores interactive scene rendering. Newton GL also restores picking.

Without an explicit cameras list, Newton GL starts in perspective and offers scene cameras that provide every requested streaming_gt_types channel. An explicit streaming_sensor_prim_path starts with that scene camera selected. For example, set streaming_gt_types=("depth",) to offer depth-only cameras. The RGB view accepts RGB or RGBA outputs; the display uses opaque alpha. No images are captured to build the selector. Explicitly configured scene-camera choices must support the requested channels; incompatible choices raise a configuration error during initialization, including inactive choices.

Newton RTX displays only the sources declared in view or cameras. With neither set, launch preparation declares one perspective camera. Scene sensors are never added to its selector automatically.

Use cameras to declare a list of selectable views; the first starts active:

from isaaclab.visualizers import PerspectiveCameraCfg, SceneCameraCfg
from isaaclab_visualizers.newton import NewtonGLVisualizerCfg

viewer_cfg = NewtonGLVisualizerCfg(
    cameras=[
        SceneCameraCfg(prim_path="{ENV_REGEX_NS}/Robot/FrontCamera"),
        SceneCameraCfg(prim_path="{ENV_REGEX_NS}/Robot/BackCamera"),
        PerspectiveCameraCfg(eye=(4.0, -4.0, 3.0)),
    ],
    streaming_envs=list(range(16)),
)

The cameras field is shared across visualizers. Kit, Rerun, and Viser use the first scene camera for their image panel; Newton GL and RTX provide the source dropdown shown above. SceneCameraCfg selects an existing sensor; PerspectiveCameraCfg configures the viewer’s interactive camera. For Newton RTX, launch_simulation(env_cfg, ...) adds a CameraCfg to the scene before any scene objects are constructed and selects it through SceneCameraCfg. Newton GL retains its native perspective camera. Newton RTX keeps its perspective render resolution fixed at the initial WindowCfg.size; resizing its window scales the displayed image. Sensor resolution remains controlled by CameraCfg. Window resizing does not reallocate sensor or RTX perspective image buffers. Newton GL perspective images follow its framebuffer dimensions. Newton windows present at up to WindowCfg.fps (30 by default), measured in wall-clock time. Changing the simulation timestep does not change that limit. Headless on-demand capture is independent of the window update rate; Kit uses its application’s update cadence.

Newton RTX displays ordinary camera output in a Newton GL window. Equal OVRTXRendererCfg configurations share one renderer and its scene. NewtonRTXVisualizerCfg.renderer_cfg selects the renderer for generated perspective cameras; explicitly selected scene cameras retain their own renderer. Newton’s ViewerRTX and its borrowed-stage interface are not used. isaaclab_ov.stage.OvstageBackend remains available for downstream consumers that need an independent native stage. Its populate(stage, plan) method imports and clones a declared scene.

The camera owns pose, calibration, image buffers, capture, and cleanup. Window navigation calls Camera.set_world_poses and Camera.set_intrinsic_matrices; image reads use Camera.data. There is no separate perspective rendering entry point. No default lights are added: scenes without authored lights remain unlit. Closing a window leaves its scene camera and renderer alive.

Scripts without an InteractiveSceneCfg use ReplicateSession to declare their scene before spawning assets and cameras, including for a single environment:

from isaaclab.app import launch_simulation
from isaaclab.assets import AssetBaseCfg
from isaaclab.cloner import ReplicateSession
from isaaclab.sensors import CameraCfg
from isaaclab.sim import CuboidCfg, DomeLightCfg, PinholeCameraCfg, SimulationCfg, SimulationContext
from isaaclab.utils import instantiate
from isaaclab.visualizers import SceneCameraCfg
from isaaclab_ov.renderers import OVRTXRendererCfg
from isaaclab_visualizers.newton import NewtonRTXVisualizerCfg

camera_cfg = CameraCfg(
    prim_path="/World/PerspectiveCamera", width=960, height=540,
    spawn=PinholeCameraCfg(focal_length=12.0), renderer_cfg=OVRTXRendererCfg(),
    offset=CameraCfg.OffsetCfg(pos=(0.0, 0.0, 5.0), convention="opengl"),
    render_settings={"omni:rtx:quality": ("Int", 0)},
)
sim_cfg = SimulationCfg(
    visualizer_cfgs=[NewtonRTXVisualizerCfg(cameras=[SceneCameraCfg(prim_path=camera_cfg.prim_path)])]
)
target_cfg = AssetBaseCfg(prim_path="/World/Target", spawn=CuboidCfg(size=(1.0, 1.0, 1.0)))
light_cfg = AssetBaseCfg(prim_path="/World/Light", spawn=DomeLightCfg(intensity=1000.0))
with launch_simulation(sim_cfg, {"physics": "ovphysx", "visualizer": ["newton_rtx"]}):
    sim = SimulationContext(sim_cfg)
    with ReplicateSession((target_cfg, light_cfg, camera_cfg), num_clones=1, env_spacing=0.0):
        instantiate(target_cfg)
        instantiate(light_cfg)
        camera = instantiate(camera_cfg)
    sim.reset()
    while sim.is_running():
        sim.step()
        camera.update(sim.get_physics_dt())

The script retains the camera and advances its sensor timing, just as it does for other directly spawned sensors. SceneCameraCfg discovers it by prim path without an InteractiveScene.

Display layout and channel mappings are compiled when the source, selection, resolution, or color settings change. Composition writes into reusable device RGBA storage. Newton GL, Newton RTX, and Kit present CUDA images without downloading pixels. Recording through render_tiled_rgb_array() and the Rerun/Viser transports explicitly read back the composed image. The GPU API, render_tiled_rgba_array(), returns borrowed storage that is reused on the next composition; consumers must copy it if they need to retain a frame.

Both scene cameras must already be declared as CameraCfg entries in the scene. Keyboard and mouse navigation moves the selected sensor’s copies by the same camera-local translation and rotation across all environments, including copies not displayed. It does not move other camera choices. These are real sensor pose changes, so policies and other viewers using that sensor observe the new viewpoint too. Navigation reads the live sensor pose, including parent-body motion, independently of CameraCfg.update_latest_camera_pose. Pausing rendering freezes the displayed image and sensor navigation; pausing training alone does not.

With InteractiveSceneCfg.lazy_sensor_update=True, the viewer reads only the selected camera. Unselected cameras are still allocated and may render if another consumer reads them or the scene uses eager sensor updates. streaming_envs selects displayed tiles, not capture work.

Examples#

Running run_tiled_camera_visualizer.py demonstrates two ways to use the streaming camera view:

  • scene-declared cameras attached to AnymalD robot bases, shown in the Kit visualizer

  • streaming from existing wrist-mounted robot cameras, shown in the Newton visualizer

Example 1: Following AnymalD Robots#

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py ^
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

The script adds a CameraCfg beneath each robot’s base before constructing the environment. Its offset defines the view relative to the base, so the camera follows the robot through the normal sensor lifecycle. Of the 256 environments, 36 are sampled for display.

Kit visualizer: interactive viewport

Kit visualizer: streaming camera view

Example 2: Streaming from Robot-Mounted Cameras#

uv run --extra teleop python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl

uv run --extra teleop python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py ^
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl

The Galbot cube-stacking environment ships with wrist-mounted cameras giving an egocentric view of the gripper, table, and cubes. The script’s NewtonGLVisualizerCfg streams from the existing sensor at /World/envs/env_.*/Robot/head_camera_sim_view_frame/head_camera; edit streaming_sensor_prim_path to show a different camera. Of the 25 environments, 12 camera feeds are shown by default.

Newton visualizer: interactive viewport

Newton visualizer: streaming camera view

Configuration#

Declare camera resolution, pose, renderer, and output types on CameraCfg in the scene. Then choose what to display in VisualizerCfg:

  • cameras=[SceneCameraCfg(prim_path=...)] selects a scene camera; view=ImageViewCfg(source="name") selects a named sensor and its image layout. Newton RTX uses these explicit declarations.

  • For other viewers, streaming_sensor_prim_path selects a scene Camera by its configured prim path. The {ENV_REGEX_NS} macro uses the scene’s environment template. If omitted, Newton GL starts in perspective mode and offers compatible scene cameras in the Camera View dropdown. Kit and web viewers select the first compatible scene camera; without one, their panel stays empty.

  • streaming_envs controls how many environment tiles are shown. Pass an int to randomly sample that many environments, or a list[int] to pin specific environment indices.

  • streaming_gt_types selects which ground-truth types are shown, e.g. ["rgb", "depth", "segmentation", "normals"].

  • streaming_depth_min / streaming_depth_max set the depth colormap range in metres.

Image layouts and color tables are prepared when the selection or layout changes. Colorization and tiling run on the source device and reuse the output buffer. Kit presents CUDA images directly; recording and web consumers read back the composed RGB image through render_tiled_rgb_array().

Display sources#

VisualizerCfg.cameras accepts PerspectiveCameraCfg for the interactive view and SceneCameraCfg(prim_path=...) for a camera sensor already declared in the scene. Newton GL and RTX provide a source dropdown; Kit, Rerun, and Viser use the first scene source in their camera panel. Every explicit scene source must provide the requested streaming_gt_types channels.

Troubleshooting#

--video viz:newton_gl:streaming_view records the selected scene-camera tiles, not a network stream. Without --viz, capture runs on demand without opening a window. Adding --viz newton_gl displays the visualizer; it is not required to advance recorded frames. A new physics step refreshes the image, while repeated captures within the same step reuse it.

  • If a view reports no matching camera, declare a CameraCfg in the scene and select it with cameras=[SceneCameraCfg(prim_path=...)] or view=ImageViewCfg(source="name").

  • Each streaming_gt_types entry must have a corresponding output in the selected camera’s data_types; rgba also satisfies an rgb request. Automatically offered cameras support every requested channel. Explicitly selecting an incompatible camera raises an error.

  • If the depth panel shows a flat color, adjust streaming_depth_min and streaming_depth_max to bracket the expected depth range in your scene.

  • If the view is too expensive, reduce streaming_envs, --num_envs, or the camera resolution.

Migrating from generated cameras#

Visualizers no longer create or destroy camera sensors. Replace streaming_cam_target_prim_path, streaming_cam_eye, and streaming_cam_renderer_cfg with a scene CameraCfg: place its prim_path under the desired parent, set its offset, and supply renderer_cfg there. Select that camera through cameras=[SceneCameraCfg(...)]. This puts the camera in the clone plan and the normal sensor initialization, update, and teardown lifecycle. The full option mapping is in the 3.0 migration guide.

Closing a visualizer does not affect the camera or other visualizers reading it. streaming_envs selects displayed tiles, not camera allocation or capture resolution.

See also#