Recording video clips during training#
Isaac Lab supports recording video clips during training using the
gymnasium.wrappers.RecordVideo class.
When the --video flag is enabled, Isaac Lab captures a perspective view of the scene. If a Kit or
Newton visualizer is active, that visualizer selects the video backend by default. Otherwise, the
backend is chosen automatically from the active physics and renderer stack: an Isaac Sim Kit camera or
a Newton GL headless viewer.
This feature can be enabled by installing ffmpeg and using the following command line arguments with the training
script:
--video: enables video recording during training--video_length: length of each recorded video (in steps)--video_interval: interval between each video recording (in steps)
Note that enabling recording is equivalent to enabling rendering during training, which will slow down both startup and runtime performance.
Example usage:
./isaaclab.sh train --rl_library rl_games --task=Isaac-Cartpole --video --video_length 100 --video_interval 500
The recorded videos will be saved in the same directory as the training checkpoints, under
IsaacLab/logs/<rl_workflow>/<task>/<run>/videos/train.
Overview#
The video recording feature is implemented using the VideoRecorder class. This class is responsible for resolving the video backend from the scene, capturing the video frames, and saving them to a file.
VideoRecorderCfg(isaaclab.envs.utils.video_recorder_cfg) holds resolution, backend source, and world-space perspective parameterseyeandlookat(defaults to a diagonal view of the scene).VideoRecorder(isaaclab.envs.utils.video_recorder) picks a video backend from the scene (Kit vs Newton GL), reuses an active Newton visualizer when available, and returns RGB frames viarender_rgb_array().Direct RL, Direct MARL and manager-based RL environments copy the task’s
ViewerCfgeyeandlookatinto those fields before the recorder is constructed, so training clips align with the task’s intended viewport whenorigin_typeis"world".
Configuration: VideoRecorderCfg#
The dataclass lives in isaaclab.envs.utils.video_recorder_cfg. Fields eye and lookat are
the perspective camera position and target in meters.
@configclass
class VideoRecorderCfg:
"""Configuration for :class:`~isaaclab.envs.utils.video_recorder.VideoRecorder`."""
class_type: type = VideoRecorder
"""Recorder class to instantiate; must accept ``(cfg, scene)``."""
env_render_mode: str | None = None
"""Gym render mode forwarded from the environment constructor (``"rgb_array"`` when ``--video`` is active).
Set automatically by the environment base classes; do not set manually.
"""
eye: tuple[float, float, float] = (7.5, 7.5, 7.5)
"""Perspective camera position in world space (metres).
Direct RL / MARL and manager-based RL environments overwrite this from
:attr:`~isaaclab.envs.common.ViewerCfg.eye`. Kit and renderer-selected Newton capture use this
value. Visualizer-selected Newton capture uses the active visualizer camera instead.
"""
lookat: tuple[float, float, float] = (0.0, 0.0, 0.0)
"""Perspective camera look-at target in world space (metres).
Visualizer-selected Newton capture uses the active visualizer camera instead.
"""
backend_source: Literal["visualizer", "renderer"] = "visualizer"
"""Source used to resolve the video capture backend.
``"visualizer"`` records from the active Kit or Newton visualizer when one is enabled, and falls back to the
physics/renderer stack otherwise. ``"renderer"`` ignores active visualizers and records from the backend implied by
the physics/renderer stack.
"""
window_width: int = 1280
"""Width of the recorded frame in pixels.
Task framing: ViewerCfg#
Tasks define the interactive viewer with ViewerCfg. The eye and
lookat tuples are the same values the RL base classes copy into VideoRecorderCfg (see below).
If your task uses origin_type="world", those tuples are world-space positions and match what the
perspective recorder expects.
def _viewer_cfg_value_matches_default(current: object, default: object) -> bool:
"""Return True if ``current`` matches the dataclass field default (including list/tuple equivalence)."""
if current == default:
return True
if isinstance(current, (list, tuple)) and isinstance(default, (list, tuple)):
if len(current) != len(default):
return False
Backend selection: Kit vs Newton GL#
VideoRecorder resolves the implementation from the live InteractiveScene.
With the default VideoRecorderCfg.backend_source = "visualizer", an active --visualizer kit
selects the Kit path (omni.replicator on /OmniverseKit_Persp), and an active
--visualizer newton selects the Newton GL path. If both visualizers are active, Kit takes
precedence and only one --video stream is recorded. Rerun records .rrd replay data through
the Rerun visualizer rather than producing --video clips, and Viser does not currently provide a
--video recording backend.
When the Newton visualizer selects the backend, video capture reuses its framebuffer directly. For
example,
visible_env_indices=[0, 1, 2, 3] makes both the live Newton view and the recorded clip contain
only those four simulation worlds. The same framebuffer also preserves the live camera and
viewer-side scene markers without a second Newton rendering pass. Newton UI panels, including the
tiled camera panel, are not part of the recorded scene framebuffer. This does not apply when
VideoRecorderCfg.backend_source = "renderer", because renderer-selected capture is independent
of active visualizers.
Set VideoRecorderCfg.backend_source = "renderer" to ignore active visualizers and choose from the
physics/renderer stack instead. In that mode, PhysX physics (physics=physx) or Isaac RTX
(renderer=isaacsim_rtx) selects the Kit path. Newton physics (physics=newton_mjwarp) or
the Newton Warp renderer (renderer=newton_renderer) selects the Newton GL path when no Kit
signal is present. OVRTX (renderer=ovrtx from isaaclab_ov) can pair with IsaacSim
or Newton physics; in that case the video backend is selected via the physics preset. If both Kit and
Newton GL signals are present, the Kit path is chosen.
def _select_video_backend(scene: InteractiveScene, backend_source: str) -> tuple[_VideoBackend, VisualizerCfg | None]:
"""Resolve the capture backend and visualizer configuration that selected it.
Args:
scene: The interactive scene that owns the simulation context.
backend_source: Source used to select the capture backend.
Returns:
Backend identifier and the visualizer configuration that selected it, if any.
Raises:
ValueError: If backend_source is invalid.
RuntimeError: If no supported backend is available.
"""
if backend_source not in ("visualizer", "renderer"):
raise ValueError("VideoRecorderCfg.backend_source must be either 'visualizer' or 'renderer'.")
if backend_source == "visualizer":
visualizer_cfgs = scene.sim._resolve_visualizer_cfgs()
for visualizer_type, backend in (("kit", "kit"), ("newton", "newton_gl")):
for visualizer_cfg in visualizer_cfgs:
if visualizer_cfg.visualizer_type == visualizer_type:
return backend, visualizer_cfg
physics_name = scene.sim.physics_manager.__name__.lower()
renderer_types = scene._sensor_renderer_types()
if "physx" in physics_name or "isaac_rtx" in renderer_types:
return "kit", None
if "newton" in physics_name or "newton_warp" in renderer_types:
return "newton_gl", None
raise RuntimeError(
"Video recording (--video) requires a supported backend: "
"PhysX or Isaac RTX renderer (Kit camera), or Newton physics / Newton Warp renderer (GL viewer). "
"No supported backend detected; do not use --video for this setup."
)
Construction and dispatch#
When env_render_mode is "rgb_array" (as when wrappers or scripts request RGB frames for
video), Kit and renderer-selected Newton captures are created before simulation reset. This lets the
Kit path register its fallback camera before physics initializes. Visualizer-selected Newton capture
instead binds to the initialized visualizer on the first frame and reuses its framebuffer.
def __init__(self, cfg: VideoRecorderCfg, scene: InteractiveScene):
self.cfg = cfg
self._scene = scene
self._capture: NewtonGlPerspectiveVideo | IsaacsimKitPerspectiveVideo | NewtonVisualizer | None = None
self._use_newton_visualizer = False
if cfg.env_render_mode != "rgb_array":
return
backend, visualizer_cfg = _select_video_backend(scene, cfg.backend_source)
eye = cfg.eye if visualizer_cfg is None else visualizer_cfg.eye
lookat = cfg.lookat if visualizer_cfg is None else visualizer_cfg.lookat
if backend == "newton_gl" and visualizer_cfg is not None:
self._use_newton_visualizer = True
elif backend == "newton_gl":
from isaaclab_newton.video_recording.newton_gl_perspective_video import (
create_newton_gl_perspective_video,
)
from isaaclab_newton.video_recording.newton_gl_perspective_video_cfg import NewtonGlPerspectiveVideoCfg
self._capture = create_newton_gl_perspective_video(
NewtonGlPerspectiveVideoCfg(
window_width=cfg.window_width,
window_height=cfg.window_height,
eye=eye,
lookat=lookat,
)
)
else:
from isaaclab_physx.video_recording.isaacsim_kit_perspective_video import (
create_isaacsim_kit_perspective_video,
)
from isaaclab_physx.video_recording.isaacsim_kit_perspective_video_cfg import (
IsaacsimKitPerspectiveVideoCfg,
)
self._capture = create_isaacsim_kit_perspective_video(
IsaacsimKitPerspectiveVideoCfg(
eye=eye,
lookat=lookat,
window_width=cfg.window_width,
window_height=cfg.window_height,
)
)
Customising the camera view#
When --video is passed, active Newton visualization supplies the camera and framebuffer used by
the recording, including the visualizer’s configured resolution. The Kit path copies the active Kit
visualizer’s configured position and look-at target when it drives backend selection. Otherwise, the
defaults come from
ViewerCfg:
eye = (7.5, 7.5, 7.5)— camera position in world space (metres)lookat = (0.0, 0.0, 0.0)— camera look-at target in world space (metres)Resolution
1280x720
To change the recording angle without a visualizer, override the viewer field in your task’s
environment config. The RL base classes automatically copy eye and lookat into
VideoRecorderCfg before recording starts (when origin_type is "world"), so the video clip
uses the same configured viewpoint as the interactive viewport:
from isaaclab.envs import ManagerBasedRLEnvCfg
from isaaclab.envs.common import ViewerCfg
from isaaclab.utils.configclass import configclass
@configclass
class MyTaskCfg(ManagerBasedRLEnvCfg):
viewer: ViewerCfg = ViewerCfg(
eye=(5.0, 5.0, 5.0),
lookat=(0.0, 0.0, 1.0),
)
Summary#
Stack example ( |
Video backend |
Capture mechanism |
|---|---|---|
|
Kit ( |
|
|
Newton GL ( |
|
|
Newton GL ( |
|
|
Kit ( |
Visualizer |
|
Newton GL ( |
Active visualizer framebuffer (live camera, selected worlds, and markers) |
See also#
Capturing sensor frames during training - save image-like scene sensor outputs during training
Visualization - interactive visualizers