Renderers

Renderers produce camera-sensor buffers for policy observations and synthetic-data workflows. They are distinct from visualizers, which provide human-facing, interactive views for inspection, debugging, and recording. Isaac Lab uses a pluggable renderer architecture. All implementations follow the interface defined by BaseRenderer.

Isaac Lab supports three rendering backends. The table below summarizes their runtime requirements and primary use cases. See Renderer details for their execution models, output coverage, and limitations, and Renderer support for the output-by-backend support matrix.

Choosing a renderer backend

Backend	Requires Isaac Sim?	Best For
Isaac RTX	Yes	Full sensor fidelity, RTX photorealism, PhysX backend
OVRTX	No (kit-less; needs isaaclab_ov + ovrtx)	RTX-quality rendering without requiring Isaac Sim
Newton Warp	No (kit-less)	Newton backend, fast training

Renderer outputs at a glance

The galleries below use the same authored scene, camera, lights, materials, and initial conditions. Six spheres exercise mirror-like, transparent, semi-transparent, matte, glossy, and emissive materials. The RGB output is animated to show the spheres falling onto the table; the remaining outputs are still frames from the same run. Stills use the sixth rendered frame so temporal outputs, such as motion vectors, show useful motion while the spheres remain near their initial poses.

Treat the images as a qualitative comparison of feature coverage and image character, not a performance benchmark. The kit-less renderers use Newton physics while Isaac RTX uses PhysX, so the exact sphere poses can differ. Display-only color maps make scalar, vector, and label outputs readable here; camera sensors still return their documented raw tensors. Closely related aliases and distance or ID variants are omitted because they do not add a visually distinct mode.

Newton Warp

RGB animation

Albedo

Depth. Turbo colors progress from near (purple) to far (red).

Normals

Semantic segmentation

Instance segmentation

OVRTX

RGB animation

Albedo

Depth. Turbo colors progress from near (purple) to far (red).

Normals

Semantic segmentation

Instance segmentation

Motion vectors. Color encodes direction and relative magnitude; arrows show sampled image-space motion.

Constant diffuse shading

Diffuse MDL shading

Full MDL shading

Isaac RTX

RGB animation

Albedo

Depth. Turbo colors progress from near (purple) to far (red).

Normals

Semantic segmentation

Instance segmentation

Motion vectors. Color encodes direction and relative magnitude; arrows show sampled image-space motion.

Constant diffuse shading

Diffuse MDL shading

Full MDL shading

Renderer details

Open a renderer below for its execution model, output coverage, limitations, and typical use case. The gallery above remains a visual comparison; the camera support matrix is the authoritative output-by-backend reference.

Newton Warp renderer

• Runtime and physics: A lightweight, kit-less rasterizer built on NVIDIA Warp and paired with Newton physics.

• Output coverage: Produces RGB, albedo, depth, normals, semantic segmentation, and instance segmentation. It does not produce motion vectors or RTX material transport. Its focused raster pipeline exposes fewer ground-truth outputs than the RTX renderers, which integrate broader RTX and annotator capabilities.

• Best for: Training workflows where throughput matters more than full RTX fidelity.

OVRTX renderer

• Runtime and physics: Provides kit-less RTX rendering through the isaaclab_ov extension and pairs with Newton physics.

• Output coverage: Provides RTX Minimal and photo-real rendering with geometry, motion, and label outputs.

• Best for: RTX image quality without running Isaac Sim.

Isaac RTX renderer

• Runtime and physics: Runs NVIDIA’s Omniverse RTX rendering pipeline inside Isaac Sim and pairs with PhysX.

• Output coverage: Provides RTX Minimal and photo-real rendering, plus the broadest camera-output coverage.

• Best for: Full RTX fidelity and workflows that already depend on Isaac Sim or Kit.

Choosing a rendering capability

The overview above shows the complete range of visually distinct outputs in one place. The sections below regroup those outputs by purpose: simplified rendering for throughput-oriented training, photo-real rendering for full RTX image quality, and advanced outputs for geometry, motion, and labels. The detailed captions and commands use the suffixless Isaac-Cartpole-Camera task so each shown mode can be tried without editing Python.

Simplified rendering

Simplified rendering prioritizes throughput and predictable image formation over full light transport. Newton Warp provides a lightweight rasterized RGB path. OVRTX and Isaac RTX provide RTX Minimal mode, which disables indirect lighting and offers three levels of material evaluation:

• Constant diffuse uses one constant surface color.

• Diffuse MDL is Isaac Lab’s stable name for textured diffuse shading.

• Full MDL is Isaac Lab’s stable name for diffuse, glossy, and emissive material evaluation.

RTX Minimal uses the first distant light in the scene and hard shadows. See the upstream OVRTX Minimal mode and RTX Minimal renderer documentation for the renderer-level settings and limitations.

Newton Warp RGB

Without shadows (default)

With shadows

Newton Warp is the kit-less choice when a lightweight RGB observation is sufficient.

uv run isaaclab train --rl_library rsl_rl \
   --task Isaac-Cartpole-Camera \
   physics=newton_mjwarp renderer=newton_renderer presets=rgb

Enable directional-light shadows explicitly:

uv run isaaclab train --rl_library rsl_rl \
   --task Isaac-Cartpole-Camera \
   physics=newton_mjwarp renderer=newton_renderer presets=rgb \
   env.scene.tiled_camera.renderer_cfg.enable_shadows=true

OVRTX Minimal

Constant diffuse — presets=simple_shading_constant_diffuse

Diffuse MDL — presets=simple_shading_diffuse_mdl

Full MDL — presets=simple_shading_full_mdl

Constant diffuse

uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole-Camera \
   physics=newton_mjwarp renderer=ovrtx presets=simple_shading_constant_diffuse

Textured diffuse

uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole-Camera \
   physics=newton_mjwarp renderer=ovrtx presets=simple_shading_diffuse_mdl

Diffuse, glossy, and emissive material evaluation

uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole-Camera \
   physics=newton_mjwarp renderer=ovrtx presets=simple_shading_full_mdl

Isaac RTX Minimal

Constant diffuse — presets=simple_shading_constant_diffuse

Diffuse MDL — presets=simple_shading_diffuse_mdl

Full MDL — presets=simple_shading_full_mdl

Constant diffuse

uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole-Camera \
   physics=isaacsim_physx renderer=isaacsim_rtx presets=simple_shading_constant_diffuse

Textured diffuse

uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole-Camera \
   physics=isaacsim_physx renderer=isaacsim_rtx presets=simple_shading_diffuse_mdl

Diffuse, glossy, and emissive material evaluation

uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole-Camera \
   physics=isaacsim_physx renderer=isaacsim_rtx presets=simple_shading_full_mdl

Photo-real rendering

Here, photo-real rendering means the regular RGB path from the full RTX Real-Time Path-Tracing mode; it is a capability grouping, not an Isaac Lab preset name. Choose it when material appearance, reflections, transparency, lighting, or the accompanying RTX AOVs matter more than the lowest possible render latency. OVRTX provides this path without Kit, while Isaac RTX provides it inside Isaac Sim. See the upstream OVRTX render modes and Isaac Sim rendering modes.

OVRTX

OVRTX photo-real RGB — renderer=ovrtx presets=rgb

The same renderer also produces the albedo, depth, normals, segmentation, and motion-vector outputs shown in the overview gallery.

uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole-Camera \
   physics=newton_mjwarp renderer=ovrtx presets=rgb

Isaac RTX

Isaac RTX photo-real RGB — renderer=isaacsim_rtx presets=rgb

The same renderer also produces the albedo, depth, normals, segmentation, and motion-vector outputs shown in the overview gallery.

uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole-Camera \
   physics=isaacsim_physx renderer=isaacsim_rtx presets=rgb

Advanced rendering outputs

Beyond RGB, renderer-produced buffers expose material, geometry, motion, and labeling information:

• Albedo isolates the material base color from lighting.

• Depth measures optical-axis distance for geometric perception and reconstruction.

• Normals encode local surface orientation.

• Semantic segmentation groups pixels by class, while instance segmentation separates individual objects.

• Motion vectors encode image-space motion per pixel and require prior-frame history.

Request one or more buffers through data_types. See Configure a camera for a configuration example and Output types for the available names, tensor shapes, data types, and meanings. Environment-level presets=... selectors are task-defined convenience shortcuts, not an exhaustive list of camera outputs.

Output availability differs by backend. Check the camera renderer support matrix before choosing a renderer; for example, OVRTX and Isaac RTX produce motion vectors, while Newton Warp does not. The renderer visual comparison above shows these outputs on the same scene.

Note

Visualization markers are debug overlays provided by visualizers, not camera outputs. Their support is independent of the camera renderer: the Kit, Newton GL, Rerun, and Viser visualizers support markers, while the experimental Newton RTX visualizer does not. See Visualization for the visualizer support matrix.

Note

Temporal information for camera-based RL. Unlike RTX modes with temporal anti-aliasing (DLSS, DLAA, TAA), the Newton Warp renderer does not inject prior-frame information into the current image. Camera-control tasks that depend on velocity-like visual cues should add explicit temporal observations (e.g. task-local frame stacking) rather than relying on renderer-specific artifacts.

Per-environment Isaac RTX scene partitioning

The Isaac RTX renderer enables per-environment scene partitioning by default. It assigns matching scene-partition tokens to each /World/envs/env_<index> hierarchy and its camera so tiled views render only that environment’s geometry.

Configure the behavior through IsaacRtxRendererCfg:

from isaaclab_physx.renderers import IsaacRtxRendererCfg

renderer_cfg = IsaacRtxRendererCfg(enable_scene_partitioning=False)

Scene partitioning and the all-environment spectator view are separate controls. KitLauncher enables spectator support before RTX startup only when the Kit viewport is enabled or Kit visualization, recording, livestreaming, or XR is requested. Regular headless training and camera-sensor runs keep it disabled so tiled cameras are not exposed to the spectator mode’s world-space layout constraints.

global_settings.show_all_partitions_by_default maps to that same process-global RTX setting; it is not a separate feature. Its default value of None preserves the launch-time choice made by KitLauncher. An explicit value overrides that setting when the Isaac RTX renderer is constructed. When enabled, environments must remain spatially separated because overlapping partition bounds can make content leak into another environment or disappear. When disabled, the Kit viewport displays only the selected environment.

This setting does not affect OVRTX, which always partitions multi-environment scenes.

Prims outside the environment hierarchies remain in the shared background partition. Environment-owned PointInstancer markers can carry one matching scene-partition token per instance; markers without that ownership information remain shared.

Warning

The Isaac RTX and OVRTX renderers cap the number of scene partitions at 15625. Requesting more than 15625 environments with scene partitioning enabled discards the additional partitions, and rtx.scenedb.plugin logs SceneDbContext : Maximum number of scene partitions (15625) reached. Additional scene partitions will be discarded. Environments beyond that count then share a partition with another environment, so their tiled camera views can show another environment’s geometry. See Scene partitioning is capped at 15625 partitions for details.

Architecture Overview

The renderer system consists of:

1. BaseRenderer — Abstract base class defining the rendering lifecycle and interface

2. RendererCfg — A BackendCfg subclass; each backend extends it with backend-specific options and declares its implementation in class_type

3. Concrete implementations — Backend-specific renderers in extension packages

4. SimulationContext — Owns renderer instances in its backend registry and shares them across equal configurations of the same concrete type through get_or_create_backend(renderer_cfg).

5. RenderContext — Coordinates initialization, stage preparation, scene updates, and material writers using a filtered view of the simulation registry. It does not construct, cache, or close renderer instances.

import isaaclab.sim as sim_utils
from isaaclab.renderers import BaseRenderer
from isaaclab_newton.renderers import NewtonWarpRendererCfg

# Create a Newton Warp renderer (no Isaac Sim required)
sim_ctx = sim_utils.SimulationContext.instance()
# Reuse a matching renderer or construct one through instantiate(cfg).
renderer: BaseRenderer = sim_ctx.get_or_create_backend(NewtonWarpRendererCfg())
assert isinstance(renderer, BaseRenderer)

For the RTX renderer (requires Isaac Sim):

import isaaclab.sim as sim_utils
from isaaclab.renderers import BaseRenderer
from isaaclab_physx.renderers import IsaacRtxRendererCfg

# Create an RTX renderer
sim_ctx = sim_utils.SimulationContext.instance()
# Reuse a matching renderer or construct one through instantiate(cfg).
renderer: BaseRenderer = sim_ctx.get_or_create_backend(IsaacRtxRendererCfg())

For RTX renderer settings, see Select and configure a Renderer.

Batching camera renders

render_batch() accepts a sequence of render-data objects owned by the same renderer. Prepare the camera poses, intrinsics, and shared scene state before rendering, then read each camera’s output. An empty sequence performs no rendering.

renderer.render_batch([first_render_data, second_render_data])
renderer.read_output(first_render_data, first_camera_data)
renderer.read_output(second_render_data, second_camera_data)

render() continues to accept a single render-data object. The default render_batch() implementation calls render() for each entry, so existing custom renderers and single-camera callers need no changes. OVRTX steps all registered render products together but publishes only the requested cameras’ observations.

With eager sensor updates (scene.cfg.lazy_sensor_update=False), scene.update() advances sensor clocks in scene order and collects batch-capable sensors. After the loop, the camera batch implementation prepares the remaining due captures for submission. render_into_cameras() groups these requests by renderer instance, renders each group, and reads its outputs. The context does not retain a pending-camera queue. Each camera retains its own update period and reset state.

With lazy sensor updates, reading a camera’s data refreshes only that camera’s sensor buffers and capture timestamps. It does not refresh peer camera sensors sharing the renderer.

Core concepts

• Use the simulation registry: Always acquire renderers with a renderer-specific config class through sim_ctx.get_or_create_backend(IsaacRtxRendererCfg()). Do not import or instantiate concrete backend classes (e.g. IsaacRtxRenderer, OVRTXRenderer) directly—their names and package locations are implementation details and may change without notice.

• Lightweight config imports: Importing a renderer configuration class does not pull in backend-specific dependencies. class_type is resolved lazily when the renderer is constructed, and construction may fail if the backend is not installed.

  import isaaclab.sim as sim_utils
  from isaaclab.renderers import BaseRenderer
  # Lightweight: does not import OVRTX backend dependencies
  from isaaclab_ov.renderers import OVRTXRendererCfg
  
  # Lazily loads ovrtx when instantiated; may fail if isaaclab_ov / ovrtx is not installed
  sim_ctx = sim_utils.SimulationContext.instance()
  renderer: BaseRenderer = sim_ctx.get_or_create_backend(OVRTXRendererCfg())
  

Installing the OVRTX renderer

The OVRTX renderer is provided by the isaaclab_ov extension. The extension’s source package ships with the core install, but the renderer’s ovrtx runtime wheel (the ovrtx package, published on public PyPI) is not installed by default. You must request it explicitly — OVRTX does not require Isaac Sim.

Install via the Isaac Lab CLI using the ov[ovrtx] token:

# Install the ovrtx runtime wheel on top of an existing install
./isaaclab.sh -i ov[ovrtx]

Note

The bare ov token does not install any runtime wheel (the source packages are already part of the core install). Use ov[ovrtx] (or ov[all]) to pull in the ovrtx dependency.

Or install the public ovrtx package directly from PyPI:

pip install "ovrtx==0.5.0.377615"

• Opaque render data: The render data object returned by create_render_data() is passed to subsequent renderer methods. It should be completely opaque to the caller: inspecting or modifying it via get/set attributes is an anti-pattern and breaks the API contract.

Note

The BaseRenderer class is under active development and may change without notice.

Asynchronous Rendering

async_rendering returns a camera’s previous capture while rendering its next image. This allows rendering to overlap simulation and Python work. The first capture, including after reset, waits for a fresh image. The ovstage path stays synchronous.

Latency is one capture, not necessarily one physics or control step. State observations remain current, so enabling this option changes the policy’s observation timing; training may need to account for that delay. Physics and policy computation may also use the GPU.

For a camera capturing once per environment step, steady-state operation is:

Synchronous:   physics k -> submit image k -> wait/read image k -> policy
Asynchronous:  physics k -> submit image k -> wait/read image k-1 -> policy
                                |                                 |
                                +---- image k renders ------------+--> physics k+1

The first image is returned immediately after priming and repeated once on the next capture.

Each camera owns its pending observations. Reading camera A never changes camera B’s pixels or metadata, even when they share a native render submission. Reset discards that camera’s pending observations. A partial environment reset re-primes the whole tiled camera product, without invalidating other cameras.

Live fields such as camera.data.pos_w and camera.data.intrinsic_matrices remain current. Use the capture metadata when pairing delayed pixels with a pose or calibration, for example when deprojecting depth:

data = camera.data
depth = data.output["distance_to_image_plane"]
capture = data.info["distance_to_image_plane"]["capture"]
position, orientation = capture["pos_w"], capture["quat_w_world"]
intrinsics, frame = capture["intrinsic_matrices"], capture["frame"]

Capture metadata remains unchanged on later captures. Direct renderer callers use prepare_capture() before submitting a capture and read_output() to publish it.

SDP fills renderer-owned transform and geometry buffers, including deformable meshes, particles, and cables. Two buffers alternate per write and are reused only after OVRTX consumes them. Physics can therefore update its live arrays while the previous capture renders.

See Also

• Scene Data Provider: how scene data flows from physics backends to renderers

• Visualization — lightweight visualizer backends for interactive feedback