isaaclab.renderers

Sub-package for renderer configurations and implementations.

This sub-package contains configuration classes and implementations for different renderer backends that can be used with Isaac Lab.

Classes

BaseRenderer	Abstract base class for renderer implementations.
RendererCfg	Configuration for a renderer.

Base Renderer

class isaaclab.renderers.BaseRenderer

Bases: ABC

Abstract base class for renderer implementations.

Methods:

initialize()	Post-physics one-time initialization hook.
prepare_cameras(stage, spec)	Pre-render per-camera setup the backend needs.
supported_output_types()	Per-output layout (channels + dtype) this renderer can produce.
prepare_stage(stage, num_envs)	Prepare the stage for rendering before create_render_data() is called.
create_render_data(spec)	Create render data for the given camera CameraRenderSpec.
set_outputs(render_data, output_data)	Store reference to output buffers for writing during render.
prepare_capture(render_data, camera_data, frame)	Snapshot metadata for the next capture when its image will be delivered asynchronously.
update_transforms()	Update scene transforms before rendering.
update_geometries()	Update mutable geometry attributes before rendering.
update_camera(render_data, positions, ...)	Update camera poses and supply initial calibration for the next render.
update_camera_intrinsics(render_data, ...)	Apply a proposed runtime calibration without accessing the authored USD stage.
render(render_data)	Submit a capture; read_output() publishes its camera's available observation.
render_batch(render_data)	Submit captures for a collection of cameras.
read_output(render_data, camera_data)	Read rendered outputs from the renderer into the camera data container.
cleanup(render_data)	Release renderer resources associated with the given render data.
reset(render_data[, env_ids])	Reset the renderer-owned state of a camera when its environments reset.
close()	Release resources owned by the renderer itself rather than by a render data.

Attributes:

visual_material_writer

Return the backend's shared material-writer factory, if supported.

initialize() → None

Post-physics one-time initialization hook. Called only once.

property visual_material_writer: Callable[[tuple[VisualMaterialBatch, ...]], Any] | None

Return the backend’s shared material-writer factory, if supported.

Its writer accepts None for a full sync or channel-to-material-offset device arrays plus one environment-id device array for partial writes, and provides an idempotent close().

prepare_cameras(stage: Any, spec: CameraRenderSpec) → None

Pre-render per-camera setup the backend needs.

The default implementation is a no-op. Renderer subclasses override to perform whatever per-camera initialization their backend requires — e.g. authoring stage attributes on the resolved camera prims, configuring per-tile GPU buffers, or any other state setup.

Parameters:

• stage – Scene stage the camera prims live on, or None when no stage context applies. Stage-less backends ignore it.

• spec – Immutable description of the tiled camera bundle.

abstractmethod supported_output_types() → dict[RenderBufferKind, RenderBufferSpec]

Per-output layout (channels + dtype) this renderer can produce.

Outputs absent from the mapping are not produced by this backend.

Returns:

Mapping from supported RenderBufferKind to its RenderBufferSpec.

abstractmethod prepare_stage(stage: Any, num_envs: int) → None

Prepare the stage for rendering before create_render_data() is called.

Some renderers need to export or preprocess the USD stage before creating render data. This method is called after the renderer is instantiated and before create_render_data().

Parameters:

• stage – USD stage to prepare, or None if not applicable.

• num_envs – Number of environments.

abstractmethod create_render_data(spec: CameraRenderSpec) → Any

Create render data for the given camera CameraRenderSpec.

Parameters:

spec – Immutable description of the tiled camera (paths, config, device).

Returns:

Renderer-specific data for subsequent render() / read_output() calls.

abstractmethod set_outputs(render_data: Any, output_data: dict[str, ProxyArray]) → None

Store reference to output buffers for writing during render.

Parameters:

• render_data – The render data object from create_render_data().

• output_data – Dictionary mapping output names (e.g. "rgb", "depth") to pre-allocated ProxyArray wrappers where rendered data will be written. Use .warp for the underlying warp array or .torch for a zero-copy tensor view.

prepare_capture(render_data: Any, camera_data: CameraData, frame: ProxyArray) → None

Snapshot metadata for the next capture when its image will be delivered asynchronously.

Synchronous renderers need no snapshot. Delayed captures publish their pose, calibration, and frame indices through camera_data.info[output_name]["capture"] without changing the live camera fields.

Parameters:

• render_data – Renderer-owned camera resources.

• camera_data – Current camera pose and calibration.

• frame – Current per-environment capture indices, shape (N,), dtype wp.int64.

abstractmethod update_transforms() → None

Update scene transforms before rendering.

Called to sync physics/asset pose state into the renderer’s scene representation.

abstractmethod update_geometries() → None

Update mutable geometry attributes before rendering.

Called to sync physics-driven geometry such as mesh points, extents, or other per-frame geometry buffers into the renderer’s scene representation.

abstractmethod update_camera(render_data: Any, positions: ProxyArray, orientations: ProxyArray, intrinsics: ProxyArray) → None

Update camera poses and supply initial calibration for the next render.

Backends may use intrinsics to initialize projection state. Runtime calibration changes are submitted separately through update_camera_intrinsics().

Parameters:

• render_data – The render data object from create_render_data().

• positions – Camera positions in world frame. Shape (N,), dtype wp.vec3f. Use .torch for a (N, 3) tensor view.

• orientations – Camera orientations as quaternions (x, y, z, w). Shape (N,), dtype wp.quatf. Use .torch for a (N, 4) tensor view.

• intrinsics – Camera intrinsic matrices. Shape (N,), dtype wp.mat33f. Use .torch for a (N, 3, 3) tensor view.

update_camera_intrinsics(render_data: Any, intrinsics: wp.array, parameters: wp.array) → None

Apply a proposed runtime calibration without accessing the authored USD stage.

Called only for calibration changes, independently of pose updates. The camera commits its public buffers after this succeeds. Backends validate restrictions before modifying runtime state, and consume the arrays before returning or order their reads on the producing Warp stream; the camera reuses their storage on the next call.

Parameters:

• render_data – The camera’s renderer-owned state.

• intrinsics – Complete proposed calibration, shape (N,), dtype wp.mat33f.

• parameters – Complete physical projection parameters, shape (5, N), dtype wp.float32. Rows are focal length, horizontal/vertical aperture, and horizontal/vertical aperture offsets, in the scene’s camera length units. Unselected cameras retain their values.

abstractmethod render(render_data: Any) → None

Submit a capture; read_output() publishes its camera’s available observation.

Parameters:

render_data – The render data object from create_render_data().

render_batch(render_data: Sequence[Any]) → None

Submit captures for a collection of cameras.

All camera poses and shared scene state must be prepared before calling this method. An empty sequence is a no-op. Each object must belong to this renderer and appear once. The default implementation calls render() for each camera; subclasses may override this method to submit all cameras together.

Parameters:

render_data – Renderer-specific objects from create_render_data().

abstractmethod read_output(render_data: Any, camera_data: CameraData) → None

Read rendered outputs from the renderer into the camera data container.

Asynchronous renderers may return the previous capture with its matching metadata in camera_data.info. Publishing one camera must not change another camera’s observations.

Parameters:

• render_data – The render data object from create_render_data().

• camera_data – The CameraData instance to populate.

abstractmethod cleanup(render_data: Any) → None

Release renderer resources associated with the given render data.

Parameters:

render_data – The render data object to clean up, or None.

reset(render_data: Any, env_ids: Sequence[int] | None = None) → None

Reset the renderer-owned state of a camera when its environments reset.

A renderer implementation drops whatever per-camera state must not survive a reset. Examples are pending asynchronous observations and accumulated temporal render history. The default does nothing.

Parameters:

• render_data – The render data object from create_render_data().

• env_ids – Environments being reset, or None for all. An implementation may reset more than the given environments when its state is not separable per environment.

close() → None

Release resources owned by the renderer itself rather than by a render data.

A renderer is shared by every camera whose configuration resolves to it (see get_or_create_backend()), so state it owns outlives any single camera and cannot be released from cleanup(). clear_instance() calls this once at simulation teardown, while the stage and the underlying renderer backend are still alive.

The default implementation is a no-op, for backends whose state lives entirely on the render data. Implementations must be idempotent.

Renderer Configuration

class isaaclab.renderers.RendererCfg

Bases: BackendCfg

Configuration for a renderer.

Attributes:

class_type	Renderer implementation class.
launcher_type	The launcher that starts the runtime this renderer needs, as "module:Class", or None if none is needed.
cloning_contexts	Clone contexts that build this renderer's scene representation from the asset plan.

Methods:

supported_output_types()

Return the camera output layouts supported by this renderer configuration.

class_type: type[BaseRenderer] | str | None

Renderer implementation class. Concrete configs must set this field.

launcher_type: ClassVar[str | None] = None

The launcher that starts the runtime this renderer needs, as "module:Class", or None if none is needed.

cloning_contexts: tuple[type | str, ...]

Clone contexts that build this renderer’s scene representation from the asset plan.

supported_output_types() → dict[RenderBufferKind, RenderBufferSpec] | None

Return the camera output layouts supported by this renderer configuration.

Concrete renderer configurations override this method when their output contract can be determined without importing or instantiating the renderer implementation. Returning None defers compatibility validation until the renderer is created.

Returns:

Mapping from supported output types to their buffer layouts, or None when the renderer is selected dynamically.

Additional Public Classes

The following classes are part of the public isaaclab.renderers API.

CameraRenderSpec	Stable inputs for create_render_data().
RenderBufferKind	Canonical names for the per-pixel render buffer kinds a renderer can publish.
RenderBufferSpec	Per-pixel layout (channels + dtype) for one render buffer kind.
RenderContext	Orchestrate simulation-owned renderers and own flat runtime material buffers.

class isaaclab.renderers.CameraRenderSpec

Bases: object

Stable inputs for create_render_data().

Backends use this instead of holding a reference to the Camera sensor instance, avoiding circular dependencies between sensors and render data.

Parameters:

• cfg – Camera configuration (data types, resolution, filters, etc.).

• device – Torch device string (e.g. "cuda:0") used by GPU annotators and Warp.

• num_instances – Number of tiled camera instances (environments).

• camera_prim_paths – Absolute paths of the authored camera prims. When the renderer clones environments internally, this may contain only the source camera path; the renderer resolves its logical per-environment paths during registration.

• view_count – Number of logical camera instances in the sensor view.

Methods:

__new__(*args, **kwargs)
__init__(cfg, device, num_instances, ...)

classmethod __new__(*args, **kwargs)

__init__(cfg: CameraCfg, device: str, num_instances: int, camera_prim_paths: tuple[str, ...], view_count: int) → None

class isaaclab.renderers.RenderBufferKind

Bases: StrEnum

Canonical names for the per-pixel render buffer kinds a renderer can publish.

String values match the vocabulary used by isaaclab.sensors.camera.CameraCfg.data_types.

Methods:

__new__(value)
__init__(*args, **kwds)

__new__(value)

__init__(*args, **kwds)

class isaaclab.renderers.RenderBufferSpec

Bases: object

Per-pixel layout (channels + dtype) for one render buffer kind.

Methods:

__new__(*args, **kwargs)
__init__(channels, dtype)

classmethod __new__(*args, **kwargs)

__init__(channels: int, dtype: type) → None

class isaaclab.renderers.RenderContext

Bases: object

Orchestrate simulation-owned renderers and own flat runtime material buffers.

Renderer instances are borrowed from the simulation’s backend registry. SDP owns transform freshness, including pose writes that do not advance the physics-step counter.

Methods:

__init__(backend_registry)
__new__(*args, **kwargs)

__init__(backend_registry: list[tuple[Any, Any]]) → None

classmethod __new__(*args, **kwargs)