Scene Data Provider

Scene Data Provider#

SceneDataProvider bridges physics simulation backends and the visualizers/renderers that consume scene data. It exposes a single Warp-native read path for body transforms regardless of which physics backend (PhysX or Newton) is active, so renderers and visualizers can stay backend-agnostic.

Overview#

Isaac Lab supports multiple physics backends (PhysX and Newton) and multiple visualizers (Omniverse Kit, Newton, Rerun, Viser). Each combination needs scene data to flow from the physics engine into the renderer or visualizer. SceneDataProvider owns this flow: the physics manager provides a SceneDataBackend that wraps its native tensor views, and the provider handles format conversion and re-mapping on top of it.

from isaaclab.sim import SimulationContext

# The SimulationContext owns the active provider; consumers fetch it instead of
# constructing one directly.
provider = SimulationContext.instance().get_scene_data_provider()

Architecture#

The system has three layers:

SceneDataBackend: a small interface implemented by each physics manager. It exposes the backend’s transform array directly as one of the SceneDataFormat Warp structs, plus the per-transform prim paths and total count. There is no per-frame “update” call; the property accessors return live views into the underlying tensor each time they’re read.
- SceneDataBackend.transforms: current transforms as a Warp struct (one of SceneDataFormat.Vec3_Quat, SceneDataFormat.Transform, SceneDataFormat.Matrix44, SceneDataFormat.Vec3_Matrix33).
- SceneDataBackend.transform_count: number of transforms.
- SceneDataBackend.transform_paths: list of USD prim paths, one per transform.
- SceneDataBackend.points: flattened deformable nodal positions as SceneDataFormat.Points (optional; rigid-only backends return an empty buffer).
- SceneDataBackend.point_count: total number of geometry points.
- SceneDataBackend.geometry_paths: one USD prim path per deformable body instance.
- SceneDataBackend.geometry_counts: unpadded nodal count per geometry entity.
SceneDataProvider: wraps a backend and offers format conversion plus index re-mapping.
- SceneDataProvider.get_transforms(): writes the backend’s transforms into a consumer-provided SceneDataFormat struct, optionally converting format (e.g. Vec3_Quat to Transform) and applying an index mapping. When the backend format matches the output format and no mapping is provided, the result is a zero-copy passthrough.
- SceneDataProvider.create_mapping(): builds a remap array from the backend’s prim paths to a consumer’s desired ordering. Used when a renderer or visualizer wants transforms indexed by its own body list rather than by the physics view order.
- SceneDataProvider.get_points(): copies backend deformable nodal positions into a consumer buffer, optionally remapping entity slices via SceneDataProvider.create_geometry_mapping().
- SceneDataProvider.create_geometry_mapping(): maps backend deformable entities to consumer particle offsets in a shadow Newton particle_q buffer.
- SceneDataProvider.get_camera_transforms(): discovers per-camera, per-env world transforms from the USD stage.
- SceneDataProvider.usd_stage: USD stage handle for stage-walking consumers.
- SceneDataProvider.num_envs: environment count inferred from /World/envs/env_<id> prims.
Backend implementations:
- PhysxSceneDataBackend (internal to isaaclab_physx.physics) wraps PhysX’s RigidBodyView and exposes its transforms as SceneDataFormat.Transform. When deformable bodies are present it also exposes flattened simulation nodal positions through SceneDataFormat.Points.
- OvPhysxSceneDataBackend (internal to isaaclab_ov.physics) mirrors the PhysX contract for rigid transforms and OVPhysX deformable nodal tensors.
- NewtonSceneDataBackend (internal to isaaclab_newton.physics) wraps the Newton model’s body_q and exposes it as SceneDataFormat.Transform.

PhysX backend#

When PhysX is the active physics backend, the provider reads transforms directly from PhysX’s RigidBodyView (a wildcard-expanded tensor view covering every rigid body across all envs). The transforms are returned as SceneDataFormat.Transform (Warp transformf array), so consumers that want this format get them zero-copy.

Newton-native consumers (Newton visualizer, Rerun, Viser, Newton Warp renderer, OVRTX renderer) also need a Newton Model/State to render against. To provide that, NewtonManager builds a shadow Newton model from the USD stage on first access and updates its body_q from the PhysX backend each render frame. When the scene has PhysX or OVPhysX deformables, the shadow model also allocates particle_q render slots for soft/cloth meshes, syncs simulation nodal positions through SceneDataProvider.get_points() with allow_passthrough=False into a separate sim-sized buffer, and remaps or copies those positions into the render-sized particle_q buffer each frame. Volume deformables with mismatched sim and visual vertex counts use a barycentric sim-to-visual remap so Newton Warp and OVRTX render the paired visual mesh rather than tet simulation topology. The shadow deformable registry exposes render-slot offsets and particles_per_body counts for OVRTX point bindings.

This is hidden behind NewtonManager.get_model() / NewtonManager.get_state(), so renderers don’t need to know which physics backend is active.

Newton backend#

When Newton is the active physics backend, the backend wraps the Newton model’s body_q directly. No shadow model or per-frame sync is needed: Newton already owns the authoritative model and state, and the provider exposes that state as SceneDataFormat.Transform.

Data requirements#

Visualizers and renderers declare what they need from the scene data path. This is resolved at simulation-context construction time and is what triggers the shadow-model build for PhysX:

Component	Requires Newton model	Requires USD stage
Kit visualizer	No	Yes
Newton visualizer	Yes	No
Rerun visualizer	Yes	No
Viser visualizer	Yes	No
Isaac RTX renderer	No	Yes
Newton Warp renderer	Yes	No
OVRTX renderer	Yes	Yes