Scene Data Provider

Scene Data Provider#

SceneDataProvider bridges physics simulation backends and the visualizers/renderers that consume scene data. It exposes a single Warp-native read path for body transforms and visual geometry regardless of which physics backend is active, so renderers and visualizers can stay backend-agnostic.

Overview#

Isaac Lab supports multiple physics backends (PhysX and Newton) and multiple visualizers (Omniverse Kit, Newton, Rerun, Viser). Each combination needs scene data to flow from the physics engine into the renderer or visualizer. SceneDataProvider owns this flow: the physics manager provides a SceneDataBackend that wraps its native tensor views, and the provider handles format conversion and re-mapping on top of it.

from isaaclab.sim import SimulationContext

# The SimulationContext owns the active provider; consumers fetch it instead of
# constructing one directly.
provider = SimulationContext.instance().get_scene_data_provider()

Architecture#

Lazy-read contract#

Use TimestampedBuffer to pair cached data with its last successful update timestamp. Owners allocate storage; the container never allocates or converts it. Same-step writes invalidate affected asset caches with reset_timestamps and advance SDP’s publication timestamp. Each consumer compares its own timestamp after requesting data, since resolving a native pointer can itself detect a change. Consumers tracking only an upload need a timestamp, not another copy of SDP’s data reference.

SDP readers cache buffers and mapping data with the same timestamp guard used by asset data, without binding factories or stored update callbacks. Matching native arrays bypass conversion. Caller-owned destinations are weakly referenced, so their cached mappings expire when the caller releases them.

Pending work belongs to its executor. Newton’s forward() and PhysX/OVPhysX’s update_kinematics() check kinematics_dirty internally and clear it after success; readers call the operation without inspecting its guard. Reordered articulation views still refresh their own derived arrays. Sensors and Newton’s shared BVH similarly use dirty flags, while device masks select the environments requiring work. Keep eager contact reads to detect contact loss.

Sampling periods and finite differences use simulation time [s]; Fabric geometry cadence uses render frames. Neither is a publication counter. Python guards do not execute during CUDA graph replay: captured work must remain in the graph or use device-side invalidation. Newton retains its selective reset masks and conservative reads after externally replayed writes.

Data flow#

The system has three layers:

SceneDataBackend: a small interface implemented by each physics manager. It exposes the backend’s transform array directly as one of the SceneDataFormat Warp structs, plus the per-transform prim paths and total count. Producers increment transforms_timestamp after native state writes or buffer swaps; SDP calls get_transforms(output_format) before reading the timestamp, since resolving the pointer can itself detect a swap. The default implementation returns the existing transforms property. This is a logical publication timestamp, not elapsed time. It never resets within the backend’s lifetime, so independent readers cannot hide changes from one another.
- SceneDataBackend.transforms: the native data as a Warp struct (one of SceneDataFormat.Vec3_Quat, SceneDataFormat.Transform, SceneDataFormat.Matrix44, SceneDataFormat.Vec3_Matrix33).
- SceneDataBackend.transforms_timestamp: logical update timestamp of the native transforms.
- SceneDataBackend.transform_count: number of transforms.
- SceneDataBackend.transform_paths: list of USD prim paths, one per transform.
- SceneDataBackend.native_transform_formats: formats published without conversion. PhysX publishes either packed poses or Fabric matrices and refreshes only the requested representation.
- SceneDataBackend.get_geometry_batches(): native point arrays or interpolation inputs, paired with exact visual prim paths and ranges compiled during backend construction. It returns the requested native representation when available, otherwise the primary representations for SDP to convert. The return type is always a list of batches, including native Fabric.
- SceneDataBackend.geometry_timestamp: logical update timestamp, advanced for same-step writes and native pointer swaps. Cached outputs record the timestamp they contain, like asset data buffers. This is not elapsed simulation time or a shared dirty flag that a reader clears.
- SceneDataBackend.native_geometry_formats: geometry formats available without conversion.
SceneDataProvider: wraps a backend and offers format conversion plus index re-mapping.
- SceneDataProvider.get_transforms(): binds native arrays when format and ordering match, or SDP-owned buffers converted once per producer timestamp and destination layout. These shared arrays are read-only, including when they replace preallocated output fields. Pass allow_passthrough=False to write directly into caller-owned arrays instead.
- SceneDataProvider.create_mapping(): builds a remap array from the backend’s prim paths to a consumer’s desired ordering. Used when a renderer or visualizer wants transforms indexed by its own body list rather than by the physics view order.
- SceneDataProvider.get_geometry_points(): read-only world-space point views keyed by exact visual prim path. Native point ranges alias the producer; interpolation and destination reordering are fused into one cached conversion. Consumers with fixed native storage pass that array or FabricPoints as output and visual-path offsets as offsets. These calls return the supplied destination. Destination caches retain indexing metadata, not the consumer’s buffers, and expire with the destination.
- SceneDataProvider.get_camera_transforms(): discovers per-camera, per-env world transforms from the USD stage.
- SceneDataProvider.num_envs: environment count inferred from /World/envs/env_<id> prims.
Backend implementations:
- PhysxSceneDataBackend (internal to isaaclab_physx.physics) wraps PhysX’s RigidBodyView and exposes its transforms as SceneDataFormat.Transform. When deformable bodies are present it also exposes flattened simulation nodal positions through SceneDataFormat.Points.
- OvPhysxSceneDataBackend (internal to isaaclab_ov.physics) mirrors the PhysX contract for rigid transforms and OVPhysX deformable nodal tensors.
- NewtonSceneDataBackend (internal to isaaclab_newton.physics) wraps the Newton model’s body_q and exposes it as SceneDataFormat.Transform.

PhysX backend#

When PhysX is the active physics backend, the provider reads transforms directly from PhysX’s RigidBodyView (a wildcard-expanded tensor view covering every rigid body across all envs). The transforms are returned as SceneDataFormat.Transform (Warp transformf array), so consumers that want this format get them zero-copy.

Newton-native consumers (Newton visualizer, Rerun, Viser, Newton Warp renderer) also need a Newton Model/State. Their declared cloning contexts construct that representation from the shared clone plan before initialization. Its rigid body_q binds to SDP’s requested Transform array; no intermediate per-frame copy into a second state buffer is required. OVRTX requests TransposedMatrix44d directly from SDP, including destination ordering and static scale in the same conversion. It no longer reads Newton state for rigid transforms. For deformables with different simulation and visual meshes, the producer compiles barycentric indices and weights from the declared prototype once. SDP applies that interpolation directly into the Newton representation’s final particle_q slots. There is no intermediate simulation-sized buffer and no consumer-owned remap. Native PhysX padded nodal arrays are borrowed without packing.

OVRTX receives the same exact visual-path publications through SDP. It neither imports Newton managers nor requests a Newton model. Meshes, particle clouds, and cable curves use one point-binding path.

PhysX owns its native Fabric refresh and publishes the resulting matrices through SDP without fetching packed poses. isaaclab_physx.renderers.fabric.FabricBackend owns the shared native stage and hierarchy handles. Its identity is the stage and device, not the SDP source or attribute type. After physics initializes, Kit, Isaac RTX, and explicit Fabric synchronization obtain the same resource through sim.get_or_create_backend(FabricBackendCfg(stage=sim.stage, device=sim.device)). The simulation registry does not need to know which backends exist. Transform bindings are state on that resource, not a separate backend. Consumers pass the simulation’s SDP to update_transforms(provider); for foreign physics it converts directly into Fabric local matrices, then propagates the GPU hierarchy. Core RenderContext owns no Fabric bindings. It binds rigid destinations as Fabric-only reset-stack roots because physics publishes absolute poses, including for nested bodies. Visual descendants still inherit their body’s transform; authored USD is unchanged. Native source indices and world scales are bound once. Fabric’s selection reuse API reports scene-wide structural changes; the resource refreshes array views without repeating path matching or scale capture. Otherwise GPU propagation reuses the hierarchy topology. Clean requests never acquire writable Fabric arrays. Renderers do not select a physics-specific synchronization path. The same Fabric resource receives geometry through update_geometries(provider, frame). PhysX publishes its native FabricPoints without a conversion or rewrite. Foreign mesh points are interpolated directly into GPU Fabric storage. The current Kit Hydra path requires CPU Fabric destinations for Points and BasisCurves; SDP handles their device transfer without USD attribute writes. Only destinations whose update interval has elapsed are transferred. World-space point destinations reset their transform stack to avoid applying the environment or body pose twice. FabricMatrix44 and FabricPoints contain only array storage, not bindings or native engine handles.

Newton backend#

When Newton is the active physics backend, the backend wraps the Newton model’s body_q directly. No shadow model or per-frame sync is needed: Newton already owns the authoritative model and state, and the provider exposes that state as SceneDataFormat.Transform.

Native reads reconcile pending authored state writes once. A new physics publication does not itself request forward kinematics. Kit/RTX requests current Fabric transforms through SDP before rendering, without issuing an additional physics forward(). Headless viewport capture requests these transforms on demand rather than on every visualizer step.

Externally replayed CUDA graphs do not call Python write hooks. After writes have been captured, Newton conservatively republishes transforms when read so an unannounced replay cannot leave rendering stale. Those reads do not benefit from clean-publication caching.

Geometry publication#

Newton deformable and MPM positions are direct views of native particle_q ranges. Cable publications borrow native body poses and capsule parameters; SDP derives curve endpoints once per update timestamp. Both camera renderers and viewers consume the same cached result.

ClonePlan remains a generic replication and routing description. Asset construction authors prototype geometry; native import combines those prototypes with the plan and records native ranges. Consumers bind to those completed resources, never rediscovering the completed stage.

# Default: read-only native or converted views, cached by producer timestamp.
points_by_path = provider.get_geometry_points()

# A consumer with fixed native storage receives the conversion directly.
provider.get_geometry_points(output=state.particle_q, offsets=visual_path_offsets)

# A Fabric consumer supplies native storage and its exact visual-path row indices.
provider.get_geometry_points(output=fabric_points, offsets=visual_path_rows)

The internal flat-node queries and physics-owned geometry sync methods were removed. Rendering consumers use get_geometry_points; physics managers no longer run geometry writers from pre_render.

Transform conversion caches use TimestampedBuffer, the same data-and-timestamp container used by asset data. Storage is allocated only when conversion is needed; freshness is committed after conversion succeeds. Native matching formats still pass through without allocation. Fabric destination replacement invalidates the cached binding even if physics is unchanged.

As in articulation and rigid-object data, the current timestamp and a cached buffer’s timestamp serve different purposes: one identifies current state; the other identifies the state in that buffer. Asset data advances _sim_timestamp with time and invalidates dependent buffers on same-step writes. SDP instead advances geometry_timestamp on those writes, so independently updated consumers all see the change. The output’s cache owns its freshness check; a downstream upload or BVH may need its own invalidation, but should not repeat the conversion’s cache bookkeeping.

Data requirements#

Visualizers and renderers declare what they need from the scene data path. This is resolved at consumer construction time, before the shared clone plan is built:

Component	Requires Newton model	Requires USD stage
Kit visualizer	No	Yes
Newton visualizer	Yes	No
Rerun visualizer	Yes	No
Viser visualizer	Yes	No
Isaac RTX renderer	No	Yes
Newton Warp renderer	Yes	No
OVRTX renderer	No	Yes