isaaclab.benchmark

Benchmarking utilities for IsaacLab.

This package provides the public benchmark framework and workflow API.

Request and result classes

BenchmarkLauncherConfig	Simulation launcher configuration shared by benchmark workflows.
BenchmarkOutputConfig	Output configuration shared by benchmark workflows.
BenchmarkPlayRequest	Request an RL checkpoint playback benchmark.
BenchmarkResult	Completed benchmark result.
BenchmarkRuntimeRequest	Request an environment runtime benchmark.
BenchmarkStartupRequest	Request a startup profiling benchmark.
BenchmarkTrainingRequest	Request an RL training benchmark.

Workflow functions

run_benchmark()	Run a benchmark workflow from a typed request.
run_play_benchmark(request)	Run an RL checkpoint playback benchmark.
run_runtime_benchmark(request)	Run an environment runtime benchmark.
run_startup_benchmark(request)	Run a startup profiling benchmark.
run_training_benchmark(request)	Run an RL training benchmark.

Benchmark framework

BaseIsaacLabBenchmark	Base benchmark class for IsaacLab's benchmarks.
BenchmarkMonitor	Background thread that periodically updates benchmark recorders.
MethodBenchmarkDefinition	Definition of a method benchmark.
MethodBenchmarkRunner	Runner for method-level benchmarks using the new benchmark tooling.
MethodBenchmarkRunnerConfig	Configuration for MethodBenchmarkRunner.
PlayBundle	Top-level shape of play.json — a checkpoint-driven inference rollout.
RuntimeBundle	Top-level shape of runtime.json (environment stepping, no learning).
StartupBundle	Top-level shape of startup.json.
TrainingBundle	Top-level shape of training.json — a runtime bundle plus learning metrics.

Request and Result Classes

class isaaclab.benchmark.BenchmarkLauncherConfig

Simulation launcher configuration shared by benchmark workflows.

Parameters:

• device – Simulation device identifier, such as "cpu" or "cuda:0".

• enable_cameras – Whether to enable camera rendering.

• visualizers – Visualizers to enable. An empty tuple explicitly disables all visualizers; None preserves task and environment defaults.

• max_visible_envs – Maximum number of environments shown by visualizers.

• experience – Isaac Sim experience file.

• deterministic – Whether to request deterministic rendering and backend behavior.

• animation_recording – Whether to record time-sampled USD animations.

• animation_recording_start_time – Simulation time when animation recording starts [s].

• animation_recording_stop_time – Simulation time when animation recording stops [s].

• kit_args – Arguments forwarded directly to Omniverse Kit.

• livestream – Livestream mode, where 0 disables and 1 or 2 enables WebRTC.

• xr – Whether to enable XR mode.

• verbose – Whether to enable verbose simulator logging.

• info – Whether to enable informational simulator logging.

Methods:

__init__([device, enable_cameras, ...])

__init__(device: str | None = None, enable_cameras: bool = False, visualizers: tuple[Literal['kit', 'newton', 'rerun', 'viser'], ...] | None = None, max_visible_envs: int | None = None, experience: str | None = None, deterministic: bool = False, animation_recording: bool = False, animation_recording_start_time: float | None = None, animation_recording_stop_time: float | None = None, kit_args: str | None = None, livestream: Literal[0, 1, 2] | None = None, xr: bool = False, verbose: bool = False, info: bool = False) → None

class isaaclab.benchmark.BenchmarkOutputConfig

Output configuration shared by benchmark workflows.

Parameters:

• path – Directory receiving benchmark output files.

• formatters – Output formatters to run.

Methods:

__init__([path, formatters])

__init__(path: Path = PosixPath('.'), formatters: tuple[Literal['schema', 'omniperf', 'osmo', 'json', 'summary'], ...] = ('schema',)) → None

class isaaclab.benchmark.BenchmarkPlayRequest

Request an RL checkpoint playback benchmark.

Parameters:

• backend – Reinforcement-learning backend to benchmark.

• task – Registered Gym task identifier.

• checkpoint – Local or Nucleus checkpoint path. When omitted, the backend may use a published checkpoint.

• agent – Optional task agent configuration entry point.

• num_envs – Number of parallel environments.

• num_frames – Number of measured inference steps.

• warmup_frames – Number of initial environment steps excluded from environment-step timing.

• seed – Environment seed.

• measure_synchronized_step_breakdown – Whether to collect serialized synchronized environment/simulation step diagnostics.

• presets – Typed preset names applied to the task configuration.

• backend_args – Backend-specific command-line arguments.

• hydra_args – Additional Hydra overrides.

• output – Output configuration.

• launcher – Simulation launcher configuration.

Attributes:

workflow

Return the workflow dispatcher key.

Methods:

__init__(backend, task[, checkpoint, agent, ...])

property workflow: Literal['play']

Return the workflow dispatcher key.

__init__(backend: ~typing.Literal['rl_games', 'rsl_rl', 'sb3', 'skrl'], task: str, checkpoint: str | None = None, agent: str | None = None, num_envs: int | None = None, num_frames: int = 100, warmup_frames: int = 1, seed: int | None = None, measure_synchronized_step_breakdown: bool = False, presets: tuple[str, ...] = <factory>, backend_args: tuple[str, ...] = <factory>, hydra_args: tuple[str, ...] = <factory>, output: ~isaaclab.benchmark.api.BenchmarkOutputConfig = <factory>, launcher: ~isaaclab.benchmark.api.BenchmarkLauncherConfig = <factory>) → None

class isaaclab.benchmark.BenchmarkResult

Completed benchmark result.

Parameters:

• bundle – Typed benchmark result bundle.

• output_paths – Files written by the selected formatters.

Methods:

__init__(bundle, output_paths)

__init__(bundle: BenchmarkBundleT, output_paths: tuple[Path, ...]) → None

class isaaclab.benchmark.BenchmarkRuntimeRequest

Request an environment runtime benchmark.

Parameters:

• task – Registered Gym task identifier.

• num_envs – Number of parallel environments.

• num_frames – Number of measured environment steps.

• warmup_frames – Number of warm-up steps excluded from throughput measurements.

• seed – Environment seed.

• measure_synchronized_step_breakdown – Whether to collect serialized synchronized environment/simulation step diagnostics.

• presets – Typed preset names applied to the task configuration.

• hydra_args – Additional Hydra overrides.

• output – Output configuration.

• launcher – Simulation launcher configuration.

Attributes:

workflow

Return the workflow dispatcher key.

Methods:

__init__(task[, num_envs, num_frames, ...])

property workflow: Literal['runtime']

Return the workflow dispatcher key.

__init__(task: str, num_envs: int | None = None, num_frames: int = 1000, warmup_frames: int = 50, seed: int | None = None, measure_synchronized_step_breakdown: bool = False, presets: tuple[str, ...] = <factory>, hydra_args: tuple[str, ...] = <factory>, output: ~isaaclab.benchmark.api.BenchmarkOutputConfig = <factory>, launcher: ~isaaclab.benchmark.api.BenchmarkLauncherConfig = <factory>) → None

class isaaclab.benchmark.BenchmarkStartupRequest

Request a startup profiling benchmark.

Parameters:

• task – Registered Gym task identifier.

• num_envs – Number of parallel environments.

• seed – Environment seed.

• top_n – Number of top cProfile functions retained per phase.

• whitelist_config – Optional YAML whitelist for phase-specific functions.

• presets – Typed preset names applied to the task configuration.

• hydra_args – Additional Hydra overrides.

• output – Output configuration.

• launcher – Simulation launcher configuration.

Attributes:

workflow

Return the workflow dispatcher key.

Methods:

__init__(task[, num_envs, seed, top_n, ...])

property workflow: Literal['startup']

Return the workflow dispatcher key.

__init__(task: str, num_envs: int | None = None, seed: int | None = None, top_n: int | None = None, whitelist_config: ~pathlib.Path | None = None, presets: tuple[str, ...] = <factory>, hydra_args: tuple[str, ...] = <factory>, output: ~isaaclab.benchmark.api.BenchmarkOutputConfig = <factory>, launcher: ~isaaclab.benchmark.api.BenchmarkLauncherConfig = <factory>) → None

class isaaclab.benchmark.BenchmarkTrainingRequest

Request an RL training benchmark.

Parameters:

• backend – Reinforcement-learning backend to benchmark.

• task – Registered Gym task identifier.

• agent – Optional task agent configuration entry point.

• num_envs – Number of parallel environments.

• seed – Environment and agent seed.

• max_iterations – Maximum training iterations.

• warmup_steps – Number of initial environment steps excluded from environment-step timing.

• ray_proc_id – Ray worker process identifier.

• video – Whether to record training videos.

• video_length – Recorded video length [steps].

• video_interval – Interval between video recordings [steps].

• export_io_descriptors – Whether to export environment IO descriptors.

• capture_env_sensors – Number of environment views captured from each image-like sensor.

• capture_env_sensors_length – Length of each sensor capture window [steps].

• capture_env_sensors_interval – Interval between sensor capture windows [steps].

• capture_env_sensors_format – Storage format for captured sensor frames.

• ema_alpha – Learning-curve exponential moving-average coefficient.

• keep_series – Whether to retain full per-iteration learning series.

• check_success – Whether supported backends should stop after success convergence.

• success_threshold – Optional success threshold override.

• success_window – Optional success convergence window override [iterations].

• measure_synchronized_step_breakdown – Whether to collect serialized synchronized environment/simulation step diagnostics.

• presets – Typed preset names applied to the task configuration.

• backend_args – Backend-specific command-line arguments.

• hydra_args – Additional Hydra overrides.

• output – Output configuration.

• launcher – Simulation launcher configuration.

Attributes:

workflow

Return the workflow dispatcher key.

Methods:

__init__(backend, task[, agent, num_envs, ...])

property workflow: Literal['training']

Return the workflow dispatcher key.

__init__(backend: ~typing.Literal['rl_games', 'rsl_rl', 'sb3', 'skrl'], task: str, agent: str | None = None, num_envs: int | None = None, seed: int | None = None, max_iterations: int | None = None, warmup_steps: int = 1, ray_proc_id: int | None = None, video: bool = False, video_length: int = 200, video_interval: int = 2000, export_io_descriptors: bool = False, capture_env_sensors: int = 0, capture_env_sensors_length: int = 200, capture_env_sensors_interval: int = 2000, capture_env_sensors_format: ~typing.Literal['tensorboard', 'file'] = 'tensorboard', ema_alpha: float = 0.1, keep_series: bool = True, check_success: bool = False, success_threshold: float | None = None, success_window: int | None = None, measure_synchronized_step_breakdown: bool = False, presets: tuple[str, ...] = <factory>, backend_args: tuple[str, ...] = <factory>, hydra_args: tuple[str, ...] = <factory>, output: ~isaaclab.benchmark.api.BenchmarkOutputConfig = <factory>, launcher: ~isaaclab.benchmark.api.BenchmarkLauncherConfig = <factory>) → None

Workflow Functions

isaaclab.benchmark.run_benchmark(request: BenchmarkRuntimeRequest) → BenchmarkResult[RuntimeBundle]

isaaclab.benchmark.run_benchmark(request: BenchmarkStartupRequest) → BenchmarkResult[StartupBundle]

isaaclab.benchmark.run_benchmark(request: BenchmarkTrainingRequest) → BenchmarkResult[TrainingBundle]

isaaclab.benchmark.run_benchmark(request: BenchmarkPlayRequest) → BenchmarkResult[PlayBundle]

Run a benchmark workflow from a typed request.

Parameters:

request – Runtime, startup, training, or play benchmark request.

Returns:

Completed benchmark bundle and output paths.

isaaclab.benchmark.run_play_benchmark(request: BenchmarkPlayRequest) → BenchmarkResult[PlayBundle]

Run an RL checkpoint playback benchmark.

Parameters:

request – Playback benchmark request.

Returns:

Completed benchmark bundle and output paths.

isaaclab.benchmark.run_runtime_benchmark(request: BenchmarkRuntimeRequest) → BenchmarkResult[RuntimeBundle]

Run an environment runtime benchmark.

Parameters:

request – Runtime benchmark request.

Returns:

Completed benchmark bundle and output paths.

isaaclab.benchmark.run_startup_benchmark(request: BenchmarkStartupRequest) → BenchmarkResult[StartupBundle]

Run a startup profiling benchmark.

Parameters:

request – Startup benchmark request.

Returns:

Completed benchmark bundle and output paths.

isaaclab.benchmark.run_training_benchmark(request: BenchmarkTrainingRequest) → BenchmarkResult[TrainingBundle]

Run an RL training benchmark.

Parameters:

request – Training benchmark request.

Returns:

Completed benchmark bundle and output paths.

Benchmark Framework

class isaaclab.benchmark.BaseIsaacLabBenchmark

Base benchmark class for IsaacLab’s benchmarks.

Methods:

__init__(benchmark_name[, formatter_type, ...])	Initialize common benchmark state and recorders.
attach_bundle(bundle)	Attach a typed bundle for schema serialization and flat-formatter projection.
update_manual_recorders()	Update manual recorders that don't depend on the kit timeline.
add_measurement(phase_name[, measurement, ...])	Add a measurement to the benchmark.
finalize()	Finalize metric collection and write selected formatter outputs.

Attributes:

output_file_path

Get the full path to the output file.

__init__(benchmark_name: str, formatter_type: str | list[str] | None = None, output_path: str | None = None, use_recorders: bool = True, output_prefix: str | None = None, workflow_metadata: dict | None = None, frametime_recorders: bool = False, backend_type: str | list[str] | None = None)

Initialize common benchmark state and recorders.

Parameters:

• benchmark_name – Name of benchmark to use in outputs.

• formatter_type – Formatter(s) used to collect and print metrics. Accepts a single type name, a list of type names, or a comma-separated string (e.g. "schema,omniperf"); each selected formatter writes its own output file.

• output_path – Path to output directory.

• use_recorders – Whether to use recorders to collect metrics. Defaults to True.

• output_prefix – Prefix used to generate the output filename. Defaults to None.

• workflow_metadata – Metadata describing benchmark, defaults to None.

• frametime_recorders – Whether to use frametime recorders to collect metrics. Defaults to False.

• backend_type – Alias for formatter_type.

property output_file_path: str

Get the full path to the output file.

attach_bundle(bundle: RuntimeBundle | TrainingBundle | StartupBundle | PlayBundle | None) → None

Attach a typed bundle for schema serialization and flat-formatter projection.

Parameters:

bundle – Runtime, training, startup, or play benchmark bundle.

update_manual_recorders() → None

Update manual recorders that don’t depend on the kit timeline.

add_measurement(phase_name: str, measurement: Measurement | Sequence[Measurement] | None = None, metadata: MetadataBase | Sequence[MetadataBase] | None = None) → None

Add a measurement to the benchmark.

Parameters:

• phase_name – The name of the phase to add the measurement to.

• measurement – The measurement to add.

• metadata – The metadata to add.

finalize() → tuple[Path, ...]

Finalize metric collection and write selected formatter outputs.

Returns:

Files written by the selected formatters.

class isaaclab.benchmark.BenchmarkMonitor

Background thread that periodically updates benchmark recorders.

This utility enables continuous system resource monitoring during blocking RL training loops (RSL-RL, RL-Games) where update_manual_recorders() would otherwise only be called once after training completes.

Usage:

with BenchmarkMonitor(benchmark, interval=1.0):

runner.learn(…) # Blocking training call

Methods:

__init__(benchmark[, interval])	Initialize the benchmark monitor.
start()	Start the monitoring thread.
stop()	Stop the monitoring thread and wait for it to finish.

__init__(benchmark: BaseIsaacLabBenchmark, interval: float = 1.0)

Initialize the benchmark monitor.

Parameters:

• benchmark – The benchmark instance to monitor.

• interval – Time between recorder updates in seconds. Defaults to 1.0.

start() → None

Start the monitoring thread.

stop() → None

Stop the monitoring thread and wait for it to finish.

class isaaclab.benchmark.MethodBenchmarkDefinition

Definition of a method benchmark.

name

Display name for the benchmark.

Type:

str

method_name

Name of the method to benchmark on the target object.

Type:

str

input_generators

Dict mapping mode names to input generator functions.

Type:

dict[str, collections.abc.Callable]

category

Category for grouping results into phases.

Type:

str

Methods:

__init__(name, method_name, input_generators)

__init__(name: str, method_name: str, input_generators: dict[str, Callable], category: str = 'default') → None

class isaaclab.benchmark.MethodBenchmarkRunner

Runner for method-level benchmarks using the new benchmark tooling.

This class extends BaseIsaacLabBenchmark to provide method-level benchmarking with automatic hardware/version info collection, multiple backend support, and organized output by category phases.

Methods:

__init__(benchmark_name, config[, ...])	Initialize the method benchmark runner.
run_benchmarks(benchmarks, target_object)	Run all defined benchmarks on the target object.
run_property_benchmarks(target_data, ...[, ...])	Run benchmarks for data class properties.

Attributes:

config

Return the benchmark configuration.

__init__(benchmark_name: str, config: MethodBenchmarkRunnerConfig, backend_type: str = 'json', output_path: str = '.', use_recorders: bool = True)

Initialize the method benchmark runner.

Parameters:

• benchmark_name – Name of the benchmark (used in output files).

• config – Benchmark configuration.

• backend_type – Output backend type (“json”, “osmo”, “omni_perf”).

• output_path – Directory to write output files.

• use_recorders – Whether to collect hardware/version info.

property config: MethodBenchmarkRunnerConfig

Return the benchmark configuration.

run_benchmarks(benchmarks: list[MethodBenchmarkDefinition], target_object: object) → None

Run all defined benchmarks on the target object.

Parameters:

• benchmarks – List of benchmark definitions to run.

• target_object – Object containing the methods to benchmark.

run_property_benchmarks(target_data: object, properties: list[str], gen_mock_data: Callable, dependencies: dict[str, list[str]] | None = None, category: str = 'property') → None

Run benchmarks for data class properties.

This is a convenience method for benchmarking properties on data classes where the test involves generating mock data and accessing properties.

Parameters:

• target_data – Data object containing the properties to benchmark.

• properties – List of property names to benchmark.

• gen_mock_data – Function that generates/updates mock data.

• dependencies – Optional dict mapping property names to their dependencies.

• category – Category name for grouping results.

class isaaclab.benchmark.MethodBenchmarkRunnerConfig

Configuration for MethodBenchmarkRunner.

num_iterations

Number of timing iterations per method.

Type:

int

warmup_steps

Number of warmup iterations before timing.

Type:

int

num_instances

Number of environment instances.

Type:

int

num_bodies

Number of bodies per instance.

Type:

int

num_joints

Number of joints per instance.

Type:

int

device

Device to run benchmarks on.

Type:

str

mode

Which input modes to run (“all” or specific mode name).

Type:

str | list[str]

Methods:

__init__([num_iterations, warmup_steps, ...])

__init__(num_iterations: int = 1000, warmup_steps: int = 10, num_instances: int = 4096, num_bodies: int = 12, num_joints: int = 11, device: str = 'cuda:0', mode: str | list[str] = 'all') → None

class isaaclab.benchmark.PlayBundle

Top-level shape of play.json — a checkpoint-driven inference rollout.

Mirrors RuntimeBundle (with RunIdentity.framework set to the RL library that produced the checkpoint) and adds the inference-evaluation aggregates: a success rate plus scalar reward and episode-length statistics. Unlike TrainingBundle, reward and ep_length are scalar MeanStd aggregates over completed episodes, not per-iteration learning curves.

Parameters:

• success_rate – Mean success rate [0..1] over completed episodes, or None when the task does not report one.

• reward – Episode-return aggregate over completed episodes, or None when no episode completed.

• ep_length – Episode-length aggregate over completed episodes, or None when no episode completed.

• checkpoint_path – Path to the policy checkpoint that was rolled out.

• video_path – Path to a recorded rollout video/gif, if any.

• extra – Optional free-form scalar values (experimental or producer-specific) that are not part of the stable schema contract. Consumers must tolerate its absence and must not depend on specific keys; promote a key to a typed field once it is stable and broadly useful.

Methods:

__init__(run, versions, hardware, runtime, ...)

__init__(run: RunIdentity, versions: Versions, hardware: Hardware, runtime: Runtime, resources: Resources, success_rate: float | None = None, reward: MeanStd | None = None, ep_length: MeanStd | None = None, checkpoint_path: str | None = None, video_path: str | None = None, extra: dict[str, float | int | str | bool] | None = None, schema_version: str = '1.2') → None

class isaaclab.benchmark.RuntimeBundle

Top-level shape of runtime.json (environment stepping, no learning).

Mirrors TrainingBundle without the learning metrics.

Parameters:

extra – Optional free-form scalar values (experimental or producer-specific) that are not part of the stable schema contract. Consumers must tolerate its absence and must not depend on specific keys; promote a key to a typed field once it is stable and broadly useful.

Methods:

__init__(run, versions, hardware, runtime, ...)

__init__(run: RunIdentity, versions: Versions, hardware: Hardware, runtime: Runtime, resources: Resources, extra: dict[str, float | int | str | bool] | None = None, schema_version: str = '1.2') → None

class isaaclab.benchmark.StartupBundle

Top-level shape of startup.json.

Reuses RunIdentity with framework/num_envs/max_iterations left unset, since they are not meaningful for a startup profile.

Parameters:

extra – Optional free-form scalar values (experimental or producer-specific) that are not part of the stable schema contract. Consumers must tolerate its absence and must not depend on specific keys; promote a key to a typed field once it is stable and broadly useful.

Methods:

__init__(run, versions, hardware, phases, config)

__init__(run: RunIdentity, versions: Versions, hardware: Hardware, phases: dict[str, StartupPhase], config: StartupConfig, extra: dict[str, float | int | str | bool] | None = None, schema_version: str = '1.2') → None

class isaaclab.benchmark.TrainingBundle

Top-level shape of training.json — a runtime bundle plus learning metrics.

Parameters:

• success_rate – Final success rate [0..1] when the task tracks one, else None.

• checkpoint_path – Path to the final saved policy checkpoint, if any.

• video_path – Path to a recorded rollout video/gif, if any.

• extra – Optional free-form scalar values (experimental or producer-specific) that are not part of the stable schema contract. Consumers must tolerate its absence and must not depend on specific keys; promote a key to a typed field once it is stable and broadly useful.

Methods:

__init__(run, versions, hardware, runtime, ...)

__init__(run: RunIdentity, versions: Versions, hardware: Hardware, runtime: Runtime, resources: Resources, learning: Learning, success_rate: float | None = None, checkpoint_path: str | None = None, video_path: str | None = None, extra: dict[str, float | int | str | bool] | None = None, schema_version: str = '1.2') → None