isaaclab.benchmark

Benchmarking utilities for IsaacLab.

This package provides the public benchmark framework and workflow API.

Request and result classes

BenchmarkLauncherConfig	Simulation launcher configuration shared by benchmark workflows.
BenchmarkOutputConfig	Output configuration shared by benchmark workflows.
BenchmarkPlayRequest	Request an RL checkpoint playback benchmark.
BenchmarkResult	Completed benchmark result.
BenchmarkRuntimeRequest	Request an environment runtime benchmark.
BenchmarkStartupRequest	Request a startup profiling benchmark.
BenchmarkTrainingRequest	Request an RL training benchmark.

Workflow functions

run_benchmark()	Run a benchmark workflow from a typed request.
run_play_benchmark(request)	Run an RL checkpoint playback benchmark.
run_runtime_benchmark(request)	Run an environment runtime benchmark.
run_startup_benchmark(request)	Run a startup profiling benchmark.
run_training_benchmark(request)	Run an RL training benchmark.

Benchmark framework

BaseIsaacLabBenchmark	Base benchmark class for IsaacLab's benchmarks.
BenchmarkMonitor	Background thread that periodically updates benchmark recorders.
MethodBenchmarkDefinition	Definition of a method benchmark.
MethodBenchmarkRunner	Runner for method-level benchmarks using the new benchmark tooling.
MethodBenchmarkRunnerConfig	Configuration for MethodBenchmarkRunner.

Micro-benchmarks

LatencyBenchmarkRunner	One-shot runner for latency micro-benchmarks.
LatencySample	One host-submission and device-synchronized latency sample.
LatencyStatistics	Aggregate statistics for a latency sample series.
MicrobenchmarkCommand	Resolved micro-benchmark child command.
MicrobenchmarkFactory	Resolve exact physics variants and components to benchmark entrypoints.
add_latency_measurements(benchmark, ...)	Add latency mean, standard deviation, and percentiles to a benchmark.
measure_latency(operation, synchronize, *[, ...])	Measure host submission and synchronized completion latency.
run_microbenchmark_cli([args])	Parse and run isaaclab microbenchmark.
summarize_latency(samples_s)	Summarize a non-empty latency sample series.

Measurements and metadata

BooleanMeasurement	Boolean measurement.
DictMeasurement	Dictionary measurement.
DictMetadata	Dictionary metadata.
FloatMetadata	Float metadata.
IntMetadata	Integer metadata.
ListMeasurement	List measurement.
Measurement	Base measurement record.
MetadataBase	Base metadata record.
SingleMeasurement	Single floating-point measurement.
StatisticalMeasurement	Statistical measurement.
StringMetadata	String metadata.
TestPhase	Represent a single test phase with associated metrics and metadata.

Result schema

EnvironmentStepTiming	Environment-step wall time and optional synchronized simulation breakdown.
Learning	Learning curves for a training run, plus their EMA smoothing factor.
LearningCurve	One learning curve (reward, episode length, or success rate).
MeanStd	Scalar aggregate with mean, standard deviation, and optional peak.
PlayBundle	Top-level shape of play.json — a checkpoint-driven inference rollout.
Resources	Aggregated resource-utilisation metrics for a run.
RunConfig	Physics/rendering backend and active presets for a run.
RunIdentity	Identity of a benchmark run (training, runtime, or startup).
Runtime	Aggregated runtime metrics for a run.
RuntimeBundle	Top-level shape of runtime.json (environment stepping, no learning).
StartupBundle	Top-level shape of startup.json.
StartupPhase	Wall-clock total plus top cProfile functions for one startup phase.
StartupTime	Wall-clock duration of each startup phase [s].
TrainingBundle	Top-level shape of training.json — a runtime bundle plus learning metrics.

Request and Result Classes

class isaaclab.benchmark.BenchmarkLauncherConfig

Simulation launcher configuration shared by benchmark workflows.

Parameters:

• device – Simulation device identifier, such as "cpu" or "cuda:0".

• enable_cameras – Whether to enable camera rendering.

• visualizers – Visualizers to enable. An empty tuple explicitly disables all visualizers; None preserves task and environment defaults.

• max_visible_envs – Maximum number of environments shown by visualizers.

• experience – Isaac Sim experience file.

• deterministic – Whether to request deterministic rendering and backend behavior.

• animation_recording – Whether to record time-sampled USD animations.

• animation_recording_start_time – Simulation time when animation recording starts [s].

• animation_recording_stop_time – Simulation time when animation recording stops [s].

• kit_args – Arguments forwarded directly to Omniverse Kit.

• livestream – Livestream mode, where 0 disables and 1 or 2 enables WebRTC.

• xr – Whether to enable XR mode.

• verbose – Whether to enable verbose simulator logging.

• info – Whether to enable informational simulator logging.

Methods:

__init__([device, enable_cameras, ...])

__init__(device: str | None = None, enable_cameras: bool = False, visualizers: tuple[Literal['kit', 'newton', 'rerun', 'viser'], ...] | None = None, max_visible_envs: int | None = None, experience: str | None = None, deterministic: bool = False, animation_recording: bool = False, animation_recording_start_time: float | None = None, animation_recording_stop_time: float | None = None, kit_args: str | None = None, livestream: Literal[0, 1, 2] | None = None, xr: bool = False, verbose: bool = False, info: bool = False) → None

class isaaclab.benchmark.BenchmarkOutputConfig

Output configuration shared by benchmark workflows.

Parameters:

• path – Directory receiving benchmark output files.

• formatters – Output formatters to run.

Methods:

__init__([path, formatters])

__init__(path: Path = PosixPath('.'), formatters: tuple[Literal['schema', 'omniperf', 'osmo', 'json', 'summary'], ...] = ('schema',)) → None

class isaaclab.benchmark.BenchmarkPlayRequest

Request an RL checkpoint playback benchmark.

Parameters:

• backend – Reinforcement-learning backend to benchmark.

• task – Registered Gym task identifier.

• checkpoint – Local or Nucleus checkpoint path. When omitted, the backend may use a published checkpoint.

• agent – Optional task agent configuration entry point.

• num_envs – Number of parallel environments.

• num_steps – Number of measured inference steps.

• warmup_steps – Number of initial environment steps excluded from environment-step timing.

• seed – Environment seed.

• measure_synchronized_step_breakdown – Whether to collect serialized synchronized environment/simulation step diagnostics.

• presets – Typed preset names applied to the task configuration.

• backend_args – Backend-specific command-line arguments.

• hydra_args – Additional Hydra overrides.

• output – Output configuration.

• launcher – Simulation launcher configuration.

Attributes:

workflow

Return the workflow dispatcher key.

Methods:

__init__(backend, task[, checkpoint, agent, ...])

property workflow: Literal['play']

Return the workflow dispatcher key.

__init__(backend: ~typing.Literal['rl_games', 'rsl_rl', 'sb3', 'skrl'], task: str, checkpoint: str | None = None, agent: str | None = None, num_envs: int | None = None, num_steps: int = 100, warmup_steps: int = 1, seed: int | None = None, measure_synchronized_step_breakdown: bool = False, presets: tuple[str, ...] = <factory>, backend_args: tuple[str, ...] = <factory>, hydra_args: tuple[str, ...] = <factory>, output: ~isaaclab.benchmark.api.BenchmarkOutputConfig = <factory>, launcher: ~isaaclab.benchmark.api.BenchmarkLauncherConfig = <factory>) → None

class isaaclab.benchmark.BenchmarkResult

Completed benchmark result.

Parameters:

• bundle – Typed benchmark result bundle.

• output_paths – Files written by the selected formatters.

Methods:

__init__(bundle, output_paths)

__init__(bundle: BenchmarkBundleT, output_paths: tuple[Path, ...]) → None

class isaaclab.benchmark.BenchmarkRuntimeRequest

Request an environment runtime benchmark.

Parameters:

• task – Registered Gym task identifier.

• num_envs – Number of parallel environments.

• num_steps – Number of measured environment steps.

• warmup_steps – Number of warm-up steps excluded from throughput measurements.

• seed – Environment seed.

• measure_synchronized_step_breakdown – Whether to collect serialized synchronized environment/simulation step diagnostics.

• presets – Typed preset names applied to the task configuration.

• hydra_args – Additional Hydra overrides.

• output – Output configuration.

• launcher – Simulation launcher configuration.

Attributes:

workflow

Return the workflow dispatcher key.

Methods:

__init__(task[, num_envs, num_steps, ...])

property workflow: Literal['runtime']

Return the workflow dispatcher key.

__init__(task: str, num_envs: int | None = None, num_steps: int = 1000, warmup_steps: int = 50, seed: int | None = None, measure_synchronized_step_breakdown: bool = False, presets: tuple[str, ...] = <factory>, hydra_args: tuple[str, ...] = <factory>, output: ~isaaclab.benchmark.api.BenchmarkOutputConfig = <factory>, launcher: ~isaaclab.benchmark.api.BenchmarkLauncherConfig = <factory>) → None

class isaaclab.benchmark.BenchmarkStartupRequest

Request a startup profiling benchmark.

Parameters:

• task – Registered Gym task identifier.

• num_envs – Number of parallel environments.

• seed – Environment seed.

• top_n – Number of top cProfile functions retained per phase.

• whitelist_config – Optional YAML whitelist for phase-specific functions.

• presets – Typed preset names applied to the task configuration.

• hydra_args – Additional Hydra overrides.

• output – Output configuration.

• launcher – Simulation launcher configuration.

Attributes:

workflow

Return the workflow dispatcher key.

Methods:

__init__(task[, num_envs, seed, top_n, ...])

property workflow: Literal['startup']

Return the workflow dispatcher key.

__init__(task: str, num_envs: int | None = None, seed: int | None = None, top_n: int | None = None, whitelist_config: ~pathlib.Path | None = None, presets: tuple[str, ...] = <factory>, hydra_args: tuple[str, ...] = <factory>, output: ~isaaclab.benchmark.api.BenchmarkOutputConfig = <factory>, launcher: ~isaaclab.benchmark.api.BenchmarkLauncherConfig = <factory>) → None

class isaaclab.benchmark.BenchmarkTrainingRequest

Request an RL training benchmark.

Parameters:

• backend – Reinforcement-learning backend to benchmark.

• task – Registered Gym task identifier.

• agent – Optional task agent configuration entry point.

• num_envs – Number of parallel environments.

• seed – Environment and agent seed.

• max_iterations – Maximum training iterations.

• warmup_steps – Number of initial environment steps excluded from environment-step timing.

• ray_proc_id – Ray worker process identifier.

• video – Whether to record training videos.

• video_length – Recorded video length [steps].

• video_interval – Interval between video recordings [steps].

• export_io_descriptors – Whether to export environment IO descriptors.

• capture_env_sensors – Number of environment views captured from each image-like sensor.

• capture_env_sensors_length – Length of each sensor capture window [steps].

• capture_env_sensors_interval – Interval between sensor capture windows [steps].

• capture_env_sensors_format – Storage format for captured sensor frames.

• ema_alpha – Learning-curve exponential moving-average coefficient.

• keep_series – Whether to retain full per-iteration learning series.

• check_success – Whether supported backends should stop after success convergence.

• success_threshold – Optional success threshold override.

• success_window – Optional success convergence window override [iterations].

• measure_synchronized_step_breakdown – Whether to collect serialized synchronized environment/simulation step diagnostics.

• presets – Typed preset names applied to the task configuration.

• backend_args – Backend-specific command-line arguments.

• hydra_args – Additional Hydra overrides.

• output – Output configuration.

• launcher – Simulation launcher configuration.

Attributes:

workflow

Return the workflow dispatcher key.

Methods:

__init__(backend, task[, agent, num_envs, ...])

property workflow: Literal['training']

Return the workflow dispatcher key.

__init__(backend: ~typing.Literal['rl_games', 'rsl_rl', 'sb3', 'skrl'], task: str, agent: str | None = None, num_envs: int | None = None, seed: int | None = None, max_iterations: int | None = None, warmup_steps: int = 1, ray_proc_id: int | None = None, video: bool = False, video_length: int = 200, video_interval: int = 2000, export_io_descriptors: bool = False, capture_env_sensors: int = 0, capture_env_sensors_length: int = 200, capture_env_sensors_interval: int = 2000, capture_env_sensors_format: ~typing.Literal['tensorboard', 'file'] = 'tensorboard', ema_alpha: float = 0.1, keep_series: bool = True, check_success: bool = False, success_threshold: float | None = None, success_window: int | None = None, measure_synchronized_step_breakdown: bool = False, presets: tuple[str, ...] = <factory>, backend_args: tuple[str, ...] = <factory>, hydra_args: tuple[str, ...] = <factory>, output: ~isaaclab.benchmark.api.BenchmarkOutputConfig = <factory>, launcher: ~isaaclab.benchmark.api.BenchmarkLauncherConfig = <factory>) → None

Micro-Benchmark Dispatch

class isaaclab.benchmark.MicrobenchmarkCommand

Resolved micro-benchmark child command.

physics

Exact physics variant selected for the workload.

Type:

str

component

Component workload to benchmark.

Type:

str

script

Benchmark entrypoint to execute.

Type:

pathlib.Path

args

Arguments passed to the benchmark entrypoint.

Type:

list[str]

Methods:

__init__(physics, component, script, args)

__init__(physics: str, component: str, script: Path, args: list[str]) → None

class isaaclab.benchmark.MicrobenchmarkFactory

Resolve exact physics variants and components to benchmark entrypoints.

Methods:

repository_root()	Return the Isaac Lab repository root.
physics_variants()	Return discoverable exact physics variants.
components()	Return discoverable component workloads.
build_command(physics, component, ...)	Resolve one exact physics/component selection.

classmethod repository_root() → Path

Return the Isaac Lab repository root.

Returns:

Repository root containing the backend benchmark entrypoints.

classmethod physics_variants() → tuple[str, ...]

Return discoverable exact physics variants.

Returns:

Exact physics selectors accepted by build_command().

classmethod components() → tuple[str, ...]

Return discoverable component workloads.

Returns:

Sorted component names accepted by build_command().

build_command(physics: str, component: str, passthrough_args: list[str]) → MicrobenchmarkCommand

Resolve one exact physics/component selection.

Parameters:

• physics – Exact physics variant.

• component – Asset or sensor component workload.

• passthrough_args – Additional arguments for the selected entrypoint.

Returns:

Child command for the selected workload.

Raises:

ValueError – If the physics variant or component is unknown, or if passthrough_args overrides the selected variant.

isaaclab.benchmark.run_microbenchmark_cli(args: list[str] | None = None) → int

Parse and run isaaclab microbenchmark.

Parameters:

args – Command arguments. Uses sys.argv when None.

Returns:

Zero after the child benchmark completes successfully.

Raises:

• SystemExit – If command arguments are invalid.

• subprocess.CalledProcessError – If the child benchmark fails.

Latency Micro-Benchmarks

class isaaclab.benchmark.LatencyBenchmarkRunner

One-shot runner for latency micro-benchmarks.

Parameters:

• benchmark_name – Name used in output metadata and filenames.

• formatter_type – Formatter used to report results.

• output_path – Directory for result files.

• metadata – Workload metadata stored with the result.

• use_recorders – Whether to collect hardware and version information.

Methods:

__init__(benchmark_name, formatter_type, ...)	Initialize common benchmark state and recorders.
add_latency_samples(phase_name, samples)	Add synchronized completion and host submission latency series.
add_synchronized_samples(phase_name, name, ...)	Add a synchronized-only latency series.
finalize()	Sample recorders and write results.

__init__(benchmark_name: str, formatter_type: str, output_path: str, metadata: dict[str, str | int | float | dict] | None = None, use_recorders: bool = True) → None

Initialize common benchmark state and recorders.

Parameters:

• benchmark_name – Name of benchmark to use in outputs.

• formatter_type – Formatter(s) used to collect and print metrics. Accepts a single type name, a list of type names, or a comma-separated string (e.g. "schema,omniperf"); each selected formatter writes its own output file.

• output_path – Path to output directory.

• use_recorders – Whether to use recorders to collect metrics. Defaults to True.

• output_prefix – Prefix used to generate the output filename. Defaults to None.

• workflow_metadata – Metadata describing benchmark, defaults to None.

• frametime_recorders – Whether to use frametime recorders to collect metrics. Defaults to False.

• backend_type – Alias for formatter_type.

add_latency_samples(phase_name: str, samples: Sequence[LatencySample]) → LatencyStatistics

Add synchronized completion and host submission latency series.

Parameters:

• phase_name – Phase that owns the measurements.

• samples – Paired host and synchronized latency samples [s].

Returns:

Synchronized completion latency statistics [s].

add_synchronized_samples(phase_name: str, name: str, samples_s: Sequence[float]) → LatencyStatistics

Add a synchronized-only latency series.

Parameters:

• phase_name – Phase that owns the measurements.

• name – Measurement name.

• samples_s – Synchronized latency samples [s].

Returns:

Synchronized latency statistics [s].

finalize() → tuple[Path, ...]

Sample recorders and write results.

Returns:

Paths written by the selected formatters.

class isaaclab.benchmark.LatencySample

One host-submission and device-synchronized latency sample.

submission_s

Time [s] until the operation returned to the host.

Type:

float

synchronized_s

Time [s] until all submitted device work completed.

Type:

float

Methods:

__init__(submission_s, synchronized_s)

__init__(submission_s: float, synchronized_s: float) → None

class isaaclab.benchmark.LatencyStatistics

Aggregate statistics for a latency sample series.

mean_s

Arithmetic mean latency [s].

Type:

float

std_s

Sample standard deviation [s], or zero for one sample.

Type:

float

p50_s

Linearly interpolated 50th percentile latency [s].

Type:

float

p95_s

Linearly interpolated 95th percentile latency [s].

Type:

float

n

Number of samples.

Type:

int

Methods:

__init__(mean_s, std_s, p50_s, p95_s, n)

__init__(mean_s: float, std_s: float, p50_s: float, p95_s: float, n: int) → None

isaaclab.benchmark.add_latency_measurements(benchmark: _MeasurementSink, phase_name: str, name: str, samples_s: Sequence[float]) → LatencyStatistics

Add latency mean, standard deviation, and percentiles to a benchmark.

Parameters:

• benchmark – Benchmark receiving the measurements.

• phase_name – Phase that owns the measurements.

• name – Base measurement name.

• samples_s – Latency samples [s].

Returns:

Computed latency statistics [s].

isaaclab.benchmark.measure_latency(operation: Callable[[], None], synchronize: Callable[[], None], *, clock_ns: Callable[[], int] | None = None) → LatencySample

Measure host submission and synchronized completion latency.

The pre-boundary synchronization prevents asynchronous work submitted before operation from being charged to the sample. The post-boundary synchronization includes all device work submitted by the operation.

Parameters:

• operation – Workload to measure.

• synchronize – Function that blocks until pending device work completes.

• clock_ns – Monotonic nanosecond clock. Defaults to time.perf_counter_ns().

Returns:

Host-submission and device-synchronized latency [s].

isaaclab.benchmark.summarize_latency(samples_s: Sequence[float]) → LatencyStatistics

Summarize a non-empty latency sample series.

Parameters:

samples_s – Latency samples [s].

Returns:

Mean, sample standard deviation, and interpolated percentiles [s].

Raises:

ValueError – If samples_s is empty.

Workflow Functions

isaaclab.benchmark.run_benchmark(request: BenchmarkRuntimeRequest) → BenchmarkResult[RuntimeBundle]

isaaclab.benchmark.run_benchmark(request: BenchmarkStartupRequest) → BenchmarkResult[StartupBundle]

isaaclab.benchmark.run_benchmark(request: BenchmarkTrainingRequest) → BenchmarkResult[TrainingBundle]

isaaclab.benchmark.run_benchmark(request: BenchmarkPlayRequest) → BenchmarkResult[PlayBundle]

Run a benchmark workflow from a typed request.

Parameters:

request – Runtime, startup, training, or play benchmark request.

Returns:

Completed benchmark bundle and output paths.

isaaclab.benchmark.run_play_benchmark(request: BenchmarkPlayRequest) → BenchmarkResult[PlayBundle]

Run an RL checkpoint playback benchmark.

Parameters:

request – Playback benchmark request.

Returns:

Completed benchmark bundle and output paths.

isaaclab.benchmark.run_runtime_benchmark(request: BenchmarkRuntimeRequest) → BenchmarkResult[RuntimeBundle]

Run an environment runtime benchmark.

Parameters:

request – Runtime benchmark request.

Returns:

Completed benchmark bundle and output paths.

isaaclab.benchmark.run_startup_benchmark(request: BenchmarkStartupRequest) → BenchmarkResult[StartupBundle]

Run a startup profiling benchmark.

Parameters:

request – Startup benchmark request.

Returns:

Completed benchmark bundle and output paths.

isaaclab.benchmark.run_training_benchmark(request: BenchmarkTrainingRequest) → BenchmarkResult[TrainingBundle]

Run an RL training benchmark.

Parameters:

request – Training benchmark request.

Returns:

Completed benchmark bundle and output paths.

Benchmark Framework

class isaaclab.benchmark.BaseIsaacLabBenchmark

Base benchmark class for IsaacLab’s benchmarks.

Methods:

__init__(benchmark_name[, formatter_type, ...])	Initialize common benchmark state and recorders.
attach_bundle(bundle)	Attach a typed bundle for schema serialization and flat-formatter projection.
update_manual_recorders()	Update manual recorders that don't depend on the kit timeline.
add_measurement(phase_name[, measurement, ...])	Add a measurement to the benchmark.
finalize()	Finalize metric collection and write selected formatter outputs.

Attributes:

output_file_path

Get the full path to the output file.

__init__(benchmark_name: str, formatter_type: str | list[str] | None = None, output_path: str | None = None, use_recorders: bool = True, output_prefix: str | None = None, workflow_metadata: dict | None = None, frametime_recorders: bool = False, backend_type: str | list[str] | None = None)

Initialize common benchmark state and recorders.

Parameters:

• benchmark_name – Name of benchmark to use in outputs.

• formatter_type – Formatter(s) used to collect and print metrics. Accepts a single type name, a list of type names, or a comma-separated string (e.g. "schema,omniperf"); each selected formatter writes its own output file.

• output_path – Path to output directory.

• use_recorders – Whether to use recorders to collect metrics. Defaults to True.

• output_prefix – Prefix used to generate the output filename. Defaults to None.

• workflow_metadata – Metadata describing benchmark, defaults to None.

• frametime_recorders – Whether to use frametime recorders to collect metrics. Defaults to False.

• backend_type – Alias for formatter_type.

property output_file_path: str

Get the full path to the output file.

attach_bundle(bundle: RuntimeBundle | TrainingBundle | StartupBundle | PlayBundle | None) → None

Attach a typed bundle for schema serialization and flat-formatter projection.

Parameters:

bundle – Runtime, training, startup, or play benchmark bundle.

update_manual_recorders() → None

Update manual recorders that don’t depend on the kit timeline.

add_measurement(phase_name: str, measurement: Measurement | Sequence[Measurement] | None = None, metadata: MetadataBase | Sequence[MetadataBase] | None = None) → None

Add a measurement to the benchmark.

Parameters:

• phase_name – The name of the phase to add the measurement to.

• measurement – The measurement to add.

• metadata – The metadata to add.

finalize() → tuple[Path, ...]

Finalize metric collection and write selected formatter outputs.

Returns:

Files written by the selected formatters.

class isaaclab.benchmark.BenchmarkMonitor

Background thread that periodically updates benchmark recorders.

This utility enables continuous system resource monitoring during blocking RL training loops (RSL-RL, RL-Games) where update_manual_recorders() would otherwise only be called once after training completes.

Usage:

with BenchmarkMonitor(benchmark, interval=1.0):

runner.learn(…) # Blocking training call

Methods:

__init__(benchmark[, interval])	Initialize the benchmark monitor.
start()	Start the monitoring thread.
stop()	Stop the monitoring thread and wait for it to finish.

__init__(benchmark: BaseIsaacLabBenchmark, interval: float = 1.0)

Initialize the benchmark monitor.

Parameters:

• benchmark – The benchmark instance to monitor.

• interval – Time between recorder updates in seconds. Defaults to 1.0.

start() → None

Start the monitoring thread.

stop() → None

Stop the monitoring thread and wait for it to finish.

class isaaclab.benchmark.MethodBenchmarkDefinition

Definition of a method benchmark.

name

Display name for the benchmark.

Type:

str

method_name

Name of the method to benchmark on the target object.

Type:

str

input_generators

Dict mapping mode names to input generator functions.

Type:

dict[str, collections.abc.Callable]

category

Category for grouping results into phases.

Type:

str

prepare_target

Optional hook that prepares the target outside the measured operation.

Type:

collections.abc.Callable[[object], None] | None

Methods:

__init__(name, method_name, input_generators)

__init__(name: str, method_name: str, input_generators: dict[str, Callable], category: str = 'default', prepare_target: Callable[[object], None] | None = None) → None

class isaaclab.benchmark.MethodBenchmarkRunner

Runner for method-level benchmarks using the new benchmark tooling.

This class extends BaseIsaacLabBenchmark to provide method-level benchmarking with automatic hardware/version info collection, multiple backend support, and organized output by category phases.

Methods:

__init__(benchmark_name, config[, ...])	Initialize the method benchmark runner.
run_benchmarks(benchmarks, target_object)	Run all defined benchmarks on the target object.
run_property_benchmarks(target_data, ...[, ...])	Run benchmarks for data class properties.

Attributes:

config

Return the benchmark configuration.

__init__(benchmark_name: str, config: MethodBenchmarkRunnerConfig, backend_type: str = 'json', output_path: str = '.', use_recorders: bool = True, physics_variant: str | None = None)

Initialize the method benchmark runner.

Parameters:

• benchmark_name – Name of the benchmark (used in output files).

• config – Benchmark configuration.

• backend_type – Output backend type (“json”, “osmo”, “omni_perf”).

• output_path – Directory to write output files.

• use_recorders – Whether to collect hardware/version info.

• physics_variant – Exact physics backend selector, when applicable.

property config: MethodBenchmarkRunnerConfig

Return the benchmark configuration.

run_benchmarks(benchmarks: list[MethodBenchmarkDefinition], target_object: object) → None

Run all defined benchmarks on the target object.

Parameters:

• benchmarks – List of benchmark definitions to run.

• target_object – Object containing the methods to benchmark.

run_property_benchmarks(target_data: object, properties: list[str], gen_mock_data: Callable, dependencies: dict[str, list[str]] | None = None, category: str = 'property') → None

Run benchmarks for data class properties.

This is a convenience method for benchmarking properties on data classes where the test involves generating mock data and accessing properties.

Parameters:

• target_data – Data object containing the properties to benchmark.

• properties – List of property names to benchmark.

• gen_mock_data – Function that generates/updates mock data.

• dependencies – Optional dict mapping property names to their dependencies.

• category – Category name for grouping results.

class isaaclab.benchmark.MethodBenchmarkRunnerConfig

Configuration for MethodBenchmarkRunner.

num_iterations

Number of timing iterations per method.

Type:

int

warmup_steps

Number of warmup iterations before timing.

Type:

int

num_instances

Number of environment instances.

Type:

int

num_bodies

Number of bodies per instance.

Type:

int

num_joints

Number of joints per instance.

Type:

int

device

Device to run benchmarks on.

Type:

str

mode

Which input modes to run (“all” or specific mode name).

Type:

str | list[str]

Methods:

__init__([num_iterations, warmup_steps, ...])

__init__(num_iterations: int = 1000, warmup_steps: int = 10, num_instances: int = 4096, num_bodies: int = 12, num_joints: int = 11, device: str = 'cuda:0', mode: str | list[str] = 'all') → None

class isaaclab.benchmark.PlayBundle

Top-level shape of play.json — a checkpoint-driven inference rollout.

Mirrors RuntimeBundle (with RunIdentity.framework set to the RL library that produced the checkpoint) and adds the inference-evaluation aggregates: a success rate plus scalar reward and episode-length statistics. Unlike TrainingBundle, reward and ep_length are scalar MeanStd aggregates over completed episodes, not per-iteration learning curves.

Parameters:

• success_rate – Mean success rate [0..1] over completed episodes, or None when the task does not report one.

• reward – Episode-return aggregate over completed episodes, or None when no episode completed.

• ep_length – Episode-length aggregate over completed episodes, or None when no episode completed.

• checkpoint_path – Path to the policy checkpoint that was rolled out.

• video_path – Path to a recorded rollout video/gif, if any.

• extra – Optional free-form scalar values (experimental or producer-specific) that are not part of the stable schema contract. Consumers must tolerate its absence and must not depend on specific keys; promote a key to a typed field once it is stable and broadly useful.

Methods:

__init__(run, versions, hardware, runtime, ...)

__init__(run: RunIdentity, versions: Versions, hardware: Hardware, runtime: Runtime, resources: Resources, success_rate: float | None = None, reward: MeanStd | None = None, ep_length: MeanStd | None = None, checkpoint_path: str | None = None, video_path: str | None = None, extra: dict[str, float | int | str | bool] | None = None, schema_version: str = '1.4') → None

class isaaclab.benchmark.RuntimeBundle

Top-level shape of runtime.json (environment stepping, no learning).

Mirrors TrainingBundle without the learning metrics.

Parameters:

extra – Optional free-form scalar values (experimental or producer-specific) that are not part of the stable schema contract. Consumers must tolerate its absence and must not depend on specific keys; promote a key to a typed field once it is stable and broadly useful.

Methods:

__init__(run, versions, hardware, runtime, ...)

__init__(run: RunIdentity, versions: Versions, hardware: Hardware, runtime: Runtime, resources: Resources, extra: dict[str, float | int | str | bool] | None = None, schema_version: str = '1.4') → None

class isaaclab.benchmark.StartupBundle

Top-level shape of startup.json.

Reuses RunIdentity with framework/num_envs/max_iterations left unset, since they are not meaningful for a startup profile.

Parameters:

extra – Optional free-form scalar values (experimental or producer-specific) that are not part of the stable schema contract. Consumers must tolerate its absence and must not depend on specific keys; promote a key to a typed field once it is stable and broadly useful.

Methods:

__init__(run, versions, hardware, phases, config)

__init__(run: RunIdentity, versions: Versions, hardware: Hardware, phases: dict[str, StartupPhase], config: StartupConfig, extra: dict[str, float | int | str | bool] | None = None, schema_version: str = '1.4') → None

class isaaclab.benchmark.TrainingBundle

Top-level shape of training.json — a runtime bundle plus learning metrics.

Parameters:

• success_rate – Final success rate [0..1] when the task tracks one, else None.

• checkpoint_path – Path to the final saved policy checkpoint, if any.

• video_path – Path to a recorded rollout video/gif, if any.

• extra – Optional free-form scalar values (experimental or producer-specific) that are not part of the stable schema contract. Consumers must tolerate its absence and must not depend on specific keys; promote a key to a typed field once it is stable and broadly useful.

Methods:

__init__(run, versions, hardware, runtime, ...)

__init__(run: RunIdentity, versions: Versions, hardware: Hardware, runtime: Runtime, resources: Resources, learning: Learning, success_rate: float | None = None, checkpoint_path: str | None = None, video_path: str | None = None, extra: dict[str, float | int | str | bool] | None = None, schema_version: str = '1.4') → None

Additional Public Classes

The following classes are part of the public isaaclab.benchmark API.

BooleanMeasurement	Boolean measurement.
CProfileFunction	One entry from a cProfile top-N table.
DictMeasurement	Dictionary measurement.
DictMetadata	Dictionary metadata.
EnvironmentStepTiming	Environment-step wall time and optional synchronized simulation breakdown.
FloatMetadata	Float metadata.
GpuDeviceInfo	Information about a single GPU device.
Hardware	Host hardware snapshot captured at run time.
IntMetadata	Integer metadata.
Learning	Learning curves for a training run, plus their EMA smoothing factor.
LearningCurve	One learning curve (reward, episode length, or success rate).
ListMeasurement	List measurement.
MeanStd	Scalar aggregate with mean, standard deviation, and optional peak.
Measurement	Base measurement record.
MetadataBase	Base metadata record.
Resources	Aggregated resource-utilisation metrics for a run.
RunConfig	Physics/rendering backend and active presets for a run.
RunIdentity	Identity of a benchmark run (training, runtime, or startup).
Runtime	Aggregated runtime metrics for a run.
SingleMeasurement	Single floating-point measurement.
StartupConfig	CLI configuration captured in a StartupBundle.
StartupPhase	Wall-clock total plus top cProfile functions for one startup phase.
StartupTime	Wall-clock duration of each startup phase [s].
StatisticalMeasurement	Statistical measurement.
StringMetadata	String metadata.
TestPhase	Represent a single test phase with associated metrics and metadata.
Versions	Software versions captured at run time.

class isaaclab.benchmark.BooleanMeasurement

Bases: Measurement

Boolean measurement.

Parameters:

• name – Measurement name.

• bvalue – Measurement value.

• type – Measurement type label. Defaults to “boolean”.

Methods:

__new__(*args, **kwargs)
__init__(name, bvalue[, type])

classmethod __new__(*args, **kwargs)

__init__(name: str, bvalue: bool, type: str = 'boolean') → None

class isaaclab.benchmark.CProfileFunction

Bases: object

One entry from a cProfile top-N table.

Parameters:

• name – Function label.

• own_time_s – Own (exclusive) time [s].

• cum_time_s – Cumulative (inclusive) time [s].

• calls – Number of calls.

Methods:

__new__(*args, **kwargs)
__init__(name, own_time_s, cum_time_s, calls)

classmethod __new__(*args, **kwargs)

__init__(name: str, own_time_s: float, cum_time_s: float, calls: int) → None

class isaaclab.benchmark.DictMeasurement

Bases: Measurement

Dictionary measurement.

Parameters:

• name – Measurement name.

• value – Measurement value.

• type – Measurement type label. Defaults to “dict”.

Methods:

__new__(*args, **kwargs)
__init__(name, value[, type])

classmethod __new__(*args, **kwargs)

__init__(name: str, value: dict, type: str = 'dict') → None

class isaaclab.benchmark.DictMetadata

Bases: MetadataBase

Dictionary metadata.

Parameters:

• name – Metadata name.

• data – Metadata value.

• type – Metadata type label. Defaults to “dict”.

Methods:

__new__(*args, **kwargs)
__init__(name, data[, type])

classmethod __new__(*args, **kwargs)

__init__(name: str, data: dict, type: str = 'dict') → None

class isaaclab.benchmark.EnvironmentStepTiming

Bases: object

Environment-step wall time and optional synchronized simulation breakdown.

host_return mode records how long the host spends inside env.step() without forcing device completion. serialized_synchronized mode drains pending work at every environment and simulation boundary so the measured environment time can be partitioned into time inside and outside nested SimulationContext.step() calls. The latter mode serializes device work and is an observer-perturbed diagnostic, not production throughput.

When the serialized mode is active, every timing and throughput aggregate in the enclosing Runtime was collected under that instrumented schedule, not only the fields in this breakdown.

Time outside simulation calls includes required action, actuator, state, manager, reset, wrapper, and synchronization work. It is not an estimate of removable Isaac Lab overhead.

Parameters:

• environment_step_time_s – Per-environment-step wall time [s], interpreted according to measurement_mode.

• environment_step_fps – Reciprocal environment-step rate [frames/s], interpreted according to measurement_mode.

• simulation_step_time_s – Synchronized simulation wall time per environment step [s], when measured.

• outside_simulation_step_time_s – Time outside simulation calls per environment step [s], when measured.

• outside_simulation_step_fraction – Fraction of synchronized environment-step time outside simulation calls.

• environment_step_calls – Number of measured environment-step calls.

• simulation_step_calls – Number of measured simulation-step calls, when measured.

• measurement_mode – Timing boundary semantics. host_return does not force device completion. serialized_synchronized explicitly synchronizes every measured boundary.

• warmup_steps – Number of initial environment-step calls excluded from timing.

Methods:

__new__(*args, **kwargs)
__init__(environment_step_time_s, ...[, ...])

classmethod __new__(*args, **kwargs)

__init__(environment_step_time_s: MeanStd, environment_step_fps: MeanStd, simulation_step_time_s: MeanStd | None, outside_simulation_step_time_s: MeanStd | None, outside_simulation_step_fraction: float | None, environment_step_calls: int, simulation_step_calls: int | None, measurement_mode: Literal['host_return', 'serialized_synchronized'], warmup_steps: int = 0) → None

class isaaclab.benchmark.FloatMetadata

Bases: MetadataBase

Float metadata.

Parameters:

• name – Metadata name.

• data – Metadata value.

• type – Metadata type label. Defaults to “float”.

Methods:

__new__(*args, **kwargs)
__init__(name, data[, type])

classmethod __new__(*args, **kwargs)

__init__(name: str, data: float, type: str = 'float') → None

class isaaclab.benchmark.GpuDeviceInfo

Bases: object

Information about a single GPU device.

Parameters:

• name – Device model name.

• mem_gb – Total device memory [GB].

• compute_cap – CUDA compute capability (e.g. "9.0").

Methods:

__new__(*args, **kwargs)
__init__(name, mem_gb, compute_cap)

classmethod __new__(*args, **kwargs)

__init__(name: str, mem_gb: float, compute_cap: str) → None

class isaaclab.benchmark.Hardware

Bases: object

Host hardware snapshot captured at run time.

Parameters:

• hostname – Host machine name.

• gpu_devices – Per-device GPU information.

• cpu_name – CPU model name.

• cpu_count – Physical CPU core count.

• ram_gb – Total host RAM [GB].

Methods:

__new__(*args, **kwargs)
__init__(hostname, gpu_devices, cpu_name, ...)

classmethod __new__(*args, **kwargs)

__init__(hostname: str, gpu_devices: list[GpuDeviceInfo], cpu_name: str, cpu_count: int, ram_gb: float) → None

class isaaclab.benchmark.IntMetadata

Bases: MetadataBase

Integer metadata.

Parameters:

• name – Metadata name.

• data – Metadata value.

• type – Metadata type label. Defaults to “int”.

Methods:

__new__(*args, **kwargs)
__init__(name, data[, type])

classmethod __new__(*args, **kwargs)

__init__(name: str, data: int, type: str = 'int') → None

class isaaclab.benchmark.Learning

Bases: object

Learning curves for a training run, plus their EMA smoothing factor.

Parameters:

• ema_alpha – EMA smoothing factor in [0, 1].

• reward – Per-iteration mean-reward learning curve.

• ep_length – Per-iteration mean episode-length learning curve.

• success_rate – Per-iteration success-rate learning curve, or None when the task does not report success.

Methods:

__new__(*args, **kwargs)
__init__(ema_alpha, reward, ep_length[, ...])

classmethod __new__(*args, **kwargs)

__init__(ema_alpha: float, reward: LearningCurve, ep_length: LearningCurve, success_rate: LearningCurve | None = None) → None

class isaaclab.benchmark.LearningCurve

Bases: object

One learning curve (reward, episode length, or success rate).

Methods:

__new__(*args, **kwargs)
__init__(final_raw, final_ema, series_per_iter)

classmethod __new__(*args, **kwargs)

__init__(final_raw: float, final_ema: float, series_per_iter: list[float] | None) → None

class isaaclab.benchmark.ListMeasurement

Bases: Measurement

List measurement.

Parameters:

• name – Measurement name.

• value – Measurement value.

• type – Measurement type label. Defaults to “list”.

Methods:

__new__(*args, **kwargs)
__init__(name, value[, type])

classmethod __new__(*args, **kwargs)

__init__(name: str, value: list, type: str = 'list') → None

class isaaclab.benchmark.MeanStd

Bases: object

Scalar aggregate with mean, standard deviation, and optional peak.

Parameters:

• mean – Central value of the aggregate. For most fields this is the arithmetic sample mean; for effective-throughput fields it is the aggregate rate (total completed work over total wall time).

• std – Ordinary sample standard deviation of the per-sample values. This remains centered on the sample mean even when mean is an effective aggregate rate.

• peak – Maximum observed value, or None where a peak is not meaningful (e.g. GPU utilisation, whose ceiling is always 100%).

Methods:

__new__(*args, **kwargs)
__init__(mean, std[, peak])

classmethod __new__(*args, **kwargs)

__init__(mean: float, std: float, peak: float | None = None) → None

class isaaclab.benchmark.Measurement

Bases: object

Base measurement record.

Parameters:

name – Measurement name.

Methods:

__new__(*args, **kwargs)
__init__(name)

classmethod __new__(*args, **kwargs)

__init__(name: str) → None

class isaaclab.benchmark.MetadataBase

Bases: object

Base metadata record.

Parameters:

name – Metadata name.

Methods:

__new__(*args, **kwargs)
__init__(name)

classmethod __new__(*args, **kwargs)

__init__(name: str) → None

class isaaclab.benchmark.Resources

Bases: object

Aggregated resource-utilisation metrics for a run.

Utilisation fields leave MeanStd.peak as None (a peak of 100% is uninformative); memory fields populate peak.

Parameters:

• gpu_util_pct – Utilisation of the device the run used [%].

• gpu_mem_gb – Memory used on the device the run used [GB].

• cpu_util_pct – CPU utilisation [%].

• ram_gb – Host RAM used [GB].

• devices – Per-device metrics keyed by logical CUDA device index, covering every device visible to the process. On a multi-GPU run this is the whole node, while gpu_util_pct and gpu_mem_gb stay scoped to the device the reporting rank used.

Methods:

__new__(*args, **kwargs)
__init__(gpu_util_pct, gpu_mem_gb, ...[, ...])

classmethod __new__(*args, **kwargs)

__init__(gpu_util_pct: ~isaaclab.benchmark.schema.MeanStd, gpu_mem_gb: ~isaaclab.benchmark.schema.MeanStd, cpu_util_pct: ~isaaclab.benchmark.schema.MeanStd, ram_gb: ~isaaclab.benchmark.schema.MeanStd, devices: dict[str, ~isaaclab.benchmark.schema.GpuResources] = <factory>) → None

class isaaclab.benchmark.RunConfig

Bases: object

Physics/rendering backend and active presets for a run.

Parameters:

• physics_backend – Physics solver preset the run used.

• rendering_backend – Rendering backend, or "none" for headless runs with no camera sensors.

• presets – Active Hydra preset tokens applied to the run (e.g. ["rgb", "ovrtx"]). Open-ended so sensor data types, resolutions, and any other domain presets are captured without a closed enum; physics_backend / rendering_backend surface the two primary grouping dimensions as typed fields.

Methods:

__new__(*args, **kwargs)
__init__(physics_backend[, ...])

classmethod __new__(*args, **kwargs)

__init__(physics_backend: ~typing.Literal['physx', 'newton_mjwarp', 'newton_kamino', 'ovphysx'], rendering_backend: ~typing.Literal['none', 'isaacsim_rtx', 'ovrtx', 'newton'] = 'none', presets: list[str] = <factory>) → None

class isaaclab.benchmark.RunIdentity

Bases: object

Identity of a benchmark run (training, runtime, or startup).

Parameters:

• run_id – Stable identifier for the run.

• framework – RL library for training runs; None for non-learning (pure runtime, startup) runs.

• config – Physics/rendering/sensor configuration.

• task – Gym task id.

• seed – Environment/agent seed.

• start_time_utc – ISO-8601 UTC start timestamp.

• end_time_utc – ISO-8601 UTC end timestamp.

• duration_s – Wall-clock run duration [s].

• status – Terminal status of the run.

• num_envs – Number of parallel environments, or None (startup).

• max_iterations – Training iteration budget, or None (startup, runtime).

Methods:

__new__(*args, **kwargs)
__init__(run_id, framework, config, task, ...)

classmethod __new__(*args, **kwargs)

__init__(run_id: str, framework: Literal['rsl_rl', 'rl_games', 'skrl', 'sb3'] | None, config: RunConfig, task: str, seed: int, start_time_utc: str, end_time_utc: str, duration_s: float, status: Literal['completed', 'interrupted', 'crashed'], num_envs: int | None = None, max_iterations: int | None = None) → None

class isaaclab.benchmark.Runtime

Bases: object

Aggregated runtime metrics for a run.

Parameters:

• startup_time_s – Per-phase startup wall-clock durations [s].

• iterations_completed – Number of completed iterations.

• total_wall_time_s – Total run wall-clock time [s].

• steps_per_iteration – Environment steps collected per iteration.

• iteration_time_s – Per-iteration wall-clock time [s].

• collection_fps – Environment-stepping (rollout) throughput [frames/s] — environment steps per second across all environments during data collection (the scripts’ “Collection FPS” / “Environment + Inference FPS”).

• total_fps – End-to-end throughput [frames/s] including the policy update — the headline FPS (the scripts’ “Total FPS” / “effective FPS”). For pure runtime runs with no learning, this equals collection_fps.

• iterations_per_s – Iteration rate [iter/s].

• environment_step_timing – Environment-step timing and optional synchronized simulation breakdown, when measured. Its measurement mode also describes the schedule used by the enclosing timing and rate fields.

Methods:

__new__(*args, **kwargs)
__init__(startup_time_s, ...[, ...])

classmethod __new__(*args, **kwargs)

__init__(startup_time_s: StartupTime, iterations_completed: int, total_wall_time_s: float, steps_per_iteration: int, iteration_time_s: MeanStd, collection_fps: MeanStd, total_fps: MeanStd, iterations_per_s: MeanStd, environment_step_timing: EnvironmentStepTiming | None = None) → None

class isaaclab.benchmark.SingleMeasurement

Bases: Measurement

Single floating-point measurement.

Parameters:

• name – Measurement name.

• value – Measurement value.

• unit – Unit string.

• type – Measurement type label. Defaults to “single”.

Methods:

__new__(*args, **kwargs)
__init__(name, value, unit[, type])

classmethod __new__(*args, **kwargs)

__init__(name: str, value: float | int | str, unit: str, type: str = 'single') → None

class isaaclab.benchmark.StartupConfig

Bases: object

CLI configuration captured in a StartupBundle.

Methods:

__new__(*args, **kwargs)
__init__(top_n, whitelist)

classmethod __new__(*args, **kwargs)

__init__(top_n: int, whitelist: str | None) → None

class isaaclab.benchmark.StartupPhase

Bases: object

Wall-clock total plus top cProfile functions for one startup phase.

Methods:

__new__(*args, **kwargs)
__init__(total_time_s, top_functions)

classmethod __new__(*args, **kwargs)

__init__(total_time_s: float, top_functions: list[CProfileFunction]) → None

class isaaclab.benchmark.StartupTime

Bases: object

Wall-clock duration of each startup phase [s].

Methods:

__new__(*args, **kwargs)
__init__(app_launch, env_creation, first_step)

classmethod __new__(*args, **kwargs)

__init__(app_launch: float, env_creation: float, first_step: float, python_imports: float | None = None, task_config: float | None = None) → None

class isaaclab.benchmark.StatisticalMeasurement

Bases: Measurement

Statistical measurement.

Parameters:

• name – Measurement name.

• mean – Mean value.

• std – Standard deviation value.

• n – Number of samples.

• unit – Unit string.

• type – Measurement type label. Defaults to “statistical”.

Methods:

__new__(*args, **kwargs)
__init__(name, mean, std, n, unit[, type])

classmethod __new__(*args, **kwargs)

__init__(name: str, mean: float, std: float, n: int, unit: str, type: str = 'statistical') → None

class isaaclab.benchmark.StringMetadata

Bases: MetadataBase

String metadata.

Parameters:

• name – Metadata name.

• data – Metadata value.

• type – Metadata type label. Defaults to “string”.

Methods:

__new__(*args, **kwargs)
__init__(name, data[, type])

classmethod __new__(*args, **kwargs)

__init__(name: str, data: str | None, type: str = 'string') → None

class isaaclab.benchmark.TestPhase

Bases: object

Represent a single test phase with associated metrics and metadata.

Parameters:

• phase_name – Name of the phase.

• measurements – Measurements recorded for the phase. Defaults to an empty list.

• metadata – Metadata recorded for the phase. Defaults to an empty list.

Methods:

__new__(*args, **kwargs)
__init__(phase_name[, measurements, metadata])

classmethod __new__(*args, **kwargs)

__init__(phase_name: str, measurements: list[~isaaclab.benchmark.measurements.Measurement] = <factory>, metadata: list[~isaaclab.benchmark.measurements.StringMetadata | ~isaaclab.benchmark.measurements.IntMetadata | ~isaaclab.benchmark.measurements.FloatMetadata | ~isaaclab.benchmark.measurements.DictMetadata] = <factory>) → None

class isaaclab.benchmark.Versions

Bases: object

Software versions captured at run time.

Version fields are None when the corresponding runtime or package is unavailable.

Methods:

__new__(*args, **kwargs)
__init__(isaaclab, isaacsim, kit, newton, ...)

classmethod __new__(*args, **kwargs)

__init__(isaaclab: str, isaacsim: str | None, kit: str | None, newton: str | None, warp: str | None, mjwarp: str | None, torch: str, rsl_rl: str | None, rl_games: str | None, skrl: str | None, sb3: str | None, git_commit: str | None, git_branch: str | None, git_dirty: bool, numpy: str | None = None, isaaclab_newton: str | None = None, isaaclab_physx: str | None = None, isaaclab_ov: str | None = None, isaaclab_tasks: str | None = None, isaaclab_rl: str | None = None, ovrtx: str | None = None, ovphysx: str | None = None, mujoco: str | None = None, cuda_bindings: str | None = None, usd_core: str | None = None, usd_exchange: str | None = None, isaaclab_release: str | None = None) → None