Benchmarking Framework

Benchmarking Framework#

Isaac Lab provides a comprehensive benchmarking framework for measuring the performance of simulations, training workflows, and system resources. The framework is designed to work without depending on Isaac Sim’s benchmark services, enabling standalone benchmarking with pluggable output formatters.

Overview#

The benchmarking framework consists of several key components:

┌─────────────────────────────────┐
│    BaseIsaacLabBenchmark        │
│    (benchmark_core.py)          │
└───────────────┬─────────────────┘
                │
    ┌───────────┼───────────┐
    │           │           │
    ▼           ▼           ▼
┌───────┐  ┌─────────┐  ┌────────────┐
│Phases │  │Recorders│  │ Formatters │
└───────┘  └─────────┘  └────────────┘

Key Components:

  • BaseIsaacLabBenchmark: Main class for orchestrating benchmark execution

  • Measurements: Data classes for recording metrics (timing, counts, statistics)

  • Metadata: Data classes for recording context (hardware, versions, parameters)

  • TestPhase: Container for organizing measurements into logical groups

  • Recorders: System information collectors (CPU, GPU, memory, versions)

  • Formatters: Output formatters (JSON, Osmo, OmniPerf, Summary, Schema)

See also

For method-level micro-benchmarks that measure asset setter/writer and property performance using mock interfaces (without running full simulations), see Micro-Benchmarks for Performance Testing.

Quick Start#

Basic usage with BaseIsaacLabBenchmark:

from isaaclab.test.benchmark import (
    BaseIsaacLabBenchmark,
    SingleMeasurement,
    StatisticalMeasurement,
    StringMetadata,
)

# Initialize benchmark
benchmark = BaseIsaacLabBenchmark(
    benchmark_name="MyBenchmark",
    formatter_type="json",
    output_path="./results",
)

# Record measurements
benchmark.add_measurement(
    phase_name="simulation",
    measurement=SingleMeasurement(
        name="fps",
        value=1234.5,
        unit="frames/sec"
    ),
)

benchmark.add_measurement(
    phase_name="simulation",
    measurement=StatisticalMeasurement(
        name="step_time",
        mean=0.82,
        std=0.05,
        n=1000,
        unit="ms"
    ),
)

# Add metadata
benchmark.add_measurement(
    phase_name="simulation",
    metadata=StringMetadata(name="task", data="Isaac-Cartpole"),
)

# Finalize and write output
benchmark._finalize_impl()

Running Benchmark Scripts#

Isaac Lab provides unified runtime.py, startup.py, training.py, and play.py entry points under scripts/benchmarks/. They default to --benchmark_formatter schema, which emits a schema-v1 JSON bundle via isaaclab.test.benchmark. --benchmark_formatter accepts a comma-separated list (e.g. schema,omniperf) to emit several formats in a single run. Each selected formatter writes timestamped output; the Osmo formatter writes one phase-suffixed JSON file per phase.

The examples below use uv run isaaclab benchmark. From an existing Isaac Lab environment, run the same workflows directly instead:

  • Runtime: ./isaaclab.sh -p scripts/benchmarks/runtime.py <arguments>

  • Startup: ./isaaclab.sh -p scripts/benchmarks/startup.py <arguments>

  • Training: ./isaaclab.sh -p scripts/benchmarks/training.py <arguments>

  • Play: ./isaaclab.sh -p scripts/benchmarks/play.py <arguments>

Non-RL / Runtime Benchmarks#

Measure environment stepping performance without any RL library:

uv run isaaclab benchmark runtime \
    --task Isaac-Cartpole \
    --num_envs 4096 \
    --num_frames 100 \
    --benchmark_formatter json \
    --output_path ./results

RL Training Benchmarks#

Measure training performance. Use --rl_library to select the RL library (rsl_rl, rl_games, skrl, or sb3):

# Benchmark with RSL-RL
uv run isaaclab benchmark training \
    --rl_library rsl_rl \
    --task Isaac-Cartpole \
    --num_envs 4096 \
    --max_iterations 500 \
    --benchmark_formatter json \
    --output_path ./results

# Benchmark with RL Games
uv run isaaclab benchmark training \
    --rl_library rl_games \
    --task Isaac-Cartpole \
    --num_envs 4096 \
    --max_iterations 500 \
    --benchmark_formatter json \
    --output_path ./results

RL Play Benchmarks#

Load a trained checkpoint and benchmark policy inference (the play workflow). The same --rl_library dispatch selects the RL library (rsl_rl, rl_games, skrl, or sb3). In addition to the inference throughput, the emitted PlayBundle reports the rolled-out policy’s reward, episode length, and success rate. The checkpoints consumed here are produced by training.py.

# Benchmark inference of a trained RSL-RL policy
uv run isaaclab benchmark play \
    --rl_library rsl_rl \
    --task Isaac-Cartpole \
    --num_envs 4096 \
    --num_frames 1000 \
    --checkpoint /path/to/model.pt \
    --benchmark_formatter json \
    --output_path ./results

The checkpoint is resolved in the following order:

  1. --checkpoint — a local filesystem path or a Nucleus URI.

  2. Otherwise, the published Nucleus checkpoint for the task is downloaded (a warning is logged).

  3. If neither is available, an error is raised.

Note

reward, ep_length, and success_rate aggregate only completed episodes. Set --num_frames larger than the task’s episode length so at least one episode finishes during the rollout; otherwise these fields remain null (the inference throughput is still reported).

PhysX Micro-Benchmarks#

Measure asset method and property performance using mock interfaces:

# Run articulation benchmarks
./isaaclab.sh -p source/isaaclab_physx/benchmark/assets/benchmark_articulation.py \
    --num_iterations 1000 \
    --num_instances 4096

For detailed documentation on micro-benchmarks, including available benchmark files, input modes, and how to add new benchmarks, see Micro-Benchmarks for Performance Testing.

Startup Profiling Benchmark#

Profile the startup sequence of an IsaacLab environment using cProfile. Each startup stage is wrapped in its own profiling session and the top functions by own-time are reported. This is useful for investigating startup regressions and understanding where time is spent during initialization.

# Basic usage — reports top 30 functions per phase
uv run isaaclab benchmark startup \
    --task Isaac-Ant \
    --num_envs 4096 \
    --benchmark_formatter summary

The script profiles five phases independently:

  • app_launch: launch_simulation() context entry (simulation runtime initialization)

  • python_imports: importing gymnasium, torch, isaaclab_tasks, etc.

  • task_config: resolve_task_config() (Hydra config resolution)

  • env_creation: gym.make() + env.reset() (scene creation, sim start)

  • first_step: a single env.step() call

Schema output records each phase wall-clock time and per-function own-time, cumulative time, and call count. Flat formatters project the same data into measurements. Only Isaac Lab functions and first-level calls into external libraries are included (deep internals of torch, USD, etc. are filtered out).

Whitelist mode — For dashboard time-series comparisons across runs, use a YAML whitelist config to report a fixed set of functions instead of top-N. Patterns use fnmatch syntax (* and ? wildcards):

# Example whitelist config
app_launch:
  - "isaaclab.utils.configclass:_custom_post_init"
  - "isaaclab.sim.*:__init__"
env_creation:
  - "isaaclab.cloner.*:usd_replicate"
  - "isaaclab.cloner.*:filter_collisions"
  - "isaaclab.scene.*:_init_scene"
first_step:
  - "isaaclab.actuators.*:compute"
  - "warp.*:launch"

uv run isaaclab benchmark startup \
    --task Isaac-Ant \
    --num_envs 4096 \
    --benchmark_formatter omniperf \
    --whitelist_config scripts/benchmarks/startup_whitelist.yaml

Phases listed in the YAML use the whitelist; phases not listed fall back to --top_n (default: 5 in whitelist mode, 30 otherwise). Patterns that match no profiled function emit 0.0 placeholders so the output always contains the same keys.

A default whitelist is provided at scripts/benchmarks/startup_whitelist.yaml.

Argument

Default

Description

--task

required

Environment task name

--num_envs

from config

Number of parallel environments

--top_n

30 (5 with whitelist)

Max functions per non-whitelisted phase

--whitelist_config

None

Path to YAML whitelist file

--benchmark_formatter

schema

Output formatter(s), comma-separated (schema, json, osmo, omniperf, summary)

--output_path

.

Directory for output files

Command Line Arguments#

Common Arguments#

Argument

Default

Description

--benchmark_formatter

schema

Output formatter(s), comma-separated (schema, json, osmo, omniperf, summary)

--output_path

./

Directory for output files

Non-RL / Runtime Benchmark Arguments#

Argument

Default

Description

--task

required

Environment task name (e.g., Isaac-Cartpole)

--num_envs

None (task config)

Number of parallel environments

--num_frames

100

Number of simulation frames to run

--enable_cameras

false

Enable camera rendering (for RGB/depth tasks)

RL Training Arguments#

Argument

Default

Description

--rl_library

required

RL library: rsl_rl, rl_games, skrl, or sb3

--task

required

Environment task name

--num_envs

None (task config)

Number of parallel environments

--max_iterations

None (task config)

Number of training iterations

RL Play Arguments#

Argument

Default

Description

--task

required

Environment task name

--rl_library

required

RL library that produced the checkpoint: rsl_rl, rl_games, skrl, or sb3

--num_envs

None (task config)

Number of parallel environments

--num_frames

100

Number of inference steps to roll out

--checkpoint

None (published Nucleus checkpoint)

Local path or Nucleus URI of the checkpoint to roll out

--benchmark_formatter

schema

Output formatter(s), comma-separated (schema, json, osmo, omniperf, summary)

Measurement Types#

The framework provides several measurement types for different data:

SingleMeasurement#

For single numeric values:

from isaaclab.test.benchmark import SingleMeasurement

measurement = SingleMeasurement(
    name="total_frames",
    value=100000,
    unit="frames"
)

StatisticalMeasurement#

For statistical summaries:

from isaaclab.test.benchmark import StatisticalMeasurement

measurement = StatisticalMeasurement(
    name="step_time",
    mean=0.82,
    std=0.05,
    n=1000,
    unit="ms"
)

BooleanMeasurement#

For pass/fail status:

from isaaclab.test.benchmark import BooleanMeasurement

measurement = BooleanMeasurement(
    name="converged",
    bvalue=True
)

DictMeasurement#

For structured data:

from isaaclab.test.benchmark import DictMeasurement

measurement = DictMeasurement(
    name="config",
    value={"learning_rate": 0.001, "batch_size": 64}
)

ListMeasurement#

For sequences of values:

from isaaclab.test.benchmark import ListMeasurement

measurement = ListMeasurement(
    name="rewards_per_episode",
    value=[100.5, 102.3, 98.7, 105.1]
)

Test Phases#

TestPhase organizes measurements and metadata into logical groups. Common phases include:

  • benchmark_info: Workflow name, timestamp, configuration

  • hardware_info: CPU, GPU, memory information

  • version_info: Software versions (Isaac Sim, PyTorch, etc.)

  • simulation: Environment stepping metrics

  • training: RL training metrics

  • runtime: Execution time and resource usage

Example:

# Measurements are automatically grouped by phase
benchmark.add_measurement("simulation", measurement=fps_measurement)
benchmark.add_measurement("simulation", metadata=task_metadata)
benchmark.add_measurement("training", measurement=reward_measurement)

Output Formatters#

JSON Formatter#

Full output with all phases, measurements, and metadata:

./isaaclab.sh -p ... --benchmark_formatter json --output_path ./results

Output structure:

[
  {
    "phase_name": "simulation",
    "measurements": [
      {
        "name": "MyBenchmark simulation fps",
        "value": 1234.5,
        "unit": "frames/sec",
        "type": "single"
      }
    ],
    "metadata": [
      {"name": "MyBenchmark simulation task", "data": "Isaac-Cartpole", "type": "string"}
    ]
  }
]

Osmo Formatter#

Simplified key-value format for CI/CD integration:

./isaaclab.sh -p ... --benchmark_formatter osmo --output_path ./results

Output structure:

{
  "workflow_name": "MyBenchmark",
  "phase": "simulation",
  "fps": 1234.5,
  "task": "Isaac-Cartpole"
}

OmniPerf Formatter#

Format for database upload and performance tracking:

./isaaclab.sh -p ... --benchmark_formatter omniperf --output_path ./results

Output structure:

{
  "simulation": {
    "workflow_name": "MyBenchmark",
    "fps": 1234.5,
    "step_time_mean": 0.82,
    "step_time_std": 0.05
  }
}

Schema Formatter#

Writes a schema-v1 bundle attached with attach_bundle(). Use it with a RuntimeBundle, TrainingBundle, or StartupBundle when a typed, stable output contract is required.

./isaaclab.sh -p ... --benchmark_formatter schema --output_path ./results

Summary Formatter#

Human-readable console report plus JSON file. Prints a formatted summary to the terminal while also writing the same data as JSON. Standard phases (runtime, startup, train, frametime, system info) are rendered with specialized formatting; any additional phases (e.g., from the startup profiling benchmark) are rendered automatically with their SingleMeasurement and StatisticalMeasurement entries. Use when you want a quick readout without opening the JSON:

./isaaclab.sh -p ... --benchmark_formatter summary --output_path ./results

When summary is selected, frametime recorders are enabled automatically when running with Isaac Sim (Kit).

BenchmarkMonitor#

BenchmarkMonitor enables continuous system monitoring during blocking operations like RL training loops:

from isaaclab.test.benchmark import BaseIsaacLabBenchmark, BenchmarkMonitor

benchmark = BaseIsaacLabBenchmark(
    benchmark_name="TrainingBenchmark",
    formatter_type="json",
    output_path="./results",
)

# Monitor system resources during blocking training call
with BenchmarkMonitor(benchmark, interval=1.0):
    runner.learn(num_learning_iterations=1000)  # Blocking call

benchmark._finalize_impl()

The monitor runs in a background thread and periodically calls update_manual_recorders() to capture CPU, GPU, and memory usage samples.

System Recorders#

The framework includes built-in recorders for system information:

CPUInfoRecorder#

Captures CPU model, core count, and usage statistics.

GPUInfoRecorder#

Captures GPU model, memory, and utilization via nvidia-smi.

MemoryInfoRecorder#

Captures system and GPU memory usage over time.

VersionInfoRecorder#

Captures software versions:

  • Isaac Sim version

  • Isaac Lab version

  • PyTorch version

  • CUDA version

  • Python version

Creating Custom Benchmarks#

Step 1: Initialize Benchmark#

import argparse
from isaaclab.test.benchmark import BaseIsaacLabBenchmark

parser = argparse.ArgumentParser()
parser.add_argument("--benchmark_formatter", default="json")
parser.add_argument("--output_path", default="./")
args = parser.parse_args()

benchmark = BaseIsaacLabBenchmark(
    benchmark_name="CustomBenchmark",
    formatter_type=args.benchmark_formatter,
    output_path=args.output_path,
)

Step 2: Run Your Workload#

import time

start_time = time.time()

# Your workload here
for i in range(num_iterations):
    env.step(actions)

elapsed = time.time() - start_time

Step 3: Record Measurements#

from isaaclab.test.benchmark import SingleMeasurement, StringMetadata

benchmark.add_measurement(
    phase_name="runtime",
    measurement=SingleMeasurement(
        name="total_time",
        value=elapsed,
        unit="seconds"
    ),
)

benchmark.add_measurement(
    phase_name="runtime",
    metadata=StringMetadata(name="num_iterations", data=str(num_iterations)),
)

Step 4: Finalize#

benchmark._finalize_impl()

Integration with CI/CD#

The benchmark entry points under scripts/benchmarks/ are designed for CI/CD integration:

# GitHub Actions / GitLab CI example
- name: Run Runtime Benchmark
  run: |
    uv run isaaclab benchmark runtime \
        --task Isaac-Cartpole --num_envs 4096 --num_frames 100 \
        --benchmark_formatter json --output_path ./benchmark_results

- name: Run Training Benchmark
  run: |
    uv run isaaclab benchmark training \
        --rl_library rsl_rl --task Isaac-Cartpole --num_envs 4096 \
        --max_iterations 500 --benchmark_formatter json \
        --output_path ./benchmark_results

- name: Upload Results
  uses: actions/upload-artifact@v3
  with:
    name: benchmark-results
    path: ./benchmark_results/

For Osmo integration, use the osmo formatter:

uv run isaaclab benchmark runtime \
    --task Isaac-Cartpole --num_envs 4096 --num_frames 100 \
    --benchmark_formatter osmo --output_path ./results
# Results are in Osmo-compatible JSON format

Troubleshooting#

Import Errors#

Ensure you’re running through the Isaac Lab launcher:

./isaaclab.sh -p your_benchmark.py

Not:

python your_benchmark.py  # Missing environment setup

Missing GPU Metrics#

Verify nvidia-smi is available and CUDA is configured:

nvidia-smi  # Should show GPU info

Empty Output Files#

Ensure _finalize_impl() is called before the script exits:

try:
    # Your benchmark code
    pass
finally:
    benchmark._finalize_impl()

Formatter Not Recognized#

Valid formatter types are: schema, json, osmo, omniperf, or summary

# Correct
--benchmark_formatter json

# Incorrect
--benchmark_formatter JSON  # Case sensitive