Arena Experiments#

An Arena Experiment groups one or more named Runs into one evaluation. Each Run selects an existing Environment Definition and combines it with a policy and the settings that control the rollout.

The Experiment Definition is the YAML file that describes those Runs. It is the main interface for evaluation: run the same definition locally with the Experiment Runner, or submit it to OSMO for managed execution.

Note

A Run can reuse either form of Environment Definition:

  • a registered Python environment, selected by name in environment.type; or

  • an existing ArenaEnvGraphSpec YAML file, selected by its path in environment.type.

Different Runs in the same Experiment can use different forms.

The Experiment Definition describes the evaluation. The Environment Definition describes the scene, embodiment, and task. Each Run builds a fresh runtime environment from that definition.

Experiment Definition

The typed YAML file. It holds shared values and one or more named Runs.

Run

One Environment Definition combined with one policy and rollout settings. A Run can contain many parallel simulated environments.

Execution

The Experiment Runner executes locally. OSMO schedules the same Runs on managed compute.

The Experiment Definition#

The YAML has a required runs mapping. A shared mapping is optional. When present, its values are reused by every Run. An optional top-level name string names the Experiment; if it is not given, the name is taken from the YAML file name (without extension). An optional top-level kpi_exporters list names the registered KPI exporters (see the KPI exporter registry) that write the Experiment’s KPI records once every Run has completed. An empty list exports no KPIs. KPIs are aggregated over all Runs of the Experiment. Each key below runs is a Run name:

name: my_experiment
kpi_exporters: [success_rate, timing]
shared:
  environment:
    type: pick_and_place_maple_table
    embodiment: droid_rel_joint_pos
    hdr: home_office_robolab
  policy:
    type: zero_action
  rollout_limit:
    num_steps: 50

runs:
  baseline: {}

  billiard_hall:
    environment:
      hdr: billiard_hall_robolab

baseline inherits all shared values. billiard_hall inherits them too, then replaces only the background. The Run name comes from its key below runs; there is no separate name field. Arena uses this name in command-line overrides, output directories, and reports.

rollout_limit sets when a Run stops. num_rebuilds is a Run field next to it that sets how many times the Run builds a fresh environment. These settings give:

Settings

Result

num_steps: 50

Each rebuild runs 50 environment steps.

num_episodes: 30, num_rebuilds: 3

10 / 10 / 10 episodes. With 31, the first rebuild gets the extra one: 11 / 10 / 10.

num_episodes_per_rebuild: [20, 5, 5], num_rebuilds: 3

20 / 5 / 5 episodes.

num_episodes: 30, num_episodes_per_rebuild: [20, 5, 5], num_rebuilds: 3

20 / 5 / 5 episodes. num_episodes matches the sum of the list.

No limit

The policy’s own length. Only a policy with a fixed action-sequence length supports this.

These combinations are rejected:

  • num_steps with num_episodes or num_episodes_per_rebuild.

  • num_episodes smaller than num_rebuilds.

  • num_episodes_per_rebuild with a length other than num_rebuilds.

  • num_episodes that differs from the sum of num_episodes_per_rebuild.

  • num_episodes_per_rebuild with replay from recorded variation samples.

num_rebuilds sits at the Run level, beside rollout_limit:

num_rebuilds: 3
rollout_limit:
  num_episodes_per_rebuild: [20, 5, 5]   # rebuild 0: 20, rebuild 1: 5, rebuild 2: 5

See Record and Replay Experiments for variation recording, partial and complete replay, and episode scheduling across one or more rebuilds.

Runs keep their YAML order and execute locally in that order.

One definition, two execution paths#

Local — Experiment Runner

Loads the YAML once and executes its Runs in order in one process and one SimulationApp. Every Run builds a fresh environment. The runner stops at the first failure unless --continue_on_error is set.

Managed — OSMO

Turns every Run into an independently scheduled group. Runs can execute at the same time when resources are available. OSMO then collects their outputs into one combined result.

The CLI override root depends on the execution path:

The Experiment Runner receives the Experiment Definition directly.

  • Use shared.<path>=<value> to change a shared value.

  • Use runs.<name>.<path>=<value> to change one Run.

  • Use kpi_exporters=[<name>,<name>,...] to change the KPI exporters.

python isaaclab_arena/evaluation/experiment_runner.py \
  --experiment_config path/to/experiment.yaml \
  shared.rollout_limit.num_steps=100

OSMO nests the Experiment below experiment_cfg, so Experiment overrides need that prefix.

  • Use experiment_cfg.shared.<path>=<value> to change a shared value.

  • Use experiment_cfg.runs.<name>.<path>=<value> to change one Run.

  • Use experiment_cfg.kpi_exporters=[<name>,<name>,...] to change the KPI exporters.

python -m osmo.submit_arena_experiment \
  --experiment_cfg path/to/experiment.yaml \
  --dry_run \
  experiment_cfg.shared.rollout_limit.num_steps=100

If Hydra suggests +shared.<path>=<value>, do not add +. Use experiment_cfg.shared.<path>=<value> instead.

Both commands override the same shared.rollout_limit.num_steps field declared in the Experiment Definition. A value written inside an individual Run still takes priority over the shared override.

For remote policies, configure the policy client inside its Run like any other policy. Start the policy server separately for local execution. OSMO can co-schedule a server for supported policy types.

Follow First Arena Experiment for a complete local example. For OSMO setup and submission options, see Multi-node Evaluation.

Reuse values with shared#

shared removes repetition. Arena applies these values to every Run, then places each Run’s own values on top. A Run only needs to declare what is different.

A Run that declares rollout_limit replaces the shared one as a whole, so a Run can switch from a shared num_steps to num_episodes or num_episodes_per_rebuild without clearing the shared field. A runs.<name>.rollout_limit.* CLI override is applied after this step, so it is still merged with the Run’s limit.

Values are applied in this order, from lowest to highest priority:

typed configuration defaults
              ↓
shared values, including shared.* CLI overrides
              ↓
values written in an individual Run
              ↓
runs.<name>.* CLI overrides
Configuration rules

  • The only top-level fields are name, kpi_exporters, shared, and runs.

  • runs must contain at least one Run.

  • A Run name starts with a letter or underscore. It can then contain letters, numbers, underscores, or hyphens.

  • Each kpi_exporters name must be a registered KPI exporter and appear once.

  • An override can change a field on a Run declared in the YAML, but it cannot add another Run.

  • A shared override must point to a field already present below shared in the Experiment YAML. The Hydra operators +, ++, and ~ are not supported for shared values.

Choosing a runner#

Note

Use the Experiment Runner for policy evaluations defined in YAML. Use the Environment Runner (isaaclab_arena/scripts/environment_runner.py) to inspect and physically manipulate an environment without a policy. The Policy Runner is currently needed only for torchrun multi-GPU execution or policy evaluation of an external environment loaded with --external_environment_class_path.