Run an Evaluation with Recorded Variations#

Recorded variations let an evaluation reproduce the same placement, lighting, object-presence, and camera conditions with a different number of parallel environments.

The workflow has three stages:

  1. record eight settled placement layouts;

  2. run 128 episodes across two rebuilds while recording all enabled variations;

  3. replay the two 64-row recordings in three environments and match each variation payload to its indexed source row.

This workflow uses the clamp_in_right_bin YAML environment as an example. The generated JSONL files stay below outputs/recorded_variations_workflow and are not source files.

Three-stage workflow that records eight live-solved placement rows, replays placement while recording four live-sampled variations into two 64-row JSONL files, then replays all five conditions in three parallel environments and verifies each replay payload against its indexed source row.

Record Settled Placement Layouts#

Note

Before starting or restarting all three stages, remove the workflow’s previous generated outputs. This also clears partial recordings and nonempty Experiment directories that would otherwise cause a retry to fail. Copy any results you want to keep elsewhere first:

rm -rf -- outputs/recorded_variations_workflow

From the repository root, record eight layouts that pass the configured post-physics checks:

python isaaclab_arena/scripts/record_placement_layouts.py \
  env_spec=isaaclab_arena_environments/robolab/tasks/clamp_in_right_bin.yaml \
  output=outputs/recorded_variations_workflow/placements.jsonl \
  num_envs=4 env_spacing=2 layouts_per_env=2 \
  min_layouts=8 max_batches=5 seed=42 presets=physx \
  settle.num_steps=120 \
  settle.validators.pose_shift.max_translation_m=0.015

The verified run accepted all eight requested layouts. Each output row records the complete scene-root poses under the top-level placement key.

Record Variations with Replayed Placement Layouts#

The maintained Experiment Definition configures four parallel environments, the zero-action policy, two rebuilds, and 128 total episodes. Both rebuilds load the eight-row placement recording through environment_builder.recorded_variation_samples_path; the replay scheduler wraps as needed. The total budget is split evenly, so each rebuild records 64 episodes:

# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0

runs:
  recorded_variations:
    environment:
      type: isaaclab_arena_environments/robolab/tasks/clamp_in_right_bin.yaml
      enable_cameras: true
    environment_builder:
      num_envs: 4
      presets: PHYSX
      recorded_variation_samples_path: outputs/recorded_variations_workflow/placements.jsonl
    policy:
      type: zero_action
    rollout_limit:
      num_episodes: 128
    num_rebuilds: 2
    variations:
      light.hdr_image.enabled: true
      light.color.enabled: true
      red_hammer.disappear.enabled: true
      droid_abs_joint_pos.camera_extrinsics_wrist_camera.enabled: true

It enables two build-time variations:

  • light.hdr_image

  • light.color

It also enables two run-time variations:

  • red_hammer.disappear for a distractor that is not part of the task;

  • droid_abs_joint_pos.camera_extrinsics_wrist_camera.

The placement recording supplies the scene-level relation-placement samples. Other enabled variations are sampled live and recorded in each episode result.

Run the Experiment:

python isaaclab_arena/evaluation/experiment_runner.py \
  --experiment_config isaaclab_arena_environments/experiment_configs/placement_record_replay/recorded_variations_workflow_experiment.yaml \
  --viz none \
  --experiment_output_directory outputs/recorded_variations_workflow/record \
  '+runs.recorded_variations.environment_builder.env_cfg_override={episode_length_s:1.0}'

The final argument shortens each episode from 60 seconds to 1 second for this run to speed up sampling. The replay command below uses the same value. Omit it for normal evaluations.

The canonical recordings are written to:

outputs/recorded_variations_workflow/record/recorded_variations/episode_results_rebuild0.jsonl
outputs/recorded_variations_workflow/record/recorded_variations/episode_results_rebuild1.jsonl

A complete run writes exactly 64 rows to each file, for 128 episodes total. Zero action does not solve the task; task failure is expected and does not indicate an execution failure.

Run Experiment with Replayed Variations and Placement Layouts#

The replay Experiment Definition lists the two step-2 episode recordings, configures three parallel environments, and runs 128 total episodes. The two-file replay list supplies one file per rebuild, so each rebuild replays all 64 rows:

# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0

runs:
  recorded_variations_replay:
    environment:
      type: isaaclab_arena_environments/robolab/tasks/clamp_in_right_bin.yaml
      enable_cameras: true
    environment_builder:
      num_envs: 3
      presets: PHYSX
    recorded_variation_samples_paths:
      - outputs/recorded_variations_workflow/record/recorded_variations/episode_results_rebuild0.jsonl
      - outputs/recorded_variations_workflow/record/recorded_variations/episode_results_rebuild1.jsonl
    policy:
      type: zero_action
    rollout_limit:
      num_episodes: 128
    variations:
      light.hdr_image.enabled: true
      light.color.enabled: true
      red_hammer.disappear.enabled: true
      droid_abs_joint_pos.camera_extrinsics_wrist_camera.enabled: true

Run the replay:

python isaaclab_arena/evaluation/experiment_runner.py \
  --experiment_config isaaclab_arena_environments/experiment_configs/placement_record_replay/recorded_variations_replay_experiment.yaml \
  --viz none \
  --experiment_output_directory outputs/recorded_variations_workflow/replay \
  '+runs.recorded_variations_replay.environment_builder.env_cfg_override={episode_length_s:1.0}'

Verify the episode count, source-index coverage, and variation payloads:

python - <<'PY'
import json
from pathlib import Path

root = Path("outputs/recorded_variations_workflow")
for rebuild_index in range(2):
    filename = f"episode_results_rebuild{rebuild_index}.jsonl"
    source_path = root / "record/recorded_variations" / filename
    replay_path = root / "replay/recorded_variations_replay" / filename
    source = [json.loads(line) for line in source_path.read_text().splitlines()]
    replay = [json.loads(line) for line in replay_path.read_text().splitlines()]

    assert len(source) == len(replay) == 64
    source_indices = [row["replay_source_record_index"] for row in replay]
    assert sorted(source_indices) == list(range(len(source)))
    for replay_row in replay:
        index = replay_row["replay_source_record_index"]
        source_row = source[index]
        assert replay_row["placement"] == source_row["placement"]
        assert replay_row["variations"] == source_row["variations"]
print("Matched all 128 recorded placement and variation rows.")
PY

A complete replay writes exactly 64 rows per rebuild. Every replayed row’s placement and variations mappings should exactly match the corresponding indexed source row. The replay_source_record_index values should collectively cover every source row exactly once; their output order may differ. Episode metadata such as env_id and timestamp can differ because the replay uses a different parallelization.

The matching replay payload contains:

  • the relation-placement scene-root poses under placement;

  • HDR selection, light color, red-hammer presence, and wrist-camera extrinsics under variations.

This verifies that recorded variation order is independent of the number of parallel simulation environments.