Run an Evaluation with Recorded Variations#
Recorded variations let an evaluation reproduce the same placement, lighting, object-presence, and camera conditions with a different number of parallel environments.
The workflow has three stages:
record eight settled placement layouts;
run 128 episodes across two rebuilds while recording all enabled variations;
replay the two 64-row recordings in three environments and match each variation payload to its indexed source row.
This workflow uses the clamp_in_right_bin YAML environment as an example.
The generated JSONL files stay below outputs/recorded_variations_workflow
and are not source files.
Record Settled Placement Layouts#
Note
Before starting or restarting all three stages, remove the workflow’s previous generated outputs. This also clears partial recordings and nonempty Experiment directories that would otherwise cause a retry to fail. Copy any results you want to keep elsewhere first:
rm -rf -- outputs/recorded_variations_workflow
From the repository root, record eight layouts that pass the configured post-physics checks:
python isaaclab_arena/scripts/record_placement_layouts.py \
env_spec=isaaclab_arena_environments/robolab/tasks/clamp_in_right_bin.yaml \
output=outputs/recorded_variations_workflow/placements.jsonl \
num_envs=4 env_spacing=2 layouts_per_env=2 \
min_layouts=8 max_batches=5 seed=42 presets=physx \
settle.num_steps=120 \
settle.validators.pose_shift.max_translation_m=0.015
The verified run accepted all eight requested layouts. Each output row records
the complete scene-root poses under the top-level placement key.
Record Variations with Replayed Placement Layouts#
The maintained Experiment Definition configures four parallel environments,
the zero-action policy, two rebuilds, and 128 total episodes. Both rebuilds
load the eight-row placement recording through
environment_builder.recorded_variation_samples_path; the replay scheduler
wraps as needed.
The total budget is split evenly, so each rebuild records 64 episodes:
# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
runs:
recorded_variations:
environment:
type: isaaclab_arena_environments/robolab/tasks/clamp_in_right_bin.yaml
enable_cameras: true
environment_builder:
num_envs: 4
presets: PHYSX
recorded_variation_samples_path: outputs/recorded_variations_workflow/placements.jsonl
policy:
type: zero_action
rollout_limit:
num_episodes: 128
num_rebuilds: 2
variations:
light.hdr_image.enabled: true
light.color.enabled: true
red_hammer.disappear.enabled: true
droid_abs_joint_pos.camera_extrinsics_wrist_camera.enabled: true
It enables two build-time variations:
light.hdr_imagelight.color
It also enables two run-time variations:
red_hammer.disappearfor a distractor that is not part of the task;droid_abs_joint_pos.camera_extrinsics_wrist_camera.
The placement recording supplies the scene-level relation-placement samples. Other enabled variations are sampled live and recorded in each episode result.
Run the Experiment:
python isaaclab_arena/evaluation/experiment_runner.py \
--experiment_config isaaclab_arena_environments/experiment_configs/placement_record_replay/recorded_variations_workflow_experiment.yaml \
--viz none \
--experiment_output_directory outputs/recorded_variations_workflow/record \
'+runs.recorded_variations.environment_builder.env_cfg_override={episode_length_s:1.0}'
The final argument shortens each episode from 60 seconds to 1 second for this run to speed up sampling. The replay command below uses the same value. Omit it for normal evaluations.
The canonical recordings are written to:
outputs/recorded_variations_workflow/record/recorded_variations/episode_results_rebuild0.jsonl
outputs/recorded_variations_workflow/record/recorded_variations/episode_results_rebuild1.jsonl
A complete run writes exactly 64 rows to each file, for 128 episodes total. Zero action does not solve the task; task failure is expected and does not indicate an execution failure.
Run Experiment with Replayed Variations and Placement Layouts#
The replay Experiment Definition lists the two step-2 episode recordings, configures three parallel environments, and runs 128 total episodes. The two-file replay list supplies one file per rebuild, so each rebuild replays all 64 rows:
# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
runs:
recorded_variations_replay:
environment:
type: isaaclab_arena_environments/robolab/tasks/clamp_in_right_bin.yaml
enable_cameras: true
environment_builder:
num_envs: 3
presets: PHYSX
recorded_variation_samples_paths:
- outputs/recorded_variations_workflow/record/recorded_variations/episode_results_rebuild0.jsonl
- outputs/recorded_variations_workflow/record/recorded_variations/episode_results_rebuild1.jsonl
policy:
type: zero_action
rollout_limit:
num_episodes: 128
variations:
light.hdr_image.enabled: true
light.color.enabled: true
red_hammer.disappear.enabled: true
droid_abs_joint_pos.camera_extrinsics_wrist_camera.enabled: true
Run the replay:
python isaaclab_arena/evaluation/experiment_runner.py \
--experiment_config isaaclab_arena_environments/experiment_configs/placement_record_replay/recorded_variations_replay_experiment.yaml \
--viz none \
--experiment_output_directory outputs/recorded_variations_workflow/replay \
'+runs.recorded_variations_replay.environment_builder.env_cfg_override={episode_length_s:1.0}'
Verify the episode count, source-index coverage, and variation payloads:
python - <<'PY'
import json
from pathlib import Path
root = Path("outputs/recorded_variations_workflow")
for rebuild_index in range(2):
filename = f"episode_results_rebuild{rebuild_index}.jsonl"
source_path = root / "record/recorded_variations" / filename
replay_path = root / "replay/recorded_variations_replay" / filename
source = [json.loads(line) for line in source_path.read_text().splitlines()]
replay = [json.loads(line) for line in replay_path.read_text().splitlines()]
assert len(source) == len(replay) == 64
source_indices = [row["replay_source_record_index"] for row in replay]
assert sorted(source_indices) == list(range(len(source)))
for replay_row in replay:
index = replay_row["replay_source_record_index"]
source_row = source[index]
assert replay_row["placement"] == source_row["placement"]
assert replay_row["variations"] == source_row["variations"]
print("Matched all 128 recorded placement and variation rows.")
PY
A complete replay writes exactly 64 rows per rebuild. Every replayed row’s
placement and variations mappings should exactly match the corresponding
indexed source row. The replay_source_record_index values should collectively
cover every source row exactly once; their output order may differ. Episode metadata
such as env_id and timestamp can differ because the replay uses a different
parallelization.
The matching replay payload contains:
the relation-placement scene-root poses under
placement;HDR selection, light color, red-hammer presence, and wrist-camera extrinsics under
variations.
This verifies that recorded variation order is independent of the number of parallel simulation environments.