Evaluation#

Which tested environment conditions are associated with a policy’s success? Arena’s evaluation workflows turn controlled evaluation sweeps into an answer. Vary factors such as camera pose, lighting, object mass, or table material, record the outcome of each episode, then estimate a joint posterior that highlights which factor values are associated with success or failure.

This walkthrough uses an OpenPI policy and varies its wrist-camera position. The Run an Evaluation workflow records the episode results and presents them in an HTML report. The Sensitivity Analysis workflow then uses the same results to show which camera offsets were associated with success. For deterministic comparisons across different parallel environment counts, follow Run an Evaluation with Recorded Variations.

Prepare the Arena runtime from the repository root, using either a native uv environment or the base Docker container (see Installation for the full setup):

Native uv

uv sync --extra dev
source .venv/bin/activate
export OMNI_KIT_ACCEPT_EULA=YES ACCEPT_EULA=Y

Docker Container

./docker/run_docker.sh

In another terminal on the host, start the OpenPI server from the repository root:

./isaaclab_arena_openpi/docker/run_openpi_server.sh

Leave the server running while you complete the evaluation workflow. You can stop it before starting Sensitivity Analysis, which reads the saved episode results directly. For installation, model variants, and server options, see OpenPI.

Create the output directory used throughout this walkthrough:

export CAMERA_SENSITIVITY_OUTPUT_DIR="$HOME/eval/camera_sensitivity_workflow"
mkdir -p "${CAMERA_SENSITIVITY_OUTPUT_DIR}"

Arena requires the workflow directory to be empty.