GUI Runner

The agentic environment-generation GUI is a Streamlit live editor for creating, reviewing, editing, saving, visualizing, and simulation-previewing ArenaEnvGraphSpec YAML files.

What comes back depends on the model behind the selected endpoint — see Inference Model and Spec Quality. Pick the endpoint in the GUI generation panel (only endpoints whose API key is set are listed), or set ARENA_INFERENCE_ENDPOINT as the default. Since the model is non-deterministic, review and correct what it returns.

Run the GUI from inside the Isaac Lab-Arena development container:

python isaaclab_arena_examples/agentic_environment_generation/gui_runner.py

You can also open an existing environment graph spec:

python isaaclab_arena_examples/agentic_environment_generation/gui_runner.py \
   --env_spec isaaclab_arena/tests/test_data/pick_and_place_maple_table_env_graph.yaml

By default, generated YAML files are written under isaaclab_arena_environments/agent_generated. Use --out_dir to choose a different output directory, or --port to run Streamlit on a different port.

Run the full pipeline from a natural-language prompt to generated YAML, automatically updated asset snapshots and graph visualization, and a simulation preview.

UI Panels

The page is split into a left editing column and a right preview column.

Generate from prompt

Choose an inference endpoint (only endpoints whose API key is set are listed), enter a natural-language task and scene description, then click Generate spec. The GUI calls the environment-generation agent and loads the returned ArenaEnvGraphSpec YAML into the editor. When validation fails, the invalid YAML is still loaded into the editor and the validation traces are shown alongside it.

YAML editor

Edit the generated or loaded ArenaEnvGraphSpec directly. The editor validates the YAML as you work and shows either a valid-spec summary or the parse/validation error. The Save YAML button writes the spec to <env_name>.yaml in the configured output directory. A searchable background prim-tree panel helps identify prim paths while editing.

Visualization

Shows an automatically refreshed dashboard for valid YAML. The dashboard includes graph nodes, node thumbnails when available, the graph layout, task rows, and initial-state information. Snapshot axis overlays use red for \(+X\), green for \(+Y\), and blue for \(+Z\). If the YAML is invalid, the panel waits until the error is fixed before rendering.

When the spec contains an entry under object_references, expand Background prim tree to search the background USD for the referenced prim. The searchable, collapsible tree helps verify or correct the reference’s prim_path.

Sim preview

Runs the full Arena environment construction from YAML, relation solving, and zero-action rollout in a SimulationApp side process. Controls let you set the number of parallel environments, zero-action steps, and environment spacing.

Note

The preview uses the task’s default viewer configuration and records the Kit viewport camera. Increase Zero-action steps (num_steps) to extend the rollout, and move the viewport camera in the Kit window to change the recorded view.

Editing and Update Flow

The main update flow is:

1. Type a prompt and click Generate spec.

2. The agent receives the prompt and returns an ArenaEnvGraphSpec. The generated YAML is loaded into the editor and saved as <env_name>.yaml.

3. The user can manually edit the YAML in the editor. Once the edited YAML passes validation, click Save YAML to write it to the output directory. Use Change output directory to choose a different output location. The filename is derived from env_name and can be changed by editing env_name in the YAML editor.

4. The graph visualization refreshes automatically when the valid YAML text changes.

5. Click Run relation solver preview to manually trigger the simulation preview. This action sends the current editor text to the SimApp preview service, builds the Arena environment, solves relations, runs the configured zero-action rollout, and displays the recorded viewport video.

Edit the YAML with live validation, save it to a file, and review the asset snapshots and graph visualization as they update automatically.