Inference Model and Spec Quality#

How Arena Uses the Model#

Arena does not reason about the scene. It builds a prompt from the catalogs, asks the model for a JSON object matching the ArenaEnvGraphSpec schema, and validates the result. The model decides what the environment contains. Arena only checks that the answer is admissible: each registry_name is registered, each relation kind exists, and each prim_path is in the background’s prim tree. A bad answer is rejected and saved as invalid_<name>.yaml with trace lines, but Arena cannot fix it. The model must support OpenAI-compatible structured outputs (response_format={"type": "json_schema", ...}).

Why Spec Quality Varies#

Spec quality tracks model capability. Context length is usually the limiting factor: each pass sends the full catalog in one request, and a model that cannot hold it answers from the part it saw. This shows up as an unregistered asset name or an out-of-tree prim_path, not as a length error. Weaker models also:

  • invent asset names

  • drop objects

  • anchor the scene to the background instead of the counter

  • collapse five parallel pick-and-places into one atomic task

  • pick a plausible-but-wrong prim out of several similar ones

Prompt Size at Scale#

Spec inference sends roughly 15 000 characters of catalogs. Prim path resolution sends the background’s entire prim tree — about 30 000 characters (10 000 tokens) for lightwheel_robocasa_kitchen (886 prims) — so a short-context model fails on that pass first. Print the current catalog with --mode catalog.

Selecting the Model#

Each endpoint preset has its own default model, so switching endpoints switches models (see Prerequisites). The CLI runner overrides it per run with --model and --temperature; the GUI runner selects the endpoint in the generation panel and uses that endpoint’s default model. For the public endpoint, any model in the NVIDIA NIM LLM API reference works, as long as it supports strict structured outputs — a larger context window buys more reliable prim path resolution.

ARENA_INFERENCE_ENDPOINT

Accessibility

Default model

API key variable

Pass rate

Mean runtime

public (default)

Public (free)

deepseek-ai/deepseek-v4-pro-0813

NVIDIA_API_KEY

15/15 (100%)

150.66 s

internal

NVIDIA internal

openai/openai/gpt-5.6-terra

NV_API_KEY

15/15 (100%)

12.57 s

openai

Public (charged)

gpt-5.6-terra

OPENAI_API_KEY

15/15 (100%)

10.15 s

Note

The benchmark ran each of five documented prompts three times. Pass rate is the fraction of generated specs that matched the expected structure; runtime is the mean end-to-end generate_spec runtime. These results are snapshots rather than guarantees: model output is non-deterministic, and service load affects runtime.

How Model and Endpoint Affect Generation Metrics#

The model and endpoint both affect generation speed and reliability:

  • Model: affects whether a spec passes validation and how often Arena must retry. Each retry adds another model call and increases runtime.

  • Endpoint: affects response time through network and server performance. The same model can be faster or slower on different endpoints.

For example, Kimi K3 passed all 15 attempts without retries on both endpoints. Its mean runtime was 25.61 seconds on the internal endpoint and 112.15 seconds on the public endpoint. Pass rates and retry counts also varied across models.

Model and endpoint comparison#

Endpoint

Model

Pass rate

Mean runtime

Validation retries

Internal

azure/anthropic/claude-opus-5

15/15

11.71 s

0

Internal

nvidia/zai-org/glm-5.2

15/15

18.37 s

5

Internal

nvidia/moonshotai/kimi-k3

15/15

25.61 s

0

Internal

nvidia/nvidia/nemotron-3-ultra

9/15

24.92 s

9

Public

moonshotai/kimi-k3

15/15

112.15 s

0

Public

deepseek-ai/deepseek-v4-pro-0813

15/15

150.66 s

1

Public

nvidia/nemotron-3-ultra-550b-a55b

9/15

149.86 s

8

Public

openai/gpt-oss-20b

7/15

333.17 s

4

Each row summarizes 15 attempts: five prompts run three times. The retry count is the total across those attempts. Mean runtime covers the full generate_spec call, so it is different from the p50 time-to-first-spec metric on the Performance and scaling page. Results can change with model output and endpoint load.

Reviewing the Generated Spec#

Output is non-deterministic, even at --temperature 0. Validation only proves a spec is admissible, not that it is the environment you asked for. Review every generated spec in the GUI live editor before generating data or evaluating a policy against it:

  • Objects — every object the prompt asked for is there, with no invented extras, and each registry_name is the asset you meant rather than a same-sounding sibling.

  • Anchor and relations — the is_anchor subject is what the scene should be built around (the counter, not the background), and each on / next_to reference points at the intended node. This decides where the robot ends up.

  • Task — the composition (atomic / sequential / parallel) matches the prompt, with one subtask per action and pick_up_object / destination_location not swapped.

  • Object references — the resolved prim_path is the right prim out of the several similar ones a background offers, and an opened articulation carries the correct openable_joint_name.

  • Sim preview — run it. Objects intersecting geometry, or a robot that cannot reach the task objects, show up here and nowhere in validation.

The reviewed YAML can be used to reproduce the environment, but re-running the prompt to create the environment is not.