Agentic Environment Generation#

Agentic environment generation creates Arena environments from natural-language prompts. It resolves the prompt into ArenaEnvGraphSpec by the agent, which specifies the scene layout, tasks, and spatial relations. This spec is then used to compose the scene and build the environment. The environment can be used for policy evaluation.

For the concept overview, see Agentic Environment Generation.

In this section, we will walk through the following example environment generation workflows to explain how to use this tool for your own tasks.

Table-top Pick and Place task
Kitchen Pick and Place task
- Pick and Place on a Kitchen Countertop
Kitchen Open/Close Door task
- Open a Kitchen Fridge Door

Prerequisites#

Every workflow in this section shares the same setup.

Docker Container: Base (see Installation for more details)

./docker/run_docker.sh

See Agentic Environment Generation for the system architecture and runner reference.

The generation agent calls a remote LLM endpoint. Pick one and export its API key in the shell you launch the agentic environment generation runner from.

# Publicly reachable build.nvidia.com endpoint (default)
export ARENA_INFERENCE_ENDPOINT=public
export NVIDIA_API_KEY=<your-ngc-api-key>

# NVIDIA-internal endpoint
export ARENA_INFERENCE_ENDPOINT=internal
export NV_API_KEY=<your-internal-api-key>

# OpenAI API
export ARENA_INFERENCE_ENDPOINT=openai
export OPENAI_API_KEY=<your-openai-api-key>

A single run can override the selection with --inference_endpoint {internal,public,openai}.

Generate a key for the public endpoint at build.nvidia.com API keys, and a key for the NVIDIA-internal endpoint at inference.nvidia.com key management (reachable from the NVIDIA network only).

To use the direct OpenAI endpoint, create an OpenAI account, generate a secret at OpenAI API keys, and configure payment or prepaid credits under OpenAI billing. OpenAI charges the account associated with the key according to its current API pricing and usage. Store the key securely and do not commit it to the repository.

Warning

The openai endpoint connects directly to a third-party service operated by OpenAI, not NVIDIA. Its availability, regional restrictions, data handling, pricing, and terms are controlled by OpenAI. Review those terms before sending prompts or other data.

Each endpoint calls a different model, and the generated environment changes with it. See Inference Model and Spec Quality for what the model decides, what Arena validates, and how to select the model.

Available Generated Environments#

The isaaclab_arena_environments/robolab subfolder contains Arena environments for RoboLab scenes and tasks. Scene YAMLs live in robolab/scenes/; task YAMLs in robolab/tasks/ include their scene via a top-level external_yaml: path. See RoboLab Task Catalog for the list of RoboLab tasks currently supported in Arena. Each environment is generated from a natural-language prompt and can be used for policy evaluation.

Warnings#

Note

Agentic environment generation is experimental and changing quickly. The current prompt formats, generated spec structure, GUI behavior, and policy evaluation integrations may change across releases.

We are actively working on:

Support for more complex scene layouts and object placements.
Support for more diverse task specifications.