Agentic Environment Generation#

Agentic environment generation creates Arena environments from natural-language prompts. It resolves the prompt into ArenaEnvGraphSpec by the agent, which specifies the scene layout, tasks, and spatial relations. This spec is then used to compose the scene and build the environment. The environment can be used for policy evaluation.

For the concept overview, see Agentic Environment Generation.

In this section, we will walk through the following example environment generation workflows to explain how to use this tool for your own tasks.

Table-top Pick and Place task
Kitchen Pick and Place task
- Pick and Place on a Kitchen Countertop
Kitchen Open/Close Door task
- Open a Kitchen Fridge Door

Prerequisites#

Every workflow in this section shares the same setup.

Inference API key setup#

The generation agent calls a remote LLM endpoint. Export one or more API keys in the host environment before launching a native uv runner or starting the Docker container. Docker forwards the configured keys when it creates the container. This step is required only once per host environment.

NVIDIA Public Endpoint

Generate an NGC API key at build.nvidia.com API keys, then export it for the publicly reachable endpoint:

export NVIDIA_API_KEY=<your-ngc-api-key>

NVIDIA Internal Endpoint

From the NVIDIA network, generate an internal API key at inference.nvidia.com key management, then export it:

export NV_API_KEY=<your-internal-api-key>

Note

This endpoint is accessible by NVIDIA employees only and counts into your Inference Hub token usage.

OpenAI Endpoint

Create an OpenAI account, generate a secret at OpenAI API keys, and configure payment or prepaid credits under OpenAI billing. OpenAI charges the account associated with the key according to its current API pricing and usage. Store the key securely and do not commit it to the repository.

export OPENAI_API_KEY=<your-openai-api-key>

Note

The openai endpoint connects directly to a third-party service operated by OpenAI, not NVIDIA. Its availability, regional restrictions, data handling, pricing, and terms are controlled by OpenAI. Review those terms before sending prompts or other data.

Set ARENA_INFERENCE_ENDPOINT to choose the default endpoint:

export ARENA_INFERENCE_ENDPOINT=public  # internal, public, or openai

The CLI runner can override the selection per run with --inference_endpoint {internal,public,openai}. The GUI runner selects among the endpoints whose API keys are available in the generation panel.

Each endpoint calls a different model, and the generated environment changes with it. See Inference Model and Spec Quality for what the model decides, what Arena validates, and how to select the model.

Start uv or Docker#

Use either a native uv environment or the base Docker container (see Installation for more details).

Native uv source

uv sync --extra dev
source .venv/bin/activate
export OMNI_KIT_ACCEPT_EULA=YES ACCEPT_EULA=Y

Native uv wheel

uv sync --no-default-groups --group isaaclab-from-wheel --extra dev
source .venv/bin/activate
export OMNI_KIT_ACCEPT_EULA=YES ACCEPT_EULA=Y

Docker Container

./docker/run_docker.sh

For either native uv flavor, isaaclab_arena_curobo is not installed; use the Docker container with -c if you need cuRobo-based reachability validation.

Available Generated Environments#

The generated environment catalogs cover tabletop and room-scale manipulation and can be used directly for policy evaluation:

RoboLab-style tabletop manipulation — diverse tabletop scenes, objects, and manipulation tasks. See the RoboLab Task Catalog.
Room-scale kitchen benchmark — object manipulation and articulated appliance tasks across room-scale kitchen scenes. See the Kitchen Benchmark Catalog.

Warning

Agentic environment generation is experimental and changing quickly. The current prompt formats, generated spec structure, GUI behavior, and policy evaluation integrations may change across releases.

We are actively working on:

Support for more complex scene layouts and object placements.
Support for more diverse task specifications.