Deformable Pick and Place#

Pick up a deformable cube with the DROID robot and place it on a plate. The same task can run with either the Newton or PhysX deformable-body backend. This walkthrough covers previewing the environment, teleoperating it, and evaluating an OpenPI policy.

Task Overview#

Property

Value

Task ID

droid_deformable_pick_and_place

Embodiment

DROID; absolute joint-position control for OpenPI and differential inverse kinematics for teleoperation

Scene

Maple table with a deformable cube and a plate

Task

Pick up the cube, place it on the plate, and let it settle

Physics

Newton by default; select the backend-specific deformable asset for PhysX. See Newton’s VBD solver tuning guide for deformable simulation parameters.

Prerequisites#

Complete the Arena installation and start your Arena runtime. Run these commands from the repository root in the activated native environment or inside the Base Docker container (/workspaces/isaaclab_arena).

Preview the Environment#

Use a zero-action policy to inspect either backend:

python isaaclab_arena/evaluation/policy_runner.py \
  --policy_type zero_action \
  --num_steps 1000 \
  --num_envs 1 \
  --viz kit \
  droid_deformable_pick_and_place \
  --pick_object deformable_cube_newton
python isaaclab_arena/evaluation/policy_runner.py \
  --policy_type zero_action \
  --num_steps 1000 \
  --num_envs 1 \
  --viz kit \
  droid_deformable_pick_and_place \
  --pick_object deformable_cube_physx

The preview exits after 1,000 steps. Increase --num_steps for a longer preview, or press Ctrl+C in the terminal to stop it.

You should see the DROID robot with a deformable cube and a round plate on the maple table. The placement relations are the same for both physics backends, with the initial poses sampled as shown below.

Four sampled deformable pick-and-place initial placements

Teleoperate the Task#

The Isaac Lab teleoperation script can control the task through Arena’s interop callback. Choose the tab for your input device, then run the command for the simulation backend.

See SpaceMouse Setup (Optional) for the required host permissions and Docker setup.

Newton

python submodules/IsaacLab/scripts/environments/teleoperation/teleop_se3_agent.py \
  --task droid_deformable_pick_and_place \
  --external_callback isaaclab_arena.environments.isaaclab_interop.environment_registration_callback \
  --teleop_device keyboard \
  --viz kit \
  --pick_object deformable_cube_newton \
  --embodiment droid_differential_ik

PhysX

python submodules/IsaacLab/scripts/environments/teleoperation/teleop_se3_agent.py \
  --task droid_deformable_pick_and_place \
  --external_callback isaaclab_arena.environments.isaaclab_interop.environment_registration_callback \
  --teleop_device keyboard \
  --viz kit \
  --pick_object deformable_cube_physx \
  --embodiment droid_differential_ik

Controls

Action

Input

Move along the X axis

W / S

Move along the Y axis

A / D

Move along the Z axis

Q / E

Rotate about the X axis

Z / X

Rotate about the Y axis

T / G

Rotate about the Z axis

C / V

Open or close the gripper

K

Reset the current attempt

R

Newton

python submodules/IsaacLab/scripts/environments/teleoperation/teleop_se3_agent.py \
  --task droid_deformable_pick_and_place \
  --external_callback isaaclab_arena.environments.isaaclab_interop.environment_registration_callback \
  --teleop_device spacemouse \
  --viz kit \
  --pick_object deformable_cube_newton \
  --embodiment droid_differential_ik

PhysX

python submodules/IsaacLab/scripts/environments/teleoperation/teleop_se3_agent.py \
  --task droid_deformable_pick_and_place \
  --external_callback isaaclab_arena.environments.isaaclab_interop.environment_registration_callback \
  --teleop_device spacemouse \
  --viz kit \
  --pick_object deformable_cube_physx \
  --embodiment droid_differential_ik

Controls

Action

Input

Move the gripper

Move the cap laterally or vertically

Change the gripper orientation

Rotate the cap

Open or close the gripper

Press the left button

Reset the current attempt

Press the right button

Note

The cube in this example uses MeshCuboidCfg, which initially authors only a triangle surface mesh. A volume deformable also needs a tetrahedral simulation mesh, so Isaac Lab calls fTetWild during scene creation when it cannot find an existing UsdGeom.TetMesh. This on-the-fly conversion is nondeterministic between process launches and may produce different node counts or connectivity. Demonstration datasets include the initial nodal state, which can be restored only into the same topology. If deterministic topology is required, including for recording and replay, pre-generate a USD containing a UsdGeom.TetMesh and use that asset instead.

Evaluate with OpenPI#

Start the local OpenPI server in a separate terminal:

./isaaclab_arena_openpi/docker/run_openpi_server.sh

Then evaluate four parallel environments for 16 completed episodes:

python isaaclab_arena/evaluation/policy_runner.py \
  --policy_type isaaclab_arena_openpi.policy.pi0_remote_policy.Pi0RemotePolicy \
  --num_episodes 16 \
  --num_envs 4 \
  --viz kit \
  --enable_cameras \
  droid_deformable_pick_and_place \
  --pick_object deformable_cube_newton

A local 16-episode OpenPI pi05 validation run succeeded in 9 episodes (56.25%). This small stochastic run validates the workflow.

[Rank 0/1] Metrics: {'num_episodes': 16, 'success_rate': 0.5625, 'object_moved_rate': 1.0}
python isaaclab_arena/evaluation/policy_runner.py \
  --policy_type isaaclab_arena_openpi.policy.pi0_remote_policy.Pi0RemotePolicy \
  --num_episodes 16 \
  --num_envs 4 \
  --viz kit \
  --enable_cameras \
  droid_deformable_pick_and_place \
  --pick_object deformable_cube_physx

A local 16-episode OpenPI pi05 validation run succeeded in 15 episodes (93.75%). This small stochastic run validates the workflow.

[Rank 0/1] Metrics: {'num_episodes': 16, 'success_rate': 0.9375, 'object_moved_rate': 1.0}

The following rollout shows one successful Newton episode from an external camera and the robot wrist camera:

Successful Newton deformable pick-and-place rollout from external and wrist cameras