Interacting with a surface gripper

Interacting with a surface gripper#

This tutorial shows how to interact with an articulated robot with a surface gripper attached to its end-effector in the simulation. It is a continuation of the Interacting with an articulation tutorial, where we learned how to interact with an articulated robot. In Isaac Lab 3.0 with Isaac Sim 6.x, surface grippers support only CPU simulation. Run this tutorial with --device cpu.

The Code#

The tutorial corresponds to the run_surface_gripper.py script in the scripts/tutorials/01_assets directory.

Code for run_surface_gripper.py
  1# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
  2# All rights reserved.
  3#
  4# SPDX-License-Identifier: BSD-3-Clause
  5
  6"""This script demonstrates how to spawn a pick-and-place robot equipped with a surface gripper and interact with it.
  7
  8.. code-block:: bash
  9
 10    # Usage
 11    uv run python scripts/tutorials/01_assets/run_surface_gripper.py --device=cpu
 12
 13When running this script make sure the --device flag is set to cpu. This is because the surface gripper is
 14currently only supported on the CPU.
 15"""
 16
 17"""Parse the command-line arguments first."""
 18
 19import argparse
 20
 21from isaaclab.app import add_launcher_args, launch_simulation
 22from isaaclab.utils import clone
 23
 24# add argparse arguments
 25parser = argparse.ArgumentParser(description="Tutorial on spawning and interacting with a Surface Gripper.")
 26# append simulation launcher cli args
 27add_launcher_args(parser)
 28# tutorials should open Kit visualizer by default
 29parser.set_defaults(visualizer=["kit"])
 30# parse the arguments
 31args_cli = parser.parse_args()
 32
 33"""Rest everything follows."""
 34
 35import torch
 36import warp as wp
 37from isaaclab_physx.assets import SurfaceGripper, SurfaceGripperCfg
 38
 39import isaaclab.sim as sim_utils
 40from isaaclab.assets import Articulation
 41from isaaclab.sim import SimulationContext
 42
 43##
 44# Pre-defined configs
 45##
 46from isaaclab_assets import PICK_AND_PLACE_CFG  # isort:skip
 47
 48
 49def design_scene():
 50    """Designs the scene."""
 51    # Ground-plane
 52    cfg = sim_utils.GroundPlaneCfg()
 53    cfg.func("/World/defaultGroundPlane", cfg)
 54    # Lights
 55    cfg = sim_utils.DomeLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75))
 56    cfg.func("/World/Light", cfg)
 57
 58    # Create separate groups called "Origin1", "Origin2"
 59    # Each group will have a robot in it
 60    origins = [[2.75, 0.0, 0.0], [-2.75, 0.0, 0.0]]
 61    # Origin 1
 62    sim_utils.create_prim("/World/Origin1", "Xform", translation=origins[0])
 63    # Origin 2
 64    sim_utils.create_prim("/World/Origin2", "Xform", translation=origins[1])
 65
 66    # Articulation: First we define the robot config
 67    pick_and_place_robot_cfg = clone(PICK_AND_PLACE_CFG)
 68    pick_and_place_robot_cfg.prim_path = "/World/Origin.*/Robot"
 69    pick_and_place_robot = Articulation(cfg=pick_and_place_robot_cfg)
 70
 71    # Surface Gripper: Next we define the surface gripper config
 72    surface_gripper_cfg = SurfaceGripperCfg()
 73    # We need to tell the View which prim to use for the surface gripper
 74    surface_gripper_cfg.prim_path = "/World/Origin.*/Robot/picker_head/SurfaceGripper"
 75    # We can then set different parameters for the surface gripper, note that if these parameters are not set,
 76    # the View will try to read them from the prim.
 77    surface_gripper_cfg.max_grip_distance = 0.1  # [m] (Maximum distance at which the gripper can grasp an object)
 78    surface_gripper_cfg.shear_force_limit = 500.0  # [N] (Force limit in the direction perpendicular direction)
 79    surface_gripper_cfg.coaxial_force_limit = 500.0  # [N] (Force limit in the direction of the gripper's axis)
 80    surface_gripper_cfg.retry_interval = 0.1  # seconds (Time the gripper will stay in a grasping state)
 81    # We can now spawn the surface gripper
 82    surface_gripper = SurfaceGripper(cfg=surface_gripper_cfg)
 83
 84    # return the scene information
 85    scene_entities = {"pick_and_place_robot": pick_and_place_robot, "surface_gripper": surface_gripper}
 86    return scene_entities, origins
 87
 88
 89def run_simulator(
 90    sim: sim_utils.SimulationContext, entities: dict[str, Articulation | SurfaceGripper], origins: torch.Tensor
 91):
 92    """Runs the simulation loop."""
 93    # Extract scene entities
 94    robot: Articulation = entities["pick_and_place_robot"]
 95    surface_gripper: SurfaceGripper = entities["surface_gripper"]
 96
 97    # Define simulation stepping
 98    sim_dt = sim.get_physics_dt()
 99    count = 0
100    # Simulation loop
101    while sim.is_running():
102        # Reset
103        if count % 500 == 0:
104            # reset counter
105            count = 0
106            # reset the scene entities
107            # root state
108            # we offset the root state by the origin since the states are written in simulation world frame
109            # if this is not done, then the robots will be spawned at the (0, 0, 0) of the simulation world
110            root_pose = robot.data.default_root_pose.torch.clone()
111            root_pose[:, :3] += origins
112            robot.write_root_pose_to_sim_index(root_pose=root_pose)
113            root_vel = robot.data.default_root_vel.torch.clone()
114            robot.write_root_velocity_to_sim_index(root_velocity=root_vel)
115            # set joint positions with some noise
116            joint_pos, joint_vel = (
117                robot.data.default_joint_pos.torch.clone(),
118                robot.data.default_joint_vel.torch.clone(),
119            )
120            joint_pos += torch.rand_like(joint_pos) * 0.1
121            robot.write_joint_position_to_sim_index(position=joint_pos)
122            robot.write_joint_velocity_to_sim_index(velocity=joint_vel)
123            # clear internal buffers
124            robot.reset()
125            print("[INFO]: Resetting robot state...")
126            # Opens the gripper and makes sure the gripper is in the open state
127            surface_gripper.reset()
128            print("[INFO]: Resetting gripper state...")
129
130        # Sample a random command between -1 and 1.
131        gripper_commands = torch.rand(surface_gripper.num_instances) * 2.0 - 1.0
132        # The gripper behavior is as follows:
133        # -1 < command < -0.3 --> Gripper is Opening
134        # -0.3 < command < 0.3 --> Gripper is Idle
135        # 0.3 < command < 1 --> Gripper is Closing
136        print(f"[INFO]: Gripper commands: {gripper_commands}")
137        mapped_commands = [
138            "Opening" if command < -0.3 else "Closing" if command > 0.3 else "Idle" for command in gripper_commands
139        ]
140        print(f"[INFO]: Mapped commands: {mapped_commands}")
141        # Set the gripper command
142        surface_gripper.set_grippers_command(gripper_commands)
143        # Write data to sim
144        surface_gripper.write_data_to_sim()
145        # Perform step
146        sim.step()
147        # Increment counter
148        count += 1
149        # Read the gripper state from the simulation
150        surface_gripper.update(sim_dt)
151        # Read the gripper state from the buffer
152        surface_gripper_state = surface_gripper.state
153        # The gripper state is a list of integers that can be mapped to the following:
154        # -1 --> Open
155        # 0 --> Closing
156        # 1 --> Closed
157        # Print the gripper state
158        print(f"[INFO]: Gripper state: {surface_gripper_state}")
159        mapped_commands = [
160            "Open" if state == -1 else "Closing" if state == 0 else "Closed"
161            for state in wp.to_torch(surface_gripper_state).tolist()
162        ]
163        print(f"[INFO]: Mapped commands: {mapped_commands}")
164
165
166def main():
167    """Main function."""
168    # Configure the simulation
169    sim_cfg = sim_utils.SimulationCfg(device=args_cli.device)
170    # Launch the simulator runtime that the configuration needs
171    with launch_simulation(sim_cfg, args_cli):
172        # Initialize the simulation context
173        sim = SimulationContext(sim_cfg)
174        # Set main camera
175        sim.set_camera_view([2.75, 7.5, 10.0], [2.75, 0.0, 0.0])
176        # Design scene
177        scene_entities, scene_origins = design_scene()
178        scene_origins = torch.tensor(scene_origins, device=sim.device)
179        # Play the simulator
180        sim.reset()
181        # Now we are ready!
182        print("[INFO]: Setup complete...")
183        # Run the simulator
184        run_simulator(sim, scene_entities, scene_origins)
185
186
187if __name__ == "__main__":
188    # run the main function
189    main()

The Code Explained#

Designing the scene#

Similarly to the previous tutorial, we populate the scene with a ground plane and a distant light. Then, we spawn an articulation from its USD file. This time a pick-and-place robot is spawned. The pick-and-place robot is a simple robot with 3 driven axes, its gantry allows it to move along the x and y axes, as well as up and down along the z-axis. Furthermore, the robot end-effector is outfitted with a surface gripper. The USD file for the pick-and-place robot contains the robot’s geometry, joints, and other physical properties as well as the surface gripper. Before implementing a similar gripper on your own robot, we recommend to check out the USD file for the gripper found on Isaaclab’s Nucleus.

For the pick-and-place robot, we use its pre-defined configuration object, you can find out more about it in the Robot and articulation configuration tutorial. For the surface gripper, we also need to create a configuration object. This is done by instantiating a assets.SurfaceGripperCfg object and passing it the relevant parameters.

The available parameters are:

  • max_grip_distance: The maximum distance at which the gripper can grasp an object.

  • shear_force_limit: The maximum force the gripper can exert in the direction perpendicular to the gripper’s axis.

  • coaxial_force_limit: The maximum force the gripper can exert in the direction of the gripper’s axis.

  • retry_interval: The time the gripper will stay in a grasping state.

As seen in the previous tutorial, we can spawn the articulation into the scene in a similar fashion by creating an instance of the assets.Articulation class by passing the configuration object to its constructor. The same principle applies to the surface gripper. By passing the configuration object to the assets.SurfaceGripper constructor, the surface gripper is created and can be added to the scene. In practice, the object will only be initialized when the play button is pressed.

    # Create separate groups called "Origin1", "Origin2"
    # Each group will have a robot in it
    origins = [[2.75, 0.0, 0.0], [-2.75, 0.0, 0.0]]
    # Origin 1
    sim_utils.create_prim("/World/Origin1", "Xform", translation=origins[0])
    # Origin 2
    sim_utils.create_prim("/World/Origin2", "Xform", translation=origins[1])

    # Articulation: First we define the robot config
    pick_and_place_robot_cfg = clone(PICK_AND_PLACE_CFG)
    pick_and_place_robot_cfg.prim_path = "/World/Origin.*/Robot"
    pick_and_place_robot = Articulation(cfg=pick_and_place_robot_cfg)

    # Surface Gripper: Next we define the surface gripper config
    surface_gripper_cfg = SurfaceGripperCfg()
    # We need to tell the View which prim to use for the surface gripper
    surface_gripper_cfg.prim_path = "/World/Origin.*/Robot/picker_head/SurfaceGripper"
    # We can then set different parameters for the surface gripper, note that if these parameters are not set,
    # the View will try to read them from the prim.
    surface_gripper_cfg.max_grip_distance = 0.1  # [m] (Maximum distance at which the gripper can grasp an object)
    surface_gripper_cfg.shear_force_limit = 500.0  # [N] (Force limit in the direction perpendicular direction)
    surface_gripper_cfg.coaxial_force_limit = 500.0  # [N] (Force limit in the direction of the gripper's axis)
    surface_gripper_cfg.retry_interval = 0.1  # seconds (Time the gripper will stay in a grasping state)
    # We can now spawn the surface gripper
    surface_gripper = SurfaceGripper(cfg=surface_gripper_cfg)

Running the simulation loop#

Continuing from the previous tutorial, we reset the simulation at regular intervals, set commands to the articulation, step the simulation, and update the articulation’s internal buffers.

Resetting the simulation#

To reset the surface gripper, we only need to call the SurfaceGripper.reset() method which will reset the internal buffers and caches.

            # Opens the gripper and makes sure the gripper is in the open state
            surface_gripper.reset()

Stepping the simulation#

Applying commands to the surface gripper involves two steps:

  1. Setting the desired commands: This sets the desired gripper commands (Open, Close, or Idle).

  2. Writing the data to the simulation: Based on the surface gripper’s configuration, this step handles writes the converted values to the PhysX buffer.

In this tutorial, we use a random command to set the gripper’s command. The gripper behavior is as follows:

  • -1 < command < -0.3 –> Gripper is Opening

  • -0.3 < command < 0.3 –> Gripper is Idle

  • 0.3 < command < 1 –> Gripper is Closing

At every step, we randomly sample commands and set them to the gripper by calling the SurfaceGripper.set_grippers_command() method. After setting the commands, we call the SurfaceGripper.write_data_to_sim() method to write the data to the PhysX buffer. Finally, we step the simulation.

        # Sample a random command between -1 and 1.
        gripper_commands = torch.rand(surface_gripper.num_instances) * 2.0 - 1.0
        # The gripper behavior is as follows:
        # -1 < command < -0.3 --> Gripper is Opening
        # -0.3 < command < 0.3 --> Gripper is Idle
        # 0.3 < command < 1 --> Gripper is Closing
        print(f"[INFO]: Gripper commands: {gripper_commands}")
        mapped_commands = [
            "Opening" if command < -0.3 else "Closing" if command > 0.3 else "Idle" for command in gripper_commands
        ]
        print(f"[INFO]: Mapped commands: {mapped_commands}")
        # Set the gripper command
        surface_gripper.set_grippers_command(gripper_commands)
        # Write data to sim
        surface_gripper.write_data_to_sim()

Updating the state#

To know the current state of the surface gripper, we can query the assets.SurfaceGripper.state() property. This property returns a tensor of size [num_envs] where each element is either -1, 0, or 1 corresponding to the gripper state. This property is updated every time the assets.SurfaceGripper.update() method is called.

  • -1 –> Gripper is Open

  • 0 –> Gripper is Closing

  • 1 –> Gripper is Closed

        # Read the gripper state from the simulation
        surface_gripper.update(sim_dt)
        # Read the gripper state from the buffer
        surface_gripper_state = surface_gripper.state

The Code Execution#

To run the code and see the results, let’s run the script from the terminal:

uv run python scripts/tutorials/01_assets/run_surface_gripper.py --device cpu --viz kit

./isaaclab.sh -p scripts/tutorials/01_assets/run_surface_gripper.py --device cpu --viz kit

This command should open a stage with a ground plane, lights, and two pick-and-place robots. In the terminal, you should see the gripper state and the command being printed. To stop the simulation, you can either close the window, or press Ctrl+C in the terminal.

result of run_surface_gripper.py

In this tutorial, we learned how to create and interact with a surface gripper. We saw how to set commands and query the gripper state. We also saw how to update its buffers to read the latest state from the simulation.

In addition to this tutorial, we also provide a few other scripts that spawn different robots. These are included through the packaged demo command. You can run it as:

# Spawn many pick-and-place robots and perform a pick-and-place task
uv run --extra isaacsim isaaclab demo pick-and-place --viz kit

# Spawn many pick-and-place robots and perform a pick-and-place task
./isaaclab.sh demo pick-and-place --viz kit

Note that in practice, the users would be expected to register their assets.SurfaceGripper instances inside a isaaclab.InteractiveScene object, which will automatically handle the calls to the assets.SurfaceGripper.write_data_to_sim() and assets.SurfaceGripper.update() methods.

# Create a scene
scene = InteractiveScene()

# Register the surface gripper
scene.surface_grippers["gripper"] = surface_gripper