Step 2: Annotate Demonstrations

Annotation replays the recorded actions and adds data that SoftMimicGen needs: end-effector poses, rope simulation-node positions, and the grasp subtask termination signal. The second subtask, bringing the rope ends together, ends with the trajectory and needs no explicit termination signal.

Use the same physics backend that recorded the source dataset. The AutoData annotation script registers the rope environment internally, so it does not need the external callback used by Isaac Lab’s recording tool.

Automatic Annotation (Recommended)

The rope environment publishes a grasp observation. With --auto, annotation uses its first rising edge as the boundary between grasping and manipulating the rope.

Start the dev container, then run the command for your backend:

./docker/run_docker.sh

NewtonPhysX

python scripts/annotate_demos.py \
    --env_name Isaac-Rope-Franka-Arena-IK-Rel-v0 \
    --presets newton \
    --viz none \
    --device cuda \
    --enable_cameras \
    --auto \
    --no-reset_sim_buffer_each_episode \
    --task_descriptor autodata_examples/tasks/franka_rope_softmimicgen.yaml \
    --embodiment autodata_examples/embodiments/franka_ik_rel.yaml \
    --input_file ./datasets/dataset_franka_rope_newton.hdf5 \
    --output_file ./datasets/dataset_franka_rope_newton_annotated.hdf5

Note

Use --no-reset_sim_buffer_each_episode when annotating Newton demonstrations.

The script exports only episodes that complete the task during replay and have a valid grasp boundary with enough steps left for the task descriptor’s offset range. Skipped episodes are reported in the console. To watch automatic annotation, use --viz kit.

Manual Annotation (Optional)

To mark the boundary yourself, omit --auto and use --viz kit:

NewtonPhysX

python scripts/annotate_demos.py \
    --env_name Isaac-Rope-Franka-Arena-IK-Rel-v0 \
    --presets newton \
    --viz kit \
    --device cuda \
    --enable_cameras \
    --no-reset_sim_buffer_each_episode \
    --task_descriptor autodata_examples/tasks/franka_rope_softmimicgen.yaml \
    --embodiment autodata_examples/embodiments/franka_ik_rel.yaml \
    --input_file ./datasets/dataset_franka_rope_newton.hdf5 \
    --output_file ./datasets/dataset_franka_rope_newton_annotated.hdf5

Note

Use --no-reset_sim_buffer_each_episode when annotating Newton demonstrations.

Mark grasp just after the gripper closes around the rope, before the manipulation motion begins.

Each episode starts paused. Use these controls:

Key	Action
N	Begin / resume playback
B	Pause playback
S	Mark the grasp boundary at the current step
Q	Skip the current episode

Press S once per episode, just after the gripper has grasped the rope and before it starts moving the rope toward the other end. Pause and resume as needed to place the mark. Do not mark the end of the trajectory. Only fully annotated episodes that pass the replay success check are exported.

Expected Output

Check the annotated dataset:

NewtonPhysX

python scripts/validate_dataset.py ./datasets/dataset_franka_rope_newton_annotated.hdf5

Continue to Step 3: Generate the Dataset.