feat(annotate): lean GEPA-aligned subtask segmentation prompt

Replace the verbose, label-heavy segmentation prompt with a lean
adaptation of the blog's GEPA-found completed_events_duration_prior
recipe: focus on completed manipulation events, explicit no-split /
no-merge rules, a 2-10s duration prior, and an instruction to prioritize
temporally correct boundaries over label wording. The previous prompt
over-weighted label guidance, which traded away boundary precision.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Pepijn
2026-06-30 15:15:36 +02:00
parent 7cfbacdeda
commit dd4b0110d4
@@ -1,61 +1,28 @@
You are annotating a teleoperated robot demonstration shown as
timestamped contact sheets (each tile has its time in seconds burned
into the top-left corner). The operator's goal was: "{episode_task}"
Reconstruct the sequence of manipulation events in this robot
demonstration from the timestamped contact sheets (each tile has its time
in seconds burned into the top-left corner). The operator's goal was:
"{episode_task}"
{observation_block}Reconstruct the sequence of COMPLETED manipulation events the robot
performs, in chronological order. Output one segment per event with a
[start, end] time in seconds and a short action label.
GROUNDING — read first, it overrides everything below:
- Label ONLY events you can SEE in the frames. The instruction is the
goal; the VIDEO is the ground truth for what actually happened.
- Do NOT invent, anticipate, or pad steps that are not shown.
Granularity — segment by completed events, not by motion:
- Start a NEW segment whenever the world state changes: an object is
grasped, lifted, transported, placed, or released; a held object
changes; a drawer/door/lid/container opens or closes; contents move
between containers (poured); a tool starts or stops acting on a
surface. Watch the gripper open/close transitions — they usually mark
boundaries.
- Do NOT split approach, reach, grasp adjustment, small repositioning,
hesitation, or retreat into their own segments. Fold each into the
event it belongs to (the approach is part of the pick; the retreat is
part of the place).
- Do NOT merge separate completed events. Each distinct pick, place,
open, close, pour, push, wipe, or insert is its own segment, even when
they repeat on different objects or locations.
- Most segments last 2-10 seconds. Shorter segments are okay ONLY for
fast pick / place / open / close / release events. Never emit a
segment shorter than {min_subtask_seconds} seconds; merge a too-short
candidate into its neighbour instead.
- Skip idle time, pure camera motion, and tiny hand jitter.
Labels — short imperative phrases:
- One concise command naming the action and the manipulated object, e.g.
"pick up the red cup", "put the cup on the shelf", "open the top
drawer", "pour water into the glass", "insert the plug into the
socket".
- Include source, destination, side, direction, or the final
open/closed state when it is visible and central to the event.
- Prefer these verbs (extend only when none fits): pick up, put, place,
push, pull, turn, press, open, close, pour, insert, wipe, stack.
Disambiguate by what you SEE:
* STACK vs PUT: object placed ON TOP OF another object -> "stack".
* INSERT vs PUT: object pushed INTO a fitted slot/hole/socket -> "insert".
* PICK UP vs PUT (direction): gripper CLOSES and object moves WITH
the hand -> "pick up"; gripper OPENS and object stays -> "put".
* POUR vs PUT: source is tilted and contents flow -> "pour".
- Use the exact object nouns implied by the task; stay consistent across
the episode (don't switch "cube" to "block").
- Write imperative commands, never third person ("the robot ..."), and
drop articles/adverbs.
Timing:
- Use the burned-in timestamps to set start and end. Boundaries should
land on or near a printed time, and every [start, end] must lie within
[0.0, {episode_duration}] seconds, be non-overlapping, and cover the
episode in order.
{observation_block}Rules:
- Segment only completed robot manipulation events, not every visible
movement.
- Good boundaries happen when a held object changes, an object is placed
or released, a tool starts or stops changing a surface, a
container/door/lid opens or closes, or contents move between containers.
- Do not split approach, grasp adjustment, small repositioning, or retreat
unless the world state changes.
- Do not merge separate pick / place / open / close / pour / wipe events
when they complete different states.
- Most segments should be 2-10 seconds. Shorter segments are okay only for
fast pick, place, open, close, or release events. Never emit a segment
shorter than {min_subtask_seconds} seconds.
- Use the visible burned-in timestamps for start and end. Every [start,
end] must lie within [0.0, {episode_duration}] seconds, be
non-overlapping, and cover the episode in chronological order.
- Give each segment a short imperative label naming the action and the
manipulated object (add destination, side, direction, or final
open/closed state when visible), but prioritize temporally correct
boundaries over label wording.
- Emit at most {max_steps} segments.
Output strictly valid JSON of shape: