From dd4b0110d482c46b9cc9f86d6eab79b76b65c995 Mon Sep 17 00:00:00 2001 From: Pepijn Date: Tue, 30 Jun 2026 15:15:36 +0200 Subject: [PATCH] feat(annotate): lean GEPA-aligned subtask segmentation prompt Replace the verbose, label-heavy segmentation prompt with a lean adaptation of the blog's GEPA-found completed_events_duration_prior recipe: focus on completed manipulation events, explicit no-split / no-merge rules, a 2-10s duration prior, and an instruction to prioritize temporally correct boundaries over label wording. The previous prompt over-weighted label guidance, which traded away boundary precision. Co-authored-by: Cursor --- .../prompts/plan_subtasks.txt | 81 ++++++------------- 1 file changed, 24 insertions(+), 57 deletions(-) diff --git a/src/lerobot/annotations/steerable_pipeline/prompts/plan_subtasks.txt b/src/lerobot/annotations/steerable_pipeline/prompts/plan_subtasks.txt index 8227245a4..6d2b97a5c 100644 --- a/src/lerobot/annotations/steerable_pipeline/prompts/plan_subtasks.txt +++ b/src/lerobot/annotations/steerable_pipeline/prompts/plan_subtasks.txt @@ -1,61 +1,28 @@ -You are annotating a teleoperated robot demonstration shown as -timestamped contact sheets (each tile has its time in seconds burned -into the top-left corner). The operator's goal was: "{episode_task}" +Reconstruct the sequence of manipulation events in this robot +demonstration from the timestamped contact sheets (each tile has its time +in seconds burned into the top-left corner). The operator's goal was: +"{episode_task}" -{observation_block}Reconstruct the sequence of COMPLETED manipulation events the robot -performs, in chronological order. Output one segment per event with a -[start, end] time in seconds and a short action label. - -GROUNDING — read first, it overrides everything below: -- Label ONLY events you can SEE in the frames. The instruction is the - goal; the VIDEO is the ground truth for what actually happened. -- Do NOT invent, anticipate, or pad steps that are not shown. - -Granularity — segment by completed events, not by motion: -- Start a NEW segment whenever the world state changes: an object is - grasped, lifted, transported, placed, or released; a held object - changes; a drawer/door/lid/container opens or closes; contents move - between containers (poured); a tool starts or stops acting on a - surface. Watch the gripper open/close transitions — they usually mark - boundaries. -- Do NOT split approach, reach, grasp adjustment, small repositioning, - hesitation, or retreat into their own segments. Fold each into the - event it belongs to (the approach is part of the pick; the retreat is - part of the place). -- Do NOT merge separate completed events. Each distinct pick, place, - open, close, pour, push, wipe, or insert is its own segment, even when - they repeat on different objects or locations. -- Most segments last 2-10 seconds. Shorter segments are okay ONLY for - fast pick / place / open / close / release events. Never emit a - segment shorter than {min_subtask_seconds} seconds; merge a too-short - candidate into its neighbour instead. -- Skip idle time, pure camera motion, and tiny hand jitter. - -Labels — short imperative phrases: -- One concise command naming the action and the manipulated object, e.g. - "pick up the red cup", "put the cup on the shelf", "open the top - drawer", "pour water into the glass", "insert the plug into the - socket". -- Include source, destination, side, direction, or the final - open/closed state when it is visible and central to the event. -- Prefer these verbs (extend only when none fits): pick up, put, place, - push, pull, turn, press, open, close, pour, insert, wipe, stack. - Disambiguate by what you SEE: - * STACK vs PUT: object placed ON TOP OF another object -> "stack". - * INSERT vs PUT: object pushed INTO a fitted slot/hole/socket -> "insert". - * PICK UP vs PUT (direction): gripper CLOSES and object moves WITH - the hand -> "pick up"; gripper OPENS and object stays -> "put". - * POUR vs PUT: source is tilted and contents flow -> "pour". -- Use the exact object nouns implied by the task; stay consistent across - the episode (don't switch "cube" to "block"). -- Write imperative commands, never third person ("the robot ..."), and - drop articles/adverbs. - -Timing: -- Use the burned-in timestamps to set start and end. Boundaries should - land on or near a printed time, and every [start, end] must lie within - [0.0, {episode_duration}] seconds, be non-overlapping, and cover the - episode in order. +{observation_block}Rules: +- Segment only completed robot manipulation events, not every visible + movement. +- Good boundaries happen when a held object changes, an object is placed + or released, a tool starts or stops changing a surface, a + container/door/lid opens or closes, or contents move between containers. +- Do not split approach, grasp adjustment, small repositioning, or retreat + unless the world state changes. +- Do not merge separate pick / place / open / close / pour / wipe events + when they complete different states. +- Most segments should be 2-10 seconds. Shorter segments are okay only for + fast pick, place, open, close, or release events. Never emit a segment + shorter than {min_subtask_seconds} seconds. +- Use the visible burned-in timestamps for start and end. Every [start, + end] must lie within [0.0, {episode_duration}] seconds, be + non-overlapping, and cover the episode in chronological order. +- Give each segment a short imperative label naming the action and the + manipulated object (add destination, side, direction, or final + open/closed state when visible), but prioritize temporally correct + boundaries over label wording. - Emit at most {max_steps} segments. Output strictly valid JSON of shape: