diff --git a/src/lerobot/annotations/steerable_pipeline/prompts/plan_subtasks.txt b/src/lerobot/annotations/steerable_pipeline/prompts/plan_subtasks.txt index e6a5260a7..8227245a4 100644 --- a/src/lerobot/annotations/steerable_pipeline/prompts/plan_subtasks.txt +++ b/src/lerobot/annotations/steerable_pipeline/prompts/plan_subtasks.txt @@ -1,112 +1,68 @@ -You are labeling a teleoperated robot demonstration. +You are annotating a teleoperated robot demonstration shown as +timestamped contact sheets (each tile has its time in seconds burned +into the top-left corner). The operator's goal was: "{episode_task}" -The user originally asked: "{episode_task}" +{observation_block}Reconstruct the sequence of COMPLETED manipulation events the robot +performs, in chronological order. Output one segment per event with a +[start, end] time in seconds and a short action label. -You are shown the entire demonstration as a single video. Watch the -whole clip, then segment it into a list of consecutive atomic subtasks -the robot performs. +GROUNDING — read first, it overrides everything below: +- Label ONLY events you can SEE in the frames. The instruction is the + goal; the VIDEO is the ground truth for what actually happened. +- Do NOT invent, anticipate, or pad steps that are not shown. -{observation_block}GROUNDING — read this first, it overrides everything below: -- Label ONLY what the robot actually does in the video. Every subtask - you emit must correspond to motion you can SEE in specific frames. -- Do NOT invent, anticipate, or pad. If the robot only does one thing - (e.g. it just navigates to a location and the clip ends), emit - EXACTLY ONE subtask. Many demonstrations are a single atomic skill. -- ``max_steps`` below is a hard CEILING, not a target. Emitting fewer - subtasks than the ceiling is not just allowed, it is expected for - short / atomic demonstrations. One correct subtask is far better - than several invented ones. -- If the video does not clearly show the action implied by the task, - describe what you actually see — do NOT fabricate the task's steps - from the instruction text. The instruction tells you the goal; the - VIDEO is the ground truth for what happened. +Granularity — segment by completed events, not by motion: +- Start a NEW segment whenever the world state changes: an object is + grasped, lifted, transported, placed, or released; a held object + changes; a drawer/door/lid/container opens or closes; contents move + between containers (poured); a tool starts or stops acting on a + surface. Watch the gripper open/close transitions — they usually mark + boundaries. +- Do NOT split approach, reach, grasp adjustment, small repositioning, + hesitation, or retreat into their own segments. Fold each into the + event it belongs to (the approach is part of the pick; the retreat is + part of the place). +- Do NOT merge separate completed events. Each distinct pick, place, + open, close, pour, push, wipe, or insert is its own segment, even when + they repeat on different objects or locations. +- Most segments last 2-10 seconds. Shorter segments are okay ONLY for + fast pick / place / open / close / release events. Never emit a + segment shorter than {min_subtask_seconds} seconds; merge a too-short + candidate into its neighbour instead. +- Skip idle time, pure camera motion, and tiny hand jitter. -Authoring rules — Hi Robot atom granularity, pi0.7-style short prompts: +Labels — short imperative phrases: +- One concise command naming the action and the manipulated object, e.g. + "pick up the red cup", "put the cup on the shelf", "open the top + drawer", "pour water into the glass", "insert the plug into the + socket". +- Include source, destination, side, direction, or the final + open/closed state when it is visible and central to the event. +- Prefer these verbs (extend only when none fits): pick up, put, place, + push, pull, turn, press, open, close, pour, insert, wipe, stack. + Disambiguate by what you SEE: + * STACK vs PUT: object placed ON TOP OF another object -> "stack". + * INSERT vs PUT: object pushed INTO a fitted slot/hole/socket -> "insert". + * PICK UP vs PUT (direction): gripper CLOSES and object moves WITH + the hand -> "pick up"; gripper OPENS and object stays -> "put". + * POUR vs PUT: source is tilted and contents flow -> "pour". +- Use the exact object nouns implied by the task; stay consistent across + the episode (don't switch "cube" to "block"). +- Write imperative commands, never third person ("the robot ..."), and + drop articles/adverbs. -- Each subtask = one COMPOSITE atomic skill the low-level policy can - execute end-to-end. A "skill" bundles its own approach motion with - its terminal action — do NOT split the approach off as its own - subtask. The whole-arm policy already learns to reach as part of - every manipulation primitive. -- Write each subtask as an IMPERATIVE COMMAND, starting with one of - these verbs (extend only when none fits): - pick up — approach + grasp + lift in one subtask - put on/in — transport + release in one subtask - place on/in — synonym of "put"; pick one and stay consistent - push — contact + linear shove - pull — contact + linear retract - turn — rotary actuation - press