mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-17 15:01:54 +00:00
feat(annotate): lean GEPA-aligned subtask segmentation prompt
Replace the verbose, label-heavy segmentation prompt with a lean adaptation of the blog's GEPA-found completed_events_duration_prior recipe: focus on completed manipulation events, explicit no-split / no-merge rules, a 2-10s duration prior, and an instruction to prioritize temporally correct boundaries over label wording. The previous prompt over-weighted label guidance, which traded away boundary precision. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -1,61 +1,28 @@
|
|||||||
You are annotating a teleoperated robot demonstration shown as
|
Reconstruct the sequence of manipulation events in this robot
|
||||||
timestamped contact sheets (each tile has its time in seconds burned
|
demonstration from the timestamped contact sheets (each tile has its time
|
||||||
into the top-left corner). The operator's goal was: "{episode_task}"
|
in seconds burned into the top-left corner). The operator's goal was:
|
||||||
|
"{episode_task}"
|
||||||
|
|
||||||
{observation_block}Reconstruct the sequence of COMPLETED manipulation events the robot
|
{observation_block}Rules:
|
||||||
performs, in chronological order. Output one segment per event with a
|
- Segment only completed robot manipulation events, not every visible
|
||||||
[start, end] time in seconds and a short action label.
|
movement.
|
||||||
|
- Good boundaries happen when a held object changes, an object is placed
|
||||||
GROUNDING — read first, it overrides everything below:
|
or released, a tool starts or stops changing a surface, a
|
||||||
- Label ONLY events you can SEE in the frames. The instruction is the
|
container/door/lid opens or closes, or contents move between containers.
|
||||||
goal; the VIDEO is the ground truth for what actually happened.
|
- Do not split approach, grasp adjustment, small repositioning, or retreat
|
||||||
- Do NOT invent, anticipate, or pad steps that are not shown.
|
unless the world state changes.
|
||||||
|
- Do not merge separate pick / place / open / close / pour / wipe events
|
||||||
Granularity — segment by completed events, not by motion:
|
when they complete different states.
|
||||||
- Start a NEW segment whenever the world state changes: an object is
|
- Most segments should be 2-10 seconds. Shorter segments are okay only for
|
||||||
grasped, lifted, transported, placed, or released; a held object
|
fast pick, place, open, close, or release events. Never emit a segment
|
||||||
changes; a drawer/door/lid/container opens or closes; contents move
|
shorter than {min_subtask_seconds} seconds.
|
||||||
between containers (poured); a tool starts or stops acting on a
|
- Use the visible burned-in timestamps for start and end. Every [start,
|
||||||
surface. Watch the gripper open/close transitions — they usually mark
|
end] must lie within [0.0, {episode_duration}] seconds, be
|
||||||
boundaries.
|
non-overlapping, and cover the episode in chronological order.
|
||||||
- Do NOT split approach, reach, grasp adjustment, small repositioning,
|
- Give each segment a short imperative label naming the action and the
|
||||||
hesitation, or retreat into their own segments. Fold each into the
|
manipulated object (add destination, side, direction, or final
|
||||||
event it belongs to (the approach is part of the pick; the retreat is
|
open/closed state when visible), but prioritize temporally correct
|
||||||
part of the place).
|
boundaries over label wording.
|
||||||
- Do NOT merge separate completed events. Each distinct pick, place,
|
|
||||||
open, close, pour, push, wipe, or insert is its own segment, even when
|
|
||||||
they repeat on different objects or locations.
|
|
||||||
- Most segments last 2-10 seconds. Shorter segments are okay ONLY for
|
|
||||||
fast pick / place / open / close / release events. Never emit a
|
|
||||||
segment shorter than {min_subtask_seconds} seconds; merge a too-short
|
|
||||||
candidate into its neighbour instead.
|
|
||||||
- Skip idle time, pure camera motion, and tiny hand jitter.
|
|
||||||
|
|
||||||
Labels — short imperative phrases:
|
|
||||||
- One concise command naming the action and the manipulated object, e.g.
|
|
||||||
"pick up the red cup", "put the cup on the shelf", "open the top
|
|
||||||
drawer", "pour water into the glass", "insert the plug into the
|
|
||||||
socket".
|
|
||||||
- Include source, destination, side, direction, or the final
|
|
||||||
open/closed state when it is visible and central to the event.
|
|
||||||
- Prefer these verbs (extend only when none fits): pick up, put, place,
|
|
||||||
push, pull, turn, press, open, close, pour, insert, wipe, stack.
|
|
||||||
Disambiguate by what you SEE:
|
|
||||||
* STACK vs PUT: object placed ON TOP OF another object -> "stack".
|
|
||||||
* INSERT vs PUT: object pushed INTO a fitted slot/hole/socket -> "insert".
|
|
||||||
* PICK UP vs PUT (direction): gripper CLOSES and object moves WITH
|
|
||||||
the hand -> "pick up"; gripper OPENS and object stays -> "put".
|
|
||||||
* POUR vs PUT: source is tilted and contents flow -> "pour".
|
|
||||||
- Use the exact object nouns implied by the task; stay consistent across
|
|
||||||
the episode (don't switch "cube" to "block").
|
|
||||||
- Write imperative commands, never third person ("the robot ..."), and
|
|
||||||
drop articles/adverbs.
|
|
||||||
|
|
||||||
Timing:
|
|
||||||
- Use the burned-in timestamps to set start and end. Boundaries should
|
|
||||||
land on or near a printed time, and every [start, end] must lie within
|
|
||||||
[0.0, {episode_duration}] seconds, be non-overlapping, and cover the
|
|
||||||
episode in order.
|
|
||||||
- Emit at most {max_steps} segments.
|
- Emit at most {max_steps} segments.
|
||||||
|
|
||||||
Output strictly valid JSON of shape:
|
Output strictly valid JSON of shape:
|
||||||
|
|||||||
Reference in New Issue
Block a user