diff --git a/src/lerobot/annotations/steerable_pipeline/prompts/module_1_memory.txt b/src/lerobot/annotations/steerable_pipeline/prompts/module_1_memory.txt index e9b2ee136..f62fa3582 100644 --- a/src/lerobot/annotations/steerable_pipeline/prompts/module_1_memory.txt +++ b/src/lerobot/annotations/steerable_pipeline/prompts/module_1_memory.txt @@ -1,31 +1,35 @@ You are updating the robot's compressed semantic memory at the boundary of a completed subtask. -Reference (verbatim from MEM, Torne 2026): +Reference (MEM, Torne 2026): "Remove or compress information in the language memory whenever appropriate. Keep ONLY the minimal set of relevant information for future task execution. Specific object attributes (colors, precise quantities of each item) get discarded when their details won't affect subsequent actions. Functional outcomes (where items went, how many) are preserved." -Concrete example from MEM: - Before: "I put a light green bowl, a dark blue bowl and a bright yellow - bowl into the top right cabinet" - After: "I placed three bowls in the top right cabinet" - Episode task: "{episode_task}" Previous memory: {prior_memory} Just-completed subtask: "{completed_subtask}" Remaining subtasks (for relevance judgement only): {remaining_subtasks} -Update the memory as a compact state note. +Write the **shortest possible** state note that future subtasks could +need. Telegraphic style. -Rules: -- Keep only facts needed later. -- Keep WHAT changed; drop HOW it was done. -- Use fragments when clear. -- Prefer: "bowl in box; lid still open" -- Avoid: "The robot placed the bowl into the box and the lid remains open." +**Hard caps** +- ≤ 10 words total. +- No articles. No verbs in past tense ("placed", "moved"). Use + comma-separated noun→location fragments. +- Drop colors/sizes/counts unless a later subtask depends on them. +- If nothing material changed for downstream subtasks, emit "" (empty + string). + +Examples +- Good: "bowl in box, lid open" +- Good: "3 bowls in cabinet" +- Good: "cup on tray, drawer closed" +- Bad: "The bowl is now in the box and the lid is still open." +- Bad: "I placed the green bowl carefully into the cardboard box." Output strictly valid JSON: - {{ "memory": "" }} + {{ "memory": "<≤10-word telegraphic state, or empty>" }} diff --git a/src/lerobot/annotations/steerable_pipeline/prompts/module_1_subtasks.txt b/src/lerobot/annotations/steerable_pipeline/prompts/module_1_subtasks.txt index 0229dbc4c..10044fc0b 100644 --- a/src/lerobot/annotations/steerable_pipeline/prompts/module_1_subtasks.txt +++ b/src/lerobot/annotations/steerable_pipeline/prompts/module_1_subtasks.txt @@ -4,18 +4,17 @@ The user originally asked: "{episode_task}" You are shown the entire demonstration as a single video. Watch the whole clip, then segment it into a list of consecutive atomic subtasks -the robot performs. Write compact action labels, not prose. +the robot performs. Write **ultra-compact** action labels. -Authoring rules — based on Hi Robot (Shi 2025) atom granularity and -pi0.7 (Physical Intelligence 2025) compact context prompts: +Authoring rules — Hi Robot atom granularity, pi0.7-style short prompts: -- Each subtask is one atomic skill the low-level policy can execute, - e.g. "pick up one piece of lettuce", "place the bowl into the box", - "move the right arm to the left". -- Capture HOW when useful, but keep it brief — e.g. prefer - "grasp the handle of the sponge with the left hand" to "pick up the - sponge". -- Use verb phrases, not full sentences. +- Each subtask = one atomic skill the low-level policy can execute. +- **Hard length cap: ≤ 5 words per subtask.** Drop articles, modifiers, + adverbs. Verb + object (+ optional short qualifier) only. +- Prefer: "pick lettuce", "place bowl in box", "open drawer", + "grasp sponge left hand". +- Avoid: "pick up one piece of lettuce", "carefully place the bowl", + "the robot moves its arm to the left". - Subtasks are non-overlapping and cover the full episode in order. Choose the cut points yourself based on what you see in the video (gripper open/close events, contact, regrasps, transitions). @@ -28,7 +27,7 @@ Output strictly valid JSON of shape: {{ "subtasks": [ - {{"text": "", "start": , "end": }}, + {{"text": "<≤5-word verb phrase>", "start": , "end": }}, ... ] }}