mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-06 09:37:06 +00:00
annotate: tighter subtask + memory prompts (≤5 / ≤10 words)
Both feed into the high-level prompt and the plan rendering, so
keeping them short directly reduces the rendered ``${task}\nPlan:
…\nMemory: …`` prefix the model has to chew through at inference.
Subtasks
* Hard cap: ≤ 5 words. Verb + object only, drop articles/adverbs.
* Concrete good/bad examples to anchor the VLM.
Memory
* Hard cap: ≤ 10 words. Telegraphic noun→location fragments
("bowl in box, lid open"), no past-tense verbs, drop attributes
that don't matter for downstream subtasks.
* Allow empty string when no material change occurred — keeps the
rendered memory line literally blank instead of forcing a no-op
sentence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,31 +1,35 @@
|
||||
You are updating the robot's compressed semantic memory at the boundary of
|
||||
a completed subtask.
|
||||
|
||||
Reference (verbatim from MEM, Torne 2026):
|
||||
Reference (MEM, Torne 2026):
|
||||
"Remove or compress information in the language memory whenever
|
||||
appropriate. Keep ONLY the minimal set of relevant information for future
|
||||
task execution. Specific object attributes (colors, precise quantities of
|
||||
each item) get discarded when their details won't affect subsequent
|
||||
actions. Functional outcomes (where items went, how many) are preserved."
|
||||
|
||||
Concrete example from MEM:
|
||||
Before: "I put a light green bowl, a dark blue bowl and a bright yellow
|
||||
bowl into the top right cabinet"
|
||||
After: "I placed three bowls in the top right cabinet"
|
||||
|
||||
Episode task: "{episode_task}"
|
||||
Previous memory: {prior_memory}
|
||||
Just-completed subtask: "{completed_subtask}"
|
||||
Remaining subtasks (for relevance judgement only): {remaining_subtasks}
|
||||
|
||||
Update the memory as a compact state note.
|
||||
Write the **shortest possible** state note that future subtasks could
|
||||
need. Telegraphic style.
|
||||
|
||||
Rules:
|
||||
- Keep only facts needed later.
|
||||
- Keep WHAT changed; drop HOW it was done.
|
||||
- Use fragments when clear.
|
||||
- Prefer: "bowl in box; lid still open"
|
||||
- Avoid: "The robot placed the bowl into the box and the lid remains open."
|
||||
**Hard caps**
|
||||
- ≤ 10 words total.
|
||||
- No articles. No verbs in past tense ("placed", "moved"). Use
|
||||
comma-separated noun→location fragments.
|
||||
- Drop colors/sizes/counts unless a later subtask depends on them.
|
||||
- If nothing material changed for downstream subtasks, emit "" (empty
|
||||
string).
|
||||
|
||||
Examples
|
||||
- Good: "bowl in box, lid open"
|
||||
- Good: "3 bowls in cabinet"
|
||||
- Good: "cup on tray, drawer closed"
|
||||
- Bad: "The bowl is now in the box and the lid is still open."
|
||||
- Bad: "I placed the green bowl carefully into the cardboard box."
|
||||
|
||||
Output strictly valid JSON:
|
||||
{{ "memory": "<brief state note>" }}
|
||||
{{ "memory": "<≤10-word telegraphic state, or empty>" }}
|
||||
|
||||
@@ -4,18 +4,17 @@ The user originally asked: "{episode_task}"
|
||||
|
||||
You are shown the entire demonstration as a single video. Watch the
|
||||
whole clip, then segment it into a list of consecutive atomic subtasks
|
||||
the robot performs. Write compact action labels, not prose.
|
||||
the robot performs. Write **ultra-compact** action labels.
|
||||
|
||||
Authoring rules — based on Hi Robot (Shi 2025) atom granularity and
|
||||
pi0.7 (Physical Intelligence 2025) compact context prompts:
|
||||
Authoring rules — Hi Robot atom granularity, pi0.7-style short prompts:
|
||||
|
||||
- Each subtask is one atomic skill the low-level policy can execute,
|
||||
e.g. "pick up one piece of lettuce", "place the bowl into the box",
|
||||
"move the right arm to the left".
|
||||
- Capture HOW when useful, but keep it brief — e.g. prefer
|
||||
"grasp the handle of the sponge with the left hand" to "pick up the
|
||||
sponge".
|
||||
- Use verb phrases, not full sentences.
|
||||
- Each subtask = one atomic skill the low-level policy can execute.
|
||||
- **Hard length cap: ≤ 5 words per subtask.** Drop articles, modifiers,
|
||||
adverbs. Verb + object (+ optional short qualifier) only.
|
||||
- Prefer: "pick lettuce", "place bowl in box", "open drawer",
|
||||
"grasp sponge left hand".
|
||||
- Avoid: "pick up one piece of lettuce", "carefully place the bowl",
|
||||
"the robot moves its arm to the left".
|
||||
- Subtasks are non-overlapping and cover the full episode in order.
|
||||
Choose the cut points yourself based on what you see in the video
|
||||
(gripper open/close events, contact, regrasps, transitions).
|
||||
@@ -28,7 +27,7 @@ Output strictly valid JSON of shape:
|
||||
|
||||
{{
|
||||
"subtasks": [
|
||||
{{"text": "<how-not-what>", "start": <float>, "end": <float>}},
|
||||
{{"text": "<≤5-word verb phrase>", "start": <float>, "end": <float>}},
|
||||
...
|
||||
]
|
||||
}}
|
||||
|
||||
Reference in New Issue
Block a user