mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-24 02:06:15 +00:00
annotate: tighter subtask + memory prompts (≤5 / ≤10 words)
Both feed into the high-level prompt and the plan rendering, so
keeping them short directly reduces the rendered ``${task}\nPlan:
…\nMemory: …`` prefix the model has to chew through at inference.
Subtasks
* Hard cap: ≤ 5 words. Verb + object only, drop articles/adverbs.
* Concrete good/bad examples to anchor the VLM.
Memory
* Hard cap: ≤ 10 words. Telegraphic noun→location fragments
("bowl in box, lid open"), no past-tense verbs, drop attributes
that don't matter for downstream subtasks.
* Allow empty string when no material change occurred — keeps the
rendered memory line literally blank instead of forcing a no-op
sentence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,31 +1,35 @@
|
|||||||
You are updating the robot's compressed semantic memory at the boundary of
|
You are updating the robot's compressed semantic memory at the boundary of
|
||||||
a completed subtask.
|
a completed subtask.
|
||||||
|
|
||||||
Reference (verbatim from MEM, Torne 2026):
|
Reference (MEM, Torne 2026):
|
||||||
"Remove or compress information in the language memory whenever
|
"Remove or compress information in the language memory whenever
|
||||||
appropriate. Keep ONLY the minimal set of relevant information for future
|
appropriate. Keep ONLY the minimal set of relevant information for future
|
||||||
task execution. Specific object attributes (colors, precise quantities of
|
task execution. Specific object attributes (colors, precise quantities of
|
||||||
each item) get discarded when their details won't affect subsequent
|
each item) get discarded when their details won't affect subsequent
|
||||||
actions. Functional outcomes (where items went, how many) are preserved."
|
actions. Functional outcomes (where items went, how many) are preserved."
|
||||||
|
|
||||||
Concrete example from MEM:
|
|
||||||
Before: "I put a light green bowl, a dark blue bowl and a bright yellow
|
|
||||||
bowl into the top right cabinet"
|
|
||||||
After: "I placed three bowls in the top right cabinet"
|
|
||||||
|
|
||||||
Episode task: "{episode_task}"
|
Episode task: "{episode_task}"
|
||||||
Previous memory: {prior_memory}
|
Previous memory: {prior_memory}
|
||||||
Just-completed subtask: "{completed_subtask}"
|
Just-completed subtask: "{completed_subtask}"
|
||||||
Remaining subtasks (for relevance judgement only): {remaining_subtasks}
|
Remaining subtasks (for relevance judgement only): {remaining_subtasks}
|
||||||
|
|
||||||
Update the memory as a compact state note.
|
Write the **shortest possible** state note that future subtasks could
|
||||||
|
need. Telegraphic style.
|
||||||
|
|
||||||
Rules:
|
**Hard caps**
|
||||||
- Keep only facts needed later.
|
- ≤ 10 words total.
|
||||||
- Keep WHAT changed; drop HOW it was done.
|
- No articles. No verbs in past tense ("placed", "moved"). Use
|
||||||
- Use fragments when clear.
|
comma-separated noun→location fragments.
|
||||||
- Prefer: "bowl in box; lid still open"
|
- Drop colors/sizes/counts unless a later subtask depends on them.
|
||||||
- Avoid: "The robot placed the bowl into the box and the lid remains open."
|
- If nothing material changed for downstream subtasks, emit "" (empty
|
||||||
|
string).
|
||||||
|
|
||||||
|
Examples
|
||||||
|
- Good: "bowl in box, lid open"
|
||||||
|
- Good: "3 bowls in cabinet"
|
||||||
|
- Good: "cup on tray, drawer closed"
|
||||||
|
- Bad: "The bowl is now in the box and the lid is still open."
|
||||||
|
- Bad: "I placed the green bowl carefully into the cardboard box."
|
||||||
|
|
||||||
Output strictly valid JSON:
|
Output strictly valid JSON:
|
||||||
{{ "memory": "<brief state note>" }}
|
{{ "memory": "<≤10-word telegraphic state, or empty>" }}
|
||||||
|
|||||||
@@ -4,18 +4,17 @@ The user originally asked: "{episode_task}"
|
|||||||
|
|
||||||
You are shown the entire demonstration as a single video. Watch the
|
You are shown the entire demonstration as a single video. Watch the
|
||||||
whole clip, then segment it into a list of consecutive atomic subtasks
|
whole clip, then segment it into a list of consecutive atomic subtasks
|
||||||
the robot performs. Write compact action labels, not prose.
|
the robot performs. Write **ultra-compact** action labels.
|
||||||
|
|
||||||
Authoring rules — based on Hi Robot (Shi 2025) atom granularity and
|
Authoring rules — Hi Robot atom granularity, pi0.7-style short prompts:
|
||||||
pi0.7 (Physical Intelligence 2025) compact context prompts:
|
|
||||||
|
|
||||||
- Each subtask is one atomic skill the low-level policy can execute,
|
- Each subtask = one atomic skill the low-level policy can execute.
|
||||||
e.g. "pick up one piece of lettuce", "place the bowl into the box",
|
- **Hard length cap: ≤ 5 words per subtask.** Drop articles, modifiers,
|
||||||
"move the right arm to the left".
|
adverbs. Verb + object (+ optional short qualifier) only.
|
||||||
- Capture HOW when useful, but keep it brief — e.g. prefer
|
- Prefer: "pick lettuce", "place bowl in box", "open drawer",
|
||||||
"grasp the handle of the sponge with the left hand" to "pick up the
|
"grasp sponge left hand".
|
||||||
sponge".
|
- Avoid: "pick up one piece of lettuce", "carefully place the bowl",
|
||||||
- Use verb phrases, not full sentences.
|
"the robot moves its arm to the left".
|
||||||
- Subtasks are non-overlapping and cover the full episode in order.
|
- Subtasks are non-overlapping and cover the full episode in order.
|
||||||
Choose the cut points yourself based on what you see in the video
|
Choose the cut points yourself based on what you see in the video
|
||||||
(gripper open/close events, contact, regrasps, transitions).
|
(gripper open/close events, contact, regrasps, transitions).
|
||||||
@@ -28,7 +27,7 @@ Output strictly valid JSON of shape:
|
|||||||
|
|
||||||
{{
|
{{
|
||||||
"subtasks": [
|
"subtasks": [
|
||||||
{{"text": "<how-not-what>", "start": <float>, "end": <float>}},
|
{{"text": "<≤5-word verb phrase>", "start": <float>, "end": <float>}},
|
||||||
...
|
...
|
||||||
]
|
]
|
||||||
}}
|
}}
|
||||||
|
|||||||
Reference in New Issue
Block a user