annotate: tighter subtask + memory prompts (≤5 / ≤10 words)

Both feed into the high-level prompt and the plan rendering, so
keeping them short directly reduces the rendered ``${task}\nPlan:
…\nMemory: …`` prefix the model has to chew through at inference.

Subtasks
* Hard cap: ≤ 5 words. Verb + object only, drop articles/adverbs.
* Concrete good/bad examples to anchor the VLM.

Memory
* Hard cap: ≤ 10 words. Telegraphic noun→location fragments
  ("bowl in box, lid open"), no past-tense verbs, drop attributes
  that don't matter for downstream subtasks.
* Allow empty string when no material change occurred — keeps the
  rendered memory line literally blank instead of forcing a no-op
  sentence.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pepijn
2026-05-13 16:28:09 +02:00
parent a8ca5128b8
commit 99ebee4d16
2 changed files with 28 additions and 25 deletions
@@ -1,31 +1,35 @@
You are updating the robot's compressed semantic memory at the boundary of You are updating the robot's compressed semantic memory at the boundary of
a completed subtask. a completed subtask.
Reference (verbatim from MEM, Torne 2026): Reference (MEM, Torne 2026):
"Remove or compress information in the language memory whenever "Remove or compress information in the language memory whenever
appropriate. Keep ONLY the minimal set of relevant information for future appropriate. Keep ONLY the minimal set of relevant information for future
task execution. Specific object attributes (colors, precise quantities of task execution. Specific object attributes (colors, precise quantities of
each item) get discarded when their details won't affect subsequent each item) get discarded when their details won't affect subsequent
actions. Functional outcomes (where items went, how many) are preserved." actions. Functional outcomes (where items went, how many) are preserved."
Concrete example from MEM:
Before: "I put a light green bowl, a dark blue bowl and a bright yellow
bowl into the top right cabinet"
After: "I placed three bowls in the top right cabinet"
Episode task: "{episode_task}" Episode task: "{episode_task}"
Previous memory: {prior_memory} Previous memory: {prior_memory}
Just-completed subtask: "{completed_subtask}" Just-completed subtask: "{completed_subtask}"
Remaining subtasks (for relevance judgement only): {remaining_subtasks} Remaining subtasks (for relevance judgement only): {remaining_subtasks}
Update the memory as a compact state note. Write the **shortest possible** state note that future subtasks could
need. Telegraphic style.
Rules: **Hard caps**
- Keep only facts needed later. - ≤ 10 words total.
- Keep WHAT changed; drop HOW it was done. - No articles. No verbs in past tense ("placed", "moved"). Use
- Use fragments when clear. comma-separated noun→location fragments.
- Prefer: "bowl in box; lid still open" - Drop colors/sizes/counts unless a later subtask depends on them.
- Avoid: "The robot placed the bowl into the box and the lid remains open." - If nothing material changed for downstream subtasks, emit "" (empty
string).
Examples
- Good: "bowl in box, lid open"
- Good: "3 bowls in cabinet"
- Good: "cup on tray, drawer closed"
- Bad: "The bowl is now in the box and the lid is still open."
- Bad: "I placed the green bowl carefully into the cardboard box."
Output strictly valid JSON: Output strictly valid JSON:
{{ "memory": "<brief state note>" }} {{ "memory": "<≤10-word telegraphic state, or empty>" }}
@@ -4,18 +4,17 @@ The user originally asked: "{episode_task}"
You are shown the entire demonstration as a single video. Watch the You are shown the entire demonstration as a single video. Watch the
whole clip, then segment it into a list of consecutive atomic subtasks whole clip, then segment it into a list of consecutive atomic subtasks
the robot performs. Write compact action labels, not prose. the robot performs. Write **ultra-compact** action labels.
Authoring rules — based on Hi Robot (Shi 2025) atom granularity and Authoring rules — Hi Robot atom granularity, pi0.7-style short prompts:
pi0.7 (Physical Intelligence 2025) compact context prompts:
- Each subtask is one atomic skill the low-level policy can execute, - Each subtask = one atomic skill the low-level policy can execute.
e.g. "pick up one piece of lettuce", "place the bowl into the box", - **Hard length cap: ≤ 5 words per subtask.** Drop articles, modifiers,
"move the right arm to the left". adverbs. Verb + object (+ optional short qualifier) only.
- Capture HOW when useful, but keep it brief — e.g. prefer - Prefer: "pick lettuce", "place bowl in box", "open drawer",
"grasp the handle of the sponge with the left hand" to "pick up the "grasp sponge left hand".
sponge". - Avoid: "pick up one piece of lettuce", "carefully place the bowl",
- Use verb phrases, not full sentences. "the robot moves its arm to the left".
- Subtasks are non-overlapping and cover the full episode in order. - Subtasks are non-overlapping and cover the full episode in order.
Choose the cut points yourself based on what you see in the video Choose the cut points yourself based on what you see in the video
(gripper open/close events, contact, regrasps, transitions). (gripper open/close events, contact, regrasps, transitions).
@@ -28,7 +27,7 @@ Output strictly valid JSON of shape:
{{ {{
"subtasks": [ "subtasks": [
{{"text": "<how-not-what>", "start": <float>, "end": <float>}}, {{"text": "<≤5-word verb phrase>", "start": <float>, "end": <float>}},
... ...
] ]
}} }}