mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-23 09:46:00 +00:00
Add joint-sequence subtask training and collision-free FAST vocab mapping
- recipes/subtask_joint.yaml: paper-style single sequence (pi0.5 §IV-B) — the supervised subtask span gets text CE and conditions the FAST and flow losses in the same forward. - joint_subtask_conditioning config flag rebuilds the same layout at inference: state on the task turn, generated subtask as a causal assistant turn (encode_prompt_with_targets + lang_causal_marks through sample_actions), in both the policy select_action path and the runtime adapter. - fast_skip_tokens default 128 -> 1152 so FAST codes land below the <loc> range and never collide with VQA loc targets; _FAST_ACTION_VOCAB_SIZE tightened to the universal tokenizer's 1024 codes. - Strip the trailing space from the 'Assistant:' generation prefill — SentencePiece folds the space into the first target token, so the space-suffixed prefill ended in a lone '▁' never seen in training. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -146,6 +146,12 @@ The renderer does not apply a tokenizer chat template. Policy processors decide
|
||||
Blend recipes select one weighted sub-recipe deterministically from the sample index.
|
||||
`recipes/subtask_mem.yaml` trains the compact core blend — high-level subtask prediction, low-level execution, and memory. `recipes/subtask_mem_vqa_speech.yaml` is the fuller variant that also adds VQA and spoken interjection responses.
|
||||
|
||||
A message recipe with a supervised assistant turn on the `low_level` stream trains
|
||||
the π0.5 paper's joint sequence instead of a blend: the target span gets text CE
|
||||
while also conditioning the action losses in the same forward.
|
||||
`recipes/subtask_joint.yaml` is the provided example; pair it with
|
||||
`--policy.joint_subtask_conditioning=true` at inference.
|
||||
|
||||
## Graceful absence
|
||||
|
||||
If both language columns are missing, `None`, or empty, `RenderMessagesStep` is a no-op.
|
||||
|
||||
Reference in New Issue
Block a user