refactor(g05): use recipe message dropout

This commit is contained in:
Pepijn
2026-07-29 15:18:48 +02:00
parent e6ec909cc7
commit 6d2c3bb1e2
7 changed files with 166 additions and 208 deletions
+4 -9
View File
@@ -156,15 +156,10 @@ lerobot-train \
The bundled `g05_bbox_subtask.yaml` recipe resolves the active
`language_persistent` `subtask` and camera-scoped grounded `vqa` event at each
sample timestamp. It filters out unavailable formats before selecting one of
four author-compatible objectives:
| Assistant sequence | Weight |
| -------------------------- | -----: |
| Action only | 1 |
| Subtask, then action | 2 |
| BBox, then action | 1 |
| BBox, subtask, then action | 1 |
sample timestamp. It applies independent `0.5` dropout to the optional BBox and
Subtask targets, producing Action-only, Subtask+Action, BBox+Action, and joint
BBox+Subtask+Action samples. An unavailable annotation is skipped before
dropout through the recipe's `if_present` guard.
Grounded VQA boxes are converted from pixel-space `xyxy` JSON using the source
camera dimensions captured before image resizing, then serialized as G0.5
+14 -21
View File
@@ -146,29 +146,22 @@ The renderer does not apply a tokenizer chat template. Policy processors decide
Blend recipes select one weighted sub-recipe deterministically from the sample index.
`recipes/subtask_mem.yaml` trains the compact core blend — high-level subtask prediction, low-level execution, and memory. `recipes/subtask_mem_vqa_speech.yaml` is the fuller variant that also adds VQA and spoken interjection responses.
Annotation-dependent blends can set `select_from_applicable: true` and declare
`requires` on each component. The renderer first removes components whose
required bindings resolve to `None`, then performs the deterministic weighted
selection. This matches mixed-CoT policies where unavailable annotation formats
must not consume probability:
Message turns can set `dropout` to a probability between `0` and `1`. Dropout is
deterministic for a given sample index and independent between turns. Combine it
with `if_present` to sample optional supervision formats while gracefully
skipping annotations that are unavailable:
```yaml
select_from_applicable: true
blend:
subtask:
weight: 2
requires: [subtask]
messages:
- {
role: assistant,
content: "${subtask}",
stream: low_level,
target: true,
}
action:
weight: 1
messages:
- { role: user, content: "${task}", stream: low_level }
messages:
- { role: user, content: "${task}", stream: low_level }
- {
role: assistant,
content: "${subtask}",
stream: low_level,
target: true,
if_present: subtask,
dropout: 0.5,
}
```
A message recipe with a supervised assistant turn on the `low_level` stream trains