feat(g05): train with LeRobot language recipes

This commit is contained in:
Pepijn
2026-07-29 11:08:03 +02:00
parent 274ee585b4
commit 0675920df9
11 changed files with 678 additions and 14 deletions
+28 -1
View File
@@ -123,9 +123,11 @@ For example, fine-tune the private SO-101 checkpoint on a LeRobot dataset:
export HF_USER=your_hf_username
lerobot-train \
--dataset.repo_id=${HF_USER}/my_so101_dataset \
--dataset.repo_id=${HF_USER}/my_so101_dataset_annotated \
--policy.path=lerobot/g05_so101 \
--policy.device=cuda \
--policy.recipe_path=recipes/g05_bbox_subtask.yaml \
--policy.cot_bbox_camera=observation.images.exterior \
--policy.repo_id=${HF_USER}/g05_so101_finetuned \
--policy.private=true \
--output_dir=outputs/train/g05_so101 \
@@ -135,6 +137,31 @@ lerobot-train \
--save_freq=1000
```
The bundled `g05_bbox_subtask.yaml` recipe resolves the active
`language_persistent` `subtask` and camera-scoped grounded `vqa` event at each
sample timestamp. It filters out unavailable formats before selecting one of
four author-compatible objectives:
| Assistant sequence | Weight |
| -------------------------- | -----: |
| Action only | 1 |
| Subtask, then action | 2 |
| BBox, then action | 1 |
| BBox, subtask, then action | 1 |
Grounded VQA boxes are converted from pixel-space `xyxy` JSON using the source
camera dimensions captured before image resizing, then serialized as G0.5
`<locXXXX>` tokens. Joint samples preserve the released checkpoint's
`BBox → Subtask → Action` order. The user/task conditioning tokens remain
masked; the author backend applies its language/action objective to the
assistant sequence.
Generate the required `subtask` and grounded `vqa` language columns with
`lerobot-annotate` as described in the
[annotation pipeline](./annotation_pipeline). The bundled recipe targets
`observation.images.exterior`; copy the YAML and change its camera-filtered
bindings when training an embodiment with a different grounded camera.
The SO-101 recipe uses AdamW at `8e-5` with 1,000 warmup steps. The packaged
LIBERO and RoboTwin configurations use their released `1e-5` recipe, with
1,000 and 500 warmup steps respectively. All profiles preserve G0.5's six
+25
View File
@@ -146,6 +146,31 @@ The renderer does not apply a tokenizer chat template. Policy processors decide
Blend recipes select one weighted sub-recipe deterministically from the sample index.
`recipes/subtask_mem.yaml` trains the compact core blend — high-level subtask prediction, low-level execution, and memory. `recipes/subtask_mem_vqa_speech.yaml` is the fuller variant that also adds VQA and spoken interjection responses.
Annotation-dependent blends can set `select_from_applicable: true` and declare
`requires` on each component. The renderer first removes components whose
required bindings resolve to `None`, then performs the deterministic weighted
selection. This matches mixed-CoT policies where unavailable annotation formats
must not consume probability:
```yaml
select_from_applicable: true
blend:
subtask:
weight: 2
requires: [subtask]
messages:
- {
role: assistant,
content: "${subtask}",
stream: low_level,
target: true,
}
action:
weight: 1
messages:
- { role: user, content: "${task}", stream: low_level }
```
A message recipe with a supervised assistant turn on the `low_level` stream trains
the π0.5 paper's joint sequence instead of a blend: the target span gets text CE
while also conditioning the action losses in the same forward.