mirror of
https://github.com/huggingface/lerobot.git
synced 2026-05-15 16:49:55 +00:00
Pepijn
76 lines
2.8 KiB
Plaintext
76 lines
2.8 KiB
Plaintext
# Language columns and recipes
|
|
|
|
LeRobot stores reusable language annotations directly next to frame data in `data/chunk-*/file-*.parquet`.
|
|
The two optional columns are:
|
|
|
|
- `language_persistent`: a list of rows broadcast across every frame in an episode for state that remains active, such as `subtask`, `plan`, and `memory`.
|
|
- `language_events`: a list of rows only on the exact frame where an event was emitted, such as `interjection`, `vqa`, and speech tool calls.
|
|
|
|
Both columns share the same row shape:
|
|
|
|
```text
|
|
role: string
|
|
content: string | null
|
|
style: string | null
|
|
timestamp: float64
|
|
tool_calls: list[Json] | null
|
|
```
|
|
|
|
`meta/tasks.parquet` remains the canonical source for the task. The special `${task}` recipe binding always reads that task string and does not depend on language annotations.
|
|
|
|
## Architecture
|
|
|
|
The language stack has three layers:
|
|
|
|
1. `lerobot.datasets.language` defines the schema, style registry, and `column_for_style`.
|
|
2. `lerobot.datasets.language_render` resolves rows and renders messages.
|
|
3. `RenderMessagesStep` turns dataset samples into `messages`, `message_streams`, and `target_message_indices`.
|
|
|
|
`LeRobotDataset` stays recipe-agnostic. It passes `language_persistent` and `language_events` through when present, and unannotated datasets keep their existing behavior.
|
|
|
|
## Temporal semantics
|
|
|
|
Persistent styles are active after emission until replaced:
|
|
|
|
- `active_at(t, style=subtask)`
|
|
- `nth_prev(style=memory, offset=1)`
|
|
- `nth_next(style=subtask, offset=1)`
|
|
|
|
Event styles only exist on their exact timestamp:
|
|
|
|
- `emitted_at(t, style=interjection)`
|
|
- `emitted_at(t, style=vqa, role=user)`
|
|
- `emitted_at(t, role=assistant, tool_name=say)`
|
|
|
|
Exact event matching has no tolerance window, so writers must stamp event rows with frame timestamps from the parquet data.
|
|
|
|
## Recipe anatomy
|
|
|
|
Recipes are YAML files backed by `TrainingRecipe` and `MessageTurn`.
|
|
|
|
```yaml
|
|
messages:
|
|
- { role: user, content: "${task}", stream: high_level }
|
|
- { role: assistant, content: "${subtask}", stream: low_level, target: true }
|
|
```
|
|
|
|
Rendered samples use HF-style chat messages plus LeRobot sidecars:
|
|
|
|
```python
|
|
sample["messages"]
|
|
sample["message_streams"]
|
|
sample["target_message_indices"]
|
|
```
|
|
|
|
The renderer does not apply a tokenizer chat template. Policy processors decide how to serialize the messages for their backbone.
|
|
|
|
## Blends
|
|
|
|
Blend recipes select one weighted sub-recipe deterministically from the sample index.
|
|
The canonical `recipes/pi05_hirobot.yaml` combines memory updates, interjection responses, high-level subtask prediction, low-level execution, and VQA.
|
|
|
|
## Graceful absence
|
|
|
|
If both language columns are missing, `None`, or empty, `RenderMessagesStep` is a no-op.
|
|
If an event-scoped branch is selected on a frame without the required event row, rendering returns `None`, allowing a loader to retry another sample.
|