admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-24 10:16:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(annotate): friendlier rewrite + architecture diagram; drop reproducibility section

Browse Source 
Rewrite annotation_pipeline.mdx in plainer, easier-to-read language
(shorter sentences, active voice, a plain-text intro), add an ASCII
'How it fits together' architecture diagram, and remove the
'Reproducibility via seed and prompt hashes' section. Content/links are
preserved; only wording and structure change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pepijn
2026-06-04 11:48:59 +02:00
parent c6f682b3f4
commit 7bec991cdf
1 changed files with 148 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/annotation_pipeline.mdx
+147 -118
View File
@@ -1,177 +1,206 @@
# Annotation Pipeline # Annotation Pipeline
`lerobot-annotate` populates the two language columns introduced by the `lerobot-annotate` watches each episode's video with a vision-language
model (VLM) and writes natural-language annotations back into your
dataset. It fills the two language columns from the
[Language Columns and Recipes](./language_and_recipes) page — [Language Columns and Recipes](./language_and_recipes) page —
`language_persistent` and `language_events` — directly into `language_persistent` and `language_events` — straight into
`data/chunk-*/file-*.parquet`. `data/chunk-*/file-*.parquet`.
In short: point it at a LeRobot dataset, and it adds subtasks, plans,
memory, interjections, speech, and visual Q&A that a policy can be
trained on.
## How it fits together
```text
your dataset lerobot-annotate
(LeRobot v3.1) ┌──────────────────────────────────┐
│ │ read episodes │
└─────────────▶│ │ │
│ ▼ │
one shared │ ┌──────┐ ┌─────────────┐ ┌─────┐ │ each module writes
Qwen-VL server ────▶│ │ plan │ │interjections│ │ vqa │ │ raw JSONL into
(vLLM, OpenAI API) │ └──┬───┘ └──────┬──────┘ └──┬──┘ │ .annotate_staging/
│ └────────────┼───────────┘ │
│ ▼ │
│ validator │ checks everything
│ │ │
│ ▼ │
│ writer ──────────────┼─▶ data/chunk-*/file-*.parquet
└──────────────────────────────────┘ (+ meta/info.json tools)
```
Three modules (`plan`, `interjections`, `vqa`) all talk to **one** shared
VLM. Each module stages its output to disk, a validator checks it, and a
single writer rewrites the dataset shards in place.
## What the pipeline produces ## What the pipeline produces
Three modules write into a per-episode staging tree, then a single writer Each module emits a few kinds of annotation ("styles"), routed to one of
rewrites the data shards in place: the two language columns:
| Style / atom | Column | Module | | Style / atom | Column | Module |
| ------------------------------------------- | --------------------- | --------------- | | ------------------------------------------- | --------------------- | --------------- |
| `subtask` (Pi0.7-style "how, not what") | `language_persistent` | `plan` | | `subtask` (Pi0.7-style "how, not what") | `language_persistent` | `plan` |
| `plan` (initial + refresh on interjection) | `language_persistent` | `plan` | | `plan` (initial + refresh on interjection) | `language_persistent` | `plan` |
| `memory` (MEM-style compression) | `language_persistent` | `plan` | | `memory` (MEM-style compression) | `language_persistent` | `plan` |
| `task_aug` (rephrasings of canonical task) | `language_persistent` | `plan` | | `task_aug` (rephrasings of the task) | `language_persistent` | `plan` |
| `interjection` | `language_events` | `interjections` | | `interjection` | `language_events` | `interjections` |
| speech tool-call atom (`style=null`, `say`) | `language_events` | `interjections` | | speech tool-call atom (`style=null`, `say`) | `language_events` | `interjections` |
| `vqa` (user / assistant pair) | `language_events` | `vqa` | | `vqa` (user / assistant pair) | `language_events` | `vqa` |
The `plan` module generates subtasks per episode with a **describe → segment** ### How subtasks are generated
grounding flow: a first pass narrates only what is visible in the chosen
camera, and its description is fed into a second pass that segments the The `plan` module doesn't ask the VLM for subtasks in one shot. Instead
episode into consecutive atomic subtasks. The resulting spans are then it uses a two-step **describe → segment** flow:
deterministically stitched into a contiguous full-episode cover so every
frame has exactly one active subtask. See 1. **Describe** — the VLM narrates only what it actually sees in the
chosen camera (no guessing about the task).
2. **Segment** — that description is fed back in, and the VLM splits the
episode into consecutive atomic subtasks.
The resulting spans are then stitched into a gap-free, full-episode
cover, so **every frame has exactly one active subtask**. See
[`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py) [`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py)
for the production flag set (single camera, embedded frames, windowed for the production settings (single camera, embedded frames, windowed
subtask generation). subtask generation).
The writer does **not** add a `tools` column to the parquet — the tool ### Tools
catalog lives at `meta/info.json["tools"]` instead (see
[Tools](./tools)). After every annotation run the pipeline ensures the
canonical `say` schema is present in that list, preserving any tools the
user pre-declared.
If you want to declare additional tools for a dataset before annotation The writer does **not** add a `tools` column to the parquet. The tool
runs, edit `meta/info.json["tools"]` directly — the pipeline preserves catalog lives in `meta/info.json["tools"]` instead (see [Tools](./tools)).
anything already there. That makes the tool visible to the chat template After every run, the pipeline makes sure the canonical `say` schema is in
so the model can learn to _generate_ the call. The runtime layer that that list, keeping any tools you declared beforehand.
_executes_ a generated call (the `Tool` protocol / `TOOL_REGISTRY` under
`src/lerobot/tools/`) is not part of this PR — see the Want to add your own tool? Edit `meta/info.json["tools"]` directly — the
[Tools](./tools) doc, which marks those pieces as not-yet-implemented. pipeline preserves whatever is already there. That makes the tool visible
to the chat template, so the model can learn to _generate_ the call. The
runtime layer that actually _executes_ a generated call (the `Tool`
protocol / `TOOL_REGISTRY` under `src/lerobot/tools/`) is not part of
this PR — the [Tools](./tools) doc marks those pieces as
not-yet-implemented.
## Running on Hugging Face Jobs ## Running on Hugging Face Jobs
Distributed annotation is delegated to Annotation runs on [Hugging Face Jobs](https://huggingface.co/docs/hub/en/jobs).
[Hugging Face Jobs](https://huggingface.co/docs/hub/en/jobs). The repo The repo ships a launcher script you copy and tweak for your dataset:
ships a launcher script you copy and edit for your dataset:
```bash ```bash
HF_TOKEN=hf_... uv run python examples/annotations/run_hf_job.py HF_TOKEN=hf_... uv run python examples/annotations/run_hf_job.py
``` ```
[`examples/annotations/run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py) [`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py)
spawns a single-GPU `h200` job (scale up to `h200x4` for larger datasets) that: starts a single-GPU `h200` job (bump it to `h200x4` for big datasets)
that:
1. installs `lerobot` (from `main`) plus the annotation extras, 1. installs `lerobot` (from `main`) plus the annotation extras,
2. boots one vLLM server per GPU (in the `vllm/vllm-openai` image) for the 2. boots one vLLM server per GPU (using the `vllm/vllm-openai` image) and
chosen model, which the pipeline drives over the OpenAI-compatible API, drives it over the OpenAI-compatible API,
3. runs the `plan` / `interjections` / `vqa` modules across the dataset 3. runs the `plan` / `interjections` / `vqa` modules across the dataset
via `lerobot-annotate`, with `lerobot-annotate`,
4. with `--push_to_hub=true`, uploads the annotated dataset to 4. with `--push_to_hub=true`, uploads the result to `--new_repo_id` (or
`--new_repo_id` (or back to `--repo_id` in place when that is unset). back to `--repo_id` in place if you leave that unset).
To target a different dataset, model, or hub repo, edit the `CMD` block To use a different dataset, model, or hub repo, edit the `CMD` block in
inside the script — every flag in there maps directly onto a CLI flag of the script. Every flag there maps directly to a `lerobot-annotate` flag
`lerobot-annotate` (see `lerobot-annotate --help` for the full list). (run `lerobot-annotate --help` for the full list).
## Contributing new modules ## Contributing new modules
The pipeline is built to be extended, and **contributions are very The pipeline is built to grow, and **contributions are very welcome** —
welcome** — whether that's a brand-new annotation module (e.g. a a brand-new module (say, trajectory traces or affordances), a new prompt
trajectory-trace or affordance module), a new prompt template, a better template, a smarter grounding flow, or quality fixes to the existing
grounding flow, or quality improvements to the existing `plan` / `plan` / `interjections` / `vqa` modules.
`interjections` / `vqa` modules. Each module lives under
Every module lives under
`src/lerobot/annotations/steerable_pipeline/modules/`, shares the VLM `src/lerobot/annotations/steerable_pipeline/modules/`, shares the VLM
client and keyframe cache, writes its raw output to the per-episode client and the keyframe cache, writes its raw output to the staging
staging tree, and is wired into the executor as an independent phase. tree, and plugs into the executor as its own phase. Got an idea? Open an
If you have an idea for a module or an improvement, open an issue or PR issue or PR on [the repo](https://github.com/huggingface/lerobot).
on [the repo](https://github.com/huggingface/lerobot).
## Style-to-recipe consumer mapping ## How recipes consume the output
The pipeline's outputs are designed to be consumed by recipes (see The annotations are meant to be read by recipes (see
[Language Columns and Recipes](./language_and_recipes)) — typically: [Language Columns and Recipes](./language_and_recipes)). Typically:
- low-level / high-level / memory-update branches consume - low-level / high-level / memory-update branches read
`subtask` / `plan` / `memory` from `language_persistent`. `subtask` / `plan` / `memory` from `language_persistent`.
- An interjection-response branch consumes `interjection` events plus - an interjection-response branch reads `interjection` events plus the
the paired speech atom (merged into one assistant target turn via paired speech atom (merged into one assistant turn via `tool_calls_from`)
`tool_calls_from`) and the same-timestamp `plan` refresh. and the matching `plan` refresh at the same timestamp.
- A VQA branch consumes the `(vqa, user)` and `(vqa, assistant)` pairs - a VQA branch reads the `(vqa, user)` and `(vqa, assistant)` pairs from
from `language_events`. `language_events`.
## Why the design splits state from events ## Why state and events are split
Two things drive the scope: Two ideas shape the design:
1. **Persistent state vs exact-event split.** Persistent rows 1. **Persistent state vs. exact events.** Persistent rows (`subtask`,
(`subtask`, `plan`, `memory`) broadcast per episode and answer "what `plan`, `memory`) apply to the whole episode and answer "what's true
state is in force at this frame?". Event rows (`interjection`, `vqa`, right now?". Event rows (`interjection`, `vqa`, speech) appear only on
speech) only appear on the exact frame whose timestamp matches the the one frame whose timestamp matches. Timestamps are copied straight
emission. The pipeline writes timestamps taken straight from the from the source parquet — never recomputed in floating point.
source parquet — no floating-point recomputation. 2. **One VLM pass.** All three modules share a single VLM client (the
2. **One Qwen-VL pass.** All three modules share a single VLM client (the OpenAI-compatible client talking to the job's vLLM server), so you pay
OpenAI-compatible client talking to the job's vLLM server) so the cost for one model load per dataset, not three.
is one model load per dataset, not three.
## Module independence and staged reruns ## Re-running a single module
Each module writes its raw output to Each module stages its raw output to
`<root>/.annotate_staging/episode_{N:06d}/<module>.jsonl`. That makes `<root>/.annotate_staging/episode_{N:06d}/<module>.jsonl`. This makes
prompt iteration cheap — re-running one module overwrites only its own prompt iteration cheap: re-running one module overwrites only its own
JSONL file before the writer composes the final parquet. Modules can be JSONL, then the writer recomposes the final parquet. Disable modules you
disabled via `--plan.enabled=false` (and likewise `--interjections.enabled` don't want with `--plan.enabled=false` (and likewise
/ `--vqa.enabled`) to `--interjections.enabled` / `--vqa.enabled`) to test one at a time.
test them in isolation.
## Validation/report checks before final write ## What the validator checks
Before the writer runs, `StagingValidator` checks: Before the writer runs, `StagingValidator` confirms:
- exact frame-timestamp alignment for every event row; - every event row lands exactly on a real frame timestamp;
- no orphan speech / interjection pairs; - no speech / interjection pairs are left orphaned;
- `plan` is refreshed at every interjection timestamp; - `plan` is refreshed at every interjection timestamp;
- `memory` rows fall on subtask boundaries (warning, not error); - `memory` rows fall on subtask boundaries (a warning, not an error);
- VQA assistant `content` parses as JSON in one of the - each VQA assistant `content` is valid JSON in one of the
bbox / keypoint / count / attribute / spatial shapes; bbox / keypoint / count / attribute / spatial shapes;
- every row routes to the column dictated by `column_for_style(style)`. - every row goes to the column chosen by `column_for_style(style)`.
Errors abort the writer (`--skip_validation=true` overrides for debugging). Any error aborts the writer. Pass `--skip_validation=true` to override
while debugging.
## Paper inspirations per module ## Where each module's ideas come from
- **`plan` module — subtasks.** Hi Robot ([Shi 2025](https://arxiv.org/abs/2502.19417)) - **`plan` — subtasks.** Hi Robot ([Shi 2025](https://arxiv.org/abs/2502.19417))
atom granularity ("pick up one piece of lettuce", "place bowl to box"); for atom granularity ("pick up one piece of lettuce", "place bowl to
Pi0.7 ([Physical Intelligence 2025](https://pi.website/pi07)) "how, not box"); Pi0.7 ([Physical Intelligence 2025](https://pi.website/pi07))
what" detail. for "how, not what" detail.
- **`plan` module — memory.** MEM ([Torne 2026](https://arxiv.org/abs/2603.03596)) - **`plan` — memory.** MEM ([Torne 2026](https://arxiv.org/abs/2603.03596)):
compression directive: keep only minimal relevant information; functional keep only the minimal relevant information — preserve outcomes, drop
outcomes preserved, specific attributes dropped. specific attributes.
- **`interjections` module.** Hi Robot scenario taxonomy: negative task, - **`interjections`.** Hi Robot's scenario taxonomy: negative task,
situated correction, specific constraint, preference. Speech is a situated correction, specific constraint, preference. Speech is a
tool-call-only atom (`tool_calls=[{type:function, function:{name:"say", tool-call-only atom
arguments:{text:...}}}]`). (`tool_calls=[{type:function, function:{name:"say", arguments:{text:...}}}]`).
- **`vqa` module.** ECoT ([Zawalski 2024](https://arxiv.org/abs/2407.08693)) - **`vqa`.** ECoT ([Zawalski 2024](https://arxiv.org/abs/2407.08693)) for
grounded features (bounding boxes in pixel `[x_min, y_min, x_max, y_max]`, grounded features (pixel bounding boxes `[x_min, y_min, x_max, y_max]`,
keypoints) and Steerable VLA Policies ([Zhao 2025](https://arxiv.org/abs/2509.07626)) keypoints) and Steerable VLA Policies
multi-abstraction grounding. Pi0.7 also grounds answers across ([Zhao 2025](https://arxiv.org/abs/2509.07626)) for multi-abstraction
multiple abstraction levels. grounding. Pi0.7 also grounds answers across abstraction levels.
Future maintainers should adjust the prompt templates in When improving a module, tweak its prompt template in
`src/lerobot/annotations/steerable_pipeline/prompts/` against these `src/lerobot/annotations/steerable_pipeline/prompts/` rather than
references rather than rewriting from scratch. rewriting from scratch.
## Compute and list-size estimates ## Roughly how much it costs
Per episode, the pipeline issues O(`max_steps`) `plan`-module calls, Per episode, the pipeline makes about `max_steps` plan calls,
O(`max_interjections_per_episode`) `interjections`-module calls, and `max_interjections_per_episode` interjection calls, and
O(`vqa_emission_hz × episode_seconds`) `vqa`-module calls. With defaults `vqa_emission_hz × episode_seconds` VQA calls. With the defaults (8
(8 subtasks, 1 interjection, 1 Hz × 3 pairs) and 30-second episodes, that subtasks, 1 interjection, 1 Hz × 3 pairs) on a 30-second episode, that's
is ~50 VLM calls per episode. `language_persistent` per episode is ~10s of ~50 VLM calls.
KB at most (parquet dictionary-encodes one entry per episode);
`language_events` is empty on most frames and is bounded by the number of
emissions, not `num_frames × num_emissions`.
## Reproducibility via seed and prompt hashes Storage stays small: `language_persistent` is at most tens of KB per
episode (parquet dictionary-encodes the one entry that repeats across
`--seed` (default 1729) feeds the per-episode RNGs that select interjection frames), and `language_events` is empty on most frames — its size scales
timestamps and VQA question types. Combined with the deterministic prompt with the number of emissions, not `num_frames × num_emissions`.
templates checked into `prompts/`, two runs at the same seed against the
same dataset and the same model checkpoint produce byte-identical staging
artifacts. Prompt edits are recorded by file hash; future tooling can pin
expected `(seed, prompt_hash)` pairs into the dataset card.