lerobot

admin/lerobot

Fork 0

mirror of https://github.com/huggingface/lerobot.git synced 2026-07-24 18:26:11 +00:00

Commit Graph

Author SHA1 Message Date

Author	SHA1	Message	Date
Pepijn	9c82c39c7b	feat(annotate): run lerobot-annotate on HF Jobs via --job.target (#4095 ) * feat(annotate): run lerobot-annotate on HF Jobs via --job.target Annotation needed a hand-edited launcher script (examples/annotations/run_hf_job.py) to reach a GPU: users copied it, rewrote the embedded CMD string for their dataset, and ran it with `python`. Fold that into the CLI instead, mirroring `lerobot-train`: `lerobot-annotate --job.target=h200` submits the exact command you'd run locally. - AnnotationJobConfig extends JobConfig with the annotation runtime's defaults (vllm/vllm-openai image, 2h cap) plus --job.lerobot_ref, so an unmerged branch can be exercised remotely without editing a script. - lerobot.jobs.annotate builds the pod command by replaying the user's own CLI flags (minus --job./--root, with --repo_id re-emitted from the config) after a setup prelude that installs lerobot on top of the vLLM image. Job monitoring, log tailing and Ctrl-C-detaches reuse the training submitter's plumbing. - Remote runs require --repo_id; a local-only dataset is pushed privately first. The generated pod command is byte-for-byte the script's old CMD. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> fix(annotate): reject client-side config files on remote runs draccus exposes `--config_path` plus a `--<field>` config-file arg for every nested dataclass (`--vlm`, `--plan`, `--job`, ...). All name files on the client's disk, so forwarding them to the pod silently dropped whatever settings they carried. Reject them up front instead. Bare `--job` also slipped past the `--job.` prefix filter, so a `--job=cfg.yaml` holding `target: h200` would have reached the pod and had the job submit a job of its own, recursively. It is dropped from the forwarded args as well. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(jobs): share the submit-and-follow loop between both submitters `submit_annotate_to_hf` reused the leaf helpers (`_poll_until_done`, `_tail_logs`, `_pod_forwarded_args`) but duplicated the orchestration around them: ~40 of the 50 lines that spawn the poll/log threads, install the Ctrl-C-detaches handler and raise on a non-COMPLETED stage were identical in both files. Extract that into `follow_job(job_id, *, detach, success_marker=None) -> bool`, returning True when the job finished and False when we stopped watching without a verdict (detach or Ctrl-C). Training keeps its model-pushed marker by passing it in; annotation has no equivalent line (the CLI keeps working after the upload log to write the card and tag) so its completion stays stage-based. Kept in hf.py rather than a new module so every existing monkeypatch target in test_hf.py still resolves. Behaviour change: a training run whose job reaches COMPLETED without the marker matching now prints its completion line instead of returning silently. The marker was already documented as an optimisation with a stage-based fallback; the fallback just never reported success. Tests: adds annotate coverage for the non-detach path (completion and failure) — previously only ever exercised with detach=true — plus a detach short-circuit test. Both new annotate tests verified to fail under a mutation that stubs out follow_job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 10:30:33 +02:00
Pepijn	279c6c7af3	feat(annotate): improve VLM subtask annotation (legible contact sheets, seeded relabeling, self-hosted vLLM recipe) (#3896 ) * feat(annotate): WGO-tuned subtask prompt (atomic completed-events + duration prior) Rework the plan-module subtask segmentation prompt toward the WGO-Bench atomic annotation protocol: segment by completed world-state changes (grasp/place/open/close/pour/insert), fold approach+retreat into their event, keep separate events separate, and add a 2-10s duration prior. Drops the pi0.7 "fewer larger composites preferred" bias that drove under-segmentation on the benchmark. Output JSON shape unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(annotate): seeded-relabeling second pass for subtasks Add an opt-in relabel pass (plan.subtask_seeded_relabel) that, after segmentation, re-labels each span using previous/current/next segment contact sheets and the seed label as a strong prior, minimally correcting it. Mirrors macrodata's best end-to-end labeling step. Boundaries are untouched; one extra VLM call per span. Off by default. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(annotate): robust OpenAI-compat client for hosted VLMs Guard against a choice with no message (safety filter or a thinking model that spends its whole budget before emitting content) so one empty reply no longer crashes the whole annotation run; treat it as an empty response and let the existing JSON-retry path handle it. Add an optional `reasoning_effort` knob on VlmConfig, forwarded to the server when set, to cap a thinking model's reasoning (needed for Gemini via its OpenAI-compatible endpoint). Co-authored-by: Cursor <cursoragent@cursor.com> * feat(annotate): legible tile-scaled timestamp on contact sheets The burned-in timestamp used the ~10px bitmap default font, which blurs once the model downsamples a full contact sheet into 768px tiles, so the VLM can no longer read the exact source time a boundary depends on. Scale the timestamp to the tile height (with a graceful fallback on older Pillow) so the visual time cue stays readable at sheet resolution. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(annotate): lean GEPA-aligned subtask segmentation prompt Replace the verbose, label-heavy segmentation prompt with a lean adaptation of the blog's GEPA-found completed_events_duration_prior recipe: focus on completed manipulation events, explicit no-split / no-merge rules, a 2-10s duration prior, and an instruction to prioritize temporally correct boundaries over label wording. The previous prompt over-weighted label guidance, which traded away boundary precision. Co-authored-by: Cursor <cursoragent@cursor.com> * revert: restore original subtask segmentation prompt The lean GEPA-aligned paraphrase (`dd4b0110d`) regressed Gemini on the 30-ep subset: Seg F1 0.259 -> 0.189 and E2E 0.184 -> 0.135, driven by worse under-segmentation (224 -> 188 preds). The blog's 0.306 came from the actual GEPA-search artifact, which a hand paraphrase does not reproduce. Restore the original prompt, which remains our best config. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(annotate): env-var override for prompt templates Allow LEROBOT_PROMPT_OVERRIDE_<name> to supersede the packaged prompt file at load time. Enables prompt search (GEPA) to inject candidate segmentation prompts into a remote annotate job via an env secret, without committing a branch per candidate. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(annotate): genericize hosted-VLM comments (no model name) Co-authored-by: Cursor <cursoragent@cursor.com> * docs(annotate): document seeded-relabel and reasoning_effort flags Co-authored-by: Cursor <cursoragent@cursor.com> * test(annotate): update subtask-prompt marker to match WGO-tuned prompt The three plan-module tests keyed the canned VLM responder on the literal 'atomic subtasks', which the WGO-tuned segmentation prompt no longer contains (it now segments 'COMPLETED manipulation events'). Point the fixture markers at the current wording so the subtask call is matched again. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-07-15 11:38:49 +02:00
Pepijn	cec8ee0be6	feat: language annotation pipeline (#3471 ) Steerable annotation pipeline (lerobot-annotate) that populates the language_persistent and language_events columns introduced in PR 1 (#3467) directly into data/chunk-/file-.parquet. This is PR 2 of the three-PR plan: PR 1 (Add extensive language support #3467): schema + DSL + rendering, base of this PR PR 2 (this PR): annotation pipeline writing into PR 1's columns PR 3: model with language prediction and runtime A VLM (Qwen-VL family, served on vLLM) watches each episode's video and emits grounded language annotations: subtasks, plans, memory, task rephrasings, interjections + speech, and per-camera VQA. The pipeline is built for production annotation at scale — single-camera grounding, embedded-frame inputs, a describe-then-segment grounding flow, and a deterministic full-episode coverage guarantee — informed by Scale's dense-captioning findings (representation > sampling, rules > reasoning, model capacity is the biggest lever, two-pass systems compound errors)	2026-06-12 15:12:33 +02:00

Pepijn

9c82c39c7b

feat(annotate): run lerobot-annotate on HF Jobs via --job.target (#4095 )

* feat(annotate): run lerobot-annotate on HF Jobs via --job.target

Annotation needed a hand-edited launcher script (examples/annotations/run_hf_job.py)
to reach a GPU: users copied it, rewrote the embedded CMD string for their dataset,
and ran it with `python`. Fold that into the CLI instead, mirroring `lerobot-train`:
`lerobot-annotate --job.target=h200` submits the exact command you'd run locally.

- AnnotationJobConfig extends JobConfig with the annotation runtime's defaults
  (vllm/vllm-openai image, 2h cap) plus --job.lerobot_ref, so an unmerged branch
  can be exercised remotely without editing a script.
- lerobot.jobs.annotate builds the pod command by replaying the user's own CLI
  flags (minus --job.*/--root, with --repo_id re-emitted from the config) after a
  setup prelude that installs lerobot on top of the vLLM image. Job monitoring,
  log tailing and Ctrl-C-detaches reuse the training submitter's plumbing.
- Remote runs require --repo_id; a local-only dataset is pushed privately first.

The generated pod command is byte-for-byte the script's old CMD.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(annotate): reject client-side config files on remote runs

draccus exposes `--config_path` plus a `--<field>` config-file arg for every
nested dataclass (`--vlm`, `--plan`, `--job`, ...). All name files on the
client's disk, so forwarding them to the pod silently dropped whatever settings
they carried. Reject them up front instead.

Bare `--job` also slipped past the `--job.` prefix filter, so a `--job=cfg.yaml`
holding `target: h200` would have reached the pod and had the job submit a job
of its own, recursively. It is dropped from the forwarded args as well.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(jobs): share the submit-and-follow loop between both submitters

`submit_annotate_to_hf` reused the leaf helpers (`_poll_until_done`, `_tail_logs`,
`_pod_forwarded_args`) but duplicated the orchestration around them: ~40 of the 50
lines that spawn the poll/log threads, install the Ctrl-C-detaches handler and
raise on a non-COMPLETED stage were identical in both files.

Extract that into `follow_job(job_id, *, detach, success_marker=None) -> bool`,
returning True when the job finished and False when we stopped watching without a
verdict (detach or Ctrl-C). Training keeps its model-pushed marker by passing it in;
annotation has no equivalent line (the CLI keeps working after the upload log to
write the card and tag) so its completion stays stage-based.

Kept in hf.py rather than a new module so every existing monkeypatch target in
test_hf.py still resolves.

Behaviour change: a training run whose job reaches COMPLETED without the marker
matching now prints its completion line instead of returning silently. The marker
was already documented as an optimisation with a stage-based fallback; the fallback
just never reported success.

Tests: adds annotate coverage for the non-detach path (completion and failure) —
previously only ever exercised with detach=true — plus a detach short-circuit test.
Both new annotate tests verified to fail under a mutation that stubs out follow_job.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-23 10:30:33 +02:00

Pepijn

279c6c7af3

feat(annotate): improve VLM subtask annotation (legible contact sheets, seeded relabeling, self-hosted vLLM recipe) (#3896 )

* feat(annotate): WGO-tuned subtask prompt (atomic completed-events + duration prior)

Rework the plan-module subtask segmentation prompt toward the WGO-Bench
atomic annotation protocol: segment by completed world-state changes
(grasp/place/open/close/pour/insert), fold approach+retreat into their
event, keep separate events separate, and add a 2-10s duration prior.
Drops the pi0.7 "fewer larger composites preferred" bias that drove
under-segmentation on the benchmark. Output JSON shape unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(annotate): seeded-relabeling second pass for subtasks

Add an opt-in relabel pass (plan.subtask_seeded_relabel) that, after
segmentation, re-labels each span using previous/current/next segment
contact sheets and the seed label as a strong prior, minimally correcting
it. Mirrors macrodata's best end-to-end labeling step. Boundaries are
untouched; one extra VLM call per span. Off by default.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(annotate): robust OpenAI-compat client for hosted VLMs

Guard against a choice with no message (safety filter or a thinking model
that spends its whole budget before emitting content) so one empty reply
no longer crashes the whole annotation run; treat it as an empty response
and let the existing JSON-retry path handle it.

Add an optional `reasoning_effort` knob on VlmConfig, forwarded to the
server when set, to cap a thinking model's reasoning (needed for Gemini
via its OpenAI-compatible endpoint).

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(annotate): legible tile-scaled timestamp on contact sheets

The burned-in timestamp used the ~10px bitmap default font, which blurs
once the model downsamples a full contact sheet into 768px tiles, so the
VLM can no longer read the exact source time a boundary depends on. Scale
the timestamp to the tile height (with a graceful fallback on older
Pillow) so the visual time cue stays readable at sheet resolution.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(annotate): lean GEPA-aligned subtask segmentation prompt

Replace the verbose, label-heavy segmentation prompt with a lean
adaptation of the blog's GEPA-found completed_events_duration_prior
recipe: focus on completed manipulation events, explicit no-split /
no-merge rules, a 2-10s duration prior, and an instruction to prioritize
temporally correct boundaries over label wording. The previous prompt
over-weighted label guidance, which traded away boundary precision.

Co-authored-by: Cursor <cursoragent@cursor.com>

* revert: restore original subtask segmentation prompt

The lean GEPA-aligned paraphrase (dd4b0110d) regressed Gemini on the
30-ep subset: Seg F1 0.259 -> 0.189 and E2E 0.184 -> 0.135, driven by
worse under-segmentation (224 -> 188 preds). The blog's 0.306 came from
the actual GEPA-search artifact, which a hand paraphrase does not
reproduce. Restore the original prompt, which remains our best config.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(annotate): env-var override for prompt templates

Allow LEROBOT_PROMPT_OVERRIDE_<name> to supersede the packaged prompt
file at load time. Enables prompt search (GEPA) to inject candidate
segmentation prompts into a remote annotate job via an env secret,
without committing a branch per candidate.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(annotate): genericize hosted-VLM comments (no model name)

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(annotate): document seeded-relabel and reasoning_effort flags

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(annotate): update subtask-prompt marker to match WGO-tuned prompt

The three plan-module tests keyed the canned VLM responder on the
literal 'atomic subtasks', which the WGO-tuned segmentation prompt no
longer contains (it now segments 'COMPLETED manipulation events'). Point
the fixture markers at the current wording so the subtask call is matched
again.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

2026-07-15 11:38:49 +02:00

Pepijn

cec8ee0be6

feat: language annotation pipeline (#3471 )

Steerable annotation pipeline (lerobot-annotate) that populates the language_persistent and language_events columns introduced in PR 1 (#3467) directly into data/chunk-*/file-*.parquet.

This is PR 2 of the three-PR plan:

PR 1 (Add extensive language support #3467): schema + DSL + rendering, base of this PR
PR 2 (this PR): annotation pipeline writing into PR 1's columns
PR 3: model with language prediction and runtime
A VLM (Qwen-VL family, served on vLLM) watches each episode's video and emits grounded language annotations: subtasks, plans, memory, task rephrasings, interjections + speech, and per-camera VQA. The pipeline is built for production annotation at scale — single-camera grounding, embedded-frame inputs, a describe-then-segment grounding flow, and a deterministic full-episode coverage guarantee — informed by Scale's dense-captioning findings (representation > sampling, rules > reasoning, model capacity is the biggest lever, two-pass systems compound errors)

2026-06-12 15:12:33 +02:00

3 Commits