* feat(annotate): run lerobot-annotate on HF Jobs via --job.target
Annotation needed a hand-edited launcher script (examples/annotations/run_hf_job.py)
to reach a GPU: users copied it, rewrote the embedded CMD string for their dataset,
and ran it with `python`. Fold that into the CLI instead, mirroring `lerobot-train`:
`lerobot-annotate --job.target=h200` submits the exact command you'd run locally.
- AnnotationJobConfig extends JobConfig with the annotation runtime's defaults
(vllm/vllm-openai image, 2h cap) plus --job.lerobot_ref, so an unmerged branch
can be exercised remotely without editing a script.
- lerobot.jobs.annotate builds the pod command by replaying the user's own CLI
flags (minus --job.*/--root, with --repo_id re-emitted from the config) after a
setup prelude that installs lerobot on top of the vLLM image. Job monitoring,
log tailing and Ctrl-C-detaches reuse the training submitter's plumbing.
- Remote runs require --repo_id; a local-only dataset is pushed privately first.
The generated pod command is byte-for-byte the script's old CMD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(annotate): reject client-side config files on remote runs
draccus exposes `--config_path` plus a `--<field>` config-file arg for every
nested dataclass (`--vlm`, `--plan`, `--job`, ...). All name files on the
client's disk, so forwarding them to the pod silently dropped whatever settings
they carried. Reject them up front instead.
Bare `--job` also slipped past the `--job.` prefix filter, so a `--job=cfg.yaml`
holding `target: h200` would have reached the pod and had the job submit a job
of its own, recursively. It is dropped from the forwarded args as well.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(jobs): share the submit-and-follow loop between both submitters
`submit_annotate_to_hf` reused the leaf helpers (`_poll_until_done`, `_tail_logs`,
`_pod_forwarded_args`) but duplicated the orchestration around them: ~40 of the 50
lines that spawn the poll/log threads, install the Ctrl-C-detaches handler and
raise on a non-COMPLETED stage were identical in both files.
Extract that into `follow_job(job_id, *, detach, success_marker=None) -> bool`,
returning True when the job finished and False when we stopped watching without a
verdict (detach or Ctrl-C). Training keeps its model-pushed marker by passing it in;
annotation has no equivalent line (the CLI keeps working after the upload log to
write the card and tag) so its completion stays stage-based.
Kept in hf.py rather than a new module so every existing monkeypatch target in
test_hf.py still resolves.
Behaviour change: a training run whose job reaches COMPLETED without the marker
matching now prints its completion line instead of returning silently. The marker
was already documented as an optimisation with a stage-based fallback; the fallback
just never reported success.
Tests: adds annotate coverage for the non-detach path (completion and failure) —
previously only ever exercised with detach=true — plus a detach short-circuit test.
Both new annotate tests verified to fail under a mutation that stubs out follow_job.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Strip everything except the OpenHLM/pi0.5 -> SONIC encoder/decoder rollout
path so this branch does exactly that and nothing more:
- Remove the SONIC motion planner (planner ONNX + subprocess worker, PlannerMotion,
replanning, MovementState/LocomotionMode, joystick) from sonic_pipeline; keep the
encoder/decoder and the caller-fed reference buffer (PlannerController) intact.
- Slim SonicRuntime to load only the encoder/decoder; SonicWholeBodyController now
runs solely the 34-D whole-body command path (drop SMPL/VR3/keyboard teleop).
- Delete the pico_headset teleoperator (SONIC's SMPL/VR3 teleop source).
- Move WB action constants into g1_utils; repoint imports.
GR00T/Holosoma locomotion controllers are left untouched.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a dense 34-D whole-body command path so lerobot-rollout can drive the
G1 directly with an OpenHLM / pi0.5 policy through the SONIC encoder/decoder:
- SonicWholeBodyController: wb.{i}.pos action interface, mode-0 reference with
a rolling 50-frame trajectory (finite-diff velocities) and first-tick anchor
init; correct MuJoCo->IsaacLab joint reordering.
- unitree_g1: expose 34-D wb_state.{i}.pos proprio; empty/replay camera feeds
for image-conditioned policies; Dex3 hand publishing from the grip scalars.
- g1_utils: obs_to_wb34_state + WB action constants.
Co-authored-by: Cursor <cursoragent@cursor.com>
Feeding the per-frame SMPL root quaternion into the mode-2 anchor produced
root-acceleration spikes (NaN QACC at DOF 0) mid-episode during replay. Keep the
anchor self-driven until the reference root trajectory is smoothed/rate-matched
(30 Hz dataset -> 50 Hz control).
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): WGO-tuned subtask prompt (atomic completed-events + duration prior)
Rework the plan-module subtask segmentation prompt toward the WGO-Bench
atomic annotation protocol: segment by completed world-state changes
(grasp/place/open/close/pour/insert), fold approach+retreat into their
event, keep separate events separate, and add a 2-10s duration prior.
Drops the pi0.7 "fewer larger composites preferred" bias that drove
under-segmentation on the benchmark. Output JSON shape unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): seeded-relabeling second pass for subtasks
Add an opt-in relabel pass (plan.subtask_seeded_relabel) that, after
segmentation, re-labels each span using previous/current/next segment
contact sheets and the seed label as a strong prior, minimally correcting
it. Mirrors macrodata's best end-to-end labeling step. Boundaries are
untouched; one extra VLM call per span. Off by default.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): robust OpenAI-compat client for hosted VLMs
Guard against a choice with no message (safety filter or a thinking model
that spends its whole budget before emitting content) so one empty reply
no longer crashes the whole annotation run; treat it as an empty response
and let the existing JSON-retry path handle it.
Add an optional `reasoning_effort` knob on VlmConfig, forwarded to the
server when set, to cap a thinking model's reasoning (needed for Gemini
via its OpenAI-compatible endpoint).
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): legible tile-scaled timestamp on contact sheets
The burned-in timestamp used the ~10px bitmap default font, which blurs
once the model downsamples a full contact sheet into 768px tiles, so the
VLM can no longer read the exact source time a boundary depends on. Scale
the timestamp to the tile height (with a graceful fallback on older
Pillow) so the visual time cue stays readable at sheet resolution.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): lean GEPA-aligned subtask segmentation prompt
Replace the verbose, label-heavy segmentation prompt with a lean
adaptation of the blog's GEPA-found completed_events_duration_prior
recipe: focus on completed manipulation events, explicit no-split /
no-merge rules, a 2-10s duration prior, and an instruction to prioritize
temporally correct boundaries over label wording. The previous prompt
over-weighted label guidance, which traded away boundary precision.
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert: restore original subtask segmentation prompt
The lean GEPA-aligned paraphrase (dd4b0110d) regressed Gemini on the
30-ep subset: Seg F1 0.259 -> 0.189 and E2E 0.184 -> 0.135, driven by
worse under-segmentation (224 -> 188 preds). The blog's 0.306 came from
the actual GEPA-search artifact, which a hand paraphrase does not
reproduce. Restore the original prompt, which remains our best config.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): env-var override for prompt templates
Allow LEROBOT_PROMPT_OVERRIDE_<name> to supersede the packaged prompt
file at load time. Enables prompt search (GEPA) to inject candidate
segmentation prompts into a remote annotate job via an env secret,
without committing a branch per candidate.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(annotate): genericize hosted-VLM comments (no model name)
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(annotate): document seeded-relabel and reasoning_effort flags
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(annotate): update subtask-prompt marker to match WGO-tuned prompt
The three plan-module tests keyed the canned VLM responder on the
literal 'atomic subtasks', which the WGO-tuned segmentation prompt no
longer contains (it now segments 'COMPLETED manipulation events'). Point
the fixture markers at the current wording so the subtask call is matched
again.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
- Remove unused RawKeyboard/drain_keyboard/process_keyboard from sonic_pipeline
(dead code duplicating lerobot.utils.keyboard_input); the G1 integration uses
the joystick path. Drop now-unused sys/select/termios/tty imports.
- Add a comment explaining the smpl.0 presence check is a sentinel for a full
SMPL window (review question).
Co-authored-by: Cursor <cursoragent@cursor.com>
Address review feedback that sonic_pipeline.py was dense and hard to read.
Adds a module-level architecture overview plus class and key-function
docstrings (planner subprocess, encoder/decoder, movement state, input
helpers). No behavior change.
Co-authored-by: Cursor <cursoragent@cursor.com>
Duhyeon, Kim