* (depth image processing): excluding depth frames from the RGB to BGR image processing
* test(update): updating tests to include RGB/BGR conversion checks
* fix(deps): bump diffusers cap to <0.39.0 (security)
Diffusers 0.35.x is affected by GHSA-98h9-4798-4q5v (HIGH, CVSS 8.8):
'trust_remote_code bypass via custom_pipeline and local custom components'.
Fixed in diffusers 0.38.0.
Current cap 'diffusers<0.36.0' blocks downstream consumers (e.g.
strands-labs/robots) from picking up the security fix.
The lerobot diffusers surface area is narrow and stable across 0.36-0.38:
- diffusers.schedulers.scheduling_ddim.DDIMScheduler
- diffusers.schedulers.scheduling_ddpm.DDPMScheduler
- diffusers.optimization.get_scheduler
- diffusers.ConfigMixin / ModelMixin / register_to_config
- diffusers.models.attention.{Attention,FeedForward}
- diffusers.models.embeddings.*
None of these were removed, renamed, or had breaking changes in 0.36, 0.37,
or 0.38 release notes. Bumping the cap to <0.39.0 unblocks the security
fix while keeping a major-version safety bound.
* chore(dependecies): bump diffusers
* chore(deps): update uv.lock
---------
Co-authored-by: Cagatay Cali <cagataycali@users.noreply.github.com>
* Make SO follower P coefficient configurable
* chore(test): minimize tests
* feat(robots): expose PID coeff in SO arms
---------
Co-authored-by: taivu1998 <46636857+taivu1998@users.noreply.github.com>
* chore(agents): adding additional infos to AGENTS.md
* adding `lerobot-train` requirement inside PR checklist
* prefer using code already implemented from transformers / diffusers instead of re-implementing in tree
---------
Signed-off-by: Maxime Ellerbach <maxime.ellerbach@huggingface.co>
* fix(envs): set LiberoEnvConfig.fps default to 20 to match robosuite
LiberoEnvConfig.fps was set to 30, but the underlying robosuite
OffScreenRenderEnv always runs at its default control_freq of 20 Hz
since fps is never passed through. This mismatch silently decouples
the dataset/eval loop rate from the actual simulation step rate.
Set the default to 20 to match the real sim rate and avoid the
footgun.
Fixes#3368
* fix(libero): apply configured control frequency
---------
Co-authored-by: xinmotlanthua <275663218+xinmotlanthua@users.noreply.github.com>
* fix(damiao): make is_calibrated a plain property, not cached
`is_calibrated` was a `@cached_property`, so it froze at its first-read
value and never reflected later changes to `self.calibration` (set by
connect/calibrate/load). This caused the OpenArm teleop to re-run
calibration even when a calibration file existed, and to skip
`set_zero_position()` after a fresh calibration.
Switch to `@property` (matching the MotorsBus base contract and the
Feetech/SO-100 buses) and drop the now-unused `functools.cached_property`
import.
Co-authored-by: Cursor <cursoragent@cursor.com>
* don't set_zero_position() on connect
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(diffusion): add gradient checkpointing for memory optimization
Add gradient_checkpointing config option to DiffusionPolicy. When
enabled, wraps the UNet encoder, mid, and decoder residual blocks
with torch.utils.checkpoint.checkpoint to trade compute for memory.
Allows training with larger batch sizes or higher-resolution inputs
on memory-constrained GPUs. Disabled by default.
Usage: --policy.gradient_checkpointing=true
Part of the 0.6.0 roadmap item 3.3 (gradient checkpointing for all
policies).
* test(diffusion): verify gradient checkpointing parity
---------
Co-authored-by: Jash Shah <jashshah.999@gmail.com>
* gamepad does often work on macos
* review comments
* fix(gamepad): expose hidapi fallback in config
---------
Co-authored-by: Maxim Bonnaerens <maxim@bonnaerens.be>
* feat(annotate): run lerobot-annotate on HF Jobs via --job.target
Annotation needed a hand-edited launcher script (examples/annotations/run_hf_job.py)
to reach a GPU: users copied it, rewrote the embedded CMD string for their dataset,
and ran it with `python`. Fold that into the CLI instead, mirroring `lerobot-train`:
`lerobot-annotate --job.target=h200` submits the exact command you'd run locally.
- AnnotationJobConfig extends JobConfig with the annotation runtime's defaults
(vllm/vllm-openai image, 2h cap) plus --job.lerobot_ref, so an unmerged branch
can be exercised remotely without editing a script.
- lerobot.jobs.annotate builds the pod command by replaying the user's own CLI
flags (minus --job.*/--root, with --repo_id re-emitted from the config) after a
setup prelude that installs lerobot on top of the vLLM image. Job monitoring,
log tailing and Ctrl-C-detaches reuse the training submitter's plumbing.
- Remote runs require --repo_id; a local-only dataset is pushed privately first.
The generated pod command is byte-for-byte the script's old CMD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(annotate): reject client-side config files on remote runs
draccus exposes `--config_path` plus a `--<field>` config-file arg for every
nested dataclass (`--vlm`, `--plan`, `--job`, ...). All name files on the
client's disk, so forwarding them to the pod silently dropped whatever settings
they carried. Reject them up front instead.
Bare `--job` also slipped past the `--job.` prefix filter, so a `--job=cfg.yaml`
holding `target: h200` would have reached the pod and had the job submit a job
of its own, recursively. It is dropped from the forwarded args as well.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(jobs): share the submit-and-follow loop between both submitters
`submit_annotate_to_hf` reused the leaf helpers (`_poll_until_done`, `_tail_logs`,
`_pod_forwarded_args`) but duplicated the orchestration around them: ~40 of the 50
lines that spawn the poll/log threads, install the Ctrl-C-detaches handler and
raise on a non-COMPLETED stage were identical in both files.
Extract that into `follow_job(job_id, *, detach, success_marker=None) -> bool`,
returning True when the job finished and False when we stopped watching without a
verdict (detach or Ctrl-C). Training keeps its model-pushed marker by passing it in;
annotation has no equivalent line (the CLI keeps working after the upload log to
write the card and tag) so its completion stays stage-based.
Kept in hf.py rather than a new module so every existing monkeypatch target in
test_hf.py still resolves.
Behaviour change: a training run whose job reaches COMPLETED without the marker
matching now prints its completion line instead of returning silently. The marker
was already documented as an optimisation with a stage-based fallback; the fallback
just never reported success.
Tests: adds annotate coverage for the non-detach path (completion and failure) —
previously only ever exercised with detach=true — plus a detach short-circuit test.
Both new annotate tests verified to fail under a mutation that stubs out follow_job.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(annotate): WGO-tuned subtask prompt (atomic completed-events + duration prior)
Rework the plan-module subtask segmentation prompt toward the WGO-Bench
atomic annotation protocol: segment by completed world-state changes
(grasp/place/open/close/pour/insert), fold approach+retreat into their
event, keep separate events separate, and add a 2-10s duration prior.
Drops the pi0.7 "fewer larger composites preferred" bias that drove
under-segmentation on the benchmark. Output JSON shape unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): seeded-relabeling second pass for subtasks
Add an opt-in relabel pass (plan.subtask_seeded_relabel) that, after
segmentation, re-labels each span using previous/current/next segment
contact sheets and the seed label as a strong prior, minimally correcting
it. Mirrors macrodata's best end-to-end labeling step. Boundaries are
untouched; one extra VLM call per span. Off by default.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): robust OpenAI-compat client for hosted VLMs
Guard against a choice with no message (safety filter or a thinking model
that spends its whole budget before emitting content) so one empty reply
no longer crashes the whole annotation run; treat it as an empty response
and let the existing JSON-retry path handle it.
Add an optional `reasoning_effort` knob on VlmConfig, forwarded to the
server when set, to cap a thinking model's reasoning (needed for Gemini
via its OpenAI-compatible endpoint).
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): legible tile-scaled timestamp on contact sheets
The burned-in timestamp used the ~10px bitmap default font, which blurs
once the model downsamples a full contact sheet into 768px tiles, so the
VLM can no longer read the exact source time a boundary depends on. Scale
the timestamp to the tile height (with a graceful fallback on older
Pillow) so the visual time cue stays readable at sheet resolution.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): lean GEPA-aligned subtask segmentation prompt
Replace the verbose, label-heavy segmentation prompt with a lean
adaptation of the blog's GEPA-found completed_events_duration_prior
recipe: focus on completed manipulation events, explicit no-split /
no-merge rules, a 2-10s duration prior, and an instruction to prioritize
temporally correct boundaries over label wording. The previous prompt
over-weighted label guidance, which traded away boundary precision.
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert: restore original subtask segmentation prompt
The lean GEPA-aligned paraphrase (dd4b0110d) regressed Gemini on the
30-ep subset: Seg F1 0.259 -> 0.189 and E2E 0.184 -> 0.135, driven by
worse under-segmentation (224 -> 188 preds). The blog's 0.306 came from
the actual GEPA-search artifact, which a hand paraphrase does not
reproduce. Restore the original prompt, which remains our best config.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): env-var override for prompt templates
Allow LEROBOT_PROMPT_OVERRIDE_<name> to supersede the packaged prompt
file at load time. Enables prompt search (GEPA) to inject candidate
segmentation prompts into a remote annotate job via an env secret,
without committing a branch per candidate.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(annotate): genericize hosted-VLM comments (no model name)
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(annotate): document seeded-relabel and reasoning_effort flags
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(annotate): update subtask-prompt marker to match WGO-tuned prompt
The three plan-module tests keyed the canned VLM responder on the
literal 'atomic subtasks', which the WGO-tuned segmentation prompt no
longer contains (it now segments 'COMPLETED manipulation events'). Point
the fixture markers at the current wording so the subtask call is matched
again.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Pepijn