* feat(camera): add manual exposure, gain, and white balance options for RealSense cameras
The RealSense camera integration lacked sensor-level controls, causing
issues like unstable lighting from auto-exposure hunting. This adds
optional `exposure`, `gain`, and `white_balance` fields to
RealSenseCameraConfig that disable the corresponding auto modes and
apply fixed values when set.
* fix: support D405 stereo module for sensor options
D405 exposes color stream via "Stereo Module", not "RGB Camera".
Fall back to Stereo Module when RGB Camera is not found.
* test(camera): add unit tests + range-aware errors for RealSense sensor options
Address PR #3220 review:
- Wrap set_option calls; re-raise ValueError with option name, value,
and sensor.get_option_range() diagnostics on out-of-range values.
- Add unit tests for _get_color_sensor (RGB Camera, D405 Stereo Module
fallback, diagnostic error) and _configure_sensor_options (no-op,
all values, unsupported warns, partial config, out-of-range raise).
* fix(realsense): validate manual color controls
* refactor(camera): apply feedback
---------
Co-authored-by: Lev Kozlov <kozlov.l.a10@gmail.com>
* feat(lekiwi): support LeKiwi in the rollout/eval CLI
Register the lekiwi robot in lerobot_rollout.py so policies can be rolled out
on a LeKiwi, and keep base-velocity (.vel) features in build_rollout_context.
LeKiwi's observation.state and action are 9-dim (6 arm .pos + x/y/theta.vel)
and the policy is normalized on all 9. The old filter kept only .pos features,
so it fed a 6-dim vector into a 9-dim normalizer (RuntimeError, size 6 vs 9) and
silently dropped the base velocities from the action, leaving the base unable to
move. Keeping both .pos and .vel fixes both. Pure-arm robots have no .vel keys,
so this is a no-op for them.
* style: format LeKiwi rollout action features
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
* Add dataloader_multiprocessing_context, default to spawn
Make the DataLoader multiprocessing start method configurable on
TrainPipelineConfig and default it to 'spawn'.
The previous default (fork on Linux) is unsafe with libraries that hold
non-fork-safe state in the parent process — common ones in this codebase
are PyAV, torchcodec, and the ffmpeg shared libs they wrap. Symptoms
reported in #2488, #2209, and observed locally include:
- multiprocessing.context.AuthenticationError: digest received was wrong
- RuntimeError: Pin memory thread exited unexpectedly
- RuntimeError: DataLoader worker exited unexpectedly
- Random SIGSEGV inside worker processes during video decode
Switching to spawn re-imports modules cleanly in each worker and
eliminates these failure modes. Added the setting as a config field
rather than hard-coding so users on platforms where fork is preferred
can opt back in via --dataloader-multiprocessing-context=fork.
* Address review: shorten config comment, note spawn startup tradeoff
Per @jashshah999, mention that spawn workers re-import modules and so
add some startup time vs fork. Also trim the failure-mode dump from
the inline comment — the linked issue covers the symptoms in detail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(scripts): add multiprocessing_context safeguards
* chore(config): add libs note
---------
Co-authored-by: 0o8o0-blip <0o8o0-blip@users.noreply.github.com>
Register the `lekiwi` robot module in `lerobot_replay.py` so episodes can be
replayed on a LeKiwi via `--robot.type=lekiwi_client`. The module is already
registered in `lerobot_calibrate.py` and `lerobot_setup_motors.py`; this fills
the gap so the replay CLI recognizes the same robot.
Replayed actions are loaded from the dataset as torch tensors, which
`json.dumps` cannot serialize when `LeKiwiClient.send_action` ships them over
ZMQ. Coerce each action value to a plain float before sending. This is scoped
to the LeKiwi network client and does not affect any other robot.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(record): connect teleoperator before robot to avoid watchdog jump
lerobot-record connected the robot before the teleoperator. A robot's
connect()/reset() can leave it holding a default pose under a firmware
watchdog (e.g. Unitree G1); if teleop.connect() (model loading, IK init,
network setup) then takes longer than that watchdog, the joints drop to
damping and the first send_action() makes the robot jump.
Swap the order so the teleoperator connects first, matching the ordering
already used in lerobot_teleoperate.py. Pure ordering fix, no API change.
Fixes#3684
* fix(record): trim comment and add connect-order regression test
Address review feedback on #3684:
- Trim the verbose ordering comment down to two lines.
- Add test_record_connects_teleop_before_robot to tests/test_control_robot.py,
asserting teleop.connect() runs before robot.connect() in record().
* chore(test): remove test
---------
Co-authored-by: Jaimin Patel <jpatel@tuvalabs.com>
Co-authored-by: Martino Russi <77496684+nepyope@users.noreply.github.com>
* fix(scripts/eval): restore policy training mode after eval_policy()
`eval_policy` calls `policy.eval()` before the rollout but never restores
the prior mode on return. When called from the training loop
(`lerobot_train.py`'s `eval_policy_all -> run_one -> eval_one ->
eval_policy` chain), the policy is left in eval mode for every subsequent
training step, which silently:
* disables Dropout (no regularisation),
* freezes BatchNorm running stats (no further EMA updates).
Under DDP only `is_main_process` runs eval (lerobot_train.py:527), so the
main rank ends up in eval mode while workers stay in train mode — the
all-reduced gradients then combine forward passes computed with different
dropout masks and different BN behaviour, a real DDP-correctness issue.
Scope of impact:
* Affects every policy with Dropout in its forward path. In-tree, that
includes the default ACT (6 Dropout layers at p=0.1), Diffusion (vision
backbone), VQ-BeT, Multi-Task DiT, X-VLA, plus all VLA policies that
inherit Dropout from their pretrained HF backbone (PI0/PI0.5/PI0-FAST,
SmolVLA, GR00T-N1.5, EO1, Wall-X).
* Triggers from the first eval onward. On the default config
(steps=100k, eval_freq=20k) that's 80% of training; on the LIBERO /
RoboCasa / VLABench example commands in docs/ (eval_freq=1k–5k)
it's 95–99% of training.
* Policies using only LayerNorm/GroupNorm and no Dropout (TDMPC, RTC)
are unaffected. Policies using `FrozenBatchNorm2d` (ACT's ResNet
backbone) are immune to the BN-stat half; the Dropout half still bites.
Fix:
* Snapshot `policy.training` on entry to `eval_policy`.
* Restore it on normal return.
* Save-and-restore is a strict no-op for callers that pass an
already-eval-mode policy (e.g. the standalone `lerobot-eval` script
loading a frozen checkpoint).
* Restoration is placed before the normal return only, not in a
try/finally — exception paths leave the policy in eval mode, same as
today. A try/finally upgrade would require re-indenting ~165 lines and
can land as a separate cleanup if desired.
Tests (tests/scripts/test_eval.py, 7 tests total, ~1.6s):
* Regression gates on the lerobot_eval fix itself: training-mode
preservation, eval-mode preservation, dropout-active behavioural
check, non-crash for both entry modes.
* Quantitative mechanism demonstration
(`test_missing_mode_restoration_hurts_generalisation`): trains a tiny
Dropout+BatchNorm MLP under both the bug pattern and the fix pattern
on identical data and seed, then asserts the buggy variant generalises
at least 5% worse on a held-out val set. In repeated runs we see
10-25% deltas on this toy problem; real policies (more layers, more
Dropout, longer training) generally see larger gaps. Lives alongside
the regression tests so the empirical proof is reproducible from the
repo without adding a separate benchmarks/ directory.
* fix(scripts): keep policy train/eval
---------
Co-authored-by: ModeEric <ericjm4@illinois.edu>
* (depth image processing): excluding depth frames from the RGB to BGR image processing
* test(update): updating tests to include RGB/BGR conversion checks
* fix(deps): bump diffusers cap to <0.39.0 (security)
Diffusers 0.35.x is affected by GHSA-98h9-4798-4q5v (HIGH, CVSS 8.8):
'trust_remote_code bypass via custom_pipeline and local custom components'.
Fixed in diffusers 0.38.0.
Current cap 'diffusers<0.36.0' blocks downstream consumers (e.g.
strands-labs/robots) from picking up the security fix.
The lerobot diffusers surface area is narrow and stable across 0.36-0.38:
- diffusers.schedulers.scheduling_ddim.DDIMScheduler
- diffusers.schedulers.scheduling_ddpm.DDPMScheduler
- diffusers.optimization.get_scheduler
- diffusers.ConfigMixin / ModelMixin / register_to_config
- diffusers.models.attention.{Attention,FeedForward}
- diffusers.models.embeddings.*
None of these were removed, renamed, or had breaking changes in 0.36, 0.37,
or 0.38 release notes. Bumping the cap to <0.39.0 unblocks the security
fix while keeping a major-version safety bound.
* chore(dependecies): bump diffusers
* chore(deps): update uv.lock
---------
Co-authored-by: Cagatay Cali <cagataycali@users.noreply.github.com>
* Make SO follower P coefficient configurable
* chore(test): minimize tests
* feat(robots): expose PID coeff in SO arms
---------
Co-authored-by: taivu1998 <46636857+taivu1998@users.noreply.github.com>
* chore(agents): adding additional infos to AGENTS.md
* adding `lerobot-train` requirement inside PR checklist
* prefer using code already implemented from transformers / diffusers instead of re-implementing in tree
---------
Signed-off-by: Maxime Ellerbach <maxime.ellerbach@huggingface.co>
* fix(envs): set LiberoEnvConfig.fps default to 20 to match robosuite
LiberoEnvConfig.fps was set to 30, but the underlying robosuite
OffScreenRenderEnv always runs at its default control_freq of 20 Hz
since fps is never passed through. This mismatch silently decouples
the dataset/eval loop rate from the actual simulation step rate.
Set the default to 20 to match the real sim rate and avoid the
footgun.
Fixes#3368
* fix(libero): apply configured control frequency
---------
Co-authored-by: xinmotlanthua <275663218+xinmotlanthua@users.noreply.github.com>
* fix(damiao): make is_calibrated a plain property, not cached
`is_calibrated` was a `@cached_property`, so it froze at its first-read
value and never reflected later changes to `self.calibration` (set by
connect/calibrate/load). This caused the OpenArm teleop to re-run
calibration even when a calibration file existed, and to skip
`set_zero_position()` after a fresh calibration.
Switch to `@property` (matching the MotorsBus base contract and the
Feetech/SO-100 buses) and drop the now-unused `functools.cached_property`
import.
Co-authored-by: Cursor <cursoragent@cursor.com>
* don't set_zero_position() on connect
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(diffusion): add gradient checkpointing for memory optimization
Add gradient_checkpointing config option to DiffusionPolicy. When
enabled, wraps the UNet encoder, mid, and decoder residual blocks
with torch.utils.checkpoint.checkpoint to trade compute for memory.
Allows training with larger batch sizes or higher-resolution inputs
on memory-constrained GPUs. Disabled by default.
Usage: --policy.gradient_checkpointing=true
Part of the 0.6.0 roadmap item 3.3 (gradient checkpointing for all
policies).
* test(diffusion): verify gradient checkpointing parity
---------
Co-authored-by: Jash Shah <jashshah.999@gmail.com>
* gamepad does often work on macos
* review comments
* fix(gamepad): expose hidapi fallback in config
---------
Co-authored-by: Maxim Bonnaerens <maxim@bonnaerens.be>
* feat(annotate): run lerobot-annotate on HF Jobs via --job.target
Annotation needed a hand-edited launcher script (examples/annotations/run_hf_job.py)
to reach a GPU: users copied it, rewrote the embedded CMD string for their dataset,
and ran it with `python`. Fold that into the CLI instead, mirroring `lerobot-train`:
`lerobot-annotate --job.target=h200` submits the exact command you'd run locally.
- AnnotationJobConfig extends JobConfig with the annotation runtime's defaults
(vllm/vllm-openai image, 2h cap) plus --job.lerobot_ref, so an unmerged branch
can be exercised remotely without editing a script.
- lerobot.jobs.annotate builds the pod command by replaying the user's own CLI
flags (minus --job.*/--root, with --repo_id re-emitted from the config) after a
setup prelude that installs lerobot on top of the vLLM image. Job monitoring,
log tailing and Ctrl-C-detaches reuse the training submitter's plumbing.
- Remote runs require --repo_id; a local-only dataset is pushed privately first.
The generated pod command is byte-for-byte the script's old CMD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(annotate): reject client-side config files on remote runs
draccus exposes `--config_path` plus a `--<field>` config-file arg for every
nested dataclass (`--vlm`, `--plan`, `--job`, ...). All name files on the
client's disk, so forwarding them to the pod silently dropped whatever settings
they carried. Reject them up front instead.
Bare `--job` also slipped past the `--job.` prefix filter, so a `--job=cfg.yaml`
holding `target: h200` would have reached the pod and had the job submit a job
of its own, recursively. It is dropped from the forwarded args as well.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(jobs): share the submit-and-follow loop between both submitters
`submit_annotate_to_hf` reused the leaf helpers (`_poll_until_done`, `_tail_logs`,
`_pod_forwarded_args`) but duplicated the orchestration around them: ~40 of the 50
lines that spawn the poll/log threads, install the Ctrl-C-detaches handler and
raise on a non-COMPLETED stage were identical in both files.
Extract that into `follow_job(job_id, *, detach, success_marker=None) -> bool`,
returning True when the job finished and False when we stopped watching without a
verdict (detach or Ctrl-C). Training keeps its model-pushed marker by passing it in;
annotation has no equivalent line (the CLI keeps working after the upload log to
write the card and tag) so its completion stays stage-based.
Kept in hf.py rather than a new module so every existing monkeypatch target in
test_hf.py still resolves.
Behaviour change: a training run whose job reaches COMPLETED without the marker
matching now prints its completion line instead of returning silently. The marker
was already documented as an optimisation with a stage-based fallback; the fallback
just never reported success.
Tests: adds annotate coverage for the non-detach path (completion and failure) —
previously only ever exercised with detach=true — plus a detach short-circuit test.
Both new annotate tests verified to fail under a mutation that stubs out follow_job.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Steven Palma