* Add dataloader_multiprocessing_context, default to spawn
Make the DataLoader multiprocessing start method configurable on
TrainPipelineConfig and default it to 'spawn'.
The previous default (fork on Linux) is unsafe with libraries that hold
non-fork-safe state in the parent process — common ones in this codebase
are PyAV, torchcodec, and the ffmpeg shared libs they wrap. Symptoms
reported in #2488, #2209, and observed locally include:
- multiprocessing.context.AuthenticationError: digest received was wrong
- RuntimeError: Pin memory thread exited unexpectedly
- RuntimeError: DataLoader worker exited unexpectedly
- Random SIGSEGV inside worker processes during video decode
Switching to spawn re-imports modules cleanly in each worker and
eliminates these failure modes. Added the setting as a config field
rather than hard-coding so users on platforms where fork is preferred
can opt back in via --dataloader-multiprocessing-context=fork.
* Address review: shorten config comment, note spawn startup tradeoff
Per @jashshah999, mention that spawn workers re-import modules and so
add some startup time vs fork. Also trim the failure-mode dump from
the inline comment — the linked issue covers the symptoms in detail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(scripts): add multiprocessing_context safeguards
* chore(config): add libs note
---------
Co-authored-by: 0o8o0-blip <0o8o0-blip@users.noreply.github.com>
Register the `lekiwi` robot module in `lerobot_replay.py` so episodes can be
replayed on a LeKiwi via `--robot.type=lekiwi_client`. The module is already
registered in `lerobot_calibrate.py` and `lerobot_setup_motors.py`; this fills
the gap so the replay CLI recognizes the same robot.
Replayed actions are loaded from the dataset as torch tensors, which
`json.dumps` cannot serialize when `LeKiwiClient.send_action` ships them over
ZMQ. Coerce each action value to a plain float before sending. This is scoped
to the LeKiwi network client and does not affect any other robot.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(record): connect teleoperator before robot to avoid watchdog jump
lerobot-record connected the robot before the teleoperator. A robot's
connect()/reset() can leave it holding a default pose under a firmware
watchdog (e.g. Unitree G1); if teleop.connect() (model loading, IK init,
network setup) then takes longer than that watchdog, the joints drop to
damping and the first send_action() makes the robot jump.
Swap the order so the teleoperator connects first, matching the ordering
already used in lerobot_teleoperate.py. Pure ordering fix, no API change.
Fixes#3684
* fix(record): trim comment and add connect-order regression test
Address review feedback on #3684:
- Trim the verbose ordering comment down to two lines.
- Add test_record_connects_teleop_before_robot to tests/test_control_robot.py,
asserting teleop.connect() runs before robot.connect() in record().
* chore(test): remove test
---------
Co-authored-by: Jaimin Patel <jpatel@tuvalabs.com>
Co-authored-by: Martino Russi <77496684+nepyope@users.noreply.github.com>
* fix(scripts/eval): restore policy training mode after eval_policy()
`eval_policy` calls `policy.eval()` before the rollout but never restores
the prior mode on return. When called from the training loop
(`lerobot_train.py`'s `eval_policy_all -> run_one -> eval_one ->
eval_policy` chain), the policy is left in eval mode for every subsequent
training step, which silently:
* disables Dropout (no regularisation),
* freezes BatchNorm running stats (no further EMA updates).
Under DDP only `is_main_process` runs eval (lerobot_train.py:527), so the
main rank ends up in eval mode while workers stay in train mode — the
all-reduced gradients then combine forward passes computed with different
dropout masks and different BN behaviour, a real DDP-correctness issue.
Scope of impact:
* Affects every policy with Dropout in its forward path. In-tree, that
includes the default ACT (6 Dropout layers at p=0.1), Diffusion (vision
backbone), VQ-BeT, Multi-Task DiT, X-VLA, plus all VLA policies that
inherit Dropout from their pretrained HF backbone (PI0/PI0.5/PI0-FAST,
SmolVLA, GR00T-N1.5, EO1, Wall-X).
* Triggers from the first eval onward. On the default config
(steps=100k, eval_freq=20k) that's 80% of training; on the LIBERO /
RoboCasa / VLABench example commands in docs/ (eval_freq=1k–5k)
it's 95–99% of training.
* Policies using only LayerNorm/GroupNorm and no Dropout (TDMPC, RTC)
are unaffected. Policies using `FrozenBatchNorm2d` (ACT's ResNet
backbone) are immune to the BN-stat half; the Dropout half still bites.
Fix:
* Snapshot `policy.training` on entry to `eval_policy`.
* Restore it on normal return.
* Save-and-restore is a strict no-op for callers that pass an
already-eval-mode policy (e.g. the standalone `lerobot-eval` script
loading a frozen checkpoint).
* Restoration is placed before the normal return only, not in a
try/finally — exception paths leave the policy in eval mode, same as
today. A try/finally upgrade would require re-indenting ~165 lines and
can land as a separate cleanup if desired.
Tests (tests/scripts/test_eval.py, 7 tests total, ~1.6s):
* Regression gates on the lerobot_eval fix itself: training-mode
preservation, eval-mode preservation, dropout-active behavioural
check, non-crash for both entry modes.
* Quantitative mechanism demonstration
(`test_missing_mode_restoration_hurts_generalisation`): trains a tiny
Dropout+BatchNorm MLP under both the bug pattern and the fix pattern
on identical data and seed, then asserts the buggy variant generalises
at least 5% worse on a held-out val set. In repeated runs we see
10-25% deltas on this toy problem; real policies (more layers, more
Dropout, longer training) generally see larger gaps. Lives alongside
the regression tests so the empirical proof is reproducible from the
repo without adding a separate benchmarks/ directory.
* fix(scripts): keep policy train/eval
---------
Co-authored-by: ModeEric <ericjm4@illinois.edu>
The all-zero token is off the encoder's learned FSQ manifold and decodes to a
slightly goofy stance. Replace it with a NEUTRAL_TOKEN captured from the encoder's
own idle output in sim (stored as integer FSQ codes, rescaled by the encoder's
1/16 quantization step to an exact on-grid token). token_mode now seeds this
neutral, and the onboard sender starts observation.state from it so the first
inference sees the token the decoder is actually holding.
Move the token-hold idle logic into SonicWholeBodyController via a
token_mode flag (set by UnitreeG1 when sonic_token_action is enabled):
before any real token arrives the decoder is fed the all-zero neutral
token (stable neutral stance), and afterwards the last received token is
held between control ticks (the ~50 Hz control loop outruns the ~30 Hz
token stream). Living in the controller, this applies uniformly to
run_g1_onboard, lerobot-rollout and the sim replays, so the explicit
neutral seeding in run_g1_onboard is removed.
Run the whole-body controller (SONIC decoder / GR00T) onboard the G1 against
local DDS at full rate, with the laptop shipping only high-level actions over
ZMQ instead of 50Hz lowcmd via the socket bridge.
- config: add onboard, dds_interface, release_motion_control, physical_remote
- unitree_g1: onboard connect() branch (local DDS + MotionSwitcher release +
physical wireless remote), _release_motion_control, _wireless_remote_input,
controller-loop wireless priority; SDK channels when sim OR onboard
- run_g1_server: port Gripper/build_gripper/parse_camera_specs; add --cameras
spec supporting by-path device names (survive USB re-enumeration) + FOURCC
- run_g1_onboard: onboard entry point (ZMQ actions -> send_action), with a
--sonic-token-action flag for the 64-D latent-token interface
- infer_sonic_g1_onboard: laptop-side sender that runs nepyope/sonic_walk
(pi0.5) and PUSHes 64-D tokens to the onboard controller
Add a token-output VLA path (sonic_token_action) so a policy trained on 64-D SONIC
motion tokens (e.g. nepyope/sonic_walk) drives the decoder directly via lerobot-rollout:
the robot advertises a 64-D motion_token.{i}.pos action and echoes the last commanded
token as a 64-D observation.state (motion_token_state.{i}.pos), encoder bypassed.
Also:
- gr00t_locomotion: allow an external upper-body IK to override the 3 waist joints, and
cap ORT to 1 intra/inter thread so the 50Hz loop doesn't stutter under contention.
- sonic_pipeline: make_ort_session_options takes optional thread caps; report the
provider actually bound.
- unitree_g1: build the sim env with publish_images=False/cameras=[] to avoid the
offscreen EGL context crash (we drive image policies from recorded/live frames), and
guard the startup sim-step race (zero-norm pelvis quat) so the sim thread survives.
* (depth image processing): excluding depth frames from the RGB to BGR image processing
* test(update): updating tests to include RGB/BGR conversion checks
* fix(deps): bump diffusers cap to <0.39.0 (security)
Diffusers 0.35.x is affected by GHSA-98h9-4798-4q5v (HIGH, CVSS 8.8):
'trust_remote_code bypass via custom_pipeline and local custom components'.
Fixed in diffusers 0.38.0.
Current cap 'diffusers<0.36.0' blocks downstream consumers (e.g.
strands-labs/robots) from picking up the security fix.
The lerobot diffusers surface area is narrow and stable across 0.36-0.38:
- diffusers.schedulers.scheduling_ddim.DDIMScheduler
- diffusers.schedulers.scheduling_ddpm.DDPMScheduler
- diffusers.optimization.get_scheduler
- diffusers.ConfigMixin / ModelMixin / register_to_config
- diffusers.models.attention.{Attention,FeedForward}
- diffusers.models.embeddings.*
None of these were removed, renamed, or had breaking changes in 0.36, 0.37,
or 0.38 release notes. Bumping the cap to <0.39.0 unblocks the security
fix while keeping a major-version safety bound.
* chore(dependecies): bump diffusers
* chore(deps): update uv.lock
---------
Co-authored-by: Cagatay Cali <cagataycali@users.noreply.github.com>
* Make SO follower P coefficient configurable
* chore(test): minimize tests
* feat(robots): expose PID coeff in SO arms
---------
Co-authored-by: taivu1998 <46636857+taivu1998@users.noreply.github.com>
* chore(agents): adding additional infos to AGENTS.md
* adding `lerobot-train` requirement inside PR checklist
* prefer using code already implemented from transformers / diffusers instead of re-implementing in tree
---------
Signed-off-by: Maxime Ellerbach <maxime.ellerbach@huggingface.co>
- reset(): pause the background controller and, for full-body controllers,
publish the default pose directly (new _controller_paused flag) so reset and
the controller loop aren't both writing low commands.
- SONIC pipeline: add reset() to StandingEncoderDecoder and PlannerController
(clear token/proprio history/heading, rewind motion buffer); SonicRuntime.reset()
now calls controller.reset().
- sonic_whole_body: require the full dense 34-D command (no silent zero-fill of a
partial action) and integrate yaw-rate (idx 33) into heading.
- controllers/__init__: import the controller classes referenced in __all__.
- unitree_g1: lazy replay-frame decode + small cache instead of decoding all
frames up front; safer disconnect (longer controller-thread join + fail-safe
that skips the graceful ramp if the thread won't stop).
- lint: ruff-format config_unitree_g1 hand_closed_pose; prettier README table.
* fix(envs): set LiberoEnvConfig.fps default to 20 to match robosuite
LiberoEnvConfig.fps was set to 30, but the underlying robosuite
OffScreenRenderEnv always runs at its default control_freq of 20 Hz
since fps is never passed through. This mismatch silently decouples
the dataset/eval loop rate from the actual simulation step rate.
Set the default to 20 to match the real sim rate and avoid the
footgun.
Fixes#3368
* fix(libero): apply configured control frequency
---------
Co-authored-by: xinmotlanthua <275663218+xinmotlanthua@users.noreply.github.com>
* fix(damiao): make is_calibrated a plain property, not cached
`is_calibrated` was a `@cached_property`, so it froze at its first-read
value and never reflected later changes to `self.calibration` (set by
connect/calibrate/load). This caused the OpenArm teleop to re-run
calibration even when a calibration file existed, and to skip
`set_zero_position()` after a fresh calibration.
Switch to `@property` (matching the MotorsBus base contract and the
Feetech/SO-100 buses) and drop the now-unused `functools.cached_property`
import.
Co-authored-by: Cursor <cursoragent@cursor.com>
* don't set_zero_position() on connect
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(diffusion): add gradient checkpointing for memory optimization
Add gradient_checkpointing config option to DiffusionPolicy. When
enabled, wraps the UNet encoder, mid, and decoder residual blocks
with torch.utils.checkpoint.checkpoint to trade compute for memory.
Allows training with larger batch sizes or higher-resolution inputs
on memory-constrained GPUs. Disabled by default.
Usage: --policy.gradient_checkpointing=true
Part of the 0.6.0 roadmap item 3.3 (gradient checkpointing for all
policies).
* test(diffusion): verify gradient checkpointing parity
---------
Co-authored-by: Jash Shah <jashshah.999@gmail.com>
* gamepad does often work on macos
* review comments
* fix(gamepad): expose hidapi fallback in config
---------
Co-authored-by: Maxim Bonnaerens <maxim@bonnaerens.be>
* feat(annotate): run lerobot-annotate on HF Jobs via --job.target
Annotation needed a hand-edited launcher script (examples/annotations/run_hf_job.py)
to reach a GPU: users copied it, rewrote the embedded CMD string for their dataset,
and ran it with `python`. Fold that into the CLI instead, mirroring `lerobot-train`:
`lerobot-annotate --job.target=h200` submits the exact command you'd run locally.
- AnnotationJobConfig extends JobConfig with the annotation runtime's defaults
(vllm/vllm-openai image, 2h cap) plus --job.lerobot_ref, so an unmerged branch
can be exercised remotely without editing a script.
- lerobot.jobs.annotate builds the pod command by replaying the user's own CLI
flags (minus --job.*/--root, with --repo_id re-emitted from the config) after a
setup prelude that installs lerobot on top of the vLLM image. Job monitoring,
log tailing and Ctrl-C-detaches reuse the training submitter's plumbing.
- Remote runs require --repo_id; a local-only dataset is pushed privately first.
The generated pod command is byte-for-byte the script's old CMD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(annotate): reject client-side config files on remote runs
draccus exposes `--config_path` plus a `--<field>` config-file arg for every
nested dataclass (`--vlm`, `--plan`, `--job`, ...). All name files on the
client's disk, so forwarding them to the pod silently dropped whatever settings
they carried. Reject them up front instead.
Bare `--job` also slipped past the `--job.` prefix filter, so a `--job=cfg.yaml`
holding `target: h200` would have reached the pod and had the job submit a job
of its own, recursively. It is dropped from the forwarded args as well.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(jobs): share the submit-and-follow loop between both submitters
`submit_annotate_to_hf` reused the leaf helpers (`_poll_until_done`, `_tail_logs`,
`_pod_forwarded_args`) but duplicated the orchestration around them: ~40 of the 50
lines that spawn the poll/log threads, install the Ctrl-C-detaches handler and
raise on a non-COMPLETED stage were identical in both files.
Extract that into `follow_job(job_id, *, detach, success_marker=None) -> bool`,
returning True when the job finished and False when we stopped watching without a
verdict (detach or Ctrl-C). Training keeps its model-pushed marker by passing it in;
annotation has no equivalent line (the CLI keeps working after the upload log to
write the card and tag) so its completion stays stage-based.
Kept in hf.py rather than a new module so every existing monkeypatch target in
test_hf.py still resolves.
Behaviour change: a training run whose job reaches COMPLETED without the marker
matching now prints its completion line instead of returning silently. The marker
was already documented as an optimisation with a stage-based fallback; the fallback
just never reported success.
Tests: adds annotate coverage for the non-detach path (completion and failure) —
previously only ever exercised with detach=true — plus a detach short-circuit test.
Both new annotate tests verified to fail under a mutation that stubs out follow_job.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Strip everything except the OpenHLM/pi0.5 -> SONIC encoder/decoder rollout
path so this branch does exactly that and nothing more:
- Remove the SONIC motion planner (planner ONNX + subprocess worker, PlannerMotion,
replanning, MovementState/LocomotionMode, joystick) from sonic_pipeline; keep the
encoder/decoder and the caller-fed reference buffer (PlannerController) intact.
- Slim SonicRuntime to load only the encoder/decoder; SonicWholeBodyController now
runs solely the 34-D whole-body command path (drop SMPL/VR3/keyboard teleop).
- Delete the pico_headset teleoperator (SONIC's SMPL/VR3 teleop source).
- Move WB action constants into g1_utils; repoint imports.
GR00T/Holosoma locomotion controllers are left untouched.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a dense 34-D whole-body command path so lerobot-rollout can drive the
G1 directly with an OpenHLM / pi0.5 policy through the SONIC encoder/decoder:
- SonicWholeBodyController: wb.{i}.pos action interface, mode-0 reference with
a rolling 50-frame trajectory (finite-diff velocities) and first-tick anchor
init; correct MuJoCo->IsaacLab joint reordering.
- unitree_g1: expose 34-D wb_state.{i}.pos proprio; empty/replay camera feeds
for image-conditioned policies; Dex3 hand publishing from the grip scalars.
- g1_utils: obs_to_wb34_state + WB action constants.
Co-authored-by: Cursor <cursoragent@cursor.com>
Steven Palma