Drop the ``sonic_token_action`` config flag; the 64-D latent-token
action/observation interface now switches on automatically whenever the
SONIC whole-body controller is selected (``controller == "SonicWholeBodyController"``).
Keyed via a ``_sonic_token`` property so client, onboard and sim roles agree.
* docs: fix code fences in env processor guide
The "Flexibility and Reusability" section wrapped a duplicated example
in a four-backtick fence and left a following block unclosed, so the
stray closing fence matched a later block. Everything in between
rendered as one code block that swallowed the surrounding prose.
Remove the duplicated block, add the missing closing fence after the
first example, and normalize the four-backtick fences to three so all
fences pair correctly.
* docs(pi0fast): fix typo 40kk -> 40k steps
* docs(integrate-hardware): fix so101 follower source link
* docs(hope_jr): fix dataset example link
The "example" link in the Record section pointed at the dataset's
`/settings` page, which returns HTTP 403 for readers. Drop the
`/settings` suffix so it links to the public dataset page the
sentence describes.
* docs(lekiwi): render emoji shortcodes as unicode
MDX does not expand `🤗` / `🤖` shortcodes, so they showed as
literal text in the rendered install step. Replace them with the 🤗 and
🤖 unicode characters, matching how the other robot pages write emoji.
* docs(smolvla): anchor record link to its section
The "Record a dataset" link dropped readers at the top of the
il_robots page instead of the relevant section. Point it at the
`#record-a-dataset` anchor (the `## Record a dataset` heading in
il_robots.mdx) so the link lands on the step it names.
* fix(utils): validate precise_sleep spin/margin args
Negative spin_threshold/sleep_margin make remaining arithmetic wrong
and can overshoot. Reject them early; cover the no-op path.
* test: drop flaky wall-clock assertion in no-op test
Per review: the 50ms wall-clock check can exceed its bound on a preempted
CI worker even when precise_sleep returns immediately. The direct calls
already exercise the non-positive no-op path, so the assertion is redundant.
* chore(tests): remove precise_sleep test negative values
---------
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
* fix(utils): allow any JSON payload in write_json
The dict-only type stub blocked lists/scalars callers already dump.
Accept Any, set utf-8 encoding, and cover list roundtrip.
* fix(utils): json type
---------
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
* fix(utils): mark Transition.complementary_info NotRequired
TypedDict class-body ``= None`` does not make a key optional and confuses
type checkers. Use ``NotRequired[...]`` so transitions without metadata
are valid.
* refactor(utils): complete NotRequired
---------
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
* fix(utils): raise ValueError from get_safe_torch_device
Bare asserts vanish under python -O and look like programmer bugs.
Convert unavailable CUDA/MPS/XPU requests into clear ValueErrors.
* style: combine nested with in device util tests (ruff)
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(utils): reject zero-norm / invalid quaternions in Rotation
Zero or non-finite inputs previously slipped through and produced NaN
rotation matrices on later convert/apply. Validate shape and scept for
norm > 0 before normalizing.
* fix(teleop): degrade phone AR quat parse like missing pose
Address review on #3988: Rotation.from_quat now rejects zero/NaN
quaternions. Wrap HEBI iOS ARKit permission in ValueError and return the
existing (False, None, None, None) path so teleop does not die mid-session
before tracking is ready.
* style: ruff format long ValueError in rotation.py
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* Connect, test and disconnected camera instances sequentially
* Add warmup-s cli argument to lerobot-find-cameras script
* Reduce default record time from 6 to 2 seconds in find_cameras
* Annotate return value of save_image function
* Initialize logging configuration in find_cameras
* fix(robot): type FK-to-EE action features as ACTION not STATE
ForwardKinematicsJointsToEEAction.transform_features declared its
end-effector action features (ee.x/y/z/wx/wy/wz/gripper_pos) with
FeatureType.STATE, copied verbatim from the sibling
ForwardKinematicsJointsToEEObservation (where STATE is correct for
OBSERVATION features). Every other action-producing step in this file
(EEReferenceAndDelta, InverseKinematicsEEToJoints, InverseKinematicsRLStep)
types its ACTION-bucket features as FeatureType.ACTION.
The mismatch mis-classifies the converted EE actions as state, which
propagates a wrong feature schema to downstream consumers keyed on
FeatureType (e.g. normalization norm_map, policy input/output feature
classification).
* test(robot): FK-to-EE step feature-type contract (action vs observation)
Asserts ForwardKinematicsJointsToEEAction emits EE features in the ACTION
bucket typed FeatureType.ACTION, and ForwardKinematicsJointsToEEObservation
emits them in the OBSERVATION bucket typed FeatureType.STATE.
* chore: delete user file
* chore(processor): reduce verbosity
---------
Co-authored-by: Jaagat-P <jaagatp05@gmail.com>
* fix(evo1): re-pad normalizer stats when loading from checkpoint
reconcile_evo1_processors did not re-pad the (un)normalizer stats to
max_state_dim/max_action_dim on the checkpoint-load path. When
lerobot-train loads a checkpoint (e.g. stage2 from a stage1 checkpoint)
it injects the raw dataset stats via processor overrides, so LIBERO's
8-dim state stats normalized a 24-dim padded state and crashed with
"size of tensor a (24) must match tensor b (8)".
Restore _refresh_evo1_normalization_steps (removed in the "remove legacy
codepaths" refactor) and call it from reconcile_evo1_processors so the
loaded stats/features are re-padded to EVO1's fixed widths. Padding is a
no-op when stats are already at the target width.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(evo1): cover reconcile re-padding of overridden normalizer stats
Regression test for the stage2-from-checkpoint crash: reloading a
checkpoint with raw (unpadded) dataset stats injected via processor
overrides must be re-padded to max_state_dim/max_action_dim by
reconcile_evo1_processors, otherwise normalizing the padded state
raises a shape mismatch.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Martino Russi <martino@huggingface.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Revert pyproject.toml and import_utils.py to main: the onnxruntime-gpu
detection fallback and the _onnxruntime_available/_onnx_available flags
are unnecessary. Controllers now import onnx/onnxruntime unconditionally
(matching main) which also works with onnxruntime-gpu since the import
name is stable. Drop the require_package() calls we had added.
Remove the stdin e-stop listener from serve_onboard_controller and the
now-unused os/sys imports; Ctrl-C still triggers graceful shutdown.
* fix(robots): retry SO follower/leader bus reads on transient Feetech errors
SO-100/SO-101 teleoperation aborts when a sync_read of Present_Position
returns a corrupted status packet ("Incorrect status packet!"), which the
Feetech bus emits intermittently under load. The read path already supports
a num_retry argument but the SO follower and leader never used it, so a single
transient failure crashed the control loop.
Add a max_read_retry config option (default 3) to SOFollowerConfig and
SOLeaderConfig and forward it to every Present_Position sync_read. Retries are
immediate and only happen on failure, so the steady-state read cost is
unchanged; set max_read_retry=0 to restore the previous behavior.
Fixes#3131
* chore(robots): change defaults
---------
Co-authored-by: isaka1022 <isaka1022@gmail.com>
* perf(docker): split dep install from src copy for CI layer caching
Install third-party deps (torch + all extras) in a layer keyed only on
pyproject.toml + uv.lock via --no-install-project, then copy src and install
the local package. Editing src/ no longer busts the heavy dependency layer,
so BuildKit layer cache hits across CI builds.
Applied to both Dockerfile.user and Dockerfile.internal.
* chore(ci): less verbose comments + copy all files
---------
Co-authored-by: dongmao.zhang <dongmao.zhang@bytedance.com>
* fix(datasets): bound memory of augment_dataset_quantile_stats by sampling frames
Per-episode stats previously materialized every frame (and decoded up to 16
episodes in parallel), so peak memory scaled with episode length and OOM'd on
large datasets (#2889). Numeric features are now read in full from the table
(exact), while only image/video frames are sub-sampled per episode using the
existing sample_indices heuristic. Worker count is configurable via
LEROBOT_STATS_MAX_WORKERS; --no-sampling restores exact behavior.
* Update tests/datasets/test_augment_quantile_stats.py
Co-authored-by: Haoming Song <1847575517@qq.com>
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
---------
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Haoming Song <1847575517@qq.com>
* fix(eval): align seed list length with num_envs when unseeded
eval_policy appended a single None per batch to all_seeds on the unseeded path while the reward and success lists grew by num_envs. The per-episode zip(..., strict=True) then raised ValueError for num_envs > 1. Extend all_seeds by num_envs so the lists stay aligned.
* chore(tests): delete lerobot_eval test
---------
Co-authored-by: Devin Lai <markauto75@gmail.com>
* fix: handle missing/unresponsive TTS on Linux
spd-say may be installed but hang indefinitely when speech-dispatcher
is not running. Add a 5s timeout and catch TimeoutExpired alongside
FileNotFoundError so recording continues without audio.
* chore(utils): add log warning for say
---------
Co-authored-by: Jiwen Cai <jiwenc@nvidia.com>
* refactor(logging): replace print with logger in lerobot_info
* refactor(logging): replace print with logger in convert_dataset_v21_to_v30
* refactor(logging): replace print with logger in lerobot_annotate
* refactor(logging): replace print with logger in lerobot_dataset_viz
* refactor(logging): replace print with logger in lerobot_eval
* refactor(logging): replace print with logger in lerobot_find_cameras
* refactor(logging): replace print with logger in lerobot_find_joint_limits
* refactor(logging): replace print with logger in lerobot_find_port
* refactor(logging): replace print with logger in lerobot_imgtransform_viz
* refactor(logging): replace print with logger in lerobot_setup_can
* refactor(logging): replace print with logger in lerobot_teleoperate
* refactor(logging): replace print with logger in lerobot_train_tokenizer
* fix(logging): preserve CLI output semantics
---------
Co-authored-by: ailisilob <2248345706@qq.com>
Register three X-series Dynamixel models so they can be driven by
DynamixelMotorsBus: XH540-W150 (model 1110), XC330-T288 (1220) and
XC330-T181 (1210). All are standard Protocol 2.0 X-series motors that
share the existing X_SERIES control/baudrate/encoding tables and 4096
resolution; only the model number and operating-mode list are
model-specific. Values verified against the ROBOTIS e-manual.
These motors are used by the ROBOTIS OMY-L100 arm, among others.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* feat(record): add `--dataset.no_stamp` to opt out of repo_id timestamping
stamp_repo_id() unconditionally appended a date-time tag to repo_id for every new (non-resume) dataset, so users managing their own versioned repo names (e.g. for a later lerobot-edit-dataset merge) could not opt out. Add a no_stamp field to DatasetRecordConfig and make stamp_repo_id() a no-op when it is set. The flag covers both lerobot-record and lerobot-rollout since they share this config, and no call-site changes are needed. Fixes#3722.
* chore(tests): delete dataset config test
---------
Co-authored-by: Philipp Sinitsin <ph.sinitsin@gmail.com>
* feat(camera): add manual exposure, gain, and white balance options for RealSense cameras
The RealSense camera integration lacked sensor-level controls, causing
issues like unstable lighting from auto-exposure hunting. This adds
optional `exposure`, `gain`, and `white_balance` fields to
RealSenseCameraConfig that disable the corresponding auto modes and
apply fixed values when set.
* fix: support D405 stereo module for sensor options
D405 exposes color stream via "Stereo Module", not "RGB Camera".
Fall back to Stereo Module when RGB Camera is not found.
* test(camera): add unit tests + range-aware errors for RealSense sensor options
Address PR #3220 review:
- Wrap set_option calls; re-raise ValueError with option name, value,
and sensor.get_option_range() diagnostics on out-of-range values.
- Add unit tests for _get_color_sensor (RGB Camera, D405 Stereo Module
fallback, diagnostic error) and _configure_sensor_options (no-op,
all values, unsupported warns, partial config, out-of-range raise).
* fix(realsense): validate manual color controls
* refactor(camera): apply feedback
---------
Co-authored-by: Lev Kozlov <kozlov.l.a10@gmail.com>
* feat(lekiwi): support LeKiwi in the rollout/eval CLI
Register the lekiwi robot in lerobot_rollout.py so policies can be rolled out
on a LeKiwi, and keep base-velocity (.vel) features in build_rollout_context.
LeKiwi's observation.state and action are 9-dim (6 arm .pos + x/y/theta.vel)
and the policy is normalized on all 9. The old filter kept only .pos features,
so it fed a 6-dim vector into a 9-dim normalizer (RuntimeError, size 6 vs 9) and
silently dropped the base velocities from the action, leaving the base unable to
move. Keeping both .pos and .vel fixes both. Pure-arm robots have no .vel keys,
so this is a no-op for them.
* style: format LeKiwi rollout action features
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
* Add dataloader_multiprocessing_context, default to spawn
Make the DataLoader multiprocessing start method configurable on
TrainPipelineConfig and default it to 'spawn'.
The previous default (fork on Linux) is unsafe with libraries that hold
non-fork-safe state in the parent process — common ones in this codebase
are PyAV, torchcodec, and the ffmpeg shared libs they wrap. Symptoms
reported in #2488, #2209, and observed locally include:
- multiprocessing.context.AuthenticationError: digest received was wrong
- RuntimeError: Pin memory thread exited unexpectedly
- RuntimeError: DataLoader worker exited unexpectedly
- Random SIGSEGV inside worker processes during video decode
Switching to spawn re-imports modules cleanly in each worker and
eliminates these failure modes. Added the setting as a config field
rather than hard-coding so users on platforms where fork is preferred
can opt back in via --dataloader-multiprocessing-context=fork.
* Address review: shorten config comment, note spawn startup tradeoff
Per @jashshah999, mention that spawn workers re-import modules and so
add some startup time vs fork. Also trim the failure-mode dump from
the inline comment — the linked issue covers the symptoms in detail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(scripts): add multiprocessing_context safeguards
* chore(config): add libs note
---------
Co-authored-by: 0o8o0-blip <0o8o0-blip@users.noreply.github.com>
Register the `lekiwi` robot module in `lerobot_replay.py` so episodes can be
replayed on a LeKiwi via `--robot.type=lekiwi_client`. The module is already
registered in `lerobot_calibrate.py` and `lerobot_setup_motors.py`; this fills
the gap so the replay CLI recognizes the same robot.
Replayed actions are loaded from the dataset as torch tensors, which
`json.dumps` cannot serialize when `LeKiwiClient.send_action` ships them over
ZMQ. Coerce each action value to a plain float before sending. This is scoped
to the LeKiwi network client and does not affect any other robot.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(record): connect teleoperator before robot to avoid watchdog jump
lerobot-record connected the robot before the teleoperator. A robot's
connect()/reset() can leave it holding a default pose under a firmware
watchdog (e.g. Unitree G1); if teleop.connect() (model loading, IK init,
network setup) then takes longer than that watchdog, the joints drop to
damping and the first send_action() makes the robot jump.
Swap the order so the teleoperator connects first, matching the ordering
already used in lerobot_teleoperate.py. Pure ordering fix, no API change.
Fixes#3684
* fix(record): trim comment and add connect-order regression test
Address review feedback on #3684:
- Trim the verbose ordering comment down to two lines.
- Add test_record_connects_teleop_before_robot to tests/test_control_robot.py,
asserting teleop.connect() runs before robot.connect() in record().
* chore(test): remove test
---------
Co-authored-by: Jaimin Patel <jpatel@tuvalabs.com>
Co-authored-by: Martino Russi <77496684+nepyope@users.noreply.github.com>
* fix(scripts/eval): restore policy training mode after eval_policy()
`eval_policy` calls `policy.eval()` before the rollout but never restores
the prior mode on return. When called from the training loop
(`lerobot_train.py`'s `eval_policy_all -> run_one -> eval_one ->
eval_policy` chain), the policy is left in eval mode for every subsequent
training step, which silently:
* disables Dropout (no regularisation),
* freezes BatchNorm running stats (no further EMA updates).
Under DDP only `is_main_process` runs eval (lerobot_train.py:527), so the
main rank ends up in eval mode while workers stay in train mode — the
all-reduced gradients then combine forward passes computed with different
dropout masks and different BN behaviour, a real DDP-correctness issue.
Scope of impact:
* Affects every policy with Dropout in its forward path. In-tree, that
includes the default ACT (6 Dropout layers at p=0.1), Diffusion (vision
backbone), VQ-BeT, Multi-Task DiT, X-VLA, plus all VLA policies that
inherit Dropout from their pretrained HF backbone (PI0/PI0.5/PI0-FAST,
SmolVLA, GR00T-N1.5, EO1, Wall-X).
* Triggers from the first eval onward. On the default config
(steps=100k, eval_freq=20k) that's 80% of training; on the LIBERO /
RoboCasa / VLABench example commands in docs/ (eval_freq=1k–5k)
it's 95–99% of training.
* Policies using only LayerNorm/GroupNorm and no Dropout (TDMPC, RTC)
are unaffected. Policies using `FrozenBatchNorm2d` (ACT's ResNet
backbone) are immune to the BN-stat half; the Dropout half still bites.
Fix:
* Snapshot `policy.training` on entry to `eval_policy`.
* Restore it on normal return.
* Save-and-restore is a strict no-op for callers that pass an
already-eval-mode policy (e.g. the standalone `lerobot-eval` script
loading a frozen checkpoint).
* Restoration is placed before the normal return only, not in a
try/finally — exception paths leave the policy in eval mode, same as
today. A try/finally upgrade would require re-indenting ~165 lines and
can land as a separate cleanup if desired.
Tests (tests/scripts/test_eval.py, 7 tests total, ~1.6s):
* Regression gates on the lerobot_eval fix itself: training-mode
preservation, eval-mode preservation, dropout-active behavioural
check, non-crash for both entry modes.
* Quantitative mechanism demonstration
(`test_missing_mode_restoration_hurts_generalisation`): trains a tiny
Dropout+BatchNorm MLP under both the bug pattern and the fix pattern
on identical data and seed, then asserts the buggy variant generalises
at least 5% worse on a held-out val set. In repeated runs we see
10-25% deltas on this toy problem; real policies (more layers, more
Dropout, longer training) generally see larger gaps. Lives alongside
the regression tests so the empirical proof is reproducible from the
repo without adding a separate benchmarks/ directory.
* fix(scripts): keep policy train/eval
---------
Co-authored-by: ModeEric <ericjm4@illinois.edu>
Martino Russi