* feat(transforms): add 8 robotics-relevant image augmentations
Add GaussianNoise, MotionBlur, JPEGCompression, GaussianPatchBrightness,
RandomShadow, CoarseDropout, GammaCorrection, and PlanckianJitter.
Each transform addresses a real-world failure mode not covered by the
existing 6 defaults (sensor noise, motion blur, compression artifacts,
uneven lighting, cast shadows, partial occlusion, exposure variation,
color temperature shift).
All transforms are pure PyTorch, follow the make_params/transform
pattern, and integrate with ImageTransformConfig via a registry.
* add augmentation showcase image for PR
* update showcase with better sample frame
* tune showcase to balanced augmentation intensity
* tune showcase: softer shadow, dropout, jitter intensity
* refactor(transforms): several updates
* update image
* chore(media): remove example
* chore: add link to example
---------
Co-authored-by: Yuxian LI <liyuxian1358@gmail.com>
* Fix add_features for multi-dimensional per-frame features
* Use generic names in multidimensional add_features test
* tests(all shapes): enhancing tests to cover all possible features shapes
* chore(format): formatting code
---------
Co-authored-by: felixmin <felix.minze@live.de>
* Support task replacement mappings in modify_tasks
Part of #2326.
Signed-off-by: 陈伟 <woshei0a0a0a@qq.com>
* feat(task modification precedence): Improving task modification precedence so that all modes can be used in a single run. Adapting tests accordinginly.
* chore(fromat): formatting code
* docs(docstrings): updating docstrings
* docs(update): updating docs with the task modification features
---------
Signed-off-by: 陈伟 <woshei0a0a0a@qq.com>
Co-authored-by: vvezre <93599357+vvezre@users.noreply.github.com>
* docs: fix broken and placeholder links
Fix the two links called out in #4094 plus one more found while scanning
all docs: the NOTE:addlinktoLOC placeholder in async.mdx now points at
_aggregate_action_queues in robot_client.py; the dead
#mapping-videoencoderconfig--ffmpeg-options anchor in
video_encoding_parameters.mdx now points at get_codec_options in
video.py; and the stale ./cameras#setup-cameras fragment in il_robots.mdx
is dropped to link the cameras guide page.
Closes#4094.
* docs: apply prettier formatting to the video-encoding parameter table
Re-align the Markdown table columns with prettier (v3.6.2, --prose-wrap=preserve)
so the pre-commit 'Format Markdown with Prettier' hook passes. Whitespace only.
is_saving_step was the only step-frequency check without a `> 0` guard,
so save_freq=0 raised ZeroDivisionError from `step % 0` on the first
step. Route the decision through a should_save_checkpoint helper that
treats a non-positive save_freq as "save only the final checkpoint",
matching how log_freq/eval_freq handle non-positive values.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
The discard-path fix (#3683) deletes staging for all camera_keys in
clear_episode_buffer(). The post-save cleanup in save_episode() must keep
iterating image_keys only: with batch_encoding_size > 1 the video staging
frames of already-saved episodes stay on disk until the batch encoder
consumes and deletes them. Add a regression test pinning that behavior,
plus a comment explaining why the two paths differ.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Observations were sent as base64-in-JSON, which inflates every camera
frame 33% purely to fit binary JPEG inside a text protocol (~11 Mbps at
3 cameras x 30 Hz). Send a ZMQ multipart message instead: a JSON header
frame (state + camera order) followed by one raw JPEG frame per camera.
Benchmarked on hardware: -25% wire size (invariant across contention),
lower and tighter latency, and no dropped frames under load where the
base64 format stalled.
ZMQ_CONFLATE does not support multipart, so the observation sockets use
2-deep high-water marks; the client's existing drain-to-latest loop
preserves the keep-newest behavior.
Breaking wire change: host and client must run the same version.
Co-authored-by: Xingdong Zuo <18168681+zuoxingdong@users.noreply.github.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Without an explicit fourcc, OpenCV's V4L2 auto-negotiation selects
uncompressed YUYV when the camera offers it: ~147 Mbps per camera at
640x480@30 on the Pi's shared USB 2.0 bus, versus ~9 Mbps as MJPG for
identical frames. Two default cameras already exceed the bus, so frame
rates sag silently.
Pinning fourcc="MJPG" is backward compatible: it is an existing
OpenCVCameraConfig field, frames still arrive as BGR ndarrays, user
configs that set their own fourcc are unaffected, and a camera without
MJPG fails loudly at validation rather than degrading silently.
Co-authored-by: Xingdong Zuo <18168681+zuoxingdong@users.noreply.github.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(datasets): stop frame errors being treated as shard exhaustion in StreamingLeRobotDataset
StreamingLeRobotDataset.__iter__ caught every RuntimeError and treated all as exhausted shard. Real errors like video decode failure made each shard get dropped on the first frame, so iteration ended while yeilding zero frames with no errors.
Shard exahustion is StopIteration raised from make_frame generator, which python converts to RuntimeError with StopIteration as __cause__. Added check to tell StopIteration from everything else, consuming real shard exhaustion while re-raising everything else.
Added a test that injects a decode failure and asserts iteration raises instead of returning on empty stream.
Fixes#4066
* refactor(datasets): exception streaming
---------
Co-authored-by: Mohit Yadav <mohitydv09@gmail.com>
`can_peek_back(steps)` is documented to return whether `peek_back(steps)`
can be called "without raising an IndexError", but it guarded with `<=`:
return steps <= len(self._back_buf) + self._cursor
`peek_back(n)` needs n+1 buffered slots — it raises when
`n + 1 > len(self._back_buf) + self._cursor` and reads
`self._back_buf[self._cursor - (n + 1)]`. So at
`steps == len(self._back_buf) + self._cursor`, `can_peek_back` returns True
while `peek_back` raises LookBackError, contradicting the docstring.
Two siblings confirm the intended bound:
- `prev()` (one step back) requires `len(self._back_buf) + self._cursor > 1`.
- The forward twin is already consistent: `can_peek_ahead(n)` buffers n items
and `peek_ahead(n)` reads `_ahead_buf[n - 1]` (needs n).
Use `<` so `can_peek_back` matches `peek_back`'s guard exactly.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix: implement bimanual SO leader DAgger handover support
- Add feedback_features property: enables DAgger's teleop_supports_feedback() check
- Implement enable_torque()/disable_torque(): synchronized torque control for both arms
- Implement send_feedback(): routes bimanual feedback to left/right arms with prefix stripping
This fixes DAgger smooth handover for bimanual SO follower + SO leader setups:
when pausing from policy to human intervention, both leader arms now move smoothly
to the follower's current pose, avoiding discontinuities at the human takeover point.
* Update hil_data_collection.mdx
Signed-off-by: Baptiste Lubrano Lavadera <45080391+Mr-C4T@users.noreply.github.com>
* Update bi_so_leader.py
Signed-off-by: Baptiste Lubrano Lavadera <45080391+Mr-C4T@users.noreply.github.com>
---------
Signed-off-by: Baptiste Lubrano Lavadera <45080391+Mr-C4T@users.noreply.github.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(smolvla): freeze the intended VLM layers when train_expert_only=False
The partial-freeze patterns in set_requires_grad() used a
'text_model.model.' prefix that does not exist in SmolVLM parameter
names ('SmolVLMModel.text_model' is a bare LlamaModel, with no nested
'.model'). As a result the last VLM layer and the final norm were
silently left trainable, defeating the freeze that was added to avoid
unused-parameter errors with DDP; only lm_head was frozen by substring
luck.
Use the real flat names, and raise if any freeze pattern stops matching
so a future transformers renaming cannot silently reintroduce the bug.
Add a CPU regression test covering both last_layers branches.
Fixes#4018
* test(smolvla): drop regression test per review
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
serialize_torch_rng_state/deserialize_torch_rng_state only handled CPU
and CUDA generators. On MPS, resumed training was not bit-exact for any
stochastic op (dropout, ACT's CVAE noise) since the MPS generator's state
was never saved or restored. Mirrors the existing CUDA branch using
torch.mps.get_rng_state/set_rng_state (available since torch 2.11).
Note: get_rng_state()/set_rng_state() (used by seeded_context()) have the
same gap but are out of scope here — happy to follow up separately if
useful.
Co-authored-by: Sunny Dave <sunnydave@Sunnys-Mac-Studio.local>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* docs: fix code fences in env processor guide
The "Flexibility and Reusability" section wrapped a duplicated example
in a four-backtick fence and left a following block unclosed, so the
stray closing fence matched a later block. Everything in between
rendered as one code block that swallowed the surrounding prose.
Remove the duplicated block, add the missing closing fence after the
first example, and normalize the four-backtick fences to three so all
fences pair correctly.
* docs(pi0fast): fix typo 40kk -> 40k steps
* docs(integrate-hardware): fix so101 follower source link
* docs(hope_jr): fix dataset example link
The "example" link in the Record section pointed at the dataset's
`/settings` page, which returns HTTP 403 for readers. Drop the
`/settings` suffix so it links to the public dataset page the
sentence describes.
* docs(lekiwi): render emoji shortcodes as unicode
MDX does not expand `🤗` / `🤖` shortcodes, so they showed as
literal text in the rendered install step. Replace them with the 🤗 and
🤖 unicode characters, matching how the other robot pages write emoji.
* docs(smolvla): anchor record link to its section
The "Record a dataset" link dropped readers at the top of the
il_robots page instead of the relevant section. Point it at the
`#record-a-dataset` anchor (the `## Record a dataset` heading in
il_robots.mdx) so the link lands on the step it names.
* fix(utils): validate precise_sleep spin/margin args
Negative spin_threshold/sleep_margin make remaining arithmetic wrong
and can overshoot. Reject them early; cover the no-op path.
* test: drop flaky wall-clock assertion in no-op test
Per review: the 50ms wall-clock check can exceed its bound on a preempted
CI worker even when precise_sleep returns immediately. The direct calls
already exercise the non-positive no-op path, so the assertion is redundant.
* chore(tests): remove precise_sleep test negative values
---------
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
* fix(utils): allow any JSON payload in write_json
The dict-only type stub blocked lists/scalars callers already dump.
Accept Any, set utf-8 encoding, and cover list roundtrip.
* fix(utils): json type
---------
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
* fix(utils): mark Transition.complementary_info NotRequired
TypedDict class-body ``= None`` does not make a key optional and confuses
type checkers. Use ``NotRequired[...]`` so transitions without metadata
are valid.
* refactor(utils): complete NotRequired
---------
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
* fix(utils): raise ValueError from get_safe_torch_device
Bare asserts vanish under python -O and look like programmer bugs.
Convert unavailable CUDA/MPS/XPU requests into clear ValueErrors.
* style: combine nested with in device util tests (ruff)
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(utils): reject zero-norm / invalid quaternions in Rotation
Zero or non-finite inputs previously slipped through and produced NaN
rotation matrices on later convert/apply. Validate shape and scept for
norm > 0 before normalizing.
* fix(teleop): degrade phone AR quat parse like missing pose
Address review on #3988: Rotation.from_quat now rejects zero/NaN
quaternions. Wrap HEBI iOS ARKit permission in ValueError and return the
existing (False, None, None, None) path so teleop does not die mid-session
before tracking is ready.
* style: ruff format long ValueError in rotation.py
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* Connect, test and disconnected camera instances sequentially
* Add warmup-s cli argument to lerobot-find-cameras script
* Reduce default record time from 6 to 2 seconds in find_cameras
* Annotate return value of save_image function
* Initialize logging configuration in find_cameras
* fix(robot): type FK-to-EE action features as ACTION not STATE
ForwardKinematicsJointsToEEAction.transform_features declared its
end-effector action features (ee.x/y/z/wx/wy/wz/gripper_pos) with
FeatureType.STATE, copied verbatim from the sibling
ForwardKinematicsJointsToEEObservation (where STATE is correct for
OBSERVATION features). Every other action-producing step in this file
(EEReferenceAndDelta, InverseKinematicsEEToJoints, InverseKinematicsRLStep)
types its ACTION-bucket features as FeatureType.ACTION.
The mismatch mis-classifies the converted EE actions as state, which
propagates a wrong feature schema to downstream consumers keyed on
FeatureType (e.g. normalization norm_map, policy input/output feature
classification).
* test(robot): FK-to-EE step feature-type contract (action vs observation)
Asserts ForwardKinematicsJointsToEEAction emits EE features in the ACTION
bucket typed FeatureType.ACTION, and ForwardKinematicsJointsToEEObservation
emits them in the OBSERVATION bucket typed FeatureType.STATE.
* chore: delete user file
* chore(processor): reduce verbosity
---------
Co-authored-by: Jaagat-P <jaagatp05@gmail.com>
* fix(evo1): re-pad normalizer stats when loading from checkpoint
reconcile_evo1_processors did not re-pad the (un)normalizer stats to
max_state_dim/max_action_dim on the checkpoint-load path. When
lerobot-train loads a checkpoint (e.g. stage2 from a stage1 checkpoint)
it injects the raw dataset stats via processor overrides, so LIBERO's
8-dim state stats normalized a 24-dim padded state and crashed with
"size of tensor a (24) must match tensor b (8)".
Restore _refresh_evo1_normalization_steps (removed in the "remove legacy
codepaths" refactor) and call it from reconcile_evo1_processors so the
loaded stats/features are re-padded to EVO1's fixed widths. Padding is a
no-op when stats are already at the target width.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(evo1): cover reconcile re-padding of overridden normalizer stats
Regression test for the stage2-from-checkpoint crash: reloading a
checkpoint with raw (unpadded) dataset stats injected via processor
overrides must be re-padded to max_state_dim/max_action_dim by
reconcile_evo1_processors, otherwise normalizing the padded state
raises a shape mismatch.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Martino Russi <martino@huggingface.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(robots): retry SO follower/leader bus reads on transient Feetech errors
SO-100/SO-101 teleoperation aborts when a sync_read of Present_Position
returns a corrupted status packet ("Incorrect status packet!"), which the
Feetech bus emits intermittently under load. The read path already supports
a num_retry argument but the SO follower and leader never used it, so a single
transient failure crashed the control loop.
Add a max_read_retry config option (default 3) to SOFollowerConfig and
SOLeaderConfig and forward it to every Present_Position sync_read. Retries are
immediate and only happen on failure, so the steady-state read cost is
unchanged; set max_read_retry=0 to restore the previous behavior.
Fixes#3131
* chore(robots): change defaults
---------
Co-authored-by: isaka1022 <isaka1022@gmail.com>
* perf(docker): split dep install from src copy for CI layer caching
Install third-party deps (torch + all extras) in a layer keyed only on
pyproject.toml + uv.lock via --no-install-project, then copy src and install
the local package. Editing src/ no longer busts the heavy dependency layer,
so BuildKit layer cache hits across CI builds.
Applied to both Dockerfile.user and Dockerfile.internal.
* chore(ci): less verbose comments + copy all files
---------
Co-authored-by: dongmao.zhang <dongmao.zhang@bytedance.com>
* fix(datasets): bound memory of augment_dataset_quantile_stats by sampling frames
Per-episode stats previously materialized every frame (and decoded up to 16
episodes in parallel), so peak memory scaled with episode length and OOM'd on
large datasets (#2889). Numeric features are now read in full from the table
(exact), while only image/video frames are sub-sampled per episode using the
existing sample_indices heuristic. Worker count is configurable via
LEROBOT_STATS_MAX_WORKERS; --no-sampling restores exact behavior.
* Update tests/datasets/test_augment_quantile_stats.py
Co-authored-by: Haoming Song <1847575517@qq.com>
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
---------
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Haoming Song <1847575517@qq.com>
* fix(eval): align seed list length with num_envs when unseeded
eval_policy appended a single None per batch to all_seeds on the unseeded path while the reward and success lists grew by num_envs. The per-episode zip(..., strict=True) then raised ValueError for num_envs > 1. Extend all_seeds by num_envs so the lists stay aligned.
* chore(tests): delete lerobot_eval test
---------
Co-authored-by: Devin Lai <markauto75@gmail.com>
Steven Palma