* feat(cameras): prefer MJPG when fourcc unspecified; allow fourcc in ZMQ image server
With no fourcc set, OpenCV's V4L2 auto-negotiation selects uncompressed
YUYV when the camera offers it: ~16x the USB bandwidth of MJPG for
identical frames, which silently caps the frame rate on shared USB 2.0
buses (e.g. a Raspberry Pi with multiple cameras).
OpenCVCamera now prefers MJPG when config.fourcc is None: it requests
MJPG and keeps it only if the camera reports support (read-back),
falling back to the camera's own negotiation otherwise. An explicit
config.fourcc always takes precedence, so opting out is fourcc="YUYV".
The preference mirrors the existing pre/post size-and-fps ordering to
preserve the Windows DSHOW FOURCC-override handling.
Also adds a fourcc passthrough to the ZMQ image server, which previously
could not request a pixel format at all.
Behavior change: cameras that relied on implicit uncompressed capture
now receive camera-side JPEG unless fourcc is set explicitly.
* chore(cameras): address feedback
---------
Co-authored-by: Xingdong Zuo <18168681+zuoxingdong@users.noreply.github.com>
* fix(train): honor policy.use_amp when the policy has no dtype field
Since the Accelerate migration, accelerator.autocast() has been a no-op for
policies that do not expose a string dtype field (act, diffusion,
multi_task_dit, ...): PR #3912 wires mixed_precision from policy.dtype, so
these policies resolve to None and always train in full fp32 regardless of
--policy.use_amp=true. Meanwhile lerobot-eval does honor use_amp, and
PreTrainedConfig documents the flag as applying to training and evaluation.
Fall back to use_amp when no dtype field is present: bf16 where supported,
fp16 otherwise (matching torch.autocast's cuda default used by eval).
Measured on multi_task_dit / pusht, batch 256, H100: peak allocated drops
from ~75 GiB (fp32) to ~46.5 GiB under bf16 autocast.
* fix(train): conservative check
---------
Co-authored-by: Reece O'Mahoney <reece.omahoney3@gmail.com>
* fix(libero): don't reset inside step() on termination
LiberoEnv.step() called self.reset() when an episode terminated. Gymnasium's
vector envs default to AutoresetMode.NEXT_STEP, so the vector env resets the
sub-env again on the following step -- every termination paid two full resets.
The self-reset was also pure waste: `observation` is built from the terminal
raw_obs before it, so the reset's return value was discarded outright.
Worse, LiberoEnv.reset() advances init_state_id by _reset_stride, so the extra
reset skipped an initial state on every episode.
Measured with a counting subclass under SyncVectorEnv (gymnasium 1.3.0,
terminating every 4 steps over a 14-step loop):
n_envs=1: 3 terminations -> 6 resets = 1 initial + 3 self + 2 autoreset
n_envs=2: 6 terminations -> 12 resets = 2 initial + 6 self + 4 autoreset
(the final termination's autoreset does not fire before the loop ends)
Each LiberoEnv.reset() is a full LIBERO reset plus num_steps_wait settle steps,
so on the current default reset path this duplicates roughly 1.3 s per
termination.
Standalone (non-vectorised) users must now call reset() themselves after
termination, which is the Gymnasium contract.
* test(libero): pin the autoreset *default*, not the enum's spelling
The previous assertion checked `AutoresetMode.NEXT_STEP.value == "NextStep"`,
which is a naming detail. If a future Gymnasium flipped the vector-env default
to SAME_STEP, that assertion would still pass and the bug this fixes would come
back as a missing reset instead of a double one.
Assert the observable default instead. Verified on gymnasium 1.1.1 (the floor in
pyproject) and 1.3.0, for both SyncVectorEnv and AsyncVectorEnv. Negative control:
constructing with autoreset_mode=SAME_STEP fails the new assertion and passes the
old one.
Also document why `env._env = inner` is not redundant with the monkeypatched
factory: `LiberoEnv.__init__` defers simulator creation, so pre-binding keeps
`_ensure_env()` a no-op and avoids a stray `reset()` on the mock.
* test(libero): drop the redundant `env._env` prebind
@noron12234 flagged this as redundant. Their stated reason was wrong -- `__init__`
defers simulator creation (`self._env = None`, libero.py:180), so the monkeypatched
factory has not been called by the time it returns -- but the conclusion holds:
`_ensure_env()` pulls the same `inner` from the mocked factory on first `step()`.
I claimed the prebind was load-bearing because a stray `reset()` would pollute the
tests' call counts. That was wrong and I had not run it. Ran both variants against
the real class: the stray reset lands on `inner.reset`, which no assertion touches,
and both tests pass with or without the line. Dropping it.
* refactor(env): add explicit NEXT_STEP
* fix(style): pre-commit
---------
Co-authored-by: Dimitar Dimitrov <dvdimitrov13@gmail.com>
Co-authored-by: Dimitar Dimitrov <60075474+dvdimitrov13@users.noreply.github.com>
* perf(libero): skip the discarded scene rebuild on reset
LIBERO's OffScreenRenderEnv defaults to hard_reset=True, so every reset() frees
the MjSim, re-serialises the scene with model.get_xml(), recompiles it with
MjSim.from_xml_string(), constructs a fresh offscreen GL context and re-wires
every observable.
When init states are in use, LiberoEnv.reset() immediately calls
set_init_state(), which overwrites the whole sim state -- so all of that work is
discarded. This passes hard_reset=not init_states instead. Without init states
the randomisation reset() performs is the only thing placing the objects, so the
hard reset is kept.
Measured on an RTX 3060 Ti (EGL, robosuite 1.4.0, mujoco 3.2.7, 256x256 x2
cameras), through LiberoEnv.reset(), fresh env per arm:
suite hard soft saved
libero_spatial 1697 ms 233 ms 1464 ms
libero_object 1394 ms 172 ms 1222 ms
libero_goal 1177 ms 164 ms 1013 ms
libero_10 1528 ms 207 ms 1321 ms
Equivalence
-----------
Immediately after set_init_state, qpos, qvel, ctrl and act are bit-identical
between the two paths on every suite tested.
After the 10 settle steps that reset() runs, 9 of 41 qpos entries differ:
robot0_joint1..7 (<= 2.4e-5 rad) and gripper0_finger_joint1/2 (<= 2.1e-4 rad).
No object joint differs on any suite. The drift is driven by the gripper
component of the settle action ([0,0,0,0,0,0,-1]); replacing it with zeros keeps
the two paths bit-identical for 12 further steps, and with num_steps_wait=0 there
is no divergence at all.
Wrist-camera pixels can differ by up to ~87/255, because a sub-millimetre finger
displacement crosses rasterisation boundaries at 256x256. The pixel metric badly
overstates the physical difference here; 2.1e-4 rad is 0.012 degrees.
So this is not bit-identical end to end, and reviewers should decide whether
0.012 degrees of gripper drift is acceptable for the benchmark. It does not
change object placement, which is what the fixed init states exist to control.
* refactor(env): config param libero + docs
---------
Co-authored-by: Dimitar Dimitrov <dvdimitrov13@gmail.com>
Four kinematic processor steps read the observation as
observation = self.transition.get(TransitionKey.OBSERVATION).copy()
if observation is None:
raise ValueError("Joints observation is require for computing robot kinematics")
so `.copy()` runs first and the guard below it is unreachable. A transition
without an observation raises `AttributeError: 'NoneType' object has no
attribute 'copy'` instead of the intended message.
That transition is not hypothetical: `RobotProcessorPipeline.process_action`
builds one with `create_transition(action=action)`, which sets
`TransitionKey.OBSERVATION` to None.
Reads the value first, checks it, then copies. Affects EEReferenceAndDelta,
InverseKinematicsEEToJoints, GripperVelocityToJoint and InverseKinematicsRLStep.
Adds a parametrised regression test covering all four; each fails with the
AttributeError if the fix is reverted.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(edit-dataset): redirect dataset.root via backup_path returned from get_output_path
The in-place backup logic compared `output_dir` (run through `.resolve()`) to
`dataset.root` (unresolved), so when `HF_LEROBOT_HOME` was a symlink the swap
to the `_old` backup never fired and `_copy_and_reindex_videos` then read from
the now-empty output directory. Have `get_output_path` return the backup path
directly so callers don't rely on path equality.
* feat(samefile): making same file detection more robust using os.path.samefile
---------
Co-authored-by: Reece O'Mahoney <reece.omahoney3@gmail.com>
* extend index.mdx
* bring community section up + bigger logo
* fix formatting
* fix logo size
* text fix
* improved explanation
* fix emoji and formatting
* fix link
- Quickstart on LIBERO: finetune lerobot/pi05_libero_base on lerobot/libero
with a complete, copy-pasteable command; feature table mapping the dataset
keys to how pi05 consumes them; gated PaliGemma tokenizer tip.
- Explain --policy.path vs --policy.pretrained_path (weights+config vs
weights-only) and why n_action_steps/empty_cameras must be passed
explicitly with pretrained_path.
- Quantile statistics section: the exact error message, the
lerobot-edit-dataset recompute_stats fix (replacing the removed
augment_dataset_quantile_stats.py reference), where the result lands, and
the MEAN_STD alternative.
- Update stale link lerobot/pi05_libero -> lerobot/pi05_libero_base; add
PyPI install variant.
Co-authored-by: Xingdong Zuo <18168681+zuoxingdong@users.noreply.github.com>
* feat(rollout): add smooth_handover flag to DAgger strategy config
The DAgger phase transitions run blocking smooth handovers: on pause the
leader is driven to the follower (~2 s), and on correction start the
follower is slid to the teleop pose (~1 s), both inside the record loop.
For clutch-style teleoperators (e.g. VR controllers) that re-reference
their command frame at the current robot pose on engage, the handover is
already continuous — the interpolation only delays the start of the
correction and eats its first frames.
Add --strategy.smooth_handover (default true, existing behavior
unchanged) to let such setups skip it, mirroring the episodic strategy's
smooth_leader_to_follower_handover flag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: pre-commit auto-fix (prettier markdown table formatting)
---------
Co-authored-by: griffinaddison <griffinnosidda@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(rollout): add smooth_handover flag to episodic strategy config
Follow-up to #3985, which added the same flag to the DAgger strategy.
The episodic strategy's reset-phase handover had two gaps:
- Non-actuated teleops could not skip the blocking follower slide at all.
- Setting smooth_leader_to_follower_handover=false on an actuated teleop
swapped which arm moves instead of skipping the handover.
Add --strategy.smooth_handover (default true, existing behavior
unchanged) as a master switch that skips the interpolation entirely,
for clutch-style teleops that re-reference at the current robot pose
on engage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: fix table formatting via prettier
---------
Co-authored-by: griffinaddison <gaddison@seas.upenn.edu>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(vla_jepa): use device-safe autocast instead of hardcoded bfloat16
VLA-JEPA hardcodes torch.autocast with dtype=torch.bfloat16, which
crashes on MPS (no AMP support) and silently misbehaves on pre-Ampere
CUDA GPUs (no bf16). Add a _get_autocast_context() helper that reuses
the existing is_amp_available() utility to pick a safe strategy per
device, matching the pattern used by pi05 and molmoact2.
Fixes#3744
* fix: use _get_autocast_context for fp32 action head (MPS compat)
---------
Co-authored-by: devangpratap <115096812+devangpratap@users.noreply.github.com>
* feat(transforms): add 8 robotics-relevant image augmentations
Add GaussianNoise, MotionBlur, JPEGCompression, GaussianPatchBrightness,
RandomShadow, CoarseDropout, GammaCorrection, and PlanckianJitter.
Each transform addresses a real-world failure mode not covered by the
existing 6 defaults (sensor noise, motion blur, compression artifacts,
uneven lighting, cast shadows, partial occlusion, exposure variation,
color temperature shift).
All transforms are pure PyTorch, follow the make_params/transform
pattern, and integrate with ImageTransformConfig via a registry.
* add augmentation showcase image for PR
* update showcase with better sample frame
* tune showcase to balanced augmentation intensity
* tune showcase: softer shadow, dropout, jitter intensity
* refactor(transforms): several updates
* update image
* chore(media): remove example
* chore: add link to example
---------
Co-authored-by: Yuxian LI <liyuxian1358@gmail.com>
* Fix add_features for multi-dimensional per-frame features
* Use generic names in multidimensional add_features test
* tests(all shapes): enhancing tests to cover all possible features shapes
* chore(format): formatting code
---------
Co-authored-by: felixmin <felix.minze@live.de>
* Support task replacement mappings in modify_tasks
Part of #2326.
Signed-off-by: 陈伟 <woshei0a0a0a@qq.com>
* feat(task modification precedence): Improving task modification precedence so that all modes can be used in a single run. Adapting tests accordinginly.
* chore(fromat): formatting code
* docs(docstrings): updating docstrings
* docs(update): updating docs with the task modification features
---------
Signed-off-by: 陈伟 <woshei0a0a0a@qq.com>
Co-authored-by: vvezre <93599357+vvezre@users.noreply.github.com>
* docs: fix broken and placeholder links
Fix the two links called out in #4094 plus one more found while scanning
all docs: the NOTE:addlinktoLOC placeholder in async.mdx now points at
_aggregate_action_queues in robot_client.py; the dead
#mapping-videoencoderconfig--ffmpeg-options anchor in
video_encoding_parameters.mdx now points at get_codec_options in
video.py; and the stale ./cameras#setup-cameras fragment in il_robots.mdx
is dropped to link the cameras guide page.
Closes#4094.
* docs: apply prettier formatting to the video-encoding parameter table
Re-align the Markdown table columns with prettier (v3.6.2, --prose-wrap=preserve)
so the pre-commit 'Format Markdown with Prettier' hook passes. Whitespace only.
is_saving_step was the only step-frequency check without a `> 0` guard,
so save_freq=0 raised ZeroDivisionError from `step % 0` on the first
step. Route the decision through a should_save_checkpoint helper that
treats a non-positive save_freq as "save only the final checkpoint",
matching how log_freq/eval_freq handle non-positive values.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
The discard-path fix (#3683) deletes staging for all camera_keys in
clear_episode_buffer(). The post-save cleanup in save_episode() must keep
iterating image_keys only: with batch_encoding_size > 1 the video staging
frames of already-saved episodes stay on disk until the batch encoder
consumes and deletes them. Add a regression test pinning that behavior,
plus a comment explaining why the two paths differ.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Observations were sent as base64-in-JSON, which inflates every camera
frame 33% purely to fit binary JPEG inside a text protocol (~11 Mbps at
3 cameras x 30 Hz). Send a ZMQ multipart message instead: a JSON header
frame (state + camera order) followed by one raw JPEG frame per camera.
Benchmarked on hardware: -25% wire size (invariant across contention),
lower and tighter latency, and no dropped frames under load where the
base64 format stalled.
ZMQ_CONFLATE does not support multipart, so the observation sockets use
2-deep high-water marks; the client's existing drain-to-latest loop
preserves the keep-newest behavior.
Breaking wire change: host and client must run the same version.
Co-authored-by: Xingdong Zuo <18168681+zuoxingdong@users.noreply.github.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Without an explicit fourcc, OpenCV's V4L2 auto-negotiation selects
uncompressed YUYV when the camera offers it: ~147 Mbps per camera at
640x480@30 on the Pi's shared USB 2.0 bus, versus ~9 Mbps as MJPG for
identical frames. Two default cameras already exceed the bus, so frame
rates sag silently.
Pinning fourcc="MJPG" is backward compatible: it is an existing
OpenCVCameraConfig field, frames still arrive as BGR ndarrays, user
configs that set their own fourcc are unaffected, and a camera without
MJPG fails loudly at validation rather than degrading silently.
Co-authored-by: Xingdong Zuo <18168681+zuoxingdong@users.noreply.github.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(datasets): stop frame errors being treated as shard exhaustion in StreamingLeRobotDataset
StreamingLeRobotDataset.__iter__ caught every RuntimeError and treated all as exhausted shard. Real errors like video decode failure made each shard get dropped on the first frame, so iteration ended while yeilding zero frames with no errors.
Shard exahustion is StopIteration raised from make_frame generator, which python converts to RuntimeError with StopIteration as __cause__. Added check to tell StopIteration from everything else, consuming real shard exhaustion while re-raising everything else.
Added a test that injects a decode failure and asserts iteration raises instead of returning on empty stream.
Fixes#4066
* refactor(datasets): exception streaming
---------
Co-authored-by: Mohit Yadav <mohitydv09@gmail.com>
`can_peek_back(steps)` is documented to return whether `peek_back(steps)`
can be called "without raising an IndexError", but it guarded with `<=`:
return steps <= len(self._back_buf) + self._cursor
`peek_back(n)` needs n+1 buffered slots — it raises when
`n + 1 > len(self._back_buf) + self._cursor` and reads
`self._back_buf[self._cursor - (n + 1)]`. So at
`steps == len(self._back_buf) + self._cursor`, `can_peek_back` returns True
while `peek_back` raises LookBackError, contradicting the docstring.
Two siblings confirm the intended bound:
- `prev()` (one step back) requires `len(self._back_buf) + self._cursor > 1`.
- The forward twin is already consistent: `can_peek_ahead(n)` buffers n items
and `peek_ahead(n)` reads `_ahead_buf[n - 1]` (needs n).
Use `<` so `can_peek_back` matches `peek_back`'s guard exactly.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix: implement bimanual SO leader DAgger handover support
- Add feedback_features property: enables DAgger's teleop_supports_feedback() check
- Implement enable_torque()/disable_torque(): synchronized torque control for both arms
- Implement send_feedback(): routes bimanual feedback to left/right arms with prefix stripping
This fixes DAgger smooth handover for bimanual SO follower + SO leader setups:
when pausing from policy to human intervention, both leader arms now move smoothly
to the follower's current pose, avoiding discontinuities at the human takeover point.
* Update hil_data_collection.mdx
Signed-off-by: Baptiste Lubrano Lavadera <45080391+Mr-C4T@users.noreply.github.com>
* Update bi_so_leader.py
Signed-off-by: Baptiste Lubrano Lavadera <45080391+Mr-C4T@users.noreply.github.com>
---------
Signed-off-by: Baptiste Lubrano Lavadera <45080391+Mr-C4T@users.noreply.github.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(smolvla): freeze the intended VLM layers when train_expert_only=False
The partial-freeze patterns in set_requires_grad() used a
'text_model.model.' prefix that does not exist in SmolVLM parameter
names ('SmolVLMModel.text_model' is a bare LlamaModel, with no nested
'.model'). As a result the last VLM layer and the final norm were
silently left trainable, defeating the freeze that was added to avoid
unused-parameter errors with DDP; only lm_head was frozen by substring
luck.
Use the real flat names, and raise if any freeze pattern stops matching
so a future transformers renaming cannot silently reintroduce the bug.
Add a CPU regression test covering both last_layers branches.
Fixes#4018
* test(smolvla): drop regression test per review
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
serialize_torch_rng_state/deserialize_torch_rng_state only handled CPU
and CUDA generators. On MPS, resumed training was not bit-exact for any
stochastic op (dropout, ACT's CVAE noise) since the MPS generator's state
was never saved or restored. Mirrors the existing CUDA branch using
torch.mps.get_rng_state/set_rng_state (available since torch 2.11).
Note: get_rng_state()/set_rng_state() (used by seeded_context()) have the
same gap but are out of scope here — happy to follow up separately if
useful.
Co-authored-by: Sunny Dave <sunnydave@Sunnys-Mac-Studio.local>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Steven Palma