Merge the separate episode_filter no-match and unknown-key tests into a
single parametrized test, and add coverage for an out-of-range episodes
list raising rather than silently resolving to empty.
StreamingLeRobotDataset previously accepted an `episodes` argument but
never applied it, so the whole stream was returned regardless. Resolve
the selection against the dataset range, apply it as a lazy filter over
the streamed shards, and add an `episode_filter` predicate (mirroring
LeRobotDataset). Empty selections and non-matching filters now raise
instead of silently streaming everything, and num_frames/num_episodes
reflect the selection.
* fix(sarm): warn when dense/sparse targets silently collapse to all-zero
In dense_only/dual modes, if meta/episodes/*.parquet has no usable
subtask columns (column absent or NaN), _load_episode_annotations
returns None and find_stage_and_tau yields stage 0 / tau 0 for every
frame. Training "succeeds" but the head silently learns to predict 0
everywhere, with no warning. This complements #2880 (which restored
loading of episodes_df): there the DataFrame is loaded but the
*_subtask_names column is missing/NaN.
Add a one-time validation at processor construction that logs a clear
warning (all episodes missing -> predict-all-zero; some missing ->
partial). Purely additive logging, no change to training math.
Closes#3842
* fix(sarm): fail fast on unusable episode annotations
---------
Co-authored-by: 1thanShih <Smartshithan1620.en12@nycu.edu.tw>
* fix(cameras): add color_format config and auto-recovery for RealSense D405
The D405 delivers color from its stereo depth module, not a dedicated
RGB sensor. The driver previously hardcoded rs.format.rgb8, causing
silent frame capture failure on D405.
Changes:
- Add color_format field to RealSenseCameraConfig (default rgb8, D405
users set bgr8), validated against whitelist
- Use configured format in _configure_rs_pipeline_config
- Fix _postprocess_image to handle both rgb8 and bgr8 source formats
- Add _hardware_reset auto-recovery: if warmup times out, perform USB
hardware reset and retry once (common D405 recovery path)
- Fix thread race in _read_loop (local ref to stop_event)
Continuation of #3164 (closed due to deleted fork).
Tested on Intel RealSense D405 at 1280x720@30fps with color_format=bgr8.
* style: fix ruff lint B904 and format
* refactor(cameras): clarify RealSense connection retry
* refactor(cameras): address review, retry RealSense connect before hardware reset
Drop color_format (device-side streaming state was the actual cause), keep _open_pipeline attempt-agnostic, catch only retry-worthy errors, reset only as last resort, guard read loop against late frame publication after stop.
* refactor(cameras): restore BaseException teardown, shorten stop-check comment
* test(cameras): expect ConnectionError after retries are exhausted
rollout() runs `while not np.all(done)` with `done` latched, so a sub-env that
terminates early keeps being driven -- physics and offscreen rendering included
-- until the slowest sub-env in the batch finishes. A batch of N runs for
max(episode_lengths) iterations to complete work that only needs
mean(episode_lengths), and all of the surplus is discarded.
Adds FreezeAfterEpisodeEnd, applied to each sub-env of the eval vector env. It
caches the terminal transition and replays it for any further step(), and also
absorbs Gymnasium's autoreset -- under AutoresetMode.NEXT_STEP the vector env
otherwise rebuilds a finished sub-env and runs it through an entire extra episode
the rollout throws away. Reward is zeroed on replay so a frozen sub-env cannot
inflate a return if a caller sums over the padded tail.
Thawing is signalled explicitly: rollout() passes NEW_ROLLOUT_OPTION in
reset(options=...). Gymnasium's autoreset calls reset() with no arguments, but so
would a caller passing seeds=None, and inferring from that would strand an env
frozen for a whole rollout.
AutoresetMode.DISABLED is not an alternative -- Gymnasium asserts that no
terminated env is ever stepped in that mode, so the wrapper is never reached. An
earlier version of this patch used it and the vector-env test caught the assert.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Four kinematic processor steps read the observation as
observation = self.transition.get(TransitionKey.OBSERVATION).copy()
if observation is None:
raise ValueError("Joints observation is require for computing robot kinematics")
so `.copy()` runs first and the guard below it is unreachable. A transition
without an observation raises `AttributeError: 'NoneType' object has no
attribute 'copy'` instead of the intended message.
That transition is not hypothetical: `RobotProcessorPipeline.process_action`
builds one with `create_transition(action=action)`, which sets
`TransitionKey.OBSERVATION` to None.
Reads the value first, checks it, then copies. Affects EEReferenceAndDelta,
InverseKinematicsEEToJoints, GripperVelocityToJoint and InverseKinematicsRLStep.
Adds a parametrised regression test covering all four; each fails with the
AttributeError if the fix is reverted.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* feat(transforms): add 8 robotics-relevant image augmentations
Add GaussianNoise, MotionBlur, JPEGCompression, GaussianPatchBrightness,
RandomShadow, CoarseDropout, GammaCorrection, and PlanckianJitter.
Each transform addresses a real-world failure mode not covered by the
existing 6 defaults (sensor noise, motion blur, compression artifacts,
uneven lighting, cast shadows, partial occlusion, exposure variation,
color temperature shift).
All transforms are pure PyTorch, follow the make_params/transform
pattern, and integrate with ImageTransformConfig via a registry.
* add augmentation showcase image for PR
* update showcase with better sample frame
* tune showcase to balanced augmentation intensity
* tune showcase: softer shadow, dropout, jitter intensity
* refactor(transforms): several updates
* update image
* chore(media): remove example
* chore: add link to example
---------
Co-authored-by: Yuxian LI <liyuxian1358@gmail.com>
* Fix add_features for multi-dimensional per-frame features
* Use generic names in multidimensional add_features test
* tests(all shapes): enhancing tests to cover all possible features shapes
* chore(format): formatting code
---------
Co-authored-by: felixmin <felix.minze@live.de>
* Support task replacement mappings in modify_tasks
Part of #2326.
Signed-off-by: 陈伟 <woshei0a0a0a@qq.com>
* feat(task modification precedence): Improving task modification precedence so that all modes can be used in a single run. Adapting tests accordinginly.
* chore(fromat): formatting code
* docs(docstrings): updating docstrings
* docs(update): updating docs with the task modification features
---------
Signed-off-by: 陈伟 <woshei0a0a0a@qq.com>
Co-authored-by: vvezre <93599357+vvezre@users.noreply.github.com>
is_saving_step was the only step-frequency check without a `> 0` guard,
so save_freq=0 raised ZeroDivisionError from `step % 0` on the first
step. Route the decision through a should_save_checkpoint helper that
treats a non-positive save_freq as "save only the final checkpoint",
matching how log_freq/eval_freq handle non-positive values.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
The discard-path fix (#3683) deletes staging for all camera_keys in
clear_episode_buffer(). The post-save cleanup in save_episode() must keep
iterating image_keys only: with batch_encoding_size > 1 the video staging
frames of already-saved episodes stay on disk until the batch encoder
consumes and deletes them. Add a regression test pinning that behavior,
plus a comment explaining why the two paths differ.
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(datasets): stop frame errors being treated as shard exhaustion in StreamingLeRobotDataset
StreamingLeRobotDataset.__iter__ caught every RuntimeError and treated all as exhausted shard. Real errors like video decode failure made each shard get dropped on the first frame, so iteration ended while yeilding zero frames with no errors.
Shard exahustion is StopIteration raised from make_frame generator, which python converts to RuntimeError with StopIteration as __cause__. Added check to tell StopIteration from everything else, consuming real shard exhaustion while re-raising everything else.
Added a test that injects a decode failure and asserts iteration raises instead of returning on empty stream.
Fixes#4066
* refactor(datasets): exception streaming
---------
Co-authored-by: Mohit Yadav <mohitydv09@gmail.com>
serialize_torch_rng_state/deserialize_torch_rng_state only handled CPU
and CUDA generators. On MPS, resumed training was not bit-exact for any
stochastic op (dropout, ACT's CVAE noise) since the MPS generator's state
was never saved or restored. Mirrors the existing CUDA branch using
torch.mps.get_rng_state/set_rng_state (available since torch 2.11).
Note: get_rng_state()/set_rng_state() (used by seeded_context()) have the
same gap but are out of scope here — happy to follow up separately if
useful.
Co-authored-by: Sunny Dave <sunnydave@Sunnys-Mac-Studio.local>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(utils): raise ValueError from get_safe_torch_device
Bare asserts vanish under python -O and look like programmer bugs.
Convert unavailable CUDA/MPS/XPU requests into clear ValueErrors.
* style: combine nested with in device util tests (ruff)
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(utils): reject zero-norm / invalid quaternions in Rotation
Zero or non-finite inputs previously slipped through and produced NaN
rotation matrices on later convert/apply. Validate shape and scept for
norm > 0 before normalizing.
* fix(teleop): degrade phone AR quat parse like missing pose
Address review on #3988: Rotation.from_quat now rejects zero/NaN
quaternions. Wrap HEBI iOS ARKit permission in ValueError and return the
existing (False, None, None, None) path so teleop does not die mid-session
before tracking is ready.
* style: ruff format long ValueError in rotation.py
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(robot): type FK-to-EE action features as ACTION not STATE
ForwardKinematicsJointsToEEAction.transform_features declared its
end-effector action features (ee.x/y/z/wx/wy/wz/gripper_pos) with
FeatureType.STATE, copied verbatim from the sibling
ForwardKinematicsJointsToEEObservation (where STATE is correct for
OBSERVATION features). Every other action-producing step in this file
(EEReferenceAndDelta, InverseKinematicsEEToJoints, InverseKinematicsRLStep)
types its ACTION-bucket features as FeatureType.ACTION.
The mismatch mis-classifies the converted EE actions as state, which
propagates a wrong feature schema to downstream consumers keyed on
FeatureType (e.g. normalization norm_map, policy input/output feature
classification).
* test(robot): FK-to-EE step feature-type contract (action vs observation)
Asserts ForwardKinematicsJointsToEEAction emits EE features in the ACTION
bucket typed FeatureType.ACTION, and ForwardKinematicsJointsToEEObservation
emits them in the OBSERVATION bucket typed FeatureType.STATE.
* chore: delete user file
* chore(processor): reduce verbosity
---------
Co-authored-by: Jaagat-P <jaagatp05@gmail.com>
* fix(evo1): re-pad normalizer stats when loading from checkpoint
reconcile_evo1_processors did not re-pad the (un)normalizer stats to
max_state_dim/max_action_dim on the checkpoint-load path. When
lerobot-train loads a checkpoint (e.g. stage2 from a stage1 checkpoint)
it injects the raw dataset stats via processor overrides, so LIBERO's
8-dim state stats normalized a 24-dim padded state and crashed with
"size of tensor a (24) must match tensor b (8)".
Restore _refresh_evo1_normalization_steps (removed in the "remove legacy
codepaths" refactor) and call it from reconcile_evo1_processors so the
loaded stats/features are re-padded to EVO1's fixed widths. Padding is a
no-op when stats are already at the target width.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(evo1): cover reconcile re-padding of overridden normalizer stats
Regression test for the stage2-from-checkpoint crash: reloading a
checkpoint with raw (unpadded) dataset stats injected via processor
overrides must be re-padded to max_state_dim/max_action_dim by
reconcile_evo1_processors, otherwise normalizing the padded state
raises a shape mismatch.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Martino Russi <martino@huggingface.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix(robots): retry SO follower/leader bus reads on transient Feetech errors
SO-100/SO-101 teleoperation aborts when a sync_read of Present_Position
returns a corrupted status packet ("Incorrect status packet!"), which the
Feetech bus emits intermittently under load. The read path already supports
a num_retry argument but the SO follower and leader never used it, so a single
transient failure crashed the control loop.
Add a max_read_retry config option (default 3) to SOFollowerConfig and
SOLeaderConfig and forward it to every Present_Position sync_read. Retries are
immediate and only happen on failure, so the steady-state read cost is
unchanged; set max_read_retry=0 to restore the previous behavior.
Fixes#3131
* chore(robots): change defaults
---------
Co-authored-by: isaka1022 <isaka1022@gmail.com>
* fix(datasets): bound memory of augment_dataset_quantile_stats by sampling frames
Per-episode stats previously materialized every frame (and decoded up to 16
episodes in parallel), so peak memory scaled with episode length and OOM'd on
large datasets (#2889). Numeric features are now read in full from the table
(exact), while only image/video frames are sub-sampled per episode using the
existing sample_indices heuristic. Worker count is configurable via
LEROBOT_STATS_MAX_WORKERS; --no-sampling restores exact behavior.
* Update tests/datasets/test_augment_quantile_stats.py
Co-authored-by: Haoming Song <1847575517@qq.com>
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
---------
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Haoming Song <1847575517@qq.com>
* feat(camera): add manual exposure, gain, and white balance options for RealSense cameras
The RealSense camera integration lacked sensor-level controls, causing
issues like unstable lighting from auto-exposure hunting. This adds
optional `exposure`, `gain`, and `white_balance` fields to
RealSenseCameraConfig that disable the corresponding auto modes and
apply fixed values when set.
* fix: support D405 stereo module for sensor options
D405 exposes color stream via "Stereo Module", not "RGB Camera".
Fall back to Stereo Module when RGB Camera is not found.
* test(camera): add unit tests + range-aware errors for RealSense sensor options
Address PR #3220 review:
- Wrap set_option calls; re-raise ValueError with option name, value,
and sensor.get_option_range() diagnostics on out-of-range values.
- Add unit tests for _get_color_sensor (RGB Camera, D405 Stereo Module
fallback, diagnostic error) and _configure_sensor_options (no-op,
all values, unsupported warns, partial config, out-of-range raise).
* fix(realsense): validate manual color controls
* refactor(camera): apply feedback
---------
Co-authored-by: Lev Kozlov <kozlov.l.a10@gmail.com>
* (depth image processing): excluding depth frames from the RGB to BGR image processing
* test(update): updating tests to include RGB/BGR conversion checks
* Make SO follower P coefficient configurable
* chore(test): minimize tests
* feat(robots): expose PID coeff in SO arms
---------
Co-authored-by: taivu1998 <46636857+taivu1998@users.noreply.github.com>
* fix(envs): set LiberoEnvConfig.fps default to 20 to match robosuite
LiberoEnvConfig.fps was set to 30, but the underlying robosuite
OffScreenRenderEnv always runs at its default control_freq of 20 Hz
since fps is never passed through. This mismatch silently decouples
the dataset/eval loop rate from the actual simulation step rate.
Set the default to 20 to match the real sim rate and avoid the
footgun.
Fixes#3368
* fix(libero): apply configured control frequency
---------
Co-authored-by: xinmotlanthua <275663218+xinmotlanthua@users.noreply.github.com>
* feat(annotate): run lerobot-annotate on HF Jobs via --job.target
Annotation needed a hand-edited launcher script (examples/annotations/run_hf_job.py)
to reach a GPU: users copied it, rewrote the embedded CMD string for their dataset,
and ran it with `python`. Fold that into the CLI instead, mirroring `lerobot-train`:
`lerobot-annotate --job.target=h200` submits the exact command you'd run locally.
- AnnotationJobConfig extends JobConfig with the annotation runtime's defaults
(vllm/vllm-openai image, 2h cap) plus --job.lerobot_ref, so an unmerged branch
can be exercised remotely without editing a script.
- lerobot.jobs.annotate builds the pod command by replaying the user's own CLI
flags (minus --job.*/--root, with --repo_id re-emitted from the config) after a
setup prelude that installs lerobot on top of the vLLM image. Job monitoring,
log tailing and Ctrl-C-detaches reuse the training submitter's plumbing.
- Remote runs require --repo_id; a local-only dataset is pushed privately first.
The generated pod command is byte-for-byte the script's old CMD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(annotate): reject client-side config files on remote runs
draccus exposes `--config_path` plus a `--<field>` config-file arg for every
nested dataclass (`--vlm`, `--plan`, `--job`, ...). All name files on the
client's disk, so forwarding them to the pod silently dropped whatever settings
they carried. Reject them up front instead.
Bare `--job` also slipped past the `--job.` prefix filter, so a `--job=cfg.yaml`
holding `target: h200` would have reached the pod and had the job submit a job
of its own, recursively. It is dropped from the forwarded args as well.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(jobs): share the submit-and-follow loop between both submitters
`submit_annotate_to_hf` reused the leaf helpers (`_poll_until_done`, `_tail_logs`,
`_pod_forwarded_args`) but duplicated the orchestration around them: ~40 of the 50
lines that spawn the poll/log threads, install the Ctrl-C-detaches handler and
raise on a non-COMPLETED stage were identical in both files.
Extract that into `follow_job(job_id, *, detach, success_marker=None) -> bool`,
returning True when the job finished and False when we stopped watching without a
verdict (detach or Ctrl-C). Training keeps its model-pushed marker by passing it in;
annotation has no equivalent line (the CLI keeps working after the upload log to
write the card and tag) so its completion stays stage-based.
Kept in hf.py rather than a new module so every existing monkeypatch target in
test_hf.py still resolves.
Behaviour change: a training run whose job reaches COMPLETED without the marker
matching now prints its completion line instead of returning silently. The marker
was already documented as an optimisation with a stage-based fallback; the fallback
just never reported success.
Tests: adds annotate coverage for the non-detach path (completion and failure) —
previously only ever exercised with detach=true — plus a detach short-circuit test.
Both new annotate tests verified to fail under a mutation that stubs out follow_job.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
CarolinePascal