* add: a flexible transformation registry
* fix: image transforms can be set both at init and after
* add: tests
* fix: take in review
* feat(datasets): add image transform setters
* fix: pre-commit
* fix: CI
---------
Signed-off-by: Francesco Capuano <74058581+fracapuano@users.noreply.github.com>
* feat: HIL data collection, RTC interpolator, and action queue improvements
- Add Human-in-the-Loop (HIL) data collection examples (sync + RTC)
- Add HIL data collection documentation
- Add ActionInterpolator for smoother policy control at higher rates
- Integrate interpolator into lerobot-record and eval_with_real_robot
- Add action queue clear() and get_processed_left_over() methods
- Add rtc/__init__.py for cleaner imports
* docs: expand Related Work section with paper summaries
* fix: only record dataset frames at original fps, not at interpolated rate
The interpolator speeds up robot control (e.g. 2x) but dataset frames
should still be recorded at the original fps. Interpolated-only
iterations now only send actions to the robot without writing to the
dataset.
* refactor: merge HIL sync and RTC scripts into single file with --rtc.enabled toggle
Combines hil_data_collection.py and hil_data_collection_rtc.py into one
script. RTC is toggled via --rtc.enabled=true (defaults to off for sync
inference). Deletes the separate hil_data_collection_rtc.py and updates
docs to reflect the single-script usage.
* test: add ActionInterpolator test suite (29 tests)
Covers constructor validation, passthrough (multiplier=1), 2x and 3x
interpolation with exact value checks, reset/episode boundaries,
control interval calculation, multi-dim actions, and simulated
control loop integration.
* test: add ActionQueue + ActionInterpolator integration tests
Verifies the interpolator doesn't interfere with RTC's leftover chunk
tracking: queue consumption rate matches base fps regardless of
multiplier, get_left_over/get_processed_left_over only change on
queue.get(), merge preserves smooth interpolation across chunks,
and interpolator reset is independent of queue state.
* feat: register SO follower/leader configs in HIL script
Adds SOFollowerRobotConfig and SOLeaderTeleopConfig imports so
SO100/SO101 robots can be used via --robot.type=so_follower
and --teleop.type=so_leader. Updates docs accordingly.
Made-with: Cursor
* docs: remove em dashes from HIL documentation
Made-with: Cursor
* refactor: rename examples/rac to examples/hil
Updates directory name and all references in docs and script docstrings.
Made-with: Cursor
* fix: encorperate pr feedback comments
* refactor(tests): enhance ActionInterpolator test structure and add detailed docstrings
* feedback pr and test fix
* fix(test): pass correct real_delay in interpolator delay test
The test was passing real_delay=0 and relying on _check_delays to
silently override it with the index-based diff. Now passes real_delay=3
to match the 3 actions consumed during the simulated inference period.
* fix pr feedback
* ordering
* update hil script
* fix
* default name
* fix(bi_openarm): use kw_only=True to fix dataclass field ordering
BiOpenArmFollowerConfig overrides `id` with a default, making it
positional in the child — non-default `left_arm_config` then follows a
default field, which Python dataclasses forbid. Adding kw_only=True
(matching the parent RobotConfig) removes positional constraints.
Made-with: Cursor
* style: format long line in hil_data_collection.py
Made-with: Cursor
* pr feedback
---------
Co-authored-by: Khalil Meftah <khalil.meftah@huggingface.co>
* Add option for pi family models to train with relative actions (relative to state)
* formatting
* add recomputation of stats and option to compute delta stats
* normalzie after delta conversion
* only recompute state for stats
* calulate chunk based stats
* sample 100k
* load from parquet
* sample 1m
* stats per chunck
* fix
* use quantiles
* stats for entire dataset
* fix
* max 1m frames
* compute before dist
* fix multi gpu processor bug
* Fix RTC with delta actions and OpenArms motor_type wiring
* feat: align pi0_fast delta actions with pi0/pi05 and add RTC integration tests
- Add delta_exclude_joints and action_feature_names to PI0FastConfig
- Move to_absolute_actions from modeling to processor pipeline for pi0_fast
- Add delta action detection and logging to eval_with_real_robot.py
- Add delta actions documentation to pi0 and pi05 READMEs
- Fix ruff lint issues in test_delta_actions.py
- Add test_rtc_delta_actions.py (24 tests) covering:
- ActionQueue with delta vs absolute actions
- RTC denoise step with delta leftovers
- Full pipeline roundtrip (delta → RTC → absolute)
- State rebasing approximation bounds
- Non-delta policy compatibility
- Multi-chunk consistency
* chore: clean up test comments, add OpenPI attribution, remove debug logging
- Replace decorative comment separators in test files with plain section headers
- Add attribution comments for 1e-6 epsilon in normalize_processor.py (from OpenPI)
- Remove debug logging blocks from lerobot_train.py
* refactor: extract compute_delta_action_stats into compute_stats.py
Move the ~70-line inline delta action stats block from lerobot_train.py
into a dedicated function in compute_stats.py, where all other stats
computation already lives. The training script now calls it in 6 lines.
* refactor: remove unused get_processed_left_over from ActionQueue
This method was never called outside of tests. Leftover actions for RTC
guidance are always retrieved via get_left_over() (delta/original space).
* revert: remove logging-only changes from eval_with_real_robot.py
The delta actions detection helper and log message added no functional
value — the script already handles delta policies correctly via the
processor pipeline.
* refactor: use ACTION/OBS_STATE constants instead of hardcoded strings
Replace hardcoded "action" and "observation.state" with ACTION and
OBS_STATE from utils.constants in compute_stats.py, dataset_tools.py,
and lerobot_train.py.
* style: remove stray blank lines in training loop
* refactor: move delta action stats to preprocessing step, remove on-the-fly computation
- Remove on-the-fly compute_delta_action_stats from lerobot_train.py
- Rewrite recompute_stats to delegate action stats to compute_delta_action_stats
(chunk-based sampling matching what the model sees during training)
- Add chunk_size parameter to recompute_stats for delta action computation
- Add delta actions documentation to pi0.mdx and pi05.mdx
* feat: add recompute_stats CLI operation to lerobot-edit-dataset
* fix(tests): relax quantile normalization test tolerance for 1e-6 epsilon
* chore: remove agents_memory/pr_details.md from repo
* refactor: rename delta actions to relative actions throughout
What OpenPI calls "DeltaActions" is actually UMI's "relative trajectory"
representation: each action in the chunk is an offset from the current
state, not from the previous action. This avoids error accumulation.
Renamed across all source, tests, docs, and CLI:
- DeltaActionsProcessorStep → RelativeActionsProcessorStep
- to_delta_actions → to_relative_actions
- use_delta_actions → use_relative_actions
- delta_exclude_joints → relative_exclude_joints
- compute_delta_action_stats → compute_relative_action_stats
- delta_action_processor.py → relative_action_processor.py
- test_delta_actions.py → test_relative_actions.py
Kept as-is: AbsoluteActionsProcessorStep (converts TO absolute),
registry ID "delta_actions_processor" (backward compat), and unrelated
delta references (IK pipeline, Robosuite, RA-BC metrics, gym envs).
* docs: add Action Representations guide
Dedicated page explaining absolute, relative, and delta actions with
numerical examples, joint vs EE space, and how to use kinematics
pipelines and the relative action processor. References UMI paper
(Chi et al., 2024) for the terminology.
* docs: remove redundant OpenPI naming note from action representations
* docs: remove opinionated OpenPI reference from delta actions section
* docs: replace ASCII diagram with UMI paper figure
* docs: remove OpenPI reference from action representations
* docs: use HF-hosted image instead of local asset
* docs: clarify figure attribution
* revert: restore original normalization epsilon behavior
The 1e-6 unconditional epsilon change perturbed all normalized values,
breaking backward compatibility tests. The original approach (1e-8 eps
for MEAN_STD, conditional torch.where for QUANTILES) already handles
division by zero correctly without affecting non-degenerate cases.
* fix: restore delta_action_processor.py used by phone/RL teleop
The rename commit incorrectly deleted delta_action_processor.py and
duplicated its classes into relative_action_processor.py. Restore the
original file and import from it instead.
* fix(processor): address PR #2970 review comments
- Remove shebang from relative_action_processor.py (library module, not script)
- Add device alignment in to_relative_actions/to_absolute_actions so _last_state
on CPU doesn't cause cross-device errors when actions are on CUDA
- Rename delta_step → relative_step in AbsoluteActionsProcessorStep for naming
consistency; update factory.py, all processor files, and tests
- Expand _reconnect_relative_absolute_steps docstring to explain why post-hoc
rewiring is needed after deserialization
- Fix off-by-one in compute_stats.py: sample_upper_bound = total_frames - chunk_size + 1
so last valid start index is included and total_frames == chunk_size is not rejected
- Remove redundant NOTE comment in processor_pi05.py (duplicated two lines below)
- Fix pi0_fast processor ordering: move relative_step before NormalizerProcessorStep
so normalizer sees delta actions (matching pi0/pi05); flip postprocessor to
unnormalize → absolute accordingly. Relative stats are now required for all pi models
- Revert use_relative_joint_actions_aloha → use_delta_joint_actions_aloha in
configuration_smolvla.py (preserve existing public API)
- Update action_representations.mdx: add missing joint to 6-DOF example, fix
'based on a figure', clarify pi family ordering, add RTC compatibility section
* update rtc link
* feat: compute relative action stats over full dataset with optional parallelism
Remove the 100k sample cap from compute_relative_action_stats and process
all valid chunks. Vectorize with numpy (pre-load actions/states, fancy
indexing + broadcasting) for a large speedup over the per-index HF dataset
loop. Add num_workers param for thread-based parallelism (numpy releases
the GIL). Update docs to show --push_to_hub for recompute_stats.
* style: apply ruff formatting to compute_stats.py
* testing on real robot
* style: fix ruff format and remove redundant .keys() calls
* Add multitask diffusion transformer policy
Add multitask diffusion transformer policy
* expand the observation encoder to support differnt size encoders for vision and text
* add RoPE attention module as this is shown to help training dynamics and generation quality for DiTs
* update readme and citations for multitask dit policy
* remove dino vision encoder and simplify text and vision encoders by removing inheritance structure
* adjust factory comment
* update docstring for multitask dit policy processor file
* simplify config for multitask dit by merging and flattening everything, then adding comments to denote where some parameters are only used for specific objectives
* add references to the modeling file comments
* merge all modules files into the main modeling file
* add torch.no_grad decorators
* split up select action return statement
* remove redundant asserts
* add tutorial to training with multi_task_dit
* fix bugs when testing on hardware
* remove environment state conditioning
* update typo in test instruction comment
* add processor tests to multitask dit tests
* move policy to top of file
* use constants for indexing into batches and remove env state references
* remove the base classes since we don't need to be able to extend
* fix nit formatting in generate actions fcn
* reformat and clean up tutorial for multitask dit policy
* add more descriptions and depth to multitask dit tutorial
* note origins of each training objective
* rename config param for multiple vision encoders
* refactor code to perform task tokenization in the processor instead of in the modeling code for multitask dit
* add multitask dit to toc for docs
* add conditional transformers import to match all other policies that use transformers lib
* add test handling for multitask dit when transformers isnt available
* skip tests without transformers
* remove cropping of images smaller than the crop size
* add kwargs arg to multitask dit constructor
* add wallx dep conflict management for multitask dit policy
* use hyphens for cleanliness in pyproject.toml
* add conflict management to pyproject toml for pi conflict for mtdp as well
* update tests script to not use unnecessary uv sync call which resolves dependencies that do not need to run. This drastically reduces CI run time
* revert fast tests edits
* update docs and readme files, fixing some typos and adding multitask dit to readme
* chore(dependencies): upgrade transformers + hggingface-hub + peft + scipy
* chore(dependencies): bump pi0 family to transformers v5
* chore(dependencies): bump wall x to transformers v5
* chore(dependencies): bump gr00t to transformers v5
* chore(style): fix pre-commit
* fix(policy): xvla forced_bos_token missing
* test(rl): skip ci tests for resnet10
* Fix: full pi models support for transformer v5 (#2967)
* fix(pi): remove loss truncation
* fix(pi): remove state padding before tokenization
* fix(pi): fix image padding value
* fix from_pretrain
* add transformer v5 changes
* remove reference
* more fixes
* make it work
* add support for rest of pi family
* add pifast work
* more changes
* more changes
* more cleanup
* fix torch params
* dtype fix
* torch compile
* embed mismatch fix
* revert groot
* more nit fixes
* remove unused classes
* more fixes
* revert
* nit
* torch dtype warning fix
* but back dynamic renaming
* add tie embedding
---------
Co-authored-by: Yufei Sun <skieyfly@gmail.com>
* chore: fix XVLA in transformers v5 (#3006)
* test(policies): enable wall x CI testing
* style(test): pre-commit check
* style(test): pre-commit
---------
Signed-off-by: Bryson Jones <63133702+brysonjones@users.noreply.github.com>
Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Jade Choghari <chogharijade@gmail.com>
Co-authored-by: Yufei Sun <skieyfly@gmail.com>
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
* refactor(dataset): enhance dataset root directory handling and introduce hub cache support
- Updated DatasetConfig and LeRobotDatasetMetadata to clarify root directory behavior and introduce a dedicated hub cache for downloads.
- Refactored LeRobotDataset and StreamingLeRobotDataset to utilize the new hub cache and improve directory management.
- Added tests to ensure correct behavior when using the hub cache and handling different revisions without a specified root directory.
* refactor(dataset): improve root directory handling in LeRobotDataset
- Updated LeRobotDataset to store the requested root path separately from the actual root path.
- Adjusted metadata loading to use the requested root, enhancing clarity and consistency in directory management.
* refactor(dataset): minor improvements for hub cache support
* chore(datasets): guard in resume + assertion test
---------
Co-authored-by: AdilZouitine <adilzouitinegm@gmail.com>
Co-authored-by: mickaelChen <mickael.chen.levinson@gmail.com>
* refactor(dataset): split reader and writer
* chore(dataset): remove proxys
* refactor(dataset): better reader & writer encapsulation
* refactor(datasets): clean API + reduce leaky implementations
* refactor(dataset): API cleaning for writer, reader and meta
* refactor(dataset): expose writer & reader + other minor improvements
* refactor(dataset): improve teardown routine
* refactor(dataset): add hf_dataset property at the facade level
* chore(dataset): add init for datasset module
* docs(dataset): add docstrings for public API of the dataset classes
* tests(dataset): add tests for new classes
* fix(dataset): remove circular dependecy
* fix(vqbet): use in-place fill_ to avoid overwriting DDP GPU buffers with CPU tensors
When VQ discretization phase completes, the code was overwriting
register_buffer('discretized') and register_buffer('freeze_codebook')
with torch.tensor(True), which is created on CPU. DDP then fails in
_sync_buffers() with: RuntimeError: No backend type associated with
device type cpu. Fix by updating the buffers in-place with .fill_(True)
so device and registration are preserved.
Made-with: Cursor
* test(vqbet): add regression test for in-place buffer update during discretization
Verifies that discretize() updates the 'discretized' and 'freeze_codebook'
registered buffers in-place (via fill_()) rather than replacing them with new
CPU tensors. The test checks data_ptr() identity and that the tensors remain
registered buffers after the call. This prevents regressions of the DDP fix.
Made-with: Cursor
* test(vqbet): add GPU regression test to verify buffers stay on CUDA after discretize()
Directly catches the original DDP failure mode: when buffers are replaced with
torch.tensor(True) they land on CPU, causing NCCL to raise 'No backend type
associated with device type cpu' in _sync_buffers(). The GPU test places the
model on cuda:0 and asserts both buffers remain on CUDA after discretization.
Made-with: Cursor
* test(vqbet): simplify to single device-check test in test_policies.py
Per reviewer feedback: remove the separate test file and replace the two
CPU/GPU tests (with data_ptr checks) with a single focused test in
tests/policies/test_policies.py that only asserts the registered buffers
remain on the model device after discretize(). Uses DEVICE from tests/utils.py
so it runs on whatever device the CI/user selects (cpu, cuda, mps).
Made-with: Cursor
* style: fix import order in test_policies.py to pass ruff/pre-commit checks
Made-with: Cursor
---------
Co-authored-by: Zhan DiJia <2476100824@example.com>
Co-authored-by: Khalil Meftah <khalil.meftah@huggingface.co>
* fix(ci): skip HF log in (and tests) in forks and community PRs
* chore(test): remove comment about test meant to be only run locally
* fix(tests): no hf log in decorator for xvla
* fix(test): no decorator in yield
* chore(docstrings): updating v2.1-v3.0 conversion script docstrings to match the new task label
* chore(task): renamming the default index label in the tasks DataFrame to task
* Revert "chore(docstrings): updating v2.1-v3.0 conversion script docstrings to match the new task label"
This reverts commit f55de3255278f23f18b5d955565f6768d094951d.
* chore(docstrings): updating docstrings to match dataset v3.0 architecture
* chore(format): formatting code
* fix(root): adding proper support for the root and new_root arguments
* feat(roots): adding a roots agrument for the merge operation
* chore(clean): cleaning up code
* chore(doctrings): updating doctrings with new features
* fix(repo_id): setting repo_id to None when not needed
* fix(roots/repo_ids): making mypy happy by using repo_ids and roots for merge operation
* fix(path): fixing path related issues
* fix(repo_id): fixing issues related to repo_id
* chore(doctrings): updating docstrings + fix typo
* chore(clean): cleaning code
* fix(split new_repo_id): reverting new_repo_id addition for split operation
* docs(dosctrings): completing docstrings
* fix(repo_ids/roots): improving checks for repo_ids/roots lengths
* fix(repo_ids): making repo_ids optional in MergeConfig but raise if not given
* fix(docstrings): fixing docstrings for split operation
* fix(hints): updating get_output_path hints to accept paths as strings too
* fix(y/N prompts): removing y/N prompts in lerobot_edit_dataset
* fix(merge repo_id): fixing merge operation to use new_repo_id instead of repo_id
* fix(typo): fixing typo in doctrings
* Add New featrue to lerobot_edit_datset.py that show dataset information.
* Fix to draccus error when happen give only --operation.type=info
* Updating test and documents regarding lerobot-edit-dataset info function.
* Updating documents regarding lerobot-edit-dataset extract function. option name in document is mistake.
* feat(datasets): Update to align formatting with pre-commit.(#2917)
Update to align formatting by pre-commit.
---------
Co-authored-by: Caroline Pascal <caroline8.pascal@gmail.com>
* feat(datasets): add modify_tasks function for in-place task editing
Add a new utility function to modify tasks in LeRobotDataset in-place.
This allows users to:
- Set a single task for all episodes
- Set specific tasks for individual episodes
- Combine a default task with per-episode overrides
* feat(edit-dataset): add CLI support for modify_tasks operation
Integrate the modify_tasks function into lerobot_edit_dataset CLI.
Users can now modify dataset tasks via command line:
Supports setting a default task, per-episode tasks, or both combined.
* test(datasets): add tests for modify_tasks function
Add comprehensive test coverage for the modify_tasks utility:
- Single task for all episodes
- Episode-specific task assignment
- Default task with per-episode overrides
- Error handling for missing/invalid arguments
- Verification of task_index correctness
- In-place modification behavior
- Metadata preservation
* respond to copilot review
* fix(sac): make temperature a property to fix checkpoint resume bug
Temperature was stored as a plain float and not restored after loading
a checkpoint, causing incorrect loss computations until update_temperature()
was called. Changed to a property that always computes from log_alpha,
ensuring correct behavior after checkpoint loading.
* simplify docstrings
* feat(cameras): add new read_latest() method
* fix(cameras): fix threading bug + clear state
* refactor(cameras): multiple improvements
* feat(camera): add context manager to camera base class
* chore(camera): slight modifications to opencv
* test(cameras): update opencv tests according to the changes
* refactor(cameras): reflect desing changes to realsense + deal with depth
* test(cameras): fix realsense tests accordingly to new changes
* refactor(cameras): update reachymini and zmq accordingly
* chore: wrap resource sensitive examples into a try/finally
* test(cameras): add test for new read_latest
* test(cameras): fix problem with image artifact in opencv tests
* test(cameras): fix test_read_latest_high_frequency expectations
* Apply suggestions from code review 1
Co-authored-by: Caroline Pascal <caroline8.pascal@gmail.com>
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
* chore(cameras): address feedback
* feat(cameras): add max_age_ms check in read_latest
* test(cameras): fix read_latest tests
* chore(redundancies): removing redundancies in Reachy 2 camera class
* fix(warmup): replacing the arbitrary time.sleep in by an actual warmup in the RealSense camera class
* chore(format): formatting latest changes
* chore(warning): adding a "to be implemented" warning for read_latest() in Camera base class
* chore(warning): making read_latest() warning message shorter and clearer
---------
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Caroline Pascal <caroline8.pascal@gmail.com>
* Fix aggeregation of datasets when subdatasets are already a result of a previous merge
* docstring
* respond to copilot review + add regression test
* Remove unnecessary int conversion for indicies
* fix(motors): cleanup imports + fix signatures
* feat(motors): add damiao canbus + multiple fixes
* fix(motors): address comments -> last_state + different gains + sleep
* refactor(motors): reduce duplicated code + adressed some comments in the PR
* chore(motors): better timeouts
* tests(motors): damiao test and imports
* chore(deps): fix space
* Apply suggestions from code review
Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
* chore(motors): remove normalization tables damiao
* fix(motors): imports and signatures
* feat(motors): add motor_type_str + recv_id to motor class and _get_motor_recv_id raises if no motor_obj.recv_id
* chore(motors): remove normalize from base motor class and damaio
* tests(motors): remove bad tests (to be replaced)
* chore(motors): updated import check
* use constant for kp and kd range and check responses in mit_control_batch()
* Add docs on setting up canbus and use damiao otor bus, also add lerobot_setup_can.py and log if there is not response from a write command
* precommit format
* supress bandit as these are intentional cli commands
* fix setup-can
* add test
* skip test in ci
* nit precommit
* update doc example
* dont import can for tests
---------
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Pepijn <pepijn@huggingface.co>
* feat(async_inference): server always sends CPU tensors, client handles device conversion
* fix:fix the type annotation of RawObservation in src/lerobot/async_inference/helpers.py
* update the import of robot_client
---------
Co-authored-by: Sato shinji <wwwsatoshinji@gmail.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: KB <kevin-brian.n-diaye@epita.fr>
* improve image2video
* add episodes video encoding
* fix mypy failing
* iterate on review
* nit
* remove max, and let it be optional
* iterate more
* update docs
* fix test
---------
Co-authored-by: Michel Aractingi <michel.aractingi@huggingface.co>
* fix: use features when aggregating image based datasets
* add: test asserting for data type
* add: features param to writing dataset
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* feat(robots): consolidates bi SO setups
* fix(robots): solve circular dependecy
* fix(robots): teleop & record working
* feat(robots): only one SO
* fix(utils): rename bi so
* fix(scripts): bi so import
* fix(rl): remove imports
Francesco Capuano