* pi fixes for dependencies
* add walls sarm conflict
* also add conflicts for pi
* fix(ci): use --extra all instead of --all-extras + --no-extra
---------
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
* support wallx
* fix bugs in flow
* incorporate wallx model into lerobot
* update the policy methods
* reduce to least config and params & pass lerobot basic test
* fixed dtype bugs
* add wallx dependencies
* update
* remove flash-attn requirement && fix bug in inference and fast mode
* fix bug for inference
* add some small modifications
* fix pre-commit errors
* remove lerobot[wallx]
* fix ci
* fix precommit issues
* fix: exclude wallx extra properly in CI workflows
* fix: add uv conflicts for wallx transformers version
* fix: peft test import
* pre-commit
* only export WallXConfig from wall_x package to avoid peft import in CI
* remove torch dep
* precommit
* add import
---------
Co-authored-by: vincentchen <chenlufang@x2robot.com>
Co-authored-by: Geoffrey19 <sympathischmann35@gmail.com>
Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Pepijn <pepijn@huggingface.co>
* fix(optim): enable and resolve mypy type errors
Resolves#1729
build(deps): add mypy as dependency and update pre-commit hook
* change build's type annotation
* add initial modeling
* make rewind pretrained policy
* add annotation
* small fix
* add sarm
* subtasks
* fix spawn
* fix rewind discrepancies
* Add script to generate embedding for dataset (#2138)
* Add generate and validate script
* fix precommit
* Improve generate embeddings function by using dataset tools (#2206)
---------
Co-authored-by: Michel Aractingi <michel.aractingi@huggingface.co>
* cleanup
* change order train log
* print batch size
* update sarm processor
* add reward output
* change expected features
* add image validation
* change validation
* get state input from dataset stats
* raise if no state key is found
* pass stats
* cleanup and refactor
* add episode inddex to complementary data
* add subtask init and detection
* revert lerobot_train changes
* pass dataset metadata to policy
* change loadig subtasks
* add small logging
* fix progress conversion and adding initial frame
* use large offset for initial frame (ugly)
* Remove rewind, use clip tokenizer
* add tests, implement formula 1,2 correctly and cleanup
* use task from dataset, cleanup visualizer
* simplify
* simplify and cleanup code and move compute_temporal_proportions to utils
* fix normalization in visualization
* Fix visualization and change prompt
* fix formatting
* add visualize subtask annotations
* use qwen thinking
* try different prompt
* format
* update prompt
* higher temp, long output
* different settings
* use instruct
* show full resp
* split message
* Temp: increase tolerance dataset
* Fix RA-BC (#2572)
* Add next observation loading for RA-BC progress deltas
* Compute weights based on temporal progress deltas instead of static rewards
* Add hard-masking for negative progress deltas in weight computation
* Feat/add dual head (#2582)
* Add dual dense sparse head and annotation
* Add docs
* add dual to procesor
* cleanup
* change sampling in visualize and cleanup
* remove validation
* remove compile
* Feat/test uniform (#2587)
* test uniform
* add different string for misaligned
* Fix rewind and add tests
* uncomment text implementation
* run precommit
* Add head mode for ra-bc
* fix visalization of single task
* add
* return per sample loss
* Fix RA_BC (#2602)
* update rabc implementation
* compute rabc beforehand
* fix import
* add only progress calulation
* use precomputed progress
* multi gpu processing
* import
* fix dataset meta data extraction
* add logging
* logging
* log
* progress per episode
* split differently
* move clip to gpu
* pre decode frames for an episode
* fix cuda initalization
* fix import
* multi processing
* rename
* fix import
* fix
* fix rabc
* use last known progress if oob
* use last known progress if oob
* add misalignment loss with random embeddings
* discard previous changes
* add selection of models to docs for ra_bc
* add transformers dep
* extend tolerance
* initial commit with new codebase
* add tests
* fix
* remove temporal sampler
* drop last frame for sampler
* use original ref
* some fixes
* fix visualization
* remove smoothing and fix order subtasks
* add stride rabc computation
* add push to hub
* add explanation
* add kappa expllaination
* better rabc logging
* feedback pr
* remove dataset tolerance
* revert dataset tool
* revert dataset changes
* add credit
* run precommit
* change path for generate ra_bc
* fix type
* include sarm in all in pyproject
* fix precommit
* lazy import matplotlib
* lazy import qwen
* remove rich console
* skip if transformers is not installed?
* run only when we have faker
* place transformer lazy loading
* Dont test if low transformer version
* fix
* increase transformer
* increase as 4.57.0 is yanked
* remove pi from all
* go back
---------
Co-authored-by: Michel Aractingi <michel.aractingi@huggingface.co>
Co-authored-by: s1lent4gnt <kmeftah.khalil@gmail.com>
* upload
* feat(omx): simplify motor initialization and remove default calibration files
* feat(omx): read motor positions without normalization for improved accuracy
* update calibration method for return factory value
Signed-off-by: Junha Cha <ckwnsgk1@gachon.ac.kr>
* change the drive mode
* refactor: clean up code by removing unnecessary blank lines in omx_follower and omx_leader modules
* feat(omx): update calibration method to set drive modes for motors
* feat(pyproject): add 'ROBOTIS' to extend-ignore-identifiers-re list
* feat(omx): enhance calibration method to write default drive modes to motors
* Update src/lerobot/robots/omx_follower/__init__.py
Add informations about the robot
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Signed-off-by: Woojin Wie <dnldnwls1123@gmail.com>
---------
Signed-off-by: Junha Cha <ckwnsgk1@gachon.ac.kr>
Signed-off-by: Woojin Wie <dnldnwls1123@gmail.com>
Co-authored-by: Junha02 <chajunha2023@naver.com>
Co-authored-by: Junha Cha <ckwnsgk1@gachon.ac.kr>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* fix: update policy handling and type annotations
added typehint and addressed the error of mypy
* fix: rename should_push_to_hub to push_to_hub
I find that there are other dependencies of push_to_hub so I fix the property name back to original one.
* fix: typo
* fix: changed the position of try-except block
As the copilot said, use raise before `hf_hub_download` would stop program even it is able to download
* fix: update pre-commit configuration and mypy settings
add args: --follow-imports=silent to pass error which have no relationship with src/lerobot/configs
* fix: remove the specific path in .pre-commit-config.yaml
* feat: enhance typehint to adapt mypy strict mode.
* fix: remove duplicate FileNotFoundError check in PreTrainedConfig
* fix: make "pre-commit run --all-files" pass
* fix: replace logging with logger for better logging practices
* fix: fixed extra changes of lint and format changes
* fix: fixed extra changes out of "configs" module
* Update src/lerobot/configs/policies.py
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Signed-off-by: tetsugo02 <131431116+tetsugo02@users.noreply.github.com>
* fix: add logging for scratch job
---------
Signed-off-by: Adil Zouitine <adilzouitinegm@gmail.com>
Signed-off-by: tetsugo02 <131431116+tetsugo02@users.noreply.github.com>
Co-authored-by: Adil Zouitine <adilzouitinegm@gmail.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* feat(mypy-compliant): Ensure the model module passes MyPy type checks
* fix
* uncomment pyproject.toml for model module
* fix
* fix
---------
Co-authored-by: Adil Zouitine <adilzouitinegm@gmail.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* Enhance training and logging functionality with accelerator support
- Added support for multi-GPU training by introducing an `accelerator` parameter in training functions.
- Updated `update_policy` to handle gradient updates based on the presence of an accelerator.
- Modified logging to prevent duplicate messages in non-main processes.
- Enhanced `set_seed` and `get_safe_torch_device` functions to accommodate accelerator usage.
- Updated `MetricsTracker` to account for the number of processes when calculating metrics.
- Introduced a new feature in `pyproject.toml` for the `accelerate` library dependency.
* Initialize logging in training script for both main and non-main processes
- Added `init_logging` calls to ensure proper logging setup when using the accelerator and in standard training mode.
- This change enhances the clarity and consistency of logging during training sessions.
* add docs and only push model once
* Place logging under accelerate and update docs
* fix pre commit
* only log in main process
* main logging
* try with local rank
* add tests
* change runner
* fix test
* dont push to hub in multi gpu tests
* pre download dataset in tests
* small fixes
* fix path optimizer state
* update docs, and small improvements in train
* simplify accelerate main process detection
* small improvements in train
* fix OOM bug
* change accelerate detection
* add some debugging
* always use accelerate
* cleanup update method
* cleanup
* fix bug
* scale lr decay if we reduce steps
* cleanup logging
* fix formatting
* encorperate feedback pr
* add min memory to cpu tests
* use accelerate to determin logging
* fix precommit and fix tests
* chore: minor details
---------
Co-authored-by: AdilZouitine <adilzouitinegm@gmail.com>
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
* feat(dataset-tools): add dataset utilities and example script
- Introduced dataset tools for LeRobotDataset, including functions for deleting episodes, splitting datasets, adding/removing features, and merging datasets.
- Added an example script demonstrating the usage of these utilities.
- Implemented comprehensive tests for all new functionalities to ensure reliability and correctness.
* style fixes
* move example to dataset dir
* missing lisence
* fixes mostly path
* clean comments
* move tests to functions instead of class based
* - fix video editting, decode, delete frames and rencode video
- copy unchanged video and parquet files to avoid recreating the entire dataset
* Fortify tooling tests
* Fix type issue resulting from saving numpy arrays with shape 3,1,1
* added lerobot_edit_dataset
* - revert changes in examples
- remove hardcoded split names
* update comment
* fix comment
add lerobot-edit-dataset shortcut
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Michel Aractingi <michel.aractingi@huggingface.co>
* style nit after copilot review
* fix: bug in dataset root when editing the dataset in place (without setting new_repo_id
* Fix bug in aggregate.py when accumelating video timestamps; add tests to fortify aggregate videos
* Added missing output repo id
* migrate delete episode to using pyav instead of decoding, writing frames to disk and encoding again.
Co-authored-by: Caroline Pascal <caroline8.pascal@gmail.com>
* added modified suffix in case repo_id is not set in delete_episode
* adding docs for dataset tools
* bump av version and add back time_base assignment
* linter
* modified push_to_hub logic in lerobot_edit_dataset
* fix(progress bar): fixing the progress bar issue in dataset tools
* chore(concatenate): removing no longer needed concatenate_datasets usage
* fix(file sizes forwarding): forwarding files and chunk sizes in metadata info when splitting and aggregating datasets
* style fix
* refactor(aggregate): Fix video indexing and timestamp bugs in dataset merging
There were three critical bugs in aggregate.py that prevented correct dataset merging:
1. Video file indices: Changed from += to = assignment to correctly reference
merged video files
2. Video timestamps: Implemented per-source-file offset tracking to maintain
continuous timestamps when merging split datasets (was causing non-monotonic
timestamp warnings)
3. File rotation offsets: Store timestamp offsets after rotation decision to
prevent out-of-bounds frame access (was causing "Invalid frame index" errors
with small file size limits)
Changes:
- Updated update_meta_data() to apply per-source-file timestamp offsets
- Updated aggregate_videos() to track offsets correctly during file rotation
- Added get_video_duration_in_s import for duration calculation
* Improved docs for split dataset and added a check for the possible case that the split size results in zero episodes
* chore(docs): update merge documentation details
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
---------
Co-authored-by: CarolinePascal <caroline8.pascal@gmail.com>
Co-authored-by: Jack Vial <vialjack@gmail.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* initial commit
* change device in test
* do detailed import
* adhere to python 3.11 syntax
* fix autodocstring
* additionally
* do same in other files
* add model. prefix to all keys in state dict
* use dummy stats
* add pi05
* also shorten action_steps
* fix test
* all test pass! and fix tokenizer max length between 05 and 0
* remove test
* fix transformer dependency
* fix test
* split pi0 and pi05 policy in seperate files
* fix test
* fix push to hub test
* add some comments, license and readme
* remove warning in config
* add pi05 to factory
* remove check
* rename action_horizon to chunk_size
* clean up padding of state and action (more in line with lerobot pi0)
* add openpi image transforms for training and add more flexibility to _preprocess_images similar to lerobot pi0
* fix key match from pytorch state dict (similar keys to openpi implementation now)
* also for pi05
* update to python 3.11
* revert to openpi transformer replace python 3.11
* fix(modeling pi0): nit warning message
* use safeauto_docstring
* fix: remove unused param
* fix from pretrained
* add preprocess tests
* also compile forward method
* Do not add model prefix to normalization
* use same name for action and state dim as lerobot pi0 and remove fixed image keys
* load from pretrained_path
* temp: hardcode base model
* fix override self.pretrained_path = None overwrite
* rename to loss
* remove additional image augmentations, lerobot dataset already does this
* Add docs
* put tests in test folder
* Add test to instatiate all base models
* go back to python 3.10
* update docs
* adapt docs pi05
* change docs: finetune base model options
* minor docs fixes and dependencies
* remove todo
* cast float64 to float32 for mps
* skip if no transformers
* fix tests
* add new models to modelcard
* add back init
* fix circular input
* feat: only run pi test on GPU
* remove require_nightly_gpu
* replace decorator test_pi0_openpi
* rename action_dim, state_dim to max_action_dim, max_state_dim
* fix doc and constants
* cleanup tests
* fix from pretrained
* fix tests
* add comment pi0 pi05 tests, add image features to pi0 pi05 hub tests
* fix, state is included in language not in flow head
* Move test to specific folder
* and paligemma task with newline
* remove add_special_tokens, not needed
* feedback pr
* Remove previous pi0 and rename pi0_openpi and pi05_openpi
* Add Quantile stats to LeRobotDataset (#1985)
* - Add RunningQuantileStats class for efficient histogram-based quantile computation
- Integrate quantile parameters (compute_quantiles, quantiles) into LeRobotDataset
- Support quantile computation during episode collection and aggregation
- Add comprehensive function-based test suite (24 tests) for quantile functionality
- Maintain full backward compatibility with existing stats computation
- Enable configurable quantiles (default: [0.01, 0.99]) for robust normalization
* style fixes, make quantiles computation by default to new datasets
* fix tests
* - Added DEFAULT_QUANTILES=[0.01, 0.10, 0.50, 0.90, 0.99] to be computed for each features instead of being chosen by the user
- Fortified tests.
* - add helper functions to reshape stats
- add missing test for quantiles
* - Add QUANTILE normalization mode to normalize the data with the 1st and 99th percentiles.
- Add QUANTILE10 normalization mode to normalize the data with the 10th and 90th percentiles.
* style fixes
* Added missing lisence
* Simplify compute_stats
* - added script `augment_dataset_quantile_stats.py` so that we can add quantile stats to existing v3 datasets that dont have quatniles
- modified quantile computation instead of using the edge for the value, interpolate the values in the bin
* rename pi0/pi05 files
* Remove open pi patch and use custom transformer branch for now
* renaming
* fix
* Revert "fix"
This reverts commit 1ea65730ac.
* fix naming
* feet(pi0/pi0.5): add pipeline (#2009)
* feat(processor): convert openpi model with processor
* TODO: Make test works
* fix(modeling_pi0openpi): update attention mask value and time scaling; improve task handling in tests
- Changed the attention mask value from `self.config.attention_mask_value` to a fixed value of `-2.3819763e38`.
- Updated time scaling in the `sample_noise` method to use a constant factor of `0.999` and an offset of `0.001`.
- Enhanced task handling in tests to ensure proper formatting and batch size consistency.
- Cleaned up commented-out test code for clarity.
* refactor(pi0): rename PI0OpenPIConfig and PI0OpenPIPolicy to PI0Config and PI0Policy
- Updated imports and references throughout the codebase to reflect the new naming convention.
- Introduced a new processor file for PI0 to handle pre-processing and post-processing steps.
- Adjusted tests to utilize the renamed classes, ensuring consistency and functionality.
- Enhanced clarity and maintainability by removing outdated naming conventions.
* refactor(pi05): rename PI0OpenPIPolicy to PI0Policy and update configuration
- Renamed `PI0OpenPIPolicy` to `PI0Policy` for consistency with naming conventions.
- Updated the `PI05OpenPIConfig` to include a new `tokenizer_max_length` attribute and changed the normalization mode for state from `MEAN_STD` to `QUANTILES`.
- Simplified model initialization in `PI05OpenPIPolicy` by removing unused `dataset_stats` parameter.
- Added a new processor class for `Pi05PrepareStateTokenizerProcessorStep` with `@dataclass` for improved readability.
- Introduced a test script to compare the integration of the PI0OpenPI policy with the original implementation, ensuring local testing compatibility.
* feat(processor): convert openpi model with processor
* TODO: Make test works
* fix(modeling_pi0openpi): update attention mask value and time scaling; improve task handling in tests
- Changed the attention mask value from `self.config.attention_mask_value` to a fixed value of `-2.3819763e38`.
- Updated time scaling in the `sample_noise` method to use a constant factor of `0.999` and an offset of `0.001`.
- Enhanced task handling in tests to ensure proper formatting and batch size consistency.
- Cleaned up commented-out test code for clarity.
* refactor(pi0): rename PI0OpenPIConfig and PI0OpenPIPolicy to PI0Config and PI0Policy
- Updated imports and references throughout the codebase to reflect the new naming convention.
- Introduced a new processor file for PI0 to handle pre-processing and post-processing steps.
- Adjusted tests to utilize the renamed classes, ensuring consistency and functionality.
- Enhanced clarity and maintainability by removing outdated naming conventions.
* refactor(pi05): rename PI0OpenPIPolicy to PI0Policy and update configuration
- Renamed `PI0OpenPIPolicy` to `PI0Policy` for consistency with naming conventions.
- Updated the `PI05OpenPIConfig` to include a new `tokenizer_max_length` attribute and changed the normalization mode for state from `MEAN_STD` to `QUANTILES`.
- Simplified model initialization in `PI05OpenPIPolicy` by removing unused `dataset_stats` parameter.
- Added a new processor class for `Pi05PrepareStateTokenizerProcessorStep` with `@dataclass` for improved readability.
- Introduced a test script to compare the integration of the PI0OpenPI policy with the original implementation, ensuring local testing compatibility.
* refactor(pi05): update imports and rename configuration classes
- Changed imports to reflect the new naming convention for PI05 configuration and policy classes.
- Renamed `PI05OpenPIConfig` to `PI05Config` and `PI05OpenPIPolicy` to `PI05Policy` for consistency.
- Introduced a new processor file for PI05, implementing pre-processing and post-processing steps.
- Updated tests to utilize the renamed classes, ensuring functionality and consistency across the codebase.
* update(pi05): increase tokenizer_max_length for improved processing
- Changed the `tokenizer_max_length` from 48 to 200 to enhance the model's capability in handling longer sequences.
- This adjustment aims to improve the overall performance and flexibility of the PI05 configuration.
* add default for state (max_state_dim)
* correct naming
* fix import
* cleanup code
* remove unused test
* us quantiles for action
* move to device
* remove discrete state assert
* fix pi05 test
* move pi05 to device
* use base models in comparison tests
* small renames for tests
* change number of tokens pi05 test
* fix openpi tokenization in test
* fix hub test
* fix test
* assert lerobot vs openpi tests
---------
Co-authored-by: Pepijn <pepijn@huggingface.co>
* add headers
* add back previously removed imports
* update if statement load processor with dataset stats
* remove to avoid circular import
* inject dataset stats for pretrained models
* check normalization before applying
* add link to quantile augument script
* fix(policies): transformers import for ci in PI0 & PI05 (#2039)
* fix(policies): transformers import for ci in PI0
* fix(policies): transformers import for ci in PI05
* test(processor): fix expected raise when normalization types are missing (#2040)
* switch normalization order pipeline for pi05
* Fix/quantiles script (#2064)
* refactor augment stats with quantiles script
add parallelization for faster processing
shift the quantile normalization between -1 1
* fix replay buffer tests
* fix comment
* overwrite the pipeline normalization features with the policy features
* remove double normalization overwrite
* cleanup from pretrained
* remove typo
* also set norm_map
* fix(augment_quantiles) images incorrectly divided by 255
* clamp quantiles
* link to lerobot base models
* rename tests
* encorperate PR feedback
* update docstring for RunningQuantileStats
* update doc links
* Revert "clamp quantiles"
This reverts commit 172207471c.
* fix self.paligemma
* fix tests related to quantiles that were scaled to [0,1], the new range is [-1, 1]
* fix libero doc and use different transformer branch
* use fix branch instead of feat
* update results libero
* add new line
* fix formatting
* precommit
* update results libero
* update libero doc
* update title
* final changes
* add quantiles to test
* run pre commit
---------
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Michel Aractingi <michel.aractingi@huggingface.co>
Co-authored-by: Adil Zouitine <adilzouitinegm@gmail.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
* Fix configs.py None MyPy error
* Use img_tensor instead of img in utils.py
* Add type assertion in factory.py
* Resolve merge conflict
* Uncomment envs moodule for mypy checks in pyproject.toml
---------
Signed-off-by: Adil Zouitine <adilzouitinegm@gmail.com>
Co-authored-by: Adil Zouitine <adilzouitinegm@gmail.com>
* feat(mypy): enable type checking for envs module and configure mypy settings in pyproject.toml
* Add mypy configuration to check only the envs module.
* Exclude examples, benchmarks, and tests from type checking.
* Set ignore_missing_imports to true and follow_imports to skip.
* chore: comment out mypy configuration in pyproject.toml and pre-commit-config.yaml
* Comment out mypy settings to disable type checking for the envs module.
* Update pre-commit configuration to reflect changes in mypy settings.
* chore(scripts): rename script dataset viz
* feat(scripts): add entry point for dataset-viz
---------
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
* refactor(scripts): update system info script
* chore(scripts): rename info script
* feat(scripts): add entrypoint for info
* chore(ci): update issue report template
Steven Palma