Commit Graph

231 Commits

Author SHA1 Message Date
Bryson Jones f5f9833540 add kwargs arg to multitask dit constructor 2025-12-25 04:14:56 -08:00
Bryson Jones 2a3444a8dd Merge branch 'main' into feature/add-multitask-dit 2025-12-23 15:06:41 -08:00
Pepijn ff271e8b51 pi fixes for dependencies (#2706)
* pi fixes for dependencies

* add walls sarm conflict

* also add conflicts for pi

* fix(ci): use --extra all instead of --all-extras + --no-extra

---------

Co-authored-by: Steven Palma <steven.palma@huggingface.co>
2025-12-23 23:58:34 +01:00
Bryson Jones 3e5f31e0be Merge branch 'main' into feature/add-multitask-dit
Signed-off-by: Bryson Jones <63133702+brysonjones@users.noreply.github.com>
2025-12-23 07:57:56 -08:00
Pepijn a142c365dd use syslink for wall-x readme (#2708)
* use syslink for wall-x readme

* remove whitespace
2025-12-23 14:13:32 +01:00
Bryson Jones d653f96420 Merge branch 'main' into feature/add-multitask-dit
Signed-off-by: Bryson Jones <63133702+brysonjones@users.noreply.github.com>
2025-12-22 08:40:08 -08:00
Tong Wu 17c5a0774f feat: support wallx model (#2593)
* support wallx

* fix bugs in flow

* incorporate wallx model into lerobot

* update the policy methods

* reduce to least config and params & pass lerobot basic test

* fixed dtype bugs

* add wallx dependencies

* update

* remove flash-attn requirement && fix bug in inference and fast mode

* fix bug for inference

* add some small modifications

* fix pre-commit errors

* remove lerobot[wallx]

* fix ci

* fix precommit issues

* fix: exclude wallx extra properly in CI workflows

* fix: add uv conflicts for wallx transformers version

* fix: peft test import

* pre-commit

* only export WallXConfig from wall_x package to avoid peft import in CI

* remove torch dep

* precommit

* add import

---------

Co-authored-by: vincentchen <chenlufang@x2robot.com>
Co-authored-by: Geoffrey19 <sympathischmann35@gmail.com>
Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Pepijn <pepijn@huggingface.co>
2025-12-22 10:12:39 +01:00
Pepijn 0071b1ff6e Add readme (#2698)
* Add readme

* change ref
2025-12-22 10:04:33 +01:00
Bryson Jones 2b90763597 Merge branch 'main' into feature/add-multitask-dit 2025-12-20 09:53:01 -08:00
Clément Verrier 00b5f65752 fix(optim): enable and resolve mypy type errors (#2683)
* fix(optim): enable and resolve mypy type errors

Resolves #1729

build(deps): add mypy as dependency and update pre-commit hook

* change build's type annotation
2025-12-20 17:19:42 +01:00
Bryson Jones 77dbc951cd Merge branch 'main' into feature/add-multitask-dit 2025-12-18 08:36:25 -08:00
Pepijn f04958527e Add sarm (#2639)
* add initial modeling

* make rewind pretrained policy

* add annotation

* small fix

* add sarm

* subtasks

* fix spawn

* fix rewind discrepancies

* Add script to generate embedding for dataset (#2138)

* Add generate and validate script

* fix precommit

* Improve generate embeddings function by using dataset tools (#2206)

---------

Co-authored-by: Michel Aractingi <michel.aractingi@huggingface.co>

* cleanup

* change order train log

* print batch size

* update sarm processor

* add reward output

* change expected features

* add image validation

* change validation

* get state input from dataset stats

* raise if no state key is found

* pass stats

* cleanup and refactor

* add episode inddex to complementary data

* add subtask init and detection

* revert lerobot_train changes

* pass dataset metadata to policy

* change loadig subtasks

* add small logging

* fix progress conversion and adding initial frame

* use large offset for initial frame (ugly)

* Remove rewind, use clip tokenizer

* add tests, implement formula 1,2 correctly and cleanup

* use task from dataset, cleanup visualizer

* simplify

* simplify and cleanup code and move compute_temporal_proportions to utils

* fix normalization in visualization

* Fix visualization and change prompt

* fix formatting

* add visualize subtask annotations

* use qwen thinking

* try different prompt

* format

* update prompt

* higher temp, long output

* different settings

* use instruct

* show full resp

* split message

* Temp: increase tolerance dataset

* Fix RA-BC (#2572)

* Add next observation loading for RA-BC progress deltas

* Compute weights based on temporal progress deltas instead of static rewards

* Add hard-masking for negative progress deltas in weight computation

* Feat/add dual head (#2582)

* Add dual dense sparse head and annotation

* Add docs

* add dual to procesor

* cleanup

* change sampling in visualize and cleanup

* remove validation

* remove compile

* Feat/test uniform (#2587)

* test uniform

* add different string for misaligned

* Fix rewind and add tests

* uncomment text implementation

* run precommit

* Add head mode for ra-bc

* fix visalization of single task

* add

* return per sample loss

* Fix RA_BC (#2602)

* update rabc implementation

* compute rabc beforehand

* fix import

* add only progress calulation

* use precomputed progress

* multi gpu processing

* import

* fix dataset meta data extraction

* add logging

* logging

* log

* progress per episode

* split differently

* move clip to gpu

* pre decode frames for an episode

* fix cuda initalization

* fix import

* multi processing

* rename

* fix import

* fix

* fix rabc

* use last known progress if oob

* use last known progress if oob

* add misalignment loss with random embeddings

* discard previous changes

* add selection of models to docs for ra_bc

* add transformers dep

* extend tolerance

* initial commit with new codebase

* add tests

* fix

* remove temporal sampler

* drop last frame for sampler

* use original ref

* some fixes

* fix visualization

* remove smoothing and fix order subtasks

* add stride rabc computation

* add push to hub

* add explanation

* add kappa expllaination

* better rabc logging

* feedback pr

* remove dataset tolerance

* revert dataset tool

* revert dataset changes

* add credit

* run precommit

* change path for generate ra_bc

* fix type

* include sarm in all in pyproject

* fix precommit

* lazy import matplotlib

* lazy import qwen

* remove rich console

* skip if transformers is not installed?

* run only when we have faker

* place transformer lazy loading

* Dont test if low transformer version

* fix

* increase transformer

* increase as 4.57.0 is yanked

* remove pi from all

* go back

---------

Co-authored-by: Michel Aractingi <michel.aractingi@huggingface.co>
Co-authored-by: s1lent4gnt <kmeftah.khalil@gmail.com>
2025-12-18 12:50:32 +01:00
Bryson Jones b575632f4f Merge branch 'main' into feature/add-multitask-dit 2025-12-17 07:37:09 -08:00
Steven Palma b303d1ab38 feat(scripts): add more info to lerobot-info (#2663) 2025-12-17 14:14:23 +01:00
Steven Palma b1d162f333 fix(policies): add device back to smolvlm expert (#2662) 2025-12-17 12:12:03 +01:00
Bryson Jones 8a2f5aa6cb remove cropping of images smaller than the crop size 2025-12-15 22:20:20 -08:00
Bryson Jones afe2c4d3aa Merge branch 'main' into feature/add-multitask-dit 2025-12-15 18:08:21 -08:00
Steven Palma 2b304eeb84 feat(dataset): expose tolerance_s argument to training config (#2653) 2025-12-16 00:53:19 +01:00
Bryson Jones 534e143b0c add conditional transformers import to match all other policies that use transformers lib 2025-12-15 10:08:47 -08:00
Bryson Jones 4eda54c7fe Merge branch 'main' into feature/add-multitask-dit 2025-12-15 09:57:27 -08:00
Sota Nakamura 4e6048a221 finalize the dataset after recording (#2496)
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
2025-12-15 17:57:04 +01:00
Bryson Jones a632dd3af4 Merge branch 'main' into feature/add-multitask-dit 2025-12-15 08:22:05 -08:00
Martino Russi a6c3a0fa09 Feat/add mj env (#2613)
* add sim support

* close fix threading issues
2025-12-15 16:22:27 +01:00
Woojin Wie c2fb644613 feat(robot): Add support for OMX robot (#2614)
* upload

* feat(omx): simplify motor initialization and remove default calibration files

* feat(omx): read motor positions without normalization for improved accuracy

* update calibration method for return factory value

Signed-off-by: Junha Cha <ckwnsgk1@gachon.ac.kr>

* change the drive mode

* refactor: clean up code by removing unnecessary blank lines in omx_follower and omx_leader modules

* feat(omx): update calibration method to set drive modes for motors

* feat(pyproject): add 'ROBOTIS' to extend-ignore-identifiers-re list

* feat(omx): enhance calibration method to write default drive modes to motors

* Update src/lerobot/robots/omx_follower/__init__.py

Add informations about the robot

Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Signed-off-by: Woojin Wie <dnldnwls1123@gmail.com>

---------

Signed-off-by: Junha Cha <ckwnsgk1@gachon.ac.kr>
Signed-off-by: Woojin Wie <dnldnwls1123@gmail.com>
Co-authored-by: Junha02 <chajunha2023@naver.com>
Co-authored-by: Junha Cha <ckwnsgk1@gachon.ac.kr>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
2025-12-15 15:50:29 +01:00
Bryson Jones 71f359ca6e refactor code to perform task tokenization in the processor instead of in the modeling code for multitask dit 2025-12-11 12:09:54 -08:00
Bryson Jones 51dfee43f4 rename config param for multiple vision encoders 2025-12-11 09:58:57 -08:00
Bryson Jones dd4ef1383f fix nit formatting in generate actions fcn 2025-12-11 09:21:26 -08:00
Bryson Jones f3823e8bcd remove the base classes since we don't need to be able to extend 2025-12-11 09:20:25 -08:00
Bryson Jones c398a146b3 use constants for indexing into batches and remove env state references 2025-12-11 09:13:38 -08:00
Bryson Jones 9b47c5fac9 move policy to top of file 2025-12-11 09:04:38 -08:00
Bryson Jones 86e0ee787d remove environment state conditioning 2025-12-11 08:26:21 -08:00
Bryson Jones ba968e84f1 fix bugs when testing on hardware 2025-12-10 16:26:57 -08:00
Bryson Jones d49d3390f6 Merge branch 'main' into feature/add-multitask-dit 2025-12-10 14:44:04 -08:00
Bryson Jones 10cfc17705 remove redundant asserts 2025-12-10 14:43:27 -08:00
Bryson Jones 5524a0d7a7 split up select action return statement 2025-12-10 14:43:27 -08:00
Bryson Jones 3a16a002f8 add torch.no_grad decorators 2025-12-10 14:43:27 -08:00
Bryson Jones 3b2a4f548c merge all modules files into the main modeling file 2025-12-10 14:43:27 -08:00
Bryson Jones b92dc82ddd add references to the modeling file comments 2025-12-10 14:43:27 -08:00
Bryson Jones 103230c64c simplify config for multitask dit by merging and flattening everything, then adding comments to denote where some parameters are only used for specific objectives 2025-12-10 14:43:27 -08:00
Bryson Jones cdacc090cd update docstring for multitask dit policy processor file 2025-12-10 14:43:27 -08:00
Bryson Jones 6f856016c5 adjust factory comment 2025-12-10 14:43:27 -08:00
Bryson Jones adabb37af6 remove dino vision encoder and simplify text and vision encoders by removing inheritance structure 2025-12-10 14:43:27 -08:00
Bryson Jones 55e19ff9a7 update readme and citations for multitask dit policy 2025-12-10 14:43:27 -08:00
Michel Aractingi ce348a3460 enable variable image sizes to pi0/pi0.5 (#2609)
* enable variable image sizes to pi0/pi0.5

* add square image assertion
2025-12-10 19:41:11 +01:00
Bryson Jones 46ebcc2f7d add RoPE attention module as this is shown to help training dynamics and generation quality for DiTs 2025-12-09 08:42:56 -08:00
Bryson Jones a0d5a088e3 Merge branch 'main' into feature/add-multitask-dit 2025-12-09 07:44:05 -08:00
Jade Choghari 7f40b3bf82 feat(dataset): add tool to convert images to video datasets (#2560)
* add video encoding tool

* style

* make it work

* more fixes
2025-12-08 18:50:21 +01:00
Michel Aractingi 2e9c9fd832 Replay while loop in sample actions with for loops (#2600) 2025-12-08 14:47:54 +01:00
Michel Aractingi 0217e1e3ad Fix dataset aggreagation for multi video datasets' (#2550) 2025-12-05 16:09:25 +01:00
Steven Palma 56b43cc888 fix(scripts): missing so101 import (#2577)
* fix(scripts): missing so101 import

Co-authored-by: Skyler <skylerwiernik@gmail.com>

* fix(scripts): move urdf to cli args

* refactor(scripts): improve find_joints_limits

---------

Co-authored-by: Skyler <skylerwiernik@gmail.com>
2025-12-03 18:20:26 +01:00