From a64f2fd322226974467379cd1ca701711dffb37c Mon Sep 17 00:00:00 2001 From: Tong Wu <54630004+wut19@users.noreply.github.com> Date: Tue, 23 Dec 2025 18:35:06 +0800 Subject: [PATCH 1/3] modify the README file for wallx (#2705) * support wallx * fix bugs in flow * incorporate wallx model into lerobot * update the policy methods * reduce to least config and params & pass lerobot basic test * fixed dtype bugs * add wallx dependencies * update * remove flash-attn requirement && fix bug in inference and fast mode * fix bug for inference * add some small modifications * fix pre-commit errors * remove lerobot[wallx] * fix ci * fix precommit issues * fix: exclude wallx extra properly in CI workflows * fix: add uv conflicts for wallx transformers version * fix: peft test import * pre-commit * only export WallXConfig from wall_x package to avoid peft import in CI * remove torch dep * precommit * add import * update doc files * fix minor errors --------- Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com> Co-authored-by: vincentchen Co-authored-by: Geoffrey19 Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com> Co-authored-by: Pepijn --- docs/source/_toctree.yml | 2 ++ docs/source/walloss.mdx | 74 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100644 docs/source/walloss.mdx diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index 85a79ef17..7766b3472 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -41,6 +41,8 @@ title: NVIDIA GR00T N1.5 - local: xvla title: X-VLA + - local: walloss + title: WALL-OSS title: "Policies" - sections: - local: sarm diff --git a/docs/source/walloss.mdx b/docs/source/walloss.mdx new file mode 100644 index 000000000..12e9b1fc7 --- /dev/null +++ b/docs/source/walloss.mdx @@ -0,0 +1,74 @@ +# WALL-OSS + +WALL-OSS is an open-source foundation model for embodied intelligence, proposed by the [XSquare Robot](https://x2robot.com/en/research/68bc2cde8497d7f238dde690) team in 2025. The LeRobot implementation is adapted from their open-source [WallX](https://github.com/X-Square-Robot/wall-x) repository. + +X Square Robot’s WALL-OSS is now integrated into Hugging Face’s LeRobot ecosystem. This is an exciting collaborative project between the LeRobot and X Square Robot teams. You can now post-train, evaluate, and deploy WALL-OSS directly through LeRobot. With this, we’re aiming to make it easier for the open-source robotics community to customize and deploy WALL-OSS foundation models. Read and explore WALL-OSS [paper](https://arxiv.org/pdf/2509.11766) and [code](https://github.com/X-Square-Robot/wall-x). + +## Model Overview + +The WALL-OSS team is building the embodied foundation model to capture and compress the world's most valuable data: the continuous, high-fidelity stream of physical interaction. By creating a direct feedback loop between the model's decisions and the body's lived experience, the emergence of a truly generalizable intelligence is enabled—one that understands not just how the world works, but how to act effectively within it. + +Technically, WALL-OSS introduces a tightly coupled multimodal architecture (tightly-coupled MoE structure) that integrates both discrete and continuous action modeling strategies. Through a two-stage training pipeline (Inspiration → Integration), the model gradually unifies semantic reasoning and high-frequency action generation. Its core innovations include: + +- **Embodied perception–enhanced multimodal pretraining**: Large-scale training on unified vision–language–action data to strengthen spatial, causal, and manipulation understanding. +- **Unified Cross-Level Chain-of-Thought (Uni-CoT)**: A single differentiable framework that unifies high-level instruction reasoning, sub-task decomposition, and fine-grained action synthesis, forming a continuous chain from “understanding” to “execution.” +- **Mixture-of-Experts (MoE) action heads**: Dynamically activating experts depending on the task phase and modeling actions in discrete or continuous space to maintain stable VLM priors. +- **Two-stage training paradigm**: + - **Inspiration stage**: Injecting discrete action priors to strengthen spatial understanding and semantic-action alignment. + - **Integration stage**: Using flow matching to achieve high-frequency continuous control. + +## Installation Requirements + +1. Install LeRobot by following our [Installation Guide](./installation). +2. Install WallX dependencies by running: + + ```bash + pip install -e ".[wallx]" + ``` + +## Usage + +To use WallX in LeRobot, specify the policy type as: + +```python +policy.type=wall_x +``` + +## Training + +For training WallX, you can use the standard LeRobot training script with the appropriate configuration: + +```bash +python src/lerobot/scripts/lerobot_train.py \ + --dataset.repo_id=your_dataset \ + --policy.type=wall_x \ + --output_dir=./outputs/wallx_training \ + --job_name=wallx_training \ + --policy.repo_id=your_repo_id \ + --policy.pretrained_name_or_path=x-square-robot/wall-oss-flow \ + --policy.prediction_mode=diffusion \ + --policy.attn_implementation=eager \ + --steps=3000 \ + --policy.device=cuda \ + --batch_size=32 +``` + +### Training Arguments + +| Argument | Description | +| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--dataset.repo_id` | The Hugging Face Hub repository ID for your training dataset (e.g., `lerobot/aloha_sim_insertion_human`) | +| `--policy.type` | Specifies using the WallX policy architecture | +| `--output_dir` | Local directory where training checkpoints and logs will be saved | +| `--job_name` | A name identifier for this training run (used in logging/tracking) | +| `--policy.repo_id` | Your Hugging Face Hub repo ID where the trained model will be pushed | +| `--policy.pretrained_path` | Path to pretrained WallX weights to initialize from (the official WALL-OSS checkpoint) | +| `--policy.prediction_mode` | The action prediction strategy: `diffusion` or `fast` - `diffusion` uses iterative denoising for action generation, `fast` uses next token prediction instead | +| `--policy.attn_implementation` | Attention implementation backend - `eager` uses standard PyTorch attention (alternatives include `flash_attention_2` or `sdpa`) | +| `--steps` | Total number of training steps to run | +| `--policy.device` | Device to train on (`cuda` for GPU, `cpu` for CPU) | +| `--batch_size` | Number of samples per training batch | + +## License + +This model follows the **Apache 2.0 License**, consistent with the original [WallX repository](https://github.com/X-Square-Robot/wall-x). From b2ef6ae720f16af5ac2d03cf2972f997762ef0db Mon Sep 17 00:00:00 2001 From: Steven Palma Date: Tue, 23 Dec 2025 12:10:44 +0100 Subject: [PATCH 2/3] chore: modernize contributing.md (#2677) --- CONTRIBUTING.md | 352 ++++++++---------------------------------------- 1 file changed, 56 insertions(+), 296 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dcb5c03d8..abca0d821 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,323 +1,83 @@ -# How to contribute to 🤗 LeRobot? +# How to contribute to 🤗 LeRobot -Everyone is welcome to contribute, and we value everybody's contribution. Code -is thus not the only way to help the community. Answering questions, helping -others, reaching out and improving the documentations are immensely valuable to -the community. +Everyone is welcome to contribute, and we value everybody's contribution. Code is not the only way to help the community. Answering questions, helping others, reaching out, and improving the documentation are immensely valuable. -It also helps us if you spread the word: reference the library from blog posts -on the awesome projects it made possible, shout out on Twitter when it has -helped you, or simply ⭐️ the repo to say "thank you". +Whichever way you choose to contribute, please be mindful to respect our [code of conduct](./CODE_OF_CONDUCT.md). -Whichever way you choose to contribute, please be mindful to respect our -[code of conduct](https://github.com/huggingface/lerobot/blob/main/CODE_OF_CONDUCT.md). +## Ways to Contribute -## You can contribute in so many ways! +You can contribute in many ways: -Some of the ways you can contribute to 🤗 LeRobot: +- **Fixing issues:** Resolve bugs or improve existing code. +- **New features:** Develop new features. +- **Extend:** Implement new models/policies, robots, or simulation environments and upload datasets to the Hugging Face Hub. +- **Documentation:** Improve examples, guides, and docstrings. +- **Feedback:** Submit tickets related to bugs or desired new features. -- Fixing outstanding issues with the existing code. -- Implementing new models, datasets or simulation environments. -- Contributing to the examples or to the documentation. -- Submitting issues related to bugs or desired new features. +If you are unsure where to start, join our [Discord Channel](https://discord.gg/JkrYNdmw). -Following the guides below, feel free to open issues and PRs and to coordinate your efforts with the community on our [Discord Channel](https://discord.gg/VjFz58wn3R). For specific inquiries, reach out to [Remi Cadene](mailto:remi.cadene@huggingface.co). +## Development Setup -If you are not sure how to contribute or want to know the next features we working on, look on this project page: [LeRobot TODO](https://github.com/orgs/huggingface/projects/46) +To contribute code, you need to set up a development environment. -## Submitting a new issue or feature request +### 1. Fork and Clone -Do your best to follow these guidelines when submitting an issue or a feature -request. It will make it easier for us to come back to you quickly and with good -feedback. - -### Did you find a bug? - -The 🤗 LeRobot library is robust and reliable thanks to the users who notify us of -the problems they encounter. So thank you for reporting an issue. - -First, we would really appreciate it if you could **make sure the bug was not -already reported** (use the search bar on Github under Issues). - -Did not find it? :( So we can act quickly on it, please follow these steps: - -- Include your **OS type and version**, the versions of **Python** and **PyTorch**. -- A short, self-contained, code snippet that allows us to reproduce the bug in - less than 30s. -- The full traceback if an exception is raised. -- Attach any other additional information, like screenshots, you think may help. - -### Do you want a new feature? - -A good feature request addresses the following points: - -1. Motivation first: - -- Is it related to a problem/frustration with the library? If so, please explain - why. Providing a code snippet that demonstrates the problem is best. -- Is it related to something you would need for a project? We'd love to hear - about it! -- Is it something you worked on and think could benefit the community? - Awesome! Tell us what problem it solved for you. - -2. Write a _paragraph_ describing the feature. -3. Provide a **code snippet** that demonstrates its future use. -4. In case this is related to a paper, please attach a link. -5. Attach any additional information (drawings, screenshots, etc.) you think may help. - -If your issue is well written we're already 80% of the way there by the time you -post it. - -## Adding new policies, datasets or environments - -Look at our implementations for [datasets](./src/lerobot/datasets/), [policies](./src/lerobot/policies/), -environments ([aloha](https://github.com/huggingface/gym-aloha), -[pusht](https://github.com/huggingface/gym-pusht)) -and follow the same api design. - -When implementing a new dataset loadable with LeRobotDataset follow these steps: - -- Update `available_datasets_per_env` in `lerobot/__init__.py` - -When implementing a new environment (e.g. `gym_aloha`), follow these steps: - -- Update `available_tasks_per_env` and `available_datasets_per_env` in `lerobot/__init__.py` - -When implementing a new policy class (e.g. `DiffusionPolicy`) follow these steps: - -- Update `available_policies` and `available_policies_per_env`, in `lerobot/__init__.py` -- Set the required `name` class attribute. -- Update variables in `tests/test_available.py` by importing your new Policy class - -## Submitting a pull request (PR) - -Before writing code, we strongly advise you to search through the existing PRs or -issues to make sure that nobody is already working on the same thing. If you are -unsure, it is always a good idea to open an issue to get some feedback. - -You will need basic `git` proficiency to be able to contribute to -🤗 LeRobot. `git` is not the easiest tool to use but it has the greatest -manual. Type `git --help` in a shell and enjoy. If you prefer books, [Pro -Git](https://git-scm.com/book/en/v2) is a very good reference. - -Follow these steps to start contributing: - -1. Fork the [repository](https://github.com/huggingface/lerobot) by - clicking on the 'Fork' button on the repository's page. This creates a copy of the code - under your GitHub user account. - -2. Clone your fork to your local disk, and add the base repository as a remote. The following command - assumes you have your public SSH key uploaded to GitHub. See the following guide for more - [information](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository). - - ```bash - git clone git@github.com:/lerobot.git - cd lerobot - git remote add upstream https://github.com/huggingface/lerobot.git - ``` - -3. Create a new branch to hold your development changes, and do this for every new PR you work on. - - Start by synchronizing your `main` branch with the `upstream/main` branch (more details in the [GitHub Docs](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/syncing-a-fork)): - - ```bash - git checkout main - git fetch upstream - git rebase upstream/main - ``` - - Once your `main` branch is synchronized, create a new branch from it: - - ```bash - git checkout -b a-descriptive-name-for-my-changes - ``` - - 🚨 **Do not** work on the `main` branch. - -4. for development, we advise to use a tool like `poetry` or `uv` instead of just `pip` to easily track our dependencies. - Follow the instructions to [install poetry](https://python-poetry.org/docs/#installation) (use a version >=2.1.0) or to [install uv](https://docs.astral.sh/uv/getting-started/installation/#installation-methods) if you don't have one of them already. - - Set up a development environment with conda: - - ```bash - conda create -y -n lerobot-dev python=3.10 && conda activate lerobot-dev - ``` - - If you're using `uv`, it can manage python versions so you can instead do: - - ```bash - uv venv --python 3.10 && source .venv/bin/activate - ``` - - To develop on 🤗 LeRobot, you will at least need to install the `dev` and `test` extras dependencies along with the core library: - - using `poetry` - - ```bash - poetry sync --extras "dev test" - ``` - - using `uv` - - ```bash - uv sync --extra dev --extra test - ``` - - You can also install the project with all its dependencies (including environments): - - using `poetry` - - ```bash - poetry sync --all-extras - ``` - - using `uv` - - ```bash - uv sync --all-extras - ``` - - > **Note:** If you don't install simulation environments with `--all-extras`, the tests that require them will be skipped when running the pytest suite locally. However, they _will_ be tested in the CI. In general, we advise you to install everything and test locally before pushing. - - Whichever command you chose to install the project (e.g. `poetry sync --all-extras`), you should run it again when pulling code with an updated version of `pyproject.toml` and `poetry.lock` in order to synchronize your virtual environment with the new dependencies. - - The equivalent of `pip install some-package`, would just be: - - using `poetry` - - ```bash - poetry add some-package - ``` - - using `uv` - - ```bash - uv add some-package - ``` - - When making changes to the poetry sections of the `pyproject.toml`, you should run the following command to lock dependencies. - using `poetry` - - ```bash - poetry lock - ``` - - using `uv` - - ```bash - uv lock - ``` - -5. Develop the features on your branch. - - As you work on the features, you should make sure that the test suite - passes. You should run the tests impacted by your changes like this (see - below an explanation regarding the environment variable): - - ```bash - pytest tests/.py - ``` - -6. Follow our style. - - `lerobot` relies on `ruff` to format its source code - consistently. Set up [`pre-commit`](https://pre-commit.com/) to run these checks - automatically as Git commit hooks. - - Install `pre-commit` hooks: - - ```bash - pre-commit install - ``` - - You can run these hooks whenever you need on staged files with: - - ```bash - pre-commit - ``` - - Once you're happy with your changes, add changed files using `git add` and - make a commit with `git commit` to record your changes locally: - - ```bash - git add modified_file.py - git commit - ``` - - Note, if you already committed some changes that have a wrong formatting, you can use: - - ```bash - pre-commit run --all-files - ``` - - Please write [good commit messages](https://chris.beams.io/posts/git-commit/). - - It is a good idea to sync your copy of the code with the original - repository regularly. This way you can quickly account for changes: - - ```bash - git fetch upstream - git rebase upstream/main - ``` - - Push the changes to your account using: - - ```bash - git push -u origin a-descriptive-name-for-my-changes - ``` - -7. Once you are satisfied (**and the checklist below is happy too**), go to the - webpage of your fork on GitHub. Click on 'Pull request' to send your changes - to the project maintainers for review. - -8. It's ok if maintainers ask you for changes. It happens to core contributors - too! So everyone can see the changes in the Pull request, work in your local - branch and push the changes to your fork. They will automatically appear in - the pull request. - -### Checklist - -1. The title of your pull request should be a summary of its contribution; -2. If your pull request addresses an issue, please mention the issue number in - the pull request description to make sure they are linked (and people - consulting the issue know you are working on it); -3. To indicate a work in progress please prefix the title with `[WIP]`, or preferably mark - the PR as a draft PR. These are useful to avoid duplicated work, and to differentiate - it from PRs ready to be merged; -4. Make sure existing tests pass; - -### Tests - -An extensive test suite is included to test the library behavior and several examples. Library tests can be found in the [tests folder](https://github.com/huggingface/lerobot/tree/main/tests). - -Install [git lfs](https://git-lfs.com/) to retrieve test artifacts (if you don't have it already). - -On Mac: +Fork the repository on GitHub, then clone your fork: ```bash -brew install git-lfs -git lfs install +git clone https://github.com//lerobot.git +cd lerobot +git remote add upstream https://github.com/huggingface/lerobot.git ``` -On Ubuntu: +### 2. Environment Installation + +Please follow our [Installation Guide](./docs/source/installation.mdx) for the environment setup & installation from source. + +## Running Tests & Quality Checks + +### Code Style (Pre-commit) + +Install `pre-commit` hooks to run checks automatically before you commit: ```bash -sudo apt-get install git-lfs -git lfs install +pre-commit install ``` -Pull artifacts if they're not in [tests/artifacts](tests/artifacts) +To run checks manually on all files: ```bash +pre-commit run --all-files +``` + +### Running Tests + +We use `pytest`. First, ensure you have test artifacts by installing **git-lfs**: + +```bash +git lfs install git lfs pull ``` -We use `pytest` in order to run the tests. From the root of the -repository, here's how to run tests with `pytest` for the library: +Run the full suite (this may require extras installed): ```bash -python -m pytest -sv ./tests +pytest -sv ./tests ``` -You can specify a smaller set of tests in order to test only the feature -you're working on. +Or run a specific test file during development: + +```bash +pytest -sv tests/test_specific_feature.py +``` + +## Submitting Issues & Pull Requests + +Use the templates for required fields and examples. + +- **Issues:** Follow the [ticket template](./.github/ISSUE_TEMPLATE/bug-report.yml). +- **Pull requests:** Rebase on `upstream/main`, use a descriptive branch (don't work on `main`), run `pre-commit` and tests locally, and follow the [PR template](./.github/PULL_REQUEST_TEMPLATE.md). + +One member of the LeRobot team will then review your contribution. + +Thank you for contributing to LeRobot! From a142c365dd6cb0c019d5031d612cd6da98e8562f Mon Sep 17 00:00:00 2001 From: Pepijn <138571049+pkooij@users.noreply.github.com> Date: Tue, 23 Dec 2025 14:13:32 +0100 Subject: [PATCH 3/3] use syslink for wall-x readme (#2708) * use syslink for wall-x readme * remove whitespace --- docs/source/policy_walloss_README.md | 35 ++++++++++++++++++++++++++ src/lerobot/policies/wall_x/README.md | 36 +-------------------------- 2 files changed, 36 insertions(+), 35 deletions(-) create mode 100644 docs/source/policy_walloss_README.md mode change 100644 => 120000 src/lerobot/policies/wall_x/README.md diff --git a/docs/source/policy_walloss_README.md b/docs/source/policy_walloss_README.md new file mode 100644 index 000000000..78548bd8d --- /dev/null +++ b/docs/source/policy_walloss_README.md @@ -0,0 +1,35 @@ +# WALL-OSS + +This repository contains the Hugging Face port of **WALL-OSS**, a Vision-Language-Action model for cross-embodiment robotic control based on Qwen2.5-VL with flow matching/FAST action prediction. + +--- + +## Model Overview + +| Feature | Description | +| ------------------ | ----------------------------------------------------- | --- | +| Base Model | Qwen2.5-VL (Vision-Language Model) | +| Action Prediction | Flow Matching (diffusion) or FAST (discrete tokens) | +| Architecture | Mixture of Experts (MoE) with action-specific routing | | +| Multi-Modal Inputs | Vision (images/videos), Language, Proprioception | + +--- + +## Citation + +If you use this work, please cite: + +```bibtex +@article{zhai2025igniting, + title = {Igniting VLMs Toward the Embodied Space}, + author = {Zhai, Andy and Liu, Brae and Fang, Bruno and Cai, Chalse and Ma, Ellie and Yin, Ethan and Wang, Hao and Zhou, Hugo and Wang, James and Shi, Lights and Liang, Lucy and Wang, Make and Wang, Qian and Gan, Roy and Yu, Ryan and Li, Shalfun and Liu, Starrick and Chen, Sylas and Chen, Vincent and Xu, Zach}, + journal = {arXiv preprint arXiv:2509.11766}, + year = {2025} +} +``` + +--- + +## License + +This port follows the **Apache 2.0 License**. diff --git a/src/lerobot/policies/wall_x/README.md b/src/lerobot/policies/wall_x/README.md deleted file mode 100644 index 78548bd8d..000000000 --- a/src/lerobot/policies/wall_x/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# WALL-OSS - -This repository contains the Hugging Face port of **WALL-OSS**, a Vision-Language-Action model for cross-embodiment robotic control based on Qwen2.5-VL with flow matching/FAST action prediction. - ---- - -## Model Overview - -| Feature | Description | -| ------------------ | ----------------------------------------------------- | --- | -| Base Model | Qwen2.5-VL (Vision-Language Model) | -| Action Prediction | Flow Matching (diffusion) or FAST (discrete tokens) | -| Architecture | Mixture of Experts (MoE) with action-specific routing | | -| Multi-Modal Inputs | Vision (images/videos), Language, Proprioception | - ---- - -## Citation - -If you use this work, please cite: - -```bibtex -@article{zhai2025igniting, - title = {Igniting VLMs Toward the Embodied Space}, - author = {Zhai, Andy and Liu, Brae and Fang, Bruno and Cai, Chalse and Ma, Ellie and Yin, Ethan and Wang, Hao and Zhou, Hugo and Wang, James and Shi, Lights and Liang, Lucy and Wang, Make and Wang, Qian and Gan, Roy and Yu, Ryan and Li, Shalfun and Liu, Starrick and Chen, Sylas and Chen, Vincent and Xu, Zach}, - journal = {arXiv preprint arXiv:2509.11766}, - year = {2025} -} -``` - ---- - -## License - -This port follows the **Apache 2.0 License**. diff --git a/src/lerobot/policies/wall_x/README.md b/src/lerobot/policies/wall_x/README.md new file mode 120000 index 000000000..4426967de --- /dev/null +++ b/src/lerobot/policies/wall_x/README.md @@ -0,0 +1 @@ +../../../../docs/source/policy_walloss_README.md \ No newline at end of file