diff --git a/.github/workflows/benchmark_tests.yml b/.github/workflows/benchmark_tests.yml index b82c59a8b..3493e5048 100644 --- a/.github/workflows/benchmark_tests.yml +++ b/.github/workflows/benchmark_tests.yml @@ -167,9 +167,9 @@ jobs: # ── LIBERO TRAIN+EVAL SMOKE ────────────────────────────────────────────── # Train SmolVLA for 1 step (batch_size=1, dataset episode 0 only) then - # immediately runs eval inside the training loop (eval_freq=1, 1 episode). + # immediately runs eval inside the training loop (env_eval_freq=1, 1 episode). # Tests the full train→eval-within-training pipeline end-to-end. - - name: Run Libero train+eval smoke (1 step, eval_freq=1) + - name: Run Libero train+eval smoke (1 step, env_eval_freq=1) if: env.HF_USER_TOKEN != '' run: | docker run --name libero-train-smoke --gpus all \ @@ -196,7 +196,7 @@ jobs: --output_dir=/tmp/train-smoke \ --steps=1 \ --batch_size=1 \ - --eval_freq=1 \ + --env_eval_freq=1 \ --eval.n_episodes=1 \ --eval.batch_size=1 \ --eval.use_async_envs=false \ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index dff7416f4..8ae913e4e 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -65,6 +65,9 @@ repos: name: Format Markdown with Prettier types_or: [markdown, mdx] args: [--prose-wrap=preserve] + # Jinja2 model-card templates use a .md extension but contain {% ... %} / + # {{ ... }} tags that prettier's Markdown formatter mangles (e.g. table loops). + exclude: ^src/lerobot/templates/.*\.md$ ##### Security ##### - repo: https://github.com/gitleaks/gitleaks diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.md index 57a33fdba..03b270dce 100644 --- a/AGENT_GUIDE.md +++ b/AGENT_GUIDE.md @@ -138,7 +138,7 @@ lerobot-replay --robot.type=so101_follower --robot.port= --robot. --dataset.repo_id=${HF_USER}/my_task --dataset.episode=0 ``` -**4.9 Train** (default: ACT — fastest, lowest memory). Apple silicon: `--policy.device=mps`. See §6/§7 for policy and duration. +**4.9 Train** (default: ACT — fastest, lowest memory). Apple silicon: `--policy.device=mps`. No local GPU? Add `--job.target=` (e.g. `a10g-small`, list them with `hf jobs hardware`) to run on Hugging Face Jobs instead. See §6/§7 for policy and duration. ```bash lerobot-train \ diff --git a/Makefile b/Makefile index e02f02403..ea3b6e261 100644 --- a/Makefile +++ b/Makefile @@ -58,7 +58,7 @@ test-act-ete-train: --dataset.episodes="[0]" \ --batch_size=2 \ --steps=4 \ - --eval_freq=2 \ + --env_eval_freq=2 \ --eval.n_episodes=1 \ --eval.batch_size=1 \ --save_freq=2 \ @@ -96,7 +96,7 @@ test-diffusion-ete-train: --dataset.episodes="[0]" \ --batch_size=2 \ --steps=2 \ - --eval_freq=2 \ + --env_eval_freq=2 \ --eval.n_episodes=1 \ --eval.batch_size=1 \ --save_checkpoint=true \ @@ -126,7 +126,7 @@ test-tdmpc-ete-train: --dataset.episodes="[0]" \ --batch_size=2 \ --steps=2 \ - --eval_freq=2 \ + --env_eval_freq=2 \ --eval.n_episodes=1 \ --eval.batch_size=1 \ --save_checkpoint=true \ @@ -161,7 +161,7 @@ test-smolvla-ete-train: --dataset.episodes="[0]" \ --batch_size=2 \ --steps=4 \ - --eval_freq=2 \ + --env_eval_freq=2 \ --eval.n_episodes=1 \ --eval.batch_size=1 \ --save_freq=2 \ @@ -178,3 +178,9 @@ test-smolvla-ete-eval: --env.episode_length=5 \ --eval.n_episodes=1 \ --eval.batch_size=1 + +# E2E annotation pipeline smoke test against a tiny in-memory fixture +# dataset. Opt-in (not part of `make test-end-to-end`) and uses a stub VLM +# backend, so it does not require a real model checkpoint or GPU. +annotation-e2e: + uv run python -m tests.annotations.run_e2e_smoke diff --git a/README.md b/README.md index 9c40e8b34..f72952102 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ action = model.select_action(obs) robot.send_action(action) ``` -**Supported Hardware:** SO100, LeKiwi, Koch, HopeJR, OMX, EarthRover, Reachy2, Gamepads, Keyboards, Phones, OpenARM, Unitree G1. +**Supported Hardware:** SO100, LeKiwi, Koch, HopeJR, OMX, EarthRover, Reachy2, Gamepads, Keyboards, Phones, OpenARM, Unitree G1, reBot B601. While these devices are natively integrated into the LeRobot codebase, the library is designed to be extensible. You can easily implement the Robot interface to utilize LeRobot's data collection, training, and visualization tools for your own custom robot. @@ -97,15 +97,17 @@ Training a policy is as simple as running a script configuration: ```bash lerobot-train \ - --policy=act \ + --policy.type=act \ --dataset.repo_id=lerobot/aloha_mobile_cabinet ``` -| Category | Models | -| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Imitation Learning** | [ACT](./docs/source/policy_act_README.md), [Diffusion](./docs/source/policy_diffusion_README.md), [VQ-BeT](./docs/source/policy_vqbet_README.md), [Multitask DiT Policy](./docs/source/policy_multi_task_dit_README.md) | -| **Reinforcement Learning** | [HIL-SERL](./docs/source/hilserl.mdx), [TDMPC](./docs/source/policy_tdmpc_README.md) & QC-FQL (coming soon) | -| **VLAs Models** | [Pi0Fast](./docs/source/pi0fast.mdx), [Pi0.5](./docs/source/pi05.mdx), [GR00T N1.5](./docs/source/policy_groot_README.md), [SmolVLA](./docs/source/policy_smolvla_README.md), [XVLA](./docs/source/xvla.mdx) | +| Category | Models | +| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Imitation Learning** | [ACT](./docs/source/policy_act_README.md), [Diffusion](./docs/source/policy_diffusion_README.md), [VQ-BeT](./docs/source/policy_vqbet_README.md), [Multitask DiT Policy](./docs/source/policy_multi_task_dit_README.md) | +| **Reinforcement Learning** | [HIL-SERL](./docs/source/hilserl.mdx), [TDMPC](./docs/source/policy_tdmpc_README.md) & QC-FQL (coming soon) | +| **VLAs Models** | [Pi0](./docs/source/pi0.mdx), [Pi0Fast](./docs/source/pi0fast.mdx), [Pi0.5](./docs/source/pi05.mdx), [GR00T N1.5](./docs/source/policy_groot_README.md), [SmolVLA](./docs/source/policy_smolvla_README.md), [XVLA](./docs/source/xvla.mdx), [EO-1](./docs/source/eo1.mdx), [MolmoAct2](./docs/source/molmoact2.mdx), [WALL-OSS](./docs/source/walloss.mdx) | +| **World Models** | [VLA-JEPA](./docs/source/vla_jepa.mdx) (more coming soon) | +| **Reward Models** | [SARM](./docs/source/sarm.mdx), [TOPReward](./docs/source/topreward.mdx), [Robometer](./docs/source/robometer.mdx) | Similarly to the hardware, you can easily implement your own policy & leverage LeRobot's data collection, training, and visualization tools, and share your model to the HF Hub @@ -133,6 +135,8 @@ Learn how to implement your own simulation environment or benchmark and distribu - **[Discord](https://discord.gg/q8Dzzpym3f):** Join the `LeRobot` server to discuss with the community. - **[X](https://x.com/LeRobotHF):** Follow us on X to stay up-to-date with the latest developments. - **[Robot Learning Tutorial](https://huggingface.co/spaces/lerobot/robot-learning-tutorial):** A free, hands-on course to learn robot learning using LeRobot. +- **[T-Shirt Folding Experiment](https://huggingface.co/spaces/lerobot/robot-folding):** An end-to-end demonstration of folding t-shirts with LeRobot. +- **[LeLab](https://github.com/huggingface/leLab):** A web interface for LeRobot — teleoperate, calibrate, record datasets, replay, and train your SO arm from the browser, no CLI required. ## Citation @@ -140,7 +144,7 @@ If you use LeRobot in your project, please cite the GitHub repository to acknowl ```bibtex @misc{cadene2024lerobot, - author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas}, + author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Meftah, Khalil and Ellerbach, Maxime and Moss, Jess and Wolf, Thomas}, title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch}, howpublished = "\url{https://github.com/huggingface/lerobot}", year = {2024} diff --git a/benchmarks/video/README.md b/benchmarks/video/README.md deleted file mode 100644 index 1feee69c4..000000000 --- a/benchmarks/video/README.md +++ /dev/null @@ -1,288 +0,0 @@ -# Video benchmark - -## Questions - -What is the optimal trade-off between: - -- maximizing loading time with random access, -- minimizing memory space on disk, -- maximizing success rate of policies, -- compatibility across devices/platforms for decoding videos (e.g. video players, web browsers). - -How to encode videos? - -- Which video codec (`-vcodec`) to use? h264, h265, AV1? -- What pixel format to use (`-pix_fmt`)? `yuv444p` or `yuv420p`? -- How much compression (`-crf`)? No compression with `0`, intermediate compression with `25` or extreme with `50+`? -- Which frequency to chose for key frames (`-g`)? A key frame every `10` frames? - -How to decode videos? - -- Which `decoder`? `torchvision`, `torchaudio`, `ffmpegio`, `decord`, or `nvc`? -- What scenarios to use for the requesting timestamps during benchmark? (`timestamps_mode`) - -## Variables - -**Image content & size** -We don't expect the same optimal settings for a dataset of images from a simulation, or from real-world in an apartment, or in a factory, or outdoor, or with lots of moving objects in the scene, etc. Similarly, loading times might not vary linearly with the image size (resolution). -For these reasons, we run this benchmark on four representative datasets: - -- `lerobot/pusht_image`: (96 x 96 pixels) simulation with simple geometric shapes, fixed camera. -- `lerobot/aloha_mobile_shrimp_image`: (480 x 640 pixels) real-world indoor, moving camera. -- `lerobot/paris_street`: (720 x 1280 pixels) real-world outdoor, moving camera. -- `lerobot/kitchen`: (1080 x 1920 pixels) real-world indoor, fixed camera. - -Note: The datasets used for this benchmark need to be image datasets, not video datasets. - -**Data augmentations** -We might revisit this benchmark and find better settings if we train our policies with various data augmentations to make them more robust (e.g. robust to color changes, compression, etc.). - -### Encoding parameters - -| parameter | values | -| ----------- | ------------------------------------------------------------ | -| **vcodec** | `libx264`, `libx265`, `libsvtav1` | -| **pix_fmt** | `yuv444p`, `yuv420p` | -| **g** | `1`, `2`, `3`, `4`, `5`, `6`, `10`, `15`, `20`, `40`, `None` | -| **crf** | `0`, `5`, `10`, `15`, `20`, `25`, `30`, `40`, `50`, `None` | - -Note that `crf` value might be interpreted differently by various video codecs. In other words, the same value used with one codec doesn't necessarily translate into the same compression level with another codec. In fact, the default value (`None`) isn't the same amongst the different video codecs. Importantly, it is also the case for many other ffmpeg arguments like `g` which specifies the frequency of the key frames. - -For a comprehensive list and documentation of these parameters, see the ffmpeg documentation depending on the video codec used: - -- h264: https://trac.ffmpeg.org/wiki/Encode/H.264 -- h265: https://trac.ffmpeg.org/wiki/Encode/H.265 -- AV1: https://trac.ffmpeg.org/wiki/Encode/AV1 - -### Decoding parameters - -**Decoder** -We tested two video decoding backends from torchvision: - -- `pyav` -- `video_reader` (requires to build torchvision from source) - -**Requested timestamps** -Given the way video decoding works, once a keyframe has been loaded, the decoding of subsequent frames is fast. -This of course is affected by the `-g` parameter during encoding, which specifies the frequency of the keyframes. Given our typical use cases in robotics policies which might request a few timestamps in different random places, we want to replicate these use cases with the following scenarios: - -- `1_frame`: 1 frame, -- `2_frames`: 2 consecutive frames (e.g. `[t, t + 1 / fps]`), -- `6_frames`: 6 consecutive frames (e.g. `[t + i / fps for i in range(6)]`) - -Note that this differs significantly from a typical use case like watching a movie, in which every frame is loaded sequentially from the beginning to the end and it's acceptable to have big values for `-g`. - -Additionally, because some policies might request single timestamps that are a few frames apart, we also have the following scenario: - -- `2_frames_4_space`: 2 frames with 4 consecutive frames of spacing in between (e.g `[t, t + 5 / fps]`), - -However, due to how video decoding is implemented with `pyav`, we don't have access to an accurate seek so in practice this scenario is essentially the same as `6_frames` since all 6 frames between `t` and `t + 5 / fps` will be decoded. - -## Metrics - -**Data compression ratio (lower is better)** -`video_images_size_ratio` is the ratio of the memory space on disk taken by the encoded video over the memory space taken by the original images. For instance, `video_images_size_ratio=25%` means that the video takes 4 times less memory space on disk compared to the original images. - -**Loading time ratio (lower is better)** -`video_images_load_time_ratio` is the ratio of the time it takes to decode frames from the video at a given timestamps over the time it takes to load the exact same original images. Lower is better. For instance, `video_images_load_time_ratio=200%` means that decoding from video is 2 times slower than loading the original images. - -**Average Mean Square Error (lower is better)** -`avg_mse` is the average mean square error between each decoded frame and its corresponding original image over all requested timestamps, and also divided by the number of pixels in the image to be comparable when switching to different image sizes. - -**Average Peak Signal to Noise Ratio (higher is better)** -`avg_psnr` measures the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. Higher PSNR indicates better quality. - -**Average Structural Similarity Index Measure (higher is better)** -`avg_ssim` evaluates the perceived quality of images by comparing luminance, contrast, and structure. SSIM values range from -1 to 1, where 1 indicates perfect similarity. - -One aspect that can't be measured here with those metrics is the compatibility of the encoding across platforms, in particular on web browser, for visualization purposes. -h264, h265 and AV1 are all commonly used codecs and should not pose an issue. However, the chroma subsampling (`pix_fmt`) format might affect compatibility: - -- `yuv420p` is more widely supported across various platforms, including web browsers. -- `yuv444p` offers higher color fidelity but might not be supported as broadly. - - - -## How the benchmark works - -The benchmark evaluates both encoding and decoding of video frames on the first episode of each dataset. - -**Encoding:** for each `vcodec` and `pix_fmt` pair, we use a default value for `g` and `crf` upon which we change a single value (either `g` or `crf`) to one of the specified values (we don't test every combination of those as this would be computationally too heavy). -This gives a unique set of encoding parameters which is used to encode the episode. - -**Decoding:** Then, for each of those unique encodings, we iterate through every combination of the decoding parameters `backend` and `timestamps_mode`. For each of them, we record the metrics of a number of samples (given by `--num-samples`). This is parallelized for efficiency and the number of processes can be controlled with `--num-workers`. Ideally, it's best to have a `--num-samples` that is divisible by `--num-workers`. - -Intermediate results saved for each `vcodec` and `pix_fmt` combination in csv tables. -These are then all concatenated to a single table ready for analysis. - -## Caveats - -We tried to measure the most impactful parameters for both encoding and decoding. However, for computational reasons we can't test out every combination. - -Additional encoding parameters exist that are not included in this benchmark. In particular: - -- `-preset` which allows for selecting encoding presets. This represents a collection of options that will provide a certain encoding speed to compression ratio. By leaving this parameter unspecified, it is considered to be `medium` for libx264 and libx265 and `8` for libsvtav1. -- `-tune` which allows to optimize the encoding for certain aspects (e.g. film quality, fast decoding, etc.). - -See the documentation mentioned above for more detailed info on these settings and for a more comprehensive list of other parameters. - -Similarly on the decoding side, other decoders exist but are not implemented in our current benchmark. To name a few: - -- `torchaudio` -- `ffmpegio` -- `decord` -- `nvc` - -Note as well that since we are mostly interested in the performance at decoding time (also because encoding is done only once before uploading a dataset), we did not measure encoding times nor have any metrics regarding encoding. -However, besides the necessity to build ffmpeg from source, encoding did not pose any issue and it didn't take a significant amount of time during this benchmark. - -## Install - -Building ffmpeg from source is required to include libx265 and libaom/libsvtav1 (av1) video codecs ([compilation guide](https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu)). - -**Note:** While you still need to build torchvision with a conda-installed `ffmpeg<4.3` to use the `video_reader` decoder (as described in [#220](https://github.com/huggingface/lerobot/pull/220)), you also need another version which is custom-built with all the video codecs for encoding. For the script to then use that version, you can prepend the command above with `PATH="$HOME/bin:$PATH"`, which is where ffmpeg should be built. - -## Adding a video decoder - -Right now, we're only benchmarking the two video decoder available with torchvision: `pyav` and `video_reader`. -You can easily add a new decoder to benchmark by adding it to this function in the script: - -```diff -def decode_video_frames( - video_path: str, - timestamps: list[float], - tolerance_s: float, - backend: str, -) -> torch.Tensor: - if backend in ["pyav", "video_reader"]: - return decode_video_frames_torchvision( - video_path, timestamps, tolerance_s, backend - ) -+ elif backend == ["your_decoder"]: -+ return your_decoder_function( -+ video_path, timestamps, tolerance_s, backend -+ ) - else: - raise NotImplementedError(backend) -``` - -## Example - -For a quick run, you can try these parameters: - -```bash -python benchmark/video/run_video_benchmark.py \ - --output-dir outputs/video_benchmark \ - --repo-ids \ - lerobot/pusht_image \ - lerobot/aloha_mobile_shrimp_image \ - --vcodec libx264 libx265 \ - --pix-fmt yuv444p yuv420p \ - --g 2 20 None \ - --crf 10 40 None \ - --timestamps-modes 1_frame 2_frames \ - --backends pyav video_reader \ - --num-samples 5 \ - --num-workers 5 \ - --save-frames 0 -``` - -## Results - -### Reproduce - -We ran the benchmark with the following parameters: - -```bash -# h264 and h265 encodings -python benchmark/video/run_video_benchmark.py \ - --output-dir outputs/video_benchmark \ - --repo-ids \ - lerobot/pusht_image \ - lerobot/aloha_mobile_shrimp_image \ - lerobot/paris_street \ - lerobot/kitchen \ - --vcodec libx264 libx265 \ - --pix-fmt yuv444p yuv420p \ - --g 1 2 3 4 5 6 10 15 20 40 None \ - --crf 0 5 10 15 20 25 30 40 50 None \ - --timestamps-modes 1_frame 2_frames 6_frames \ - --backends pyav video_reader \ - --num-samples 50 \ - --num-workers 5 \ - --save-frames 1 - -# av1 encoding (only compatible with yuv420p and pyav decoder) -python benchmark/video/run_video_benchmark.py \ - --output-dir outputs/video_benchmark \ - --repo-ids \ - lerobot/pusht_image \ - lerobot/aloha_mobile_shrimp_image \ - lerobot/paris_street \ - lerobot/kitchen \ - --vcodec libsvtav1 \ - --pix-fmt yuv420p \ - --g 1 2 3 4 5 6 10 15 20 40 None \ - --crf 0 5 10 15 20 25 30 40 50 None \ - --timestamps-modes 1_frame 2_frames 6_frames \ - --backends pyav \ - --num-samples 50 \ - --num-workers 5 \ - --save-frames 1 -``` - -The full results are available [here](https://docs.google.com/spreadsheets/d/1OYJB43Qu8fC26k_OyoMFgGBBKfQRCi4BIuYitQnq3sw/edit?usp=sharing) - -### Parameters selected for LeRobotDataset - -Considering these results, we chose what we think is the best set of encoding parameter: - -- vcodec: `libsvtav1` -- pix-fmt: `yuv420p` -- g: `2` -- crf: `30` - -Since we're using av1 encoding, we're choosing the `pyav` decoder as `video_reader` does not support it (and `pyav` doesn't require a custom build of `torchvision`). - -### Summary - -These tables show the results for `g=2` and `crf=30`, using `timestamps-modes=6_frames` and `backend=pyav` - -| video_images_size_ratio | vcodec | pix_fmt | | | | -| --------------------------------- | ---------- | ------- | --------- | --------- | --------- | -| | libx264 | | libx265 | | libsvtav1 | -| repo_id | yuv420p | yuv444p | yuv420p | yuv444p | yuv420p | -| lerobot/pusht_image | **16.97%** | 17.58% | 18.57% | 18.86% | 22.06% | -| lerobot/aloha_mobile_shrimp_image | 2.14% | 2.11% | 1.38% | **1.37%** | 5.59% | -| lerobot/paris_street | 2.12% | 2.13% | **1.54%** | **1.54%** | 4.43% | -| lerobot/kitchen | 1.40% | 1.39% | **1.00%** | **1.00%** | 2.52% | - -| video_images_load_time_ratio | vcodec | pix_fmt | | | | -| --------------------------------- | ------- | ------- | -------- | ------- | --------- | -| | libx264 | | libx265 | | libsvtav1 | -| repo_id | yuv420p | yuv444p | yuv420p | yuv444p | yuv420p | -| lerobot/pusht_image | 6.45 | 5.19 | **1.90** | 2.12 | 2.47 | -| lerobot/aloha_mobile_shrimp_image | 11.80 | 7.92 | 0.71 | 0.85 | **0.48** | -| lerobot/paris_street | 2.21 | 2.05 | 0.36 | 0.49 | **0.30** | -| lerobot/kitchen | 1.46 | 1.46 | 0.28 | 0.51 | **0.26** | - -| | | vcodec | pix_fmt | | | | -| --------------------------------- | -------- | -------- | ------------ | -------- | --------- | ------------ | -| | | libx264 | | libx265 | | libsvtav1 | -| repo_id | metric | yuv420p | yuv444p | yuv420p | yuv444p | yuv420p | -| lerobot/pusht_image | avg_mse | 2.90E-04 | **2.03E-04** | 3.13E-04 | 2.29E-04 | 2.19E-04 | -| | avg_psnr | 35.44 | 37.07 | 35.49 | **37.30** | 37.20 | -| | avg_ssim | 98.28% | **98.85%** | 98.31% | 98.84% | 98.72% | -| lerobot/aloha_mobile_shrimp_image | avg_mse | 2.76E-04 | 2.59E-04 | 3.17E-04 | 3.06E-04 | **1.30E-04** | -| | avg_psnr | 35.91 | 36.21 | 35.88 | 36.09 | **40.17** | -| | avg_ssim | 95.19% | 95.18% | 95.00% | 95.05% | **97.73%** | -| lerobot/paris_street | avg_mse | 6.89E-04 | 6.70E-04 | 4.03E-03 | 4.02E-03 | **3.09E-04** | -| | avg_psnr | 33.48 | 33.68 | 32.05 | 32.15 | **35.40** | -| | avg_ssim | 93.76% | 93.75% | 89.46% | 89.46% | **95.46%** | -| lerobot/kitchen | avg_mse | 2.50E-04 | 2.24E-04 | 4.28E-04 | 4.18E-04 | **1.53E-04** | -| | avg_psnr | 36.73 | 37.33 | 36.56 | 36.75 | **39.12** | -| | avg_ssim | 95.47% | 95.58% | 95.52% | 95.53% | **96.82%** | diff --git a/benchmarks/video/run_video_benchmark.py b/benchmarks/video/run_video_benchmark.py deleted file mode 100644 index 064a84b48..000000000 --- a/benchmarks/video/run_video_benchmark.py +++ /dev/null @@ -1,488 +0,0 @@ -#!/usr/bin/env python - -# Copyright 2024 The HuggingFace Inc. team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Assess the performance of video decoding in various configurations. - -This script will benchmark different video encoding and decoding parameters. -See the provided README.md or run `python benchmark/video/run_video_benchmark.py --help` for usage info. -""" - -import argparse -import datetime as dt -import itertools -import random -import shutil -from collections import OrderedDict -from concurrent.futures import ThreadPoolExecutor, as_completed -from pathlib import Path -from threading import Lock - -import einops -import numpy as np -import pandas as pd -import PIL -import torch -from skimage.metrics import mean_squared_error, peak_signal_noise_ratio, structural_similarity -from tqdm import tqdm - -from lerobot.datasets.lerobot_dataset import LeRobotDataset -from lerobot.datasets.video_utils import ( - decode_video_frames, - encode_video_frames, -) -from lerobot.utils.constants import OBS_IMAGE -from lerobot.utils.utils import TimerManager - -BASE_ENCODING = OrderedDict( - [ - ("vcodec", "libx264"), - ("pix_fmt", "yuv444p"), - ("g", 2), - ("crf", None), - # TODO(aliberts): Add fastdecode - # ("fastdecode", 0), - ] -) - - -# TODO(rcadene, aliberts): move to `utils.py` folder when we want to refactor -def parse_int_or_none(value) -> int | None: - if value.lower() == "none": - return None - try: - return int(value) - except ValueError as e: - raise argparse.ArgumentTypeError(f"Invalid int or None: {value}") from e - - -def check_datasets_formats(repo_ids: list) -> None: - for repo_id in repo_ids: - dataset = LeRobotDataset(repo_id) - if len(dataset.meta.video_keys) > 0: - raise ValueError( - f"Use only image dataset for running this benchmark. Video dataset provided: {repo_id}" - ) - - -def get_directory_size(directory: Path) -> int: - total_size = 0 - for item in directory.rglob("*"): - if item.is_file(): - total_size += item.stat().st_size - return total_size - - -def load_original_frames(imgs_dir: Path, timestamps: list[float], fps: int) -> torch.Tensor: - frames = [] - for ts in timestamps: - idx = int(ts * fps) - frame = PIL.Image.open(imgs_dir / f"frame-{idx:06d}.png") - frame = torch.from_numpy(np.array(frame)) - frame = frame.type(torch.float32) / 255 - frame = einops.rearrange(frame, "h w c -> c h w") - frames.append(frame) - return torch.stack(frames) - - -def save_decoded_frames( - imgs_dir: Path, save_dir: Path, frames: torch.Tensor, timestamps: list[float], fps: int -) -> None: - if save_dir.exists() and len(list(save_dir.glob("frame-*.png"))) == len(timestamps): - return - - save_dir.mkdir(parents=True, exist_ok=True) - for i, ts in enumerate(timestamps): - idx = int(ts * fps) - frame_hwc = (frames[i].permute((1, 2, 0)) * 255).type(torch.uint8).cpu().numpy() - PIL.Image.fromarray(frame_hwc).save(save_dir / f"frame-{idx:06d}_decoded.png") - shutil.copyfile(imgs_dir / f"frame-{idx:06d}.png", save_dir / f"frame-{idx:06d}_original.png") - - -def save_first_episode(imgs_dir: Path, dataset: LeRobotDataset) -> None: - episode_index = 0 - ep_num_images = dataset.meta.episodes["length"][episode_index] - if imgs_dir.exists() and len(list(imgs_dir.glob("frame-*.png"))) == ep_num_images: - return - - imgs_dir.mkdir(parents=True, exist_ok=True) - hf_dataset = dataset.hf_dataset.with_format(None) - - # We only save images from the first camera - img_keys = [key for key in hf_dataset.features if key.startswith(OBS_IMAGE)] - imgs_dataset = hf_dataset.select_columns(img_keys[0]) - - for i, item in enumerate( - tqdm(imgs_dataset, desc=f"saving {dataset.repo_id} first episode images", leave=False) - ): - img = item[img_keys[0]] - img.save(str(imgs_dir / f"frame-{i:06d}.png"), quality=100) - - if i >= ep_num_images - 1: - break - - -def sample_timestamps(timestamps_mode: str, ep_num_images: int, fps: int) -> list[float]: - # Start at 5 to allow for 2_frames_4_space and 6_frames - idx = random.randint(5, ep_num_images - 1) - match timestamps_mode: - case "1_frame": - frame_indexes = [idx] - case "2_frames": - frame_indexes = [idx - 1, idx] - case "2_frames_4_space": - frame_indexes = [idx - 5, idx] - case "6_frames": - frame_indexes = [idx - i for i in range(6)][::-1] - case _: - raise ValueError(timestamps_mode) - - return [idx / fps for idx in frame_indexes] - - -def benchmark_decoding( - imgs_dir: Path, - video_path: Path, - timestamps_mode: str, - backend: str, - ep_num_images: int, - fps: int, - num_samples: int = 50, - num_workers: int = 4, - save_frames: bool = False, -) -> dict: - def process_sample(sample: int, lock: Lock): - time_benchmark = TimerManager(log=False) - timestamps = sample_timestamps(timestamps_mode, ep_num_images, fps) - num_frames = len(timestamps) - result = { - "psnr_values": [], - "ssim_values": [], - "mse_values": [], - } - - with time_benchmark, lock: - frames = decode_video_frames(video_path, timestamps=timestamps, tolerance_s=5e-1, backend=backend) - result["load_time_video_ms"] = (time_benchmark.last * 1000) / num_frames - - with time_benchmark: - original_frames = load_original_frames(imgs_dir, timestamps, fps) - result["load_time_images_ms"] = (time_benchmark.last * 1000) / num_frames - - frames_np, original_frames_np = frames.numpy(), original_frames.numpy() - for i in range(num_frames): - result["mse_values"].append(mean_squared_error(original_frames_np[i], frames_np[i])) - result["psnr_values"].append( - peak_signal_noise_ratio(original_frames_np[i], frames_np[i], data_range=1.0) - ) - result["ssim_values"].append( - structural_similarity(original_frames_np[i], frames_np[i], data_range=1.0, channel_axis=0) - ) - - if save_frames and sample == 0: - save_dir = video_path.with_suffix("") / f"{timestamps_mode}_{backend}" - save_decoded_frames(imgs_dir, save_dir, frames, timestamps, fps) - - return result - - load_times_video_ms = [] - load_times_images_ms = [] - mse_values = [] - psnr_values = [] - ssim_values = [] - - # A sample is a single set of decoded frames specified by timestamps_mode (e.g. a single frame, 2 frames, etc.). - # For each sample, we record metrics (loading time and quality metrics) which are then averaged over all samples. - # As these samples are independent, we run them in parallel threads to speed up the benchmark. - # Use a single shared lock for all worker threads - shared_lock = Lock() - with ThreadPoolExecutor(max_workers=num_workers) as executor: - futures = [executor.submit(process_sample, i, shared_lock) for i in range(num_samples)] - for future in tqdm(as_completed(futures), total=num_samples, desc="samples", leave=False): - result = future.result() - load_times_video_ms.append(result["load_time_video_ms"]) - load_times_images_ms.append(result["load_time_images_ms"]) - psnr_values.extend(result["psnr_values"]) - ssim_values.extend(result["ssim_values"]) - mse_values.extend(result["mse_values"]) - - avg_load_time_video_ms = float(np.array(load_times_video_ms).mean()) - avg_load_time_images_ms = float(np.array(load_times_images_ms).mean()) - video_images_load_time_ratio = avg_load_time_video_ms / avg_load_time_images_ms - - return { - "avg_load_time_video_ms": avg_load_time_video_ms, - "avg_load_time_images_ms": avg_load_time_images_ms, - "video_images_load_time_ratio": video_images_load_time_ratio, - "avg_mse": float(np.mean(mse_values)), - "avg_psnr": float(np.mean(psnr_values)), - "avg_ssim": float(np.mean(ssim_values)), - } - - -def benchmark_encoding_decoding( - dataset: LeRobotDataset, - video_path: Path, - imgs_dir: Path, - encoding_cfg: dict, - decoding_cfg: dict, - num_samples: int, - num_workers: int, - save_frames: bool, - overwrite: bool = False, - seed: int = 1337, -) -> list[dict]: - fps = dataset.fps - - if overwrite or not video_path.is_file(): - tqdm.write(f"encoding {video_path}") - encode_video_frames( - imgs_dir=imgs_dir, - video_path=video_path, - fps=fps, - vcodec=encoding_cfg["vcodec"], - pix_fmt=encoding_cfg["pix_fmt"], - g=encoding_cfg.get("g"), - crf=encoding_cfg.get("crf"), - # fast_decode=encoding_cfg.get("fastdecode"), - overwrite=True, - ) - - episode_index = 0 - ep_num_images = dataset.meta.episodes["length"][episode_index] - width, height = tuple(dataset[0][dataset.meta.camera_keys[0]].shape[-2:]) - num_pixels = width * height - video_size_bytes = video_path.stat().st_size - images_size_bytes = get_directory_size(imgs_dir) - video_images_size_ratio = video_size_bytes / images_size_bytes - - random.seed(seed) - benchmark_table = [] - for timestamps_mode in tqdm( - decoding_cfg["timestamps_modes"], desc="decodings (timestamps_modes)", leave=False - ): - for backend in tqdm(decoding_cfg["backends"], desc="decodings (backends)", leave=False): - benchmark_row = benchmark_decoding( - imgs_dir, - video_path, - timestamps_mode, - backend, - ep_num_images, - fps, - num_samples, - num_workers, - save_frames, - ) - benchmark_row.update( - **{ - "repo_id": dataset.repo_id, - "resolution": f"{width} x {height}", - "num_pixels": num_pixels, - "video_size_bytes": video_size_bytes, - "images_size_bytes": images_size_bytes, - "video_images_size_ratio": video_images_size_ratio, - "timestamps_mode": timestamps_mode, - "backend": backend, - }, - **encoding_cfg, - ) - benchmark_table.append(benchmark_row) - - return benchmark_table - - -def main( - output_dir: Path, - repo_ids: list[str], - vcodec: list[str], - pix_fmt: list[str], - g: list[int], - crf: list[int], - # fastdecode: list[int], - timestamps_modes: list[str], - backends: list[str], - num_samples: int, - num_workers: int, - save_frames: bool, -): - check_datasets_formats(repo_ids) - encoding_benchmarks = { - "g": g, - "crf": crf, - # "fastdecode": fastdecode, - } - decoding_benchmarks = { - "timestamps_modes": timestamps_modes, - "backends": backends, - } - headers = ["repo_id", "resolution", "num_pixels"] - headers += list(BASE_ENCODING.keys()) - headers += [ - "timestamps_mode", - "backend", - "video_size_bytes", - "images_size_bytes", - "video_images_size_ratio", - "avg_load_time_video_ms", - "avg_load_time_images_ms", - "video_images_load_time_ratio", - "avg_mse", - "avg_psnr", - "avg_ssim", - ] - file_paths = [] - for video_codec in tqdm(vcodec, desc="encodings (vcodec)"): - for pixel_format in tqdm(pix_fmt, desc="encodings (pix_fmt)", leave=False): - benchmark_table = [] - for repo_id in tqdm(repo_ids, desc="encodings (datasets)", leave=False): - dataset = LeRobotDataset(repo_id) - imgs_dir = output_dir / "images" / dataset.repo_id.replace("/", "_") - # We only use the first episode - save_first_episode(imgs_dir, dataset) - for duet in [ - dict(zip(encoding_benchmarks.keys(), unique_combination, strict=False)) - for unique_combination in itertools.product(*encoding_benchmarks.values()) - ]: - encoding_cfg = BASE_ENCODING.copy() - encoding_cfg["vcodec"] = video_codec - encoding_cfg["pix_fmt"] = pixel_format - for key, value in duet.items(): - encoding_cfg[key] = value - args_path = Path("_".join(str(value) for value in encoding_cfg.values())) - video_path = output_dir / "videos" / args_path / f"{repo_id.replace('/', '_')}.mp4" - benchmark_table += benchmark_encoding_decoding( - dataset, - video_path, - imgs_dir, - encoding_cfg, - decoding_benchmarks, - num_samples, - num_workers, - save_frames, - ) - - # Save intermediate results - benchmark_df = pd.DataFrame(benchmark_table, columns=headers) - now = dt.datetime.now() - csv_path = ( - output_dir - / f"{now:%Y-%m-%d}_{now:%H-%M-%S}_{video_codec}_{pixel_format}_{num_samples}-samples.csv" - ) - benchmark_df.to_csv(csv_path, header=True, index=False) - file_paths.append(csv_path) - del benchmark_df - - # Concatenate all results - df_list = [pd.read_csv(csv_path) for csv_path in file_paths] - concatenated_df = pd.concat(df_list, ignore_index=True) - concatenated_path = output_dir / f"{now:%Y-%m-%d}_{now:%H-%M-%S}_all_{num_samples}-samples.csv" - concatenated_df.to_csv(concatenated_path, header=True, index=False) - - -if __name__ == "__main__": - parser = argparse.ArgumentParser() - parser.add_argument( - "--output-dir", - type=Path, - default=Path("outputs/video_benchmark"), - help="Directory where the video benchmark outputs are written.", - ) - parser.add_argument( - "--repo-ids", - type=str, - nargs="*", - default=[ - "lerobot/pusht_image", - "lerobot/aloha_mobile_shrimp_image", - "lerobot/paris_street", - "lerobot/kitchen", - ], - help="Datasets repo-ids to test against. First episodes only are used. Must be images.", - ) - parser.add_argument( - "--vcodec", - type=str, - nargs="*", - default=["h264", "hevc", "libsvtav1"], - help="Video codecs to be tested", - ) - parser.add_argument( - "--pix-fmt", - type=str, - nargs="*", - default=["yuv444p", "yuv420p"], - help="Pixel formats (chroma subsampling) to be tested", - ) - parser.add_argument( - "--g", - type=parse_int_or_none, - nargs="*", - default=[1, 2, 3, 4, 5, 6, 10, 15, 20, 40, 100, None], - help="Group of pictures sizes to be tested.", - ) - parser.add_argument( - "--crf", - type=parse_int_or_none, - nargs="*", - default=[0, 5, 10, 15, 20, 25, 30, 40, 50, None], - help="Constant rate factors to be tested.", - ) - # parser.add_argument( - # "--fastdecode", - # type=int, - # nargs="*", - # default=[0, 1], - # help="Use the fastdecode tuning option. 0 disables it. " - # "For libx264 and libx265/hevc, only 1 is possible. " - # "For libsvtav1, 1, 2 or 3 are possible values with a higher number meaning a faster decoding optimization", - # ) - parser.add_argument( - "--timestamps-modes", - type=str, - nargs="*", - default=[ - "1_frame", - "2_frames", - "2_frames_4_space", - "6_frames", - ], - help="Timestamps scenarios to be tested.", - ) - parser.add_argument( - "--backends", - type=str, - nargs="*", - default=["torchcodec", "pyav"], - help="Torchvision decoding backend to be tested.", - ) - parser.add_argument( - "--num-samples", - type=int, - default=50, - help="Number of samples for each encoding x decoding config.", - ) - parser.add_argument( - "--num-workers", - type=int, - default=10, - help="Number of processes for parallelized sample processing.", - ) - parser.add_argument( - "--save-frames", - type=int, - default=0, - help="Whether to save decoded frames or not. Enter a non-zero number for true.", - ) - args = parser.parse_args() - main(**vars(args)) diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index 85452f41c..d2aadbb65 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -3,10 +3,14 @@ title: LeRobot - local: installation title: Installation + - local: cheat-sheet + title: Cheat sheet title: Get started - sections: - local: il_robots title: Imitation Learning for Robots + - local: lelab + title: LeLab - Lerobot GUI - local: bring_your_own_policies title: Adding a Policy - local: integrate_hardware @@ -37,8 +41,14 @@ title: Porting Large Datasets - local: using_dataset_tools title: Using the Dataset Tools - - local: dataset_subtask - title: Using Subtasks in the Dataset + - local: language_and_recipes + title: Language Columns and Recipes + - local: tools + title: Tools + - local: annotation_pipeline + title: Annotation Pipeline + - local: video_encoding_parameters + title: Video encoding parameters - local: streaming_video_encoding title: Streaming Video Encoding title: "Datasets" @@ -53,8 +63,14 @@ title: π₀-FAST (Pi0Fast) - local: pi05 title: π₀.₅ (Pi05) + - local: molmoact2 + title: MolmoAct2 + - local: vla_jepa + title: VLA-JEPA - local: eo1 title: EO-1 + - local: fastwam + title: FastWAM - local: evo1 title: EVO1 - local: groot @@ -69,6 +85,10 @@ - sections: - local: sarm title: SARM + - local: robometer + title: ROBOMETER + - local: topreward + title: TOPReward title: "Reward Models" - sections: - local: inference @@ -141,6 +161,8 @@ title: OMX - local: openarm title: OpenArm + - local: rebot_b601 + title: reBot B601-DM title: "Robots" - sections: - local: phone_teleop diff --git a/docs/source/act.mdx b/docs/source/act.mdx index 453bcbba8..f64246d7a 100644 --- a/docs/source/act.mdx +++ b/docs/source/act.mdx @@ -79,17 +79,13 @@ If your local computer doesn't have a powerful GPU, you can utilize Google Colab Once training is complete, you can evaluate your ACT policy using the `lerobot-record` command with your trained policy. This will run inference and record evaluation episodes: ```bash -lerobot-record \ - --robot.type=so100_follower \ +lerobot-rollout \ + --strategy.type=base \ + --policy.path=${HF_USER}/act_policy \ + --robot.type=so101_follower \ --robot.port=/dev/ttyACM0 \ - --robot.id=my_robot \ --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \ --display_data=true \ - --dataset.repo_id=${HF_USER}/eval_act_your_dataset \ - --dataset.num_episodes=10 \ - --dataset.single_task="Your task description" \ - --dataset.streaming_encoding=true \ - --dataset.encoder_threads=2 \ - # --dataset.vcodec=auto \ - --policy.path=${HF_USER}/act_policy + --task="Your task description" \ # can be skipped for ACT + --duration=60 ``` diff --git a/docs/source/annotation_pipeline.mdx b/docs/source/annotation_pipeline.mdx new file mode 100644 index 000000000..02658ec9a --- /dev/null +++ b/docs/source/annotation_pipeline.mdx @@ -0,0 +1,291 @@ +# Annotation Pipeline + +`lerobot-annotate` watches each episode's video with a vision-language +model (VLM) and writes natural-language annotations back into your +dataset. It fills the two language columns from the +[Language Columns and Recipes](./language_and_recipes) page — +`language_persistent` and `language_events` — straight into +`data/chunk-*/file-*.parquet`. + +In short: point it at a LeRobot dataset, and it adds subtasks, plans, +memory, interjections, speech, and visual Q&A that a policy can be +trained on. + +## How it fits together + +```text + your dataset lerobot-annotate + (LeRobot v3.1) + │ + ▼ + ┌─────────────────────────────────────────────────────┐ + │ read episodes │ + └──────────────────────────┬──────────────────────────┘ + │ + ┌────────────────────┼────────────────────┐ + ▼ ▼ ▼ + ┌──────────┐ ┌───────────────┐ ┌──────────┐ one shared Qwen-VL + │ plan │ │ interjections │ │ vqa │ ◀── server (vLLM, OpenAI + └────┬─────┘ └───────┬───────┘ └────┬─────┘ API) drives all three + └────────────────────┼─────────────────────┘ + │ each module stages raw JSONL + ▼ into .annotate_staging/ + ┌─────────────────┐ + │ validator │ ◀── checks everything + └────────┬────────┘ + ▼ + ┌─────────────────┐ + │ writer │ + └────────┬────────┘ + ▼ + data/chunk-*/file-*.parquet + (+ meta/info.json tools) +``` + +Three modules (`plan`, `interjections`, `vqa`) all talk to **one** shared +VLM. Each module stages its output to disk, a validator checks it, and a +single writer rewrites the dataset shards in place. + +## What the pipeline produces + +Each module emits a few kinds of annotation ("styles"), routed to one of +the two language columns: + +| Style / atom | Column | Module | +| ------------------------------------------- | --------------------- | --------------- | +| `subtask` (Pi0.7-style "how, not what") | `language_persistent` | `plan` | +| `plan` (initial + refresh on interjection) | `language_persistent` | `plan` | +| `memory` (MEM-style compression) | `language_persistent` | `plan` | +| `task_aug` (rephrasings of the task) | `language_persistent` | `plan` | +| `interjection` | `language_events` | `interjections` | +| speech tool-call atom (`style=null`, `say`) | `language_events` | `interjections` | +| `vqa` (user / assistant pair) | `language_events` | `vqa` | + +### How subtasks are generated + +The `plan` module doesn't ask the VLM for subtasks in one shot. Instead +it uses a two-step **describe → segment** flow: + +1. **Describe** — the VLM narrates only what it actually sees in the + chosen camera (no guessing about the task). +2. **Segment** — that description is fed back in, and the VLM splits the + episode into consecutive atomic subtasks. + +Both passes see the episode as **timestamped contact sheets** — frames +sampled at `frames_per_second` (0.5s by default) and packed into JPEG +grids with each frame's time burned into its corner, so the VLM cites +exact boundary times directly. This is far cheaper in vision tokens than +one image per frame, so the sampling can stay dense; episodes longer than +`max_frames_per_prompt` are split into windows at the same density and +merged. Both prompts also carry a causal **event-boundary** definition (a +new event starts when an object becomes held / is released / reaches a new +location / a lid changes state / contents move) to sharpen where cuts land. + +The resulting spans are then stitched into a gap-free, full-episode +cover, so **every frame has exactly one active subtask**. See +[`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py) +for the production settings (single camera, timestamped contact sheets, +auto-windowed subtask generation). + +### Tools + +The writer does **not** add a `tools` column to the parquet. The tool +catalog lives in `meta/info.json["tools"]` instead (see [Tools](./tools)). +After every run, the pipeline makes sure the canonical `say` schema is in +that list, keeping any tools you declared beforehand. + +Want to add your own tool? Edit `meta/info.json["tools"]` directly — the +pipeline preserves whatever is already there. That makes the tool visible +to the chat template, so the model can learn to _generate_ the call. The +runtime layer that actually _executes_ a generated call (the `Tool` +protocol / `TOOL_REGISTRY` under `src/lerobot/tools/`) is not part of +this PR — the [Tools](./tools) doc marks those pieces as +not-yet-implemented. + +## Running on Hugging Face Jobs + +Annotation runs on [Hugging Face Jobs](https://huggingface.co/docs/hub/en/jobs). +The repo ships a launcher script you copy and tweak for your dataset: + +```bash +HF_TOKEN=hf_... uv run python examples/annotations/run_hf_job.py +``` + +[`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py) +starts a single-GPU `h200` job (bump it to `h200x4` for big datasets) +that: + +1. installs `lerobot` (from `main`) plus the annotation extras, +2. boots one vLLM server per GPU (using the `vllm/vllm-openai` image) and + drives it over the OpenAI-compatible API, +3. runs the `plan` / `interjections` / `vqa` modules across the dataset + with `lerobot-annotate`, +4. with `--push_to_hub=true`, uploads the result to `--new_repo_id` (or + back to `--repo_id` in place if you leave that unset). + +To use a different dataset, model, or hub repo, edit the `CMD` block in +the script. Every flag there maps directly to a `lerobot-annotate` flag +(run `lerobot-annotate --help` for the full list). + +## Key options + +These are the flags you'll reach for most often. Run +`lerobot-annotate --help` for everything else; the defaults are tuned for +short manipulation episodes. + +### Dataset in / out + +| Flag | Default | What it does | +| ----------------- | ------- | ----------------------------------------------------------------------- | +| `--repo_id` | — | Hub dataset to annotate (downloaded if `--root` unset). | +| `--root` | — | Annotate a local dataset directory instead. | +| `--new_repo_id` | — | Push the result to a new repo (leaves the source repo untouched). | +| `--push_to_hub` | `false` | Upload after annotating (to `--new_repo_id`, else back to `--repo_id`). | +| `--only_episodes` | all | Annotate just these episode indices (handy for a test run). | +| `--seed` | `1729` | Seeds the RNGs that pick interjection timestamps + VQA question types. | + +### Which modules run + +Every module is on by default and can be toggled independently (set to +`false` to skip it, e.g. to iterate on one module at a time): + +| Flag | Default | Turns off | +| ------------------------- | ------- | ----------------------------------- | +| `--plan.enabled` | `true` | subtasks + plan + memory + task_aug | +| `--interjections.enabled` | `true` | interjections + speech atoms | +| `--vqa.enabled` | `true` | the VQA pairs | + +### The VLM (`--vlm.*`) + +| Flag | Default | What it does | +| -------------------------- | ------------------ | ----------------------------------------------------------------------------------- | +| `--vlm.model_id` | `Qwen/Qwen3.6-27B` | The model to serve and prompt. | +| `--vlm.camera_key` | first `images.*` | Which camera every prompt is grounded on. | +| `--vlm.serve_command` | auto | The exact `vllm serve …` command (set TP size, GPU memory, `--max-model-len` here). | +| `--vlm.parallel_servers` | `1` | Independent servers for round-robin routing (one per GPU). | +| `--vlm.num_gpus` | `0` | GPUs per server (`0` = one each). | +| `--vlm.client_concurrency` | `16` | In-flight requests across all servers. | +| `--vlm.max_new_tokens` | `512` | Generation cap per call. | +| `--vlm.temperature` | `0.2` | Sampling temperature. | + +### Subtasks / plan / memory (`--plan.*`) + +| Flag | Default | What it does | +| ------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------- | +| `--plan.frames_per_second` | `2.0` | Frame sampling rate for the contact sheets (`2.0` = one frame every 0.5s). | +| `--plan.max_frames_per_prompt` | `60` | Frame budget per VLM call. Episodes whose sampling exceeds this are auto-windowed at the same density, then stitched. | +| `--plan.contact_sheet_columns` | `5` | Columns per contact-sheet grid (`contact_sheet_frames_per_sheet` tiles, time row-major). | +| `--plan.plan_max_steps` | `8` | Upper bound on subtasks per episode. | +| `--plan.subtask_describe_first` | `true` | Run the describe→segment grounding pass (best subtask quality; +1 call/episode). | +| `--plan.emit_plan` | `true` | Emit the numbered `plan` rows (`false` = subtasks + memory only). | +| `--plan.emit_memory` | `true` | Emit the `memory` rows (`false` = subtasks + plan only); symmetric to `emit_plan`. | +| `--plan.n_task_rephrasings` | `10` | How many `task_aug` rephrasings to emit (`0` disables). | +| `--plan.derive_task_from_video` | `if_short` | Use the dataset task as-is (`off`), only when it's missing/short (`if_short`), or always re-derive from video (`always`). | + +### Interjections + VQA + +| Flag | Default | What it does | +| ----------------------------------------------- | ------- | ---------------------------------------------------------- | +| `--interjections.max_interjections_per_episode` | `3` | Cap on interjection/speech pairs per episode. | +| `--vqa.vqa_emission_hz` | `1.0` | How often VQA pairs are emitted. | +| `--vqa.restrict_to_default_camera` | `false` | Ground VQA only on `--vlm.camera_key` (else every camera). | +| `--executor.episode_parallelism` | `16` | Episodes processed concurrently within each phase. | + +## Contributing new modules + +The pipeline is built to grow, and **contributions are very welcome** — +a brand-new module (say, trajectory traces or affordances), a new prompt +template, a smarter grounding flow, or quality fixes to the existing +`plan` / `interjections` / `vqa` modules. + +Every module lives under +`src/lerobot/annotations/steerable_pipeline/modules/`, shares the VLM +client and the keyframe cache, writes its raw output to the staging +tree, and plugs into the executor as its own phase. Got an idea? Open an +issue or PR on [the repo](https://github.com/huggingface/lerobot). + +## How recipes consume the output + +The annotations are meant to be read by recipes (see +[Language Columns and Recipes](./language_and_recipes)). Typically: + +- low-level / high-level / memory-update branches read + `subtask` / `plan` / `memory` from `language_persistent`. +- an interjection-response branch reads `interjection` events plus the + paired speech atom (merged into one assistant turn via `tool_calls_from`) + and the matching `plan` refresh at the same timestamp. +- a VQA branch reads the `(vqa, user)` and `(vqa, assistant)` pairs from + `language_events`. + +## Why state and events are split + +Two ideas shape the design: + +1. **Persistent state vs. exact events.** Persistent rows (`subtask`, + `plan`, `memory`) apply to the whole episode and answer "what's true + right now?". Event rows (`interjection`, `vqa`, speech) appear only on + the one frame whose timestamp matches. Timestamps are copied straight + from the source parquet — never recomputed in floating point. +2. **One VLM pass.** All three modules share a single VLM client (the + OpenAI-compatible client talking to the job's vLLM server), so you pay + for one model load per dataset, not three. + +## Re-running a single module + +Each module stages its raw output to +`/.annotate_staging/episode_{N:06d}/.jsonl`. This makes +prompt iteration cheap: re-running one module overwrites only its own +JSONL, then the writer recomposes the final parquet. Disable modules you +don't want with `--plan.enabled=false` (and likewise +`--interjections.enabled` / `--vqa.enabled`) to test one at a time. + +## What the validator checks + +Before the writer runs, `StagingValidator` confirms: + +- every event row lands exactly on a real frame timestamp; +- no speech / interjection pairs are left orphaned; +- `plan` is refreshed at every interjection timestamp; +- `memory` rows fall on subtask boundaries (a warning, not an error); +- each VQA assistant `content` is valid JSON in one of the + bbox / keypoint / count / attribute / spatial shapes; +- every row goes to the column chosen by `column_for_style(style)`. + +Any error aborts the writer. Pass `--skip_validation=true` to override +while debugging. + +## Where each module's ideas come from + +- **`plan` — subtasks.** Hi Robot ([Shi 2025](https://arxiv.org/abs/2502.19417)) + for atom granularity ("pick up one piece of lettuce", "place bowl to + box"); Pi0.7 ([Physical Intelligence 2025](https://pi.website/pi07)) + for "how, not what" detail. +- **`plan` — memory.** MEM ([Torne 2026](https://arxiv.org/abs/2603.03596)): + keep only the minimal relevant information — preserve outcomes, drop + specific attributes. +- **`interjections`.** Hi Robot's scenario taxonomy: negative task, + situated correction, specific constraint, preference. Speech is a + tool-call-only atom + (`tool_calls=[{type:function, function:{name:"say", arguments:{text:...}}}]`). +- **`vqa`.** ECoT ([Zawalski 2024](https://arxiv.org/abs/2407.08693)) for + grounded features (pixel bounding boxes `[x_min, y_min, x_max, y_max]`, + keypoints) and Steerable VLA Policies + ([Zhao 2025](https://arxiv.org/abs/2509.07626)) for multi-abstraction + grounding. Pi0.7 also grounds answers across abstraction levels. + +When improving a module, tweak its prompt template in +`src/lerobot/annotations/steerable_pipeline/prompts/` rather than +rewriting from scratch. + +## Roughly how much it costs + +Per episode, the pipeline makes about `max_steps` plan calls, +`max_interjections_per_episode` interjection calls, and +`vqa_emission_hz × episode_seconds` VQA calls. With the defaults (8 +subtasks, 1 interjection, 1 Hz × 3 pairs) on a 30-second episode, that's +~50 VLM calls. + +Storage stays small: `language_persistent` is at most tens of KB per +episode (parquet dictionary-encodes the one entry that repeats across +frames), and `language_events` is empty on most frames — its size scales +with the number of emissions, not `num_frames × num_emissions`. diff --git a/docs/source/cameras.mdx b/docs/source/cameras.mdx index 2dc2859dd..02714d591 100644 --- a/docs/source/cameras.mdx +++ b/docs/source/cameras.mdx @@ -157,6 +157,14 @@ finally: +### Working with depth + +The Intel RealSense and Reachy 2 cameras can capture both color and depth in lockstep. Calling `read()` returns the **color** frame as `(H, W, 3)` `uint8`. Calling `read_depth()` returns the **depth map** as `(H, W, 1)` `uint16`, where each pixel value is the distance from the sensor expressed in **millimetres**. A pixel value of `0` typically means "no measurement available" (out-of-range, occluded, or low-confidence). + +During recording, the control loop peeks the freshest buffered frames non-blockingly via `read_latest()` (color) and `read_latest_depth()` (depth), adding the depth map as a sibling feature (e.g. `front_depth` next to `front`). + +For how depth streams are stored and encoded when recording a dataset, see the [Depth streams](./video_encoding_parameters#depth-streams) section of the video encoding guide. + ## Use your phone's camera diff --git a/docs/source/cheat-sheet.mdx b/docs/source/cheat-sheet.mdx new file mode 100644 index 000000000..0531c95bf --- /dev/null +++ b/docs/source/cheat-sheet.mdx @@ -0,0 +1,177 @@ +# Cheat sheet + +All of the LeRobot commands in one place. If you forgot how to use a specific command or want to learn about a new one you can do it here. + +> [!WARNING] +> For all of the commands listed below remember to change the ports/names/ids to your own values! + +> [!TIP] +> Another great way to look at all the commands and get them configured for your specific setup is to use this [Jupyter Notebook](https://github.com/huggingface/lerobot/blob/main/examples/notebooks/quickstart.ipynb). + +### Setup and installation + +For installation please look at [LeRobot Installation](https://huggingface.co/docs/lerobot/main/en/installation). + +### Useful tools + +###### Find port + +Use this to identify which serial ports your robots are connected to. Follow the instructions in your terminal: you will be asked to unplug the USB cable and press Enter. The script will then detect and print the correct serial port for that robot. + +```bash +lerobot-find-port +``` + +###### Find cameras + +Quickly find camera indices and verify their output. This command prints camera information to the terminal and saves test frames from each detected camera to `lerobot/outputs/captured_images` + +```bash +lerobot-find-cameras +``` + +### Calibration + +In most cases you will need to perform calibration just once for each robot and teleoperation device. Before performing the calibration make sure that all the joints are roughly in the middle position. + +```bash +lerobot-calibrate \ + --robot.type=so101_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.id=my_follower_arm +``` + +Make sure that you use the same IDs used during calibration later for the other scripts. That's how LeRobot finds the calibration files. + +### Teleoperation + +Teleoperating with two cameras and displaying the data with Rerun. + +```bash +lerobot-teleoperate \ + --robot.type=so101_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.id=my_follower_arm \ + --robot.cameras="{ top: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30} }" \ + --teleop.type=so101_leader \ + --teleop.port=/dev/ttyACM1 \ + --teleop.id=my_leader_arm \ + --display_data=true +``` + +### Recording a dataset + +The dataset is automatically uploaded to the server and saved under repo_id, make sure you are logged in to your HF account with CLI: +`hf auth login` + +You can get the token from: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) + +```bash +lerobot-record \ + --robot.type=so101_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.id=my_follower_arm \ + --robot.cameras="{ top: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30} }" \ + --teleop.type=so101_leader \ + --teleop.port=/dev/ttyACM1 \ + --teleop.id=my_leader_arm \ + --dataset.repo_id=${HF_USER}/so101_dataset_test \ + --dataset.num_episodes=30 \ + --dataset.single_task="put the red brick in a bowl" \ + --dataset.streaming_encoding=true \ + --display_data=true +``` + +While collecting the dataset you can control the process with your keyboard: +Control the data recording flow using keyboard shortcuts: + +- Press **Right Arrow (`→`)**: Save episode and move to the next. +- Press **Left Arrow (`←`)**: Delete current episode and retry. +- Press **Escape (`ESC`)**: Stop, encode videos, and upload. + +### Recording depth + +Intel RealSense cameras (`type: intelrealsense`) record a depth stream when you set `use_depth: true`. Depth is quantized to 12-bit codes and stored as its own video. + +```bash +lerobot-record \ + ... \ + --robot.cameras="{ head: {type: intelrealsense, serial_number_or_name: \"0123456789\", width: 640, height: 480, fps: 30, use_depth: true} }" \ + --dataset.repo_id=${HF_USER}/so101_depth_test \ + --dataset.single_task="put the red brick in a bowl" \ + --dataset.depth_encoder.depth_min=0.01 \ + --dataset.depth_encoder.depth_max=10.0 \ + --dataset.depth_encoder.shift=0.0 \ + --dataset.depth_encoder.use_log=true +``` + +### Video encoding parameters + +RGB and depth streams are encoded independently via the `--dataset.rgb_encoder.*` and `--dataset.depth_encoder.*` keys. + +```bash +lerobot-record \ + ... \ + --dataset.rgb_encoder.vcodec=h264 \ + --dataset.rgb_encoder.pix_fmt=yuv420p \ + --dataset.rgb_encoder.crf=23 \ + --dataset.depth_encoder.vcodec=hevc \ + --dataset.depth_encoder.extra_options='{"x265-params": "lossless=1"}' +``` + +### Training + +Depending on your hardware training the policy might take a few hours. That's how you train simple `ACT` policy: + +```bash +lerobot-train \ + --dataset.repo_id=${HF_USER}/so101_dataset_test \ + --policy.type=act \ + --output_dir=outputs/train/act_so101_test \ + --job_name=act_so101_test \ + --policy.device=cuda \ + --wandb.enable=true \ + --policy.repo_id=${HF_USER}/policy_test \ + --steps=20000 +``` + +- Policy Types: `act`, `diffusion`, `smolvla`, `pi05` +- Devices: `cuda` (NVIDIA), `mps` (Apple Silicon), `cpu` + +If you want to fine-tune a specific model you can provide the path to the model. In this case path is enough and type can be skipped. + +```bash +lerobot-train \ + --dataset.repo_id=${HF_USER}/so101_dataset_test \ + --policy.path=username/the_policy_to_finetune \ + --policy.device=cuda \ + --policy.repo_id=${HF_USER}/policy_test \ + --output_dir=outputs/train/act_so101_test \ + --steps=20000 +``` + +No local GPU? Add `--job.target=` (e.g. `a10g-small`) to either command and `lerobot-train` runs it on [Hugging Face Jobs](https://huggingface.co/docs/hub/jobs) instead — it uploads a local-only dataset for you and pushes the trained model. List flavors with `hf jobs hardware`. + +To resume, point `--config_path` at a checkpoint and add `--resume=true`. It accepts a local path or a Hub repo id (the latest checkpoint is fetched), and works locally or on a job by adding `--job.target=`: + +```bash +lerobot-train --config_path=${HF_USER}/policy_test --resume=true --job.target=a10g-small +``` + +### Inference + +Inference means running the trained policy/model on a robot. For that we use `lerobot-rollout`. You will need to provide a path to your policy. It can be a local path or a path to Hugging Face for example "lerobot/folding_latest". Your cameras configuration needs to match what was used when collecting the dataset. Duration is in seconds if unspecified, it will run forever. + +> [!TIP] +> If you are using the previous release V0.5.1 instead of `lerobot-rollout` you need to use `lerobot-record`. More information [here](https://huggingface.co/docs/lerobot/v0.5.1/en/il_robots#run-inference-and-evaluate-your-policy). + +```bash +lerobot-rollout \ + --strategy.type=base \ + --policy.path=${HF_USER}/my_policy \ + --robot.type=so101_follower \ + --robot.port=/dev/ttyACM1 \ + --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video1, width: 640, height: 480, fps: 30}, side: {type: opencv, index_or_path: /dev/video5, width: 640, height: 480, fps: 30}}" \ + --task="Put lego brick into the transparent box" \ + --duration=60 +``` diff --git a/docs/source/dataset_subtask.mdx b/docs/source/dataset_subtask.mdx deleted file mode 100644 index 6264aca22..000000000 --- a/docs/source/dataset_subtask.mdx +++ /dev/null @@ -1,277 +0,0 @@ -# Using Subtasks in LeRobot Datasets - -Subtask support in robotics datasets has proven effective in improving robot reasoning and understanding. Subtasks are particularly useful for: - -- **Hierarchical policies**: Building policies that include subtask predictions to visualize robot reasoning in real time -- **Reward modeling**: Helping reward models understand task progression (e.g., SARM-style stage-aware reward models) -- **Task decomposition**: Breaking down complex manipulation tasks into atomic, interpretable steps - -LeRobotDataset now supports subtasks as part of its dataset structure, alongside tasks. - -## What are Subtasks? - -While a **task** describes the overall goal (e.g., "Pick up the apple and place it in the basket"), **subtasks** break down the execution into finer-grained steps: - -1. "Approach the apple" -2. "Grasp the apple" -3. "Lift the apple" -4. "Move to basket" -5. "Release the apple" - -Each frame in the dataset can be annotated with its corresponding subtask, enabling models to learn and predict these intermediate stages. - -An overview of subtask annotation showing how frames are labeled with intermediate subtask stages - -

- Figure: Overview of subtask annotation. -

- -**Reference:** _Subtask-learning based for robot self-assembly in flexible collaborative assembly in manufacturing_, Original Article, Published: 19 April 2022. - -## Dataset Structure - -Subtask information is stored in the dataset metadata: - -``` -my-dataset/ -├── data/ -│ └── ... -├── meta/ -│ ├── info.json -│ ├── stats.json -│ ├── tasks.parquet -│ ├── subtasks.parquet # Subtask index → subtask string mapping -│ └── episodes/ -│ └── ... -└── videos/ - └── ... -``` - -### Subtasks Parquet File - -The `meta/subtasks.parquet` file maps subtask indices to their natural language descriptions: - -| subtask_index | subtask (index column) | -| ------------- | ---------------------- | -| 0 | "Approach the apple" | -| 1 | "Grasp the apple" | -| 2 | "Lift the apple" | -| ... | ... | - -### Frame-Level Annotations - -Each frame in the dataset can include a `subtask_index` field that references the subtasks parquet file: - -```python -# Example frame data in the parquet file -{ - "index": 42, - "timestamp": 1.4, - "episode_index": 0, - "task_index": 0, - "subtask_index": 2, # References "Lift the apple" - "observation.state": [...], - "action": [...], -} -``` - -## Annotating Datasets with Subtasks - -We provide a HuggingFace Space for easily annotating any LeRobotDataset with subtasks: - -**[https://huggingface.co/spaces/lerobot/annotate](https://huggingface.co/spaces/lerobot/annotate)** - -After completing your annotation: - -1. Click "Push to Hub" to upload your annotated dataset -2. You can also run the annotation space locally by following the instructions at [github.com/huggingface/lerobot-annotate](https://github.com/huggingface/lerobot-annotate) - -## Loading Datasets with Subtasks - -When you load a dataset with subtask annotations, the subtask information is automatically available: - -```python -from lerobot.datasets import LeRobotDataset - -# Load a dataset with subtask annotations -dataset = LeRobotDataset("jadechoghari/collect-fruit-annotated") - -# Access a sample -sample = dataset[100] - -# The sample includes both task and subtask information -print(sample["task"]) # "Collect the fruit" -print(sample["subtask"]) # "Grasp the apple" -print(sample["task_index"]) # tensor(0) -print(sample["subtask_index"]) # tensor(2) -``` - -### Checking for Subtask Support - -You can check if a dataset has subtask annotations: - -```python -# Check if subtasks are available -has_subtasks = ( - "subtask_index" in dataset.features - and dataset.meta.subtasks is not None -) - -if has_subtasks: - print(f"Dataset has {len(dataset.meta.subtasks)} unique subtasks") - print("Subtasks:", list(dataset.meta.subtasks.index)) -``` - -## Using Subtasks for Training - -### With the Tokenizer Processor - -The `TokenizerProcessor` automatically handles subtask tokenization for Vision-Language Action (VLA) models: - -```python -from lerobot.processor import TokenizerProcessorStep - -# Create a tokenizer processor step -tokenizer_processor = TokenizerProcessorStep( - tokenizer_name_or_path="google/paligemma-3b-pt-224", - padding="max_length", - max_length=64, -) - -# The processor will automatically tokenize subtasks if present in the batch -# and add them to the observation under: -# - "observation.subtask.tokens" -# - "observation.subtask.attention_mask" -``` - -When subtasks are available in the batch, the tokenizer processor adds: - -- `observation.subtask.tokens`: Tokenized subtask text -- `observation.subtask.attention_mask`: Attention mask for the subtask tokens - -### DataLoader with Subtasks - -```python -import torch -from lerobot.datasets import LeRobotDataset - -dataset = LeRobotDataset("jadechoghari/collect-fruit-annotated") - -dataloader = torch.utils.data.DataLoader( - dataset, - batch_size=16, - shuffle=True, -) - -for batch in dataloader: - # Access subtask information in the batch - subtasks = batch["subtask"] # List of subtask strings - subtask_indices = batch["subtask_index"] # Tensor of subtask indices - - # Use for training hierarchical policies or reward models - print(f"Batch subtasks: {set(subtasks)}") -``` - -## Example Datasets with Subtask Annotations - -Try loading a dataset with subtask annotations: - -```python -from lerobot.datasets import LeRobotDataset - -# Example dataset with subtask annotations -dataset = LeRobotDataset("jadechoghari/collect-fruit-annotated") - -# Explore the subtasks -print("Available subtasks:") -for subtask_name in dataset.meta.subtasks.index: - print(f" - {subtask_name}") - -# Get subtask distribution -subtask_counts = {} -for i in range(len(dataset)): - sample = dataset[i] - subtask = sample["subtask"] - subtask_counts[subtask] = subtask_counts.get(subtask, 0) + 1 - -print("\nSubtask distribution:") -for subtask, count in sorted(subtask_counts.items(), key=lambda x: -x[1]): - print(f" {subtask}: {count} frames") -``` - -## Use Cases - -### 1. Hierarchical Policy Training - -Train policies that predict both actions and current subtask: - -```python -class HierarchicalPolicy(nn.Module): - def __init__(self, num_subtasks): - super().__init__() - self.action_head = nn.Linear(hidden_dim, action_dim) - self.subtask_head = nn.Linear(hidden_dim, num_subtasks) - - def forward(self, observations): - features = self.encoder(observations) - actions = self.action_head(features) - subtask_logits = self.subtask_head(features) - return actions, subtask_logits -``` - -### 2. Stage-Aware Reward Modeling (SARM) - -Build reward models that understand task progression: - -```python -# SARM predicts: -# - Stage: Which subtask is being executed (discrete) -# - Progress: How far along the subtask (continuous 0-1) - -class SARMRewardModel(nn.Module): - def forward(self, observations): - features = self.encoder(observations) - stage_logits = self.stage_classifier(features) - progress = self.progress_regressor(features) - return stage_logits, progress -``` - -### 3. Progress Visualization - -Monitor robot execution by tracking subtask progression: - -```python -def visualize_execution(model, observations): - for t, obs in enumerate(observations): - action, subtask_logits = model(obs) - predicted_subtask = subtask_names[subtask_logits.argmax()] - print(f"t={t}: Executing '{predicted_subtask}'") -``` - -## API Reference - -### LeRobotDataset Properties - -| Property | Type | Description | -| --------------------------- | ---------------------- | ------------------------------------------ | -| `meta.subtasks` | `pd.DataFrame \| None` | DataFrame mapping subtask names to indices | -| `features["subtask_index"]` | `dict` | Feature spec for subtask_index if present | - -### Sample Keys - -When subtasks are available, each sample includes: - -| Key | Type | Description | -| --------------- | -------------- | ------------------------------------ | -| `subtask_index` | `torch.Tensor` | Integer index of the current subtask | -| `subtask` | `str` | Natural language subtask description | - -## Related Resources - -- [SARM Paper](https://arxiv.org/pdf/2509.25358) - Stage-Aware Reward Modeling for Long Horizon Robot Manipulation -- [LeRobot Annotate Space](https://huggingface.co/spaces/lerobot/annotate) - Interactive annotation tool -- [LeRobotDataset v3.0](./lerobot-dataset-v3) - Dataset format documentation diff --git a/docs/source/earthrover_mini_plus.mdx b/docs/source/earthrover_mini_plus.mdx index a87bd325b..f3b324093 100644 --- a/docs/source/earthrover_mini_plus.mdx +++ b/docs/source/earthrover_mini_plus.mdx @@ -194,7 +194,7 @@ lerobot-record \ --dataset.single_task="Navigate around obstacles" \ --dataset.streaming_encoding=true \ --dataset.encoder_threads=2 \ - # --dataset.vcodec=auto \ + # --dataset.rgb_encoder.vcodec=auto \ --display_data=true ``` diff --git a/docs/source/fastwam.mdx b/docs/source/fastwam.mdx new file mode 100644 index 000000000..18b4775f8 --- /dev/null +++ b/docs/source/fastwam.mdx @@ -0,0 +1,167 @@ +# FastWAM + +FastWAM is a World Action Model policy for robot control. The LeRobot integration exposes FastWAM through the standard policy API so it can be configured with `policy.type=fastwam`, trained with `lerobot-train`, and loaded through the LeRobot pretrained policy interface. + +## Model Overview + +FastWAM keeps video modeling during training, but uses direct action prediction at inference time instead of iteratively generating future observations. This LeRobot policy wraps the FastWAM action model, adapts LeRobot batches to FastWAM training samples, and provides the standard processor pipeline for normalization and action postprocessing. + +The implementation initializes the visual world-model components from `Wan-AI/Wan2.2-TI2V-5B` by default and predicts action chunks with shape `[batch, action_horizon, action_dim]`. + +### What the LeRobot Integration Covers + +- Standard `policy.type=fastwam` configuration through LeRobot +- Image, state, action, and language-task batch adaptation +- Action chunk inference through `select_action` and `predict_action_chunk` +- Checkpoint save/load through the LeRobot policy APIs +- Configurable LIBERO gripper action postprocessing + +## Installation Requirements + +Install LeRobot from source, then install FastWAM dependencies: + +```bash +pip install -e ".[fastwam]" +``` + +This installs the FastWAM policy extra from `pyproject.toml`: `transformers`, +`diffusers`, `ftfy`, and `regex`, plus LeRobot's base dependencies. + +For LIBERO evaluation, install the benchmark dependencies too: + +```bash +pip install -e ".[fastwam,libero]" +``` + +This installs both extras. In addition to the FastWAM dependencies above, the +`libero` extra installs LeRobot dataset dependencies, `hf-libero` on Linux, and +`scipy`. + +FastWAM uses the Wan2.2 TI2V backbone. The default model id is: + +```python +policy.model_id=Wan-AI/Wan2.2-TI2V-5B +``` + +## Data Requirements + +FastWAM expects a LeRobot dataset with: + +- one or more visual observations whose widths concatenate to `policy.image_size[1]` +- `observation.state` when `policy.proprio_dim` is not `None` +- `action` +- a language task instruction through the dataset task field, or precomputed `context` and `context_mask` tensors + +The default visual setup is one image feature named `observation.images.image` with shape `(3, 224, 448)`. If the dataset uses two cameras, configure `policy.input_features` so their heights match `224` and their widths sum to `448`. + +## Usage + +Create a new FastWAM policy with: + +```bash +lerobot-train \ + --dataset.repo_id=your-org/your-dataset \ + --policy.type=fastwam \ + --policy.action_dim=7 \ + --policy.proprio_dim=8 \ + --policy.action_horizon=32 \ + --policy.n_action_steps=10 \ + --policy.image_size='[224,448]' \ + --output_dir=./outputs/fastwam_training \ + --job_name=fastwam_training \ + --steps=300000 \ + --batch_size=8 \ + --policy.device=cuda +``` + +Evaluate an existing LeRobot-format checkpoint on LIBERO-10 with: + +```bash +lerobot-eval \ + --policy.path=ZibinDong/fastwam_libero_uncond_2cam224 \ + --policy.device=cuda \ + --policy.torch_dtype=float32 \ + --policy.n_action_steps=10 \ + --env.type=libero \ + --env.task=libero_10 \ + --env.observation_height=224 \ + --env.observation_width=224 \ + --eval.batch_size=1 \ + --eval.n_episodes=50 \ + --seed=0 \ + --env.episode_length=600 +``` + +For `libero_goal`, `libero_spatial`, and `libero_object`, use +`--env.episode_length=300`. + +For real-robot rollout, use the same checkpoint path: + +```bash +lerobot-rollout \ + --robot.type=so101_follower \ + --robot.port=/dev/ttyACM0 \ + --policy.path=your-org/fastwam-real-robot +``` + +## Configuration Notes + +### Image Features + +`policy.image_size` is the size of the concatenated FastWAM image tensor as `(height, width)`. Each configured image feature must have shape `(3, height, camera_width)`, and all camera widths must sum to the configured width. + +### Action Chunking + +`policy.action_horizon` controls the number of future actions supervised during training and predicted during inference. `policy.n_action_steps` controls how many actions are consumed before the policy predicts a fresh chunk. `policy.n_action_steps` must be less than or equal to `policy.action_horizon`. + +### Wan Components + +FastWAM loads the Wan VAE, video DiT, text encoder, and tokenizer from the configured Wan model directory or Hugging Face Hub model id. LeRobot-format FastWAM checkpoints saved by `save_pretrained` also copy the local Wan component files needed by `from_pretrained`. + +### Attention Backend + +FastWAM's DiT uses PyTorch's `scaled_dot_product_attention` (SDPA) for all attention. It does **not** use FlashAttention: its Mixture-of-Transformers (MoT) routing needs arbitrary boolean `[query, key]` attention masks, which the FlashAttention varlen API cannot express. Installing the `flash-attn` package therefore has no effect on the FastWAM path. (Note that SDPA itself may still select PyTorch's own flash / memory-efficient / math kernel internally — this is unrelated to the `flash-attn` package.) + +### LIBERO Action Toggle + +FastWAM LIBERO checkpoints use `policy.toggle_action_dimensions=[-1]` by +default to match the gripper action convention used by the original FastWAM +evaluation pipeline: + +```bash +--policy.toggle_action_dimensions='[-1]' +``` + +## Results + +Evaluated on LIBERO with [`ZibinDong/fastwam_libero_uncond_2cam224`](https://huggingface.co/ZibinDong/fastwam_libero_uncond_2cam224): + +| Suite | Success rate | n_episodes | +| -------------- | -----------: | ---------: | +| libero_spatial | 97.6% | 500 | +| libero_object | 99.0% | 500 | +| libero_goal | 95.0% | 500 | +| libero_10 | 94.0% | 500 | +| **average** | **96.4%** | 2000 | + +Reproduce: `lerobot-eval --policy.path=ZibinDong/fastwam_libero_uncond_2cam224 --policy.device=cuda --policy.torch_dtype=float32 --policy.n_action_steps=10 --env.type=libero --env.task=libero_spatial --env.observation_height=256 --env.observation_width=256 --eval.batch_size=1 --eval.n_episodes=50 --seed=0 --env.episode_length=300` (1x H20 140 GB). + +## References + +- [Fast-WAM paper](https://arxiv.org/abs/2603.16666) +- [Fast-WAM project page](https://yuantianyuan01.github.io/FastWAM/) +- [Fast-WAM code](https://github.com/yuantianyuan01/FastWAM) +- [Released upstream checkpoints](https://huggingface.co/yuanty/fastwam) +- [Wan2.2 TI2V 5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) + +## Citation + +```bibtex +@article{yuan2026fastwam, + title = {Fast-WAM: Do World Action Models Need Test-time Future Imagination?}, + author = {Tianyuan Yuan and Zibin Dong and Yicheng Liu and Hang Zhao}, + journal = {arXiv preprint arXiv:2603.16666}, + year = {2026}, + url = {https://arxiv.org/abs/2603.16666} +} +``` diff --git a/docs/source/groot.mdx b/docs/source/groot.mdx index 2f53a4d0b..3ab202fb2 100644 --- a/docs/source/groot.mdx +++ b/docs/source/groot.mdx @@ -105,10 +105,12 @@ These results demonstrate GR00T's strong generalization capabilities across dive ### Evaluate in your hardware setup -Once you have trained your model using your parameters you can run inference in your downstream task. Follow the instructions in [Imitation Learning for Robots](./il_robots). For example: +Once you have trained your model using your parameters you can run inference in your downstream task. Follow the instructions in [Policy Deployment (lerobot-rollout)](./inference). For example: ```bash -lerobot-record \ +lerobot-rollout\ + --strategy.type=sentry \ + --strategy.upload_every_n_episodes=5 \ --robot.type=bi_so_follower \ --robot.left_arm_port=/dev/ttyACM1 \ --robot.right_arm_port=/dev/ttyACM0 \ @@ -119,14 +121,12 @@ lerobot-record \ }' \ --display_data=true \ --dataset.repo_id=/eval_groot-bimanual \ - --dataset.num_episodes=10 \ --dataset.single_task="Grab and handover the red cube to the other arm" \ --dataset.streaming_encoding=true \ --dataset.encoder_threads=2 \ - # --dataset.vcodec=auto \ + # --dataset.rgb_encoder.vcodec=auto \ --policy.path=/groot-bimanual \ # your trained model - --dataset.episode_time_s=30 \ - --dataset.reset_time_s=10 + --duration=600 ``` ## License diff --git a/docs/source/hardware_guide.mdx b/docs/source/hardware_guide.mdx index 0998344ec..5f236d3e8 100644 --- a/docs/source/hardware_guide.mdx +++ b/docs/source/hardware_guide.mdx @@ -96,3 +96,4 @@ Notes: - The leading `nvidia-smi` is a quick sanity check that CUDA is visible inside the container — useful to fail fast if the flavor or driver mismatched. - The default Job timeout is 30 minutes; pass `--timeout 4h` (or longer) for real training. - `--flavor` maps onto the table above: `t4-small`/`t4-medium` (T4, ACT only), `l4x1`/`l4x4` (L4 24 GB), `a10g-small/large/largex2/largex4` (A10G 24 GB scaled out), `a100-large` (A100). For the current full catalogue + pricing see [https://huggingface.co/docs/hub/jobs](https://huggingface.co/docs/hub/jobs). +- Prefer not to write the `hf jobs run` wrapper yourself? `lerobot-train` can submit the job for you: just add `--job.target=` to a normal training command and it handles dataset upload, log streaming, and the final model push. See the [imitation-learning training guide](./il_robots). diff --git a/docs/source/hil_data_collection.mdx b/docs/source/hil_data_collection.mdx index ba68959d1..c7df0631e 100644 --- a/docs/source/hil_data_collection.mdx +++ b/docs/source/hil_data_collection.mdx @@ -57,11 +57,11 @@ The `lerobot-rollout --strategy.type=dagger` mode requires **teleoperators with **Compatible teleoperators:** -- `openarm_mini` - OpenArm Mini +- `bi_openarm_mini` - Bimanual OpenArm Mini - `so_leader` - SO100 / SO101 leader arm > [!IMPORTANT] -> The provided commands default to `bi_openarm_follower` + `openarm_mini`. +> The provided commands default to `bi_openarm_follower` + `bi_openarm_mini`. > `so_follower` + `so_leader` configs are also registered and can be used via CLI flags. --- @@ -104,9 +104,9 @@ lerobot-rollout --strategy.type=dagger \ --robot.right_arm_config.port=can0 \ --robot.right_arm_config.side=right \ --robot.cameras='{left_wrist: {type: opencv, index_or_path: "/dev/video0", width: 1280, height: 720, fps: 30}, right_wrist: {type: opencv, index_or_path: "/dev/video4", width: 1280, height: 720, fps: 30}, base: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}}' \ - --teleop.type=openarm_mini \ - --teleop.port_left=/dev/ttyACM0 \ - --teleop.port_right=/dev/ttyACM1 \ + --teleop.type=bi_openarm_mini \ + --teleop.left_arm_config.port=/dev/ttyACM0 \ + --teleop.right_arm_config.port=/dev/ttyACM1 \ --policy.path=outputs/pretrain/checkpoints/last/pretrained_model \ --dataset.repo_id=your-username/rollout_hil_dataset \ --dataset.single_task="Fold the T-shirt properly" \ @@ -131,9 +131,9 @@ lerobot-rollout --strategy.type=dagger \ --robot.right_arm_config.port=can0 \ --robot.right_arm_config.side=right \ --robot.cameras='{left_wrist: {type: opencv, index_or_path: "/dev/video0", width: 1280, height: 720, fps: 30}, right_wrist: {type: opencv, index_or_path: "/dev/video4", width: 1280, height: 720, fps: 30}, base: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}}' \ - --teleop.type=openarm_mini \ - --teleop.port_left=/dev/ttyACM0 \ - --teleop.port_right=/dev/ttyACM1 \ + --teleop.type=bi_openarm_mini \ + --teleop.left_arm_config.port=/dev/ttyACM0 \ + --teleop.right_arm_config.port=/dev/ttyACM1 \ --policy.path=outputs/pretrain/checkpoints/last/pretrained_model \ --dataset.repo_id=your-username/rollout_hil_rtc_dataset \ --dataset.single_task="Fold the T-shirt properly" \ diff --git a/docs/source/hilserl.mdx b/docs/source/hilserl.mdx index 5b9439d51..09a370f3d 100644 --- a/docs/source/hilserl.mdx +++ b/docs/source/hilserl.mdx @@ -62,7 +62,7 @@ pip install -e ".[hilserl]" ### Understanding Configuration -The training process begins with proper configuration for the HILSerl environment. The main configuration class is `GymManipulatorConfig` in `lerobot/rl/gym_manipulator.py`, which contains nested `HILSerlRobotEnvConfig` and `DatasetConfig`. The configuration is organized into focused, nested sub-configs: +The training process begins with proper configuration for the HILSERl environment. The main configuration class is `GymManipulatorConfig` in `lerobot/rl/gym_manipulator.py`, which contains nested `HILSerlRobotEnvConfig` (defined in `lerobot/envs/configs.py`) and `DatasetConfig`. The configuration is organized into focused, nested sub-configs: ```python @@ -95,6 +95,7 @@ class HILSerlProcessorConfig: class ObservationConfig: add_joint_velocity_to_observation: bool = False # Add joint velocities to state add_current_to_observation: bool = False # Add motor currents to state + add_ee_pose_to_observation: bool = False # Add end-effector pose to state display_cameras: bool = False # Display camera feeds during execution class ImagePreprocessingConfig: @@ -326,14 +327,22 @@ lerobot-find-joint-limits \ Max joint positions [-20.0, -20.0, -20.0, -20.0, -20.0, -20.0] Min joint positions [50.0, 50.0, 50.0, 50.0, 50.0, 50.0] ``` -3. Use these values in the configuration of your teleoperation device (TeleoperatorConfig) under the `end_effector_bounds` field +3. Use these values in your environment configuration under `env.processor.inverse_kinematics.end_effector_bounds` (see `InverseKinematicsConfig` in `lerobot/envs/configs.py`) **Example Configuration** ```json -"end_effector_bounds": { - "max": [0.24, 0.20, 0.10], - "min": [0.16, -0.08, 0.03] +{ + "env": { + "processor": { + "inverse_kinematics": { + "end_effector_bounds": { + "max": [0.24, 0.2, 0.1], + "min": [0.16, -0.08, 0.03] + } + } + } + } } ``` @@ -404,30 +413,24 @@ We support using a gamepad or a keyboard or the leader arm of the robot. HIL-Serl learns actions in the end-effector space of the robot. Therefore, the teleoperation will control the end-effector's x,y,z displacements. -For that we need to define a version of the robot that takes actions in the end-effector space. Check the robot class `SO100FollowerEndEffector` and its configuration `SO100FollowerEndEffectorConfig` for the default parameters related to the end-effector space. +The end-effector transformation is applied by the processor pipeline (`InverseKinematicsRLStep`, `EEBoundsAndSafety`, `EEReferenceAndDelta`, `GripperVelocityToJoint`) configured under `env.processor.inverse_kinematics` (`InverseKinematicsConfig`) and `env.processor.gripper` / `env.processor.max_gripper_pos`. The defaults related to the end-effector space are: ```python -class SO100FollowerEndEffectorConfig(SO100FollowerConfig): - """Configuration for the SO100FollowerEndEffector robot.""" +class InverseKinematicsConfig: + """Configuration for inverse kinematics processing.""" - # Default bounds for the end-effector position (in meters) - end_effector_bounds: dict[str, list[float]] = field( # bounds for the end-effector in x,y,z direction - default_factory=lambda: { - "min": [-1.0, -1.0, -1.0], # min x, y, z - "max": [1.0, 1.0, 1.0], # max x, y, z - } - ) + urdf_path: str | None = None + target_frame_name: str | None = None + # bounds for the end-effector in x,y,z direction + end_effector_bounds: dict[str, list[float]] | None = None + # maximum step size for the end-effector in x,y,z direction + end_effector_step_sizes: dict[str, float] | None = None - max_gripper_pos: float = 50 # maximum gripper position that the gripper will be open at - - end_effector_step_sizes: dict[str, float] = field( # maximum step size for the end-effector in x,y,z direction - default_factory=lambda: { - "x": 0.02, - "y": 0.02, - "z": 0.02, - } - ) +class HILSerlProcessorConfig: + ... + # maximum gripper position that the gripper will be open at + max_gripper_pos: float | None = 100.0 ``` @@ -606,11 +609,11 @@ This guide explains how to train a reward classifier for human-in-the-loop reinf **Note**: Training a reward classifier is optional. You can start the first round of RL experiments by annotating the success manually with your gamepad or keyboard device. -The reward classifier implementation in `modeling_classifier.py` uses a pretrained vision model to process the images. It can output either a single value for binary rewards to predict success/fail cases or multiple values for multi-class settings. +The reward classifier implementation in `lerobot/rewards/classifier/modeling_classifier.py` uses a pretrained vision model to process the images. It can output either a single value for binary rewards to predict success/fail cases or multiple values for multi-class settings. **Collecting a Dataset for the reward classifier** -Before training, you need to collect a dataset with labeled examples. The `record_dataset` function in `gym_manipulator.py` enables the process of collecting a dataset of observations, actions, and rewards. +Before training, you need to collect a dataset with labeled examples. Setting `mode: "record"` in your config and running `gym_manipulator.py` enables the process of collecting a dataset of observations, actions, and rewards. To collect a dataset, you need to modify some parameters in the environment configuration based on HILSerlRobotEnvConfig. @@ -658,7 +661,7 @@ Example configuration section for data collection: }, "dataset": { "repo_id": "hf_username/dataset_name", - "dataset_root": "data/your_dataset", + "root": "data/your_dataset", "task": "reward_classifier_task", "num_episodes_to_record": 20, "replay_episode": null, @@ -671,7 +674,7 @@ Example configuration section for data collection: **Reward Classifier Configuration** -The reward classifier is configured using `configuration_classifier.py`. Here are the key parameters: +The reward classifier is configured using `lerobot/rewards/classifier/configuration_classifier.py`. Here are the key parameters: - **model_name**: Base model architecture (e.g., we mainly use `"helper2424/resnet10"`) - **model_type**: `"cnn"` or `"transformer"` @@ -689,7 +692,7 @@ Example configuration for training the [reward classifier](https://huggingface.c "repo_id": "hf_username/dataset_name", "root": null }, - "policy": { + "reward_model": { "type": "reward_classifier", "model_name": "helper2424/resnet10", "model_type": "cnn", @@ -699,7 +702,6 @@ Example configuration for training the [reward classifier](https://huggingface.c "dropout_rate": 0.1, "learning_rate": 1e-4, "device": "cuda", - "use_amp": true, "input_features": { "observation.images.front": { "type": "VISUAL", @@ -717,7 +719,7 @@ Example configuration for training the [reward classifier](https://huggingface.c "num_workers": 4, "steps": 5000, "log_freq": 10, - "eval_freq": 1000, + "env_eval_freq": 1000, "save_freq": 1000, "save_checkpoint": true, "seed": 2, @@ -818,13 +820,14 @@ The LeRobot system uses a distributed actor-learner architecture for training. T **Configuration Setup** -Create a training configuration file (example available [here](https://huggingface.co/datasets/lerobot/config_examples/resolve/main/rl/train_config.json)). The training config is based on the main `TrainRLServerPipelineConfig` class in `lerobot/configs/train.py`. +Create a training configuration file (example available [here](https://huggingface.co/datasets/lerobot/config_examples/resolve/main/rl/train_config.json)). The training config is based on the main `TrainRLServerPipelineConfig` class in `lerobot/rl/train_rl.py`. -1. Configure the policy settings (`type="sac"`, `device`, etc.) -2. Set `dataset` to your cropped dataset -3. Configure environment settings with crop parameters -4. Check the other parameters related to SAC in [configuration_sac.py](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/sac/configuration_sac.py#L79). -5. Verify that the `policy` config is correct with the right `input_features` and `output_features` for your task. +1. Configure the policy settings (`type="gaussian_actor"`, `device`, etc.) +2. Configure the algorithm settings under the top-level `algorithm` block (`type="sac"`, learning rates, discount, etc., defined in `lerobot/rl/algorithms/sac/configuration_sac.py`). +3. Set `dataset` to your cropped dataset +4. Configure environment settings with crop parameters +5. Check the other parameters related to the Gaussian Actor in [configuration_gaussian_actor.py](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/gaussian_actor/configuration_gaussian_actor.py#L79). +6. Verify that the `policy` config is correct with the right `input_features` and `output_features` for your task. **Starting the Learner** @@ -926,7 +929,7 @@ The ideal behaviour is that your intervention rate should drop gradually during Some configuration values have a disproportionate impact on training stability and speed: -- **`temperature_init`** (`policy.temperature_init`) – initial entropy temperature in SAC. Higher values encourage more exploration; lower values make the policy more deterministic early on. A good starting point is `1e-2`. We observed that setting it too high can make human interventions ineffective and slow down learning. +- **`temperature_init`** (`algorithm.temperature_init`) – initial entropy temperature in SAC. Higher values encourage more exploration; lower values make the policy more deterministic early on. A good starting point is `1e-2`. We observed that setting it too high can make human interventions ineffective and slow down learning. - **`policy_parameters_push_frequency`** (`policy.actor_learner_config.policy_parameters_push_frequency`) – interval in _seconds_ between two weight pushes from the learner to the actor. The default is `4 s`. Decrease to **1-2 s** to provide fresher weights (at the cost of more network traffic); increase only if your connection is slow, as this will reduce sample efficiency. - **`storage_device`** (`policy.storage_device`) – device on which the learner keeps the policy parameters. If you have spare GPU memory, set this to `"cuda"` (instead of the default `"cpu"`). Keeping the weights on-GPU removes CPU→GPU transfer overhead and can significantly increase the number of learner updates per second. diff --git a/docs/source/hope_jr.mdx b/docs/source/hope_jr.mdx index 8826d9758..c29a9f216 100644 --- a/docs/source/hope_jr.mdx +++ b/docs/source/hope_jr.mdx @@ -232,7 +232,7 @@ lerobot-record \ --dataset.private=true \ --dataset.streaming_encoding=true \ --dataset.encoder_threads=2 \ - # --dataset.vcodec=auto \ + # --dataset.rgb_encoder.vcodec=auto \ --display_data=true ``` @@ -278,6 +278,6 @@ lerobot-record \ --dataset.num_episodes=10 \ --dataset.streaming_encoding=true \ --dataset.encoder_threads=2 \ - # --dataset.vcodec=auto \ + # --dataset.rgb_encoder.vcodec=auto \ --policy.path=outputs/train/hopejr_hand/checkpoints/last/pretrained_model ``` diff --git a/docs/source/il_robots.mdx b/docs/source/il_robots.mdx index ff0a6229e..64a39e29c 100644 --- a/docs/source/il_robots.mdx +++ b/docs/source/il_robots.mdx @@ -68,13 +68,13 @@ from lerobot.teleoperators.so_leader import SO101Leader, SO101LeaderConfig from lerobot.robots.so_follower import SO101Follower, SO101FollowerConfig robot_config = SO101FollowerConfig( - port="/dev/tty.usbmodem58760431541", - id="my_red_robot_arm", + port="/dev/tty.usbmodem5AB90687491", + id="my_follower_arm", ) teleop_config = SO101LeaderConfig( - port="/dev/tty.usbmodem58760431551", - id="my_blue_leader_arm", + port="/dev/tty.usbmodem5AB90689011", + id="my_leader_arm", ) robot = SO101Follower(robot_config) @@ -108,13 +108,13 @@ With `rerun`, you can teleoperate again while simultaneously visualizing the cam ```bash lerobot-teleoperate \ - --robot.type=koch_follower \ - --robot.port=/dev/tty.usbmodem58760431541 \ - --robot.id=my_awesome_follower_arm \ - --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \ - --teleop.type=koch_leader \ - --teleop.port=/dev/tty.usbmodem58760431551 \ - --teleop.id=my_awesome_leader_arm \ + --robot.type=so101_follower \ + --robot.port=/dev/tty.usbmodem5AB90687491 \ + --robot.id=my_follower_arm \ + --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \ + --teleop.type=so101_leader \ + --teleop.port=/dev/tty.usbmodem5AB90689011 \ + --teleop.id=my_leader_arm \ --display_data=true ``` @@ -122,34 +122,48 @@ lerobot-teleoperate \ ```python +import time +from lerobot.teleoperators.so_leader import SO101Leader, SO101LeaderConfig +from lerobot.robots.so_follower import SO101Follower, SO101FollowerConfig from lerobot.cameras.opencv import OpenCVCameraConfig -from lerobot.teleoperators.koch_leader import KochLeader, KochLeaderConfig -from lerobot.robots.koch_follower import KochFollower, KochFollowerConfig +from lerobot.utils.visualization_utils import init_visualization, log_visualization_data, shutdown_visualization -camera_config = { - "front": OpenCVCameraConfig(index_or_path=0, width=1920, height=1080, fps=30) -} - -robot_config = KochFollowerConfig( - port="/dev/tty.usbmodem585A0076841", - id="my_red_robot_arm", - cameras=camera_config +robot_config = SO101FollowerConfig( + port="/dev/tty.usbmodem5AB90687491", + id="my_follower_arm", + cameras={ + "wrist": OpenCVCameraConfig(index_or_path=0, width=640, height=480, fps=30), + "top": OpenCVCameraConfig(index_or_path=1, width=640, height=480, fps=30) + } ) -teleop_config = KochLeaderConfig( - port="/dev/tty.usbmodem58760431551", - id="my_blue_leader_arm", +teleop_config = SO101LeaderConfig( + port="/dev/tty.usbmodem5AB90689011", + id="my_leader_arm", ) -robot = KochFollower(robot_config) -teleop_device = KochLeader(teleop_config) +init_visualization("rerun", session_name="teleoperation") # pass "foxglove" to stream to Foxglove instead + +robot = SO101Follower(robot_config) +teleop_device = SO101Leader(teleop_config) robot.connect() teleop_device.connect() +TARGET_HZ = 30 +TIME_PER_FRAME = 1.0 / TARGET_HZ + while True: + start_time = time.perf_counter() + observation = robot.get_observation() action = teleop_device.get_action() robot.send_action(action) + log_visualization_data("rerun", observation=observation, action=action) + + elapsed_time = time.perf_counter() - start_time + sleep_time = TIME_PER_FRAME - elapsed_time + if sleep_time > 0: + time.sleep(sleep_time) ``` @@ -193,7 +207,7 @@ lerobot-record \ --dataset.num_episodes=5 \ --dataset.single_task="Grab the black cube" \ --dataset.streaming_encoding=true \ - # --dataset.vcodec=auto \ + # --dataset.rgb_encoder.vcodec=auto \ --dataset.encoder_threads=2 ``` @@ -202,13 +216,14 @@ lerobot-record \ ```python from lerobot.cameras.opencv import OpenCVCameraConfig -from lerobot.datasets import LeRobotDataset +from lerobot.datasets.lerobot_dataset import LeRobotDataset from lerobot.utils.feature_utils import hw_to_dataset_features -from lerobot.robots.so_follower import SO100Follower, SO100FollowerConfig -from lerobot.teleoperators.so_leader import SO100Leader, SO100LeaderConfig +from lerobot.robots.so_follower import SO101Follower, SO101FollowerConfig +from lerobot.teleoperators.so_leader.config_so_leader import SO101LeaderConfig +from lerobot.teleoperators.so_leader.so_leader import SO101Leader from lerobot.common.control_utils import init_keyboard_listener from lerobot.utils.utils import log_say -from lerobot.utils.visualization_utils import init_rerun +from lerobot.utils.visualization_utils import init_visualization from lerobot.scripts.lerobot_record import record_loop from lerobot.processor import make_default_processors @@ -218,71 +233,56 @@ EPISODE_TIME_SEC = 60 RESET_TIME_SEC = 10 TASK_DESCRIPTION = "My task description" -# Create robot configuration -robot_config = SO100FollowerConfig( - id="my_awesome_follower_arm", - cameras={ - "front": OpenCVCameraConfig(index_or_path=0, width=640, height=480, fps=FPS) # Optional: fourcc="MJPG" for troubleshooting OpenCV async error. - }, - port="/dev/tty.usbmodem58760434471", -) - -teleop_config = SO100LeaderConfig( - id="my_awesome_leader_arm", - port="/dev/tty.usbmodem585A0077581", -) - -# Initialize the robot and teleoperator -robot = SO100Follower(robot_config) -teleop = SO100Leader(teleop_config) - -# Configure the dataset features -action_features = hw_to_dataset_features(robot.action_features, "action") -obs_features = hw_to_dataset_features(robot.observation_features, "observation") -dataset_features = {**action_features, **obs_features} - -# Create the dataset -dataset = LeRobotDataset.create( - repo_id="/", - fps=FPS, - features=dataset_features, - robot_type=robot.name, - use_videos=True, - image_writer_threads=4, -) - -# Initialize the keyboard listener and rerun visualization -_, events = init_keyboard_listener() -init_rerun(session_name="recording") - -# Connect the robot and teleoperator -robot.connect() -teleop.connect() - -# Create the required processors -teleop_action_processor, robot_action_processor, robot_observation_processor = make_default_processors() - -episode_idx = 0 -while episode_idx < NUM_EPISODES and not events["stop_recording"]: - log_say(f"Recording episode {episode_idx + 1} of {NUM_EPISODES}") - - record_loop( - robot=robot, - events=events, - fps=FPS, - teleop_action_processor=teleop_action_processor, - robot_action_processor=robot_action_processor, - robot_observation_processor=robot_observation_processor, - teleop=teleop, - dataset=dataset, - control_time_s=EPISODE_TIME_SEC, - single_task=TASK_DESCRIPTION, - display_data=True, +def main(): + # Create robot configuration + robot_config = SO101FollowerConfig( + port="/dev/tty.usbmodem5AB90687491", + id="my_follower_arm", + cameras={ + "wrist": OpenCVCameraConfig(index_or_path=0, width=640, height=480, fps=30), + "top": OpenCVCameraConfig(index_or_path=1, width=640, height=480, fps=30) + } ) - # Reset the environment if not stopping or re-recording - if not events["stop_recording"] and (episode_idx < NUM_EPISODES - 1 or events["rerecord_episode"]): - log_say("Reset the environment") + teleop_config = SO101LeaderConfig( + port="/dev/tty.usbmodem5AB90689011", + id="my_leader_arm", + ) + + # Initialize the robot and teleoperator + robot = SO101Follower(robot_config) + teleop = SO101Leader(teleop_config) + + # Configure the dataset features + action_features = hw_to_dataset_features(robot.action_features, "action") + obs_features = hw_to_dataset_features(robot.observation_features, "observation") + dataset_features = {**action_features, **obs_features} + + # Create the dataset + dataset = LeRobotDataset.create( + repo_id="/", + fps=FPS, + features=dataset_features, + robot_type=robot.name, + use_videos=True, + image_writer_threads=4, + ) + + # Initialize the keyboard listener and rerun visualization + _, events = init_keyboard_listener() + init_visualization("rerun", session_name="recording") + + # Connect the robot and teleoperator + robot.connect() + teleop.connect() + + # Create the required processors + teleop_action_processor, robot_action_processor, robot_observation_processor = make_default_processors() + + episode_idx = 0 + while episode_idx < NUM_EPISODES and not events["stop_recording"]: + log_say(f"Recording episode {episode_idx + 1} of {NUM_EPISODES}") + record_loop( robot=robot, events=events, @@ -291,26 +291,50 @@ while episode_idx < NUM_EPISODES and not events["stop_recording"]: robot_action_processor=robot_action_processor, robot_observation_processor=robot_observation_processor, teleop=teleop, - control_time_s=RESET_TIME_SEC, + dataset=dataset, + control_time_s=EPISODE_TIME_SEC, single_task=TASK_DESCRIPTION, display_data=True, ) - if events["rerecord_episode"]: - log_say("Re-recording episode") - events["rerecord_episode"] = False - events["exit_early"] = False - dataset.clear_episode_buffer() - continue + # Reset the environment if not stopping or re-recording + if not events["stop_recording"] and (episode_idx < NUM_EPISODES - 1 or events["rerecord_episode"]): + log_say("Reset the environment") + record_loop( + robot=robot, + events=events, + fps=FPS, + teleop_action_processor=teleop_action_processor, + robot_action_processor=robot_action_processor, + robot_observation_processor=robot_observation_processor, + teleop=teleop, + control_time_s=RESET_TIME_SEC, + single_task=TASK_DESCRIPTION, + display_data=True, + ) - dataset.save_episode() - episode_idx += 1 + if events["rerecord_episode"]: + log_say("Re-recording episode") + events["rerecord_episode"] = False + events["exit_early"] = False + dataset.clear_episode_buffer() + continue -# Clean up -log_say("Stop recording") -robot.disconnect() -teleop.disconnect() -dataset.push_to_hub() + dataset.save_episode() + episode_idx += 1 + + # finalize dataset + log_say("Finalizing dataset...") + dataset.finalize() + # Clean up + log_say("Stop recording") + robot.disconnect() + teleop.disconnect() + dataset.push_to_hub() + + +if __name__ == "__main__": + main() ``` @@ -348,7 +372,7 @@ The `record` function provides a suite of tools for capturing and managing data ##### 2. Checkpointing and Resuming - Checkpoints are automatically created during recording. -- If an issue occurs, you can resume by re-running the same command with `--resume=true`. When resuming a recording, `--dataset.num_episodes` must be set to the **number of additional episodes to be recorded**, and not to the targeted total number of episodes in the dataset ! +- If an issue occurs or you want to record additional episodes in the same dataset, you can resume by re-running the same command with `--resume=true`. When resuming a recording, `--dataset.num_episodes` must be set to the **number of additional episodes to be recorded**, and not to the targeted total number of episodes in the dataset! Make sure that you also set `--dataset.root="local_path"`, it's a local path to save the new part of the dataset and is required to resume. - To start recording from scratch, **manually delete** the dataset directory. ##### 3. Recording Parameters @@ -366,9 +390,17 @@ Set the flow of data recording using command-line arguments: Control the data recording flow using keyboard shortcuts: -- Press **Right Arrow (`→`)**: Early stop the current episode or reset time and move to the next. -- Press **Left Arrow (`←`)**: Cancel the current episode and re-record it. -- Press **Escape (`ESC`)**: Immediately stop the session, encode videos, and upload the dataset. +- Press **Right Arrow (`→`)** or **`n`**: Early stop the current episode or reset time and move to the next. +- Press **Left Arrow (`←`)** or **`r`**: Cancel the current episode and re-record it. +- Press **Escape (`ESC`)** or **`q`**: Immediately stop the session, encode videos, and upload the dataset. + + + +These control-flow shortcuts work on **X11, Wayland, and headless/SSH** sessions. When a global keyboard backend isn't available (Wayland, a headless machine, or macOS without Accessibility permission), `lerobot-record` automatically reads the same keys from the terminal — launch it from an interactive terminal and keep it focused. You can also use the letter equivalents **`n`** (next, same as `→`), **`r`** (re-record, same as `←`) and **`q`** (quit, same as `ESC`). No `$DISPLAY` setup is required. + +This applies to the recording control flow only. Keyboard **teleoperation** (driving the robot with the keyboard) still needs a global key backend, so it works only on an X11 session, a Windows desktop, or macOS with Accessibility/Input Monitoring granted — not on Wayland or headless sessions. + + #### Tips for gathering data @@ -382,7 +414,7 @@ If you want to dive deeper into this important topic, you can check out the [blo #### Troubleshooting: -- On Linux, if the left and right arrow keys and escape key don't have any effect during data recording, make sure you've set the `$DISPLAY` environment variable. See [pynput limitations](https://pynput.readthedocs.io/en/latest/limitations.html#linux). +- On Linux, the recording control-flow keys (arrow keys, Escape) work on X11, Wayland, and headless/SSH sessions as long as `lerobot-record` runs in an interactive terminal — no `$DISPLAY` setup is needed. If the keys have no effect, make sure you are in an interactive (TTY) terminal, not a piped/non-TTY session, and that it is focused; the letter equivalents `n` / `r` / `q` also work. Keyboard _teleoperation_ (as opposed to the recording control flow) still requires a global key backend — an X11 session, a Windows desktop, or macOS with Accessibility/Input Monitoring granted — and is unavailable on Wayland or headless machines. See [pynput limitations](https://pynput.readthedocs.io/en/latest/limitations.html#linux). ## Visualize a dataset @@ -422,7 +454,7 @@ from lerobot.utils.utils import log_say episode_idx = 0 -robot_config = SO100FollowerConfig(port="/dev/tty.usbmodem58760434471", id="my_awesome_follower_arm") +robot_config = SO100FollowerConfig(port="/dev/tty.usbmodem5AB90687491", id="my_follower_arm") robot = SO100Follower(robot_config) robot.connect() @@ -482,6 +514,12 @@ lerobot-train \ --resume=true ``` +`--config_path` also accepts a **Hub repo id**: if a run pushed its checkpoints to the Hub (with `--save_checkpoint_to_hub=true`), you can resume straight from the repo — its latest checkpoint is downloaded and training continues, restoring the optimizer, scheduler, step counter and data order: + +```bash +lerobot-train --config_path=${HF_USER}/my_policy --resume=true +``` + If you do not want to push your model to the hub after training use `--policy.push_to_hub=false`. Additionally you can provide extra `tags` or specify a `license` for your model or make the model repo `private` by adding this: `--policy.private=true --policy.tags=\[ppo,rl\] --policy.license=mit` @@ -490,6 +528,130 @@ Additionally you can provide extra `tags` or specify a `license` for your model If your local computer doesn't have a powerful GPU you could utilize Google Colab to train your model by following the [ACT training notebook](./notebooks#training-act). +#### Train using Hugging Face Jobs + +Hugging Face jobs let's you easily select hardware and run the training in the cloud. So if you don't have a powerful GPU or you need more VRAM or just want to train a model much faster use HF Jobs! It's pay as you go and you simply pay for each second of use, you can see the pricing and additional information [here](https://huggingface.co/docs/hub/jobs). + +> **Tip:** if you just want to launch a standard training run, you can skip building the command below and use the integrated **Train on HF Jobs via `--job.target`** flow described further down — `lerobot-train` then submits the job, uploads a local-only dataset for you, and streams the logs. + +To run the training manually use this command: + + + +```bash +hf jobs run \ + --flavor a10g-small \ + --timeout 4h \ + --secrets HF_TOKEN \ + huggingface/lerobot-gpu:latest \ + -- \ + python -m lerobot.scripts.lerobot_train \ + --dataset.repo_id=username/dataset \ + --policy.type=act \ + --steps=5000 \ + --batch_size=16 \ + --policy.device=cuda \ + --policy.repo_id=username/your_policy \ + --log_freq=100 +``` + + + + +```python +from huggingface_hub import run_job, get_token + +run_name = "act_so101_hf_jobs" +dataset_id = "username/dataset" +user_hub_id = "username" + +command_args = [ + "python", "-m", "lerobot.scripts.lerobot_train", + "--dataset.repo_id", dataset_id, + "--policy.type", "act", + "--steps", "5000", + "--batch_size", "16", + "--num_workers", "4", + "--policy.device", "cuda", + "--log_freq", "100", + "--save_freq", "1000", + "--save_checkpoint", "true", + "--wandb.enable", "false", + "--policy.repo_id", f"{user_hub_id}/{run_name}" +] + +print(f"Submitting job '{run_name}' to Hugging Face Infrastructure...") + +job_info = run_job( + image="huggingface/lerobot-gpu:latest", + command=command_args, + flavor="a10g-small", + timeout="4h", + secrets={"HF_TOKEN": get_token()} +) + +print("\n🚀 Job successfully launched!") +print(f"🔹 Job ID: {job_info.id}") +print(f"🔗 Live UI Dashboard & Logs: {job_info.url}") +``` + + + + + +You can modify the `--flavor` to use different hardware, for example: `t4-small`, `a100-large`, `h200`. Use `hf jobs hardware` to see the full list with pricing. +Depending on the model you want to train and the hardware you selected you can also modify the `--batch_size` and `--number_of_workers`. +For longer training sessions increase the timeout. + +Once the training is started you can go to [Jobs](https://huggingface.co/settings/jobs) and see if your jobs is running as well as all the outputs. Sometimes it takes a few minutes to schedule your job so be patient. + +After training the model will be pushed to hub and you can use it as any other model with LeRobot. + +#### Train on HF Jobs via `--job.target` (integrated CLI) + +`lerobot-train` runs locally by default. To run on a HuggingFace GPU without constructing the Docker command yourself, pass `--job.target` with a hardware flavor name: + +```bash +lerobot-train \ + --dataset.repo_id=${HF_USER}/so101_test \ + --policy.type=act \ + --policy.repo_id=${HF_USER}/my_policy \ + --job.target=a10g-small +``` + +List available flavors and pricing with `hf jobs hardware`. The run streams its logs to your terminal; press Ctrl-C to detach (the job keeps running in the cloud). Re-attach or cancel with: + +```bash +hf jobs logs +hf jobs cancel +``` + +If your dataset exists only locally (not yet on the Hub), it is automatically pushed to a **private** Hub repo so the job can download it by `repo_id` (nothing is made public). The trained model is pushed to the model repo at the end of the run. To also push every intermediate checkpoint to the Hub as it is saved (so you can monitor progress mid-run), add `--save_checkpoint_to_hub=true` — this requires a runtime image that includes this feature. + +Every job (and any dataset pushed by the run) is tagged `lerobot` so it's easy to find on the Hub. Add your own with `--job.tags '["my-tag"]'`. + +By default the job is capped at `2d` (48h) of wall-clock. Override it with an HF Jobs duration string, e.g. `--job.timeout=4h` to fail faster or `--job.timeout=7d` for a longer run. + +> **Note:** the model repo is created up front (it holds the staged training config the job runs from). If a run fails before the model is pushed, that repo is left on the Hub so you can inspect it — it is not deleted automatically, so repeated failures can leave empty repos behind. Remove one with `hf repo delete `. + +**Prerequisites:** run `hf auth login` before submitting. For Weights & Biases integration, run `wandb login` or set `WANDB_API_KEY` on your machine — the key is forwarded to the job automatically. + +**Resuming on a job.** Adding `--job.target` to a resume command runs the resume in the cloud — the same command works locally or remotely. The checkpoint repo is the source of truth, and new checkpoints continue the lineage in the same repo: + +```bash +# resume a Hub run on a job (its checkpoints are already on the Hub) +lerobot-train --config_path=${HF_USER}/my_policy --resume=true --job.target=a10g-small + +# resume a LOCAL run on a job — the checkpoint is uploaded to a private Hub repo first, +# then the job resumes from it (a local-only dataset is uploaded the same way) +lerobot-train \ + --config_path=outputs/train/act_so101_test/checkpoints/last/pretrained_model/train_config.json \ + --resume=true \ + --job.target=a10g-small +``` + +Job settings come from the current command, so override `--job.target`, `--job.timeout`, etc. as needed; for the resumed run to itself be resumable later, keep `--save_checkpoint_to_hub=true`. + #### Upload policy checkpoints Once training is done, upload the latest checkpoint with: @@ -511,6 +673,8 @@ hf upload ${HF_USER}/act_so101_test${CKPT} \ Use `lerobot-rollout` to deploy a trained policy on your robot. You can choose different strategies depending on your needs: +The examples below load the model from `--policy.path`. To pin a specific pushed version — useful once `--save_checkpoint_to_hub=true` has committed several checkpoints — add `--policy.pretrained_revision` with a commit hash, branch, or tag. Each pushed checkpoint is tagged with its step (e.g. `--policy.pretrained_revision=010000`), so you can recover a checkpoint by step without looking up its commit sha. + ```bash @@ -546,5 +710,6 @@ The `--strategy.type` flag selects the execution mode: - `sentry`: Continuous recording with auto-upload (useful for large-scale evaluation) - `highlight`: Ring buffer recording with keystroke save (useful for capturing interesting events) - `dagger`: Human-in-the-loop data collection (see [HIL Data Collection](./hil_data_collection)) +- `episodic`: Episode-oriented policy recording with reset phases between episodes All strategies support `--inference.type=rtc` for smooth execution with slow VLA models (Pi0, Pi0.5, SmolVLA). diff --git a/docs/source/inference.mdx b/docs/source/inference.mdx index b2874d823..31405b5de 100644 --- a/docs/source/inference.mdx +++ b/docs/source/inference.mdx @@ -117,7 +117,7 @@ lerobot-rollout \ --strategy.num_episodes=20 \ --policy.path=outputs/pretrain/checkpoints/last/pretrained_model \ --robot.type=bi_openarm_follower \ - --teleop.type=openarm_mini \ + --teleop.type=bi_openarm_mini \ --dataset.repo_id=${HF_USER}/rollout_hil_data \ --dataset.single_task="Fold the T-shirt" ``` @@ -157,6 +157,44 @@ Foot pedal input is also supported via `--strategy.input_device=pedal`. Configur | `--strategy.input_device` | Input device: `keyboard` or `pedal` (default: keyboard) | | `--teleop.type` | **Required.** Teleoperator type | +### Episodic (`--strategy.type=episodic`) + +Episode-oriented recording that mirrors the behavior of `lerobot-record`. The policy drives the robot for each episode; an optional teleoperator can drive the robot during the reset phase between episodes. + +```bash +lerobot-rollout \ + --strategy.type=episodic \ + --policy.path=${HF_USER}/my_policy \ + --robot.type=so100_follower \ + --robot.port=/dev/ttyACM0 \ + --teleop.type=so100_leader \ + --teleop.port=/dev/ttyACM1 \ + --dataset.repo_id=${HF_USER}/my_eval_data \ + --dataset.num_episodes=20 \ + --dataset.episode_time_s=30 \ + --dataset.reset_time_s=10 \ + --dataset.single_task="Pick up the red cube" +``` + +Teleop is optional — if omitted the robot holds its position during the reset phase. + +**Keyboard controls:** + +| Key | Action | +| ----------- | -------------------------------- | +| `→` (right) | End the current episode early | +| `←` (left) | Discard episode and re-record it | +| `ESC` | Stop the recording session | + +| Flag | Description | +| ----------------------------------------------- | -------------------------------------------------------------------------- | +| `--dataset.num_episodes` | Number of episodes to record | +| `--dataset.episode_time_s` | Duration of each recording episode in seconds | +| `--dataset.reset_time_s` | Duration of the reset phase between episodes in seconds | +| `--teleop.type` | Optional. Teleoperator to drive the robot during resets | +| `--strategy.reset_to_initial_position` | Whether to reset the robot to its initial position between episodes | +| `--strategy.smooth_leader_to_follower_handover` | Whether to turn on or off the leader -> follower smooth handover behavior. | + --- ## Inference Backends diff --git a/docs/source/language_and_recipes.mdx b/docs/source/language_and_recipes.mdx new file mode 100644 index 000000000..4181dbe34 --- /dev/null +++ b/docs/source/language_and_recipes.mdx @@ -0,0 +1,147 @@ +# Language columns and recipes + +Most LeRobot datasets ship with a single `task` string per episode — fine for +short, single-instruction skills, but not enough for the longer-horizon, +multi-modal robot policies the field is moving toward (high-level planning, +memory, interjections, VQA, tool use). To support those policies without +forking the dataset format, LeRobot extends `LeRobotDataset` with two optional +language columns and a small recipe layer that turns those rows into +chat-style training samples on the fly. + +The design splits cleanly into three layers: + +1. **Data in the dataset** — language annotations stored next to frames in + `data/chunk-*/file-*.parquet` as two optional columns (`language_persistent` + and `language_events`). Datasets without these columns keep their existing + behavior. +2. **Recipe** — a YAML file that declares which annotation rows to bind and + how to lay them out as chat turns (`role`, `content`, optional images, + optional tool calls). Recipes are pure config; no Python required to add a + new one. +3. **Training format** — at sample time, `RenderMessagesStep` resolves the + recipe against the per-frame annotations and emits HF-style `messages` plus + LeRobot-specific sidecars (`message_streams`, `target_message_indices`) + that policy processors consume. + +This page describes each layer in turn. + +## Layer 1 — language columns in the dataset + +The two optional columns live next to frame data in +`data/chunk-*/file-*.parquet`: + +- `language_persistent`: a list of rows broadcast across every frame in an episode for state that remains active, such as `subtask`, `plan`, and `memory`. +- `language_events`: a list of rows only on the exact frame where an event was emitted, such as `interjection`, `vqa`, and speech tool calls. + +Both columns share the same row shape (event rows omit `timestamp` because the +frame the row sits on already provides it): + +```text +role: string +content: string | null +style: string | null +timestamp: float32 # persistent rows only +camera: string | null # observation.images.* feature key, view-dependent rows only +tool_calls: list[Json] | null +``` + +The `camera` field tags rows whose `content` is grounded in a specific camera +view. Rows of view-dependent styles (`vqa` and `trace`) MUST set `camera` to +the matching `observation.images.*` feature key. Rows of every other style — +including `motion`, which describes robot-frame primitives in joint / Cartesian +terms — MUST leave `camera` as `null`. Pipeline writers and the validator +enforce this via `validate_camera_field(style, camera)`. + +`meta/tasks.parquet` remains the canonical source for the task. The special `${task}` recipe binding always reads that task string and does not depend on language annotations. + +### Architecture + +The language stack itself has three internal modules backing layer 1: + +1. `lerobot.datasets.language` defines the schema, style registry, and `column_for_style`. +2. `lerobot.datasets.language_render` resolves rows and renders messages. +3. `RenderMessagesStep` turns dataset samples into `messages`, `message_streams`, and `target_message_indices`. + +`LeRobotDataset` stays recipe-agnostic. It passes `language_persistent` and `language_events` through when present, and unannotated datasets keep their existing behavior. + +## Layer 2 — recipe anatomy + +Recipes are YAML files backed by `TrainingRecipe` and `MessageTurn`. They +declare which annotation rows to pull (via `bindings`) and how to compose them +into chat turns (`messages`). + +```yaml +messages: + - { role: user, content: "${task}", stream: high_level } + - { role: assistant, content: "${subtask}", stream: low_level, target: true } +``` + +A recipe can also branch into a weighted **blend** of sub-recipes. At sample +time, exactly one branch is selected deterministically from the sample index, +so different frames train different objectives (e.g. memory updates vs. +low-level execution vs. VQA) without any Python wiring. + +### Temporal semantics + +Persistent styles are active after emission until replaced: + +- `active_at(t, style=subtask)` +- `nth_prev(style=memory, offset=1)` +- `nth_next(style=subtask, offset=1)` + +Event styles only exist on their exact timestamp: + +- `emitted_at(t, style=interjection)` +- `emitted_at(t, style=vqa, role=user, camera=observation.images.top)` +- `emitted_at(t, role=assistant, tool_name=say)` + +Exact event matching has no tolerance window, so writers must stamp event rows with frame timestamps from the parquet data. + +### View-dependent resolution + +For view-dependent styles (`vqa` and `trace`), the resolver gains a +`camera=` filter parallel to `role=` and `tool_name=`. Datasets with multiple +cameras typically emit one (`vqa`, `user`) + (`vqa`, `assistant`) pair per +camera at the same timestamp; without `camera=`, those resolvers see two +matches and raise an ambiguity error. Recipes consume each camera through its +own binding plus a matching image block, e.g. + +```yaml +ask_vqa_top: + bindings: + vqa_query: "emitted_at(t, style=vqa, role=user, camera=observation.images.top)" + vqa: "emitted_at(t, style=vqa, role=assistant, camera=observation.images.top)" + messages: + - role: user + stream: high_level + if_present: vqa_query + content: + - { type: image, feature: observation.images.top } + - { type: text, text: "${vqa_query}" } + - { + role: assistant, + content: "${vqa}", + stream: high_level, + target: true, + if_present: vqa, + } +``` + +Add one such sub-recipe per camera the dataset records. + +## Layer 3 — training format + +Rendered samples use HF-style chat messages plus LeRobot sidecars: + +```python +sample["messages"] +sample["message_streams"] +sample["target_message_indices"] +``` + +The renderer does not apply a tokenizer chat template. Policy processors decide how to serialize the messages for their backbone, which keeps the same dataset usable across SmolVLA, Pi0.5, and any future VLM that expects OpenAI-style chat messages. + +## Graceful absence + +If both language columns are missing, `None`, or empty, `RenderMessagesStep` is a no-op. +If an event-scoped branch is selected on a frame without the required event row, rendering returns `None`, allowing a loader to retry another sample. diff --git a/docs/source/lekiwi.mdx b/docs/source/lekiwi.mdx index 7e7c1a680..739073b65 100644 --- a/docs/source/lekiwi.mdx +++ b/docs/source/lekiwi.mdx @@ -319,7 +319,7 @@ If you want to dive deeper into this important topic, you can check out the [blo #### Troubleshooting: -- On Linux, if the left and right arrow keys and escape key don't have any effect during data recording, make sure you've set the `$DISPLAY` environment variable. See [pynput limitations](https://pynput.readthedocs.io/en/latest/limitations.html#linux). +- On Linux, the recording control-flow keys (arrow keys, Escape) work on X11, Wayland, and headless/SSH sessions as long as you run the recording from an interactive terminal (keep it focused) — no `$DISPLAY` setup is needed; the letter equivalents `n` / `r` / `q` also work. Note that **keyboard teleoperation of the LeKiwi base** is different: it relies on a global key backend and therefore works only on an X11 session, a Windows desktop, or macOS with Accessibility/Input Monitoring granted — not on Wayland or headless machines. See [pynput limitations](https://pynput.readthedocs.io/en/latest/limitations.html#linux). ## Replay an episode diff --git a/docs/source/lelab.mdx b/docs/source/lelab.mdx new file mode 100644 index 000000000..a9f28e57b --- /dev/null +++ b/docs/source/lelab.mdx @@ -0,0 +1,29 @@ +# LeLab - LeRobot Guide + +LeLab is a graphical user interface built on top of the LeRobot library, designed to make robotics accessible without needing to memorize CLI commands. From a single app you can configure your robot, teleoperate it, collect datasets, train policies locally or on cloud GPUs via HF Jobs, and deploy trained models back onto your robot. It's the easiest way to go from an unboxed SO-101 to a working policy, and a great companion for anyone learning the LeRobot workflow. Source code and issues live on GitHub: [huggingface/leLab](https://github.com/huggingface/leLab). + +> [!TIP] +> For now LeLab is compatible only with SO-ARM101 + + + +### Installation + +Requires [`uv`](https://docs.astral.sh/uv/getting-started/installation/). Install and launch in one command: + +``` +uv tool install git+https://github.com/huggingface/leLab.git && lelab +``` + +After install, run `lelab` from your terminal anytime to start the app. + +### Features + +- **Add robots** — Select arm type (leader/follower), calibrate each joint from the middle position, and attach cameras. +- **Teleoperation** — Control the follower arm with the leader and see a live 3D visualization of the arms. +- **Dataset recording** — Define a task description, number of episodes, and episode/reset durations. Press spacebar to advance between episodes. 30+ episodes recommended. +- **Local training** — Train a policy directly on your own machine with a selected dataset, policy type, batch size, and step count. +- **Cloud training with HF Jobs** — Train on powerful GPUs via [HF Jobs](https://huggingface.co/docs/huggingface_hub/en/guides/jobs) with transparent pricing. Run `hf auth login` first. See the [Compute HW Guide](hardware_guide) for hardware/batch size tips. +- **Training visualization** — Watch progress live in the app, with checkpoints saved automatically. +- **Run trained policies** — Pick any model from your jobs list and run inference on your robot with one click. +- **Use community datasets** — Provide any Hugging Face dataset ID to train on datasets you didn't record yourself. diff --git a/docs/source/lerobot-dataset-v3.mdx b/docs/source/lerobot-dataset-v3.mdx index 8ab4a5d40..0647af0b0 100644 --- a/docs/source/lerobot-dataset-v3.mdx +++ b/docs/source/lerobot-dataset-v3.mdx @@ -10,6 +10,7 @@ This docs will guide you to: - Stream datasets without downloading using `StreamingLeRobotDataset` - Apply image transforms for data augmentation during training - Migrate existing `v2.1` datasets to `v3.0` +- Experiment with other `LeRobotDataset` formats and implementations like Lance ## What’s new in `v3` @@ -43,7 +44,7 @@ lerobot-record \ --dataset.num_episodes=5 \ --dataset.single_task="Grab the black cube" \ --dataset.streaming_encoding=true \ - # --dataset.vcodec=auto \ + # --dataset.rgb_encoder.vcodec=auto \ --dataset.encoder_threads=2 ``` @@ -274,7 +275,7 @@ A converter aggregates per‑episode files into larger shards and writes episode pip install "https://github.com/huggingface/lerobot/archive/33cad37054c2b594ceba57463e8f11ee374fa93c.zip" # Convert an existing v2.1 dataset hosted on the Hub: -python -m lerobot.datasets.v30.convert_dataset_v21_to_v30 --repo-id= +python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id= ``` **What it does** @@ -315,3 +316,39 @@ Dataset v3.0 uses incremental parquet writing with buffered metadata for efficie - Ensures the dataset is valid for loading Without calling `finalize()`, your parquet files will be incomplete and the dataset won't load properly. + +## Other formats and implementations + +### Lance + +Lance is a useful format for multimodal AI datasets, especially for large-scale training requiring high performance IO and random access. + +The `lerobot-lancedb` package implements `LeRobotLanceDataset` (for JPEG images) and `LeRobotLanceVideoDataset` (for mp4 videos). +Those two storage layouts both subclass LeRobotDataset and can provide data loading speed ups. + +`LeRobotLanceDataset` is a drop-in replacement for `LeRobotDataset`: + +```python +from lerobot.datasets import LeRobotDatasetMetadata +from lerobot.policies.diffusion.configuration_diffusion import DiffusionConfig +from lerobot_lancedb import LeRobotLanceDataset, LeRobotLanceVideoDataset + +cfg = DiffusionConfig(...) +meta = LeRobotDatasetMetadata(root=local_dataset_path) # or use repo_id=... to load metadata from the Hub +delta_timestamps = {...} + +# Use LeRobotLanceDataset for image datasets +dataset = LeRobotLanceDataset( + root=local_dataset_path, # or use repo_id=... to stream from the Hub + delta_timestamps=delta_timestamps, + return_uint8=True, +) +# Or use LeRobotLanceVideoDataset for video datasets: +dataset = LeRobotLanceVideoDataset( + root=local_dataset_path, # or use repo_id=... to stream from the Hub + delta_timestamps=delta_timestamps, + return_uint8=True, +) +``` + +Join the discussion on [Github](https://github.com/huggingface/lerobot/issues/3608) and explore the `lerobot-lancedb` documentation [here](https://lancedb.github.io/lerobot-lancedb/). diff --git a/docs/source/libero.mdx b/docs/source/libero.mdx index 043348690..b95af1d27 100644 --- a/docs/source/libero.mdx +++ b/docs/source/libero.mdx @@ -143,7 +143,7 @@ lerobot-train \ --batch_size=4 \ --eval.batch_size=1 \ --eval.n_episodes=1 \ - --eval_freq=1000 + --env_eval_freq=1000 ``` ## Reproducing published results diff --git a/docs/source/libero_plus.mdx b/docs/source/libero_plus.mdx index 4249bf49e..b065649fa 100644 --- a/docs/source/libero_plus.mdx +++ b/docs/source/libero_plus.mdx @@ -173,7 +173,7 @@ lerobot-train \ --batch_size=4 \ --eval.batch_size=1 \ --eval.n_episodes=1 \ - --eval_freq=1000 + --env_eval_freq=1000 ``` ## Relationship to LIBERO diff --git a/docs/source/metaworld.mdx b/docs/source/metaworld.mdx index 8e629dea9..b7accdfa2 100644 --- a/docs/source/metaworld.mdx +++ b/docs/source/metaworld.mdx @@ -120,11 +120,11 @@ lerobot-train \ --batch_size=4 \ --eval.batch_size=1 \ --eval.n_episodes=1 \ - --eval_freq=1000 + --env_eval_freq=1000 ``` ## Practical tips - Use the one-hot task conditioning for multi-task training (MT10/MT50 conventions) so policies have explicit task context. - Inspect the dataset task descriptions and the `info["is_success"]` keys when writing post-processing or logging so your success metrics line up with the benchmark. -- Adjust `batch_size`, `steps`, and `eval_freq` to match your compute budget. +- Adjust `batch_size`, `steps`, and `env_eval_freq` to match your compute budget. diff --git a/docs/source/molmoact2.mdx b/docs/source/molmoact2.mdx new file mode 100644 index 000000000..9eb449ca9 --- /dev/null +++ b/docs/source/molmoact2.mdx @@ -0,0 +1,495 @@ +# MolmoAct2 Policy + +MolmoAct2 is the LeRobot policy implementation of +[MolmoAct2](https://allenai.org/blog/molmoact2), ported into the LeRobot +training, evaluation, checkpointing, and dataset interfaces for easier use with +LeRobot datasets. + +This implementation currently supports training and evaluation for the regular +MolmoAct2 model. MolmoAct2-Think, which supports adaptive depth reasoning, is +not included in this LeRobot policy yet and is coming soon. + +For the original MolmoAct2 training code used for the experiments reported in +the paper, see [allenai/molmoact2](https://github.com/allenai/molmoact2). + +## Installation Requirements + +Install LeRobot with the MolmoAct2 optional dependencies: + +```bash +uv sync --locked --extra molmoact2 +``` + +To run the models in this repository, you need an NVIDIA GPU. The measurements +below were taken on a single NVIDIA H100 80GB with bf16 model loading, LIBERO with two RGB cameras. MolmoAct2 rows use `chunk_size=10`, action dim 7 +padded to `expected_max_action_dim=32`, and `num_flow_timesteps=8`. Training measurements use +`gradient_checkpointing=true` and include the forward pass, backward pass, +gradient clipping, optimizer step, and optimizer state allocation. Values are +peak GPU memory sampled with `nvidia-smi`. Leave a few GiB of headroom for +dataloader workers, CUDA context, and fragmentation. + +Multi-GPU training through `accelerate` increases throughput and global batch +size, but this LeRobot port does not currently expose the original MolmoAct2 +`fsdp_devices` model-parallel training path. The current training script has +not been tested for multi-node training. + +| Mode | Peak Memory, bs=8 | Peak Memory, bs=16 | Peak Memory, bs=32 | +| ------------------------------------------------ | ----------------: | -----------------: | -----------------: | +| Inference, continuous, CUDA graph enabled (bs=1) | 12.1 GiB | - | - | +| Fine-tuning, action expert only, continuous | 16.5 GiB | 18.3 GiB | 21.4 GiB | +| Fine-tuning, LoRA VLM, both action modes | 20.2 GiB | 26.8 GiB | 41.3 GiB | +| Fine-tuning, full model, both action modes | 48.3 GiB | 49.8 GiB | 60.1 GiB | + +The repo has been tested with Ubuntu 22.04. + +## Usage + +To use MolmoAct2 in a LeRobot training config, set: + +```bash +--policy.type=molmoact2 +``` + +## Training + +MolmoAct2 can be fine-tuned from either the released MolmoAct2 Hugging Face +checkpoint format or from a checkpoint already saved by LeRobot. Both routes use +the same LeRobot training loop, dataset transforms, checkpoint saving, and +logging. The difference is only how the initial policy weights and processor +state are loaded. + +### Training With Original MolmoAct2 Weight + +Use `policy.checkpoint_path` when starting from a released MolmoAct2 checkpoint, +for example `allenai/MolmoAct2` or `allenai/MolmoAct2-LIBERO`. LeRobot will load +the original HF model files, then build its own policy processor from the +dataset metadata and the policy options below. + +The command below shows full fine-tuning on the merged LIBERO dataset. It uses +bf16 model loading, 8 flow timesteps, LeRobot dataset statistics, image +augmentation, and LeRobot's checkpointing/logging path. + +```bash +accelerate launch \ + --num_processes=8 \ + --mixed_precision=bf16 \ + -m lerobot.scripts.lerobot_train \ + --dataset.repo_id=allenai/MolmoAct2-LIBERO-Dataset \ + --dataset.root=/path/to/lerobot/data/allenai/MolmoAct2-LIBERO-Dataset \ + --dataset.video_backend=pyav \ + --dataset.image_transforms.enable=true \ + --policy.type=molmoact2 \ + --policy.checkpoint_path=allenai/MolmoAct2-LIBERO \ + --policy.device=cuda \ + --policy.action_mode=both \ + --policy.chunk_size=10 \ + --policy.n_action_steps=10 \ + --policy.setup_type="single franka robotic arm in libero" \ + --policy.control_mode="delta end-effector pose" \ + --policy.image_keys='["observation.images.image","observation.images.wrist_image"]' \ + --policy.model_dtype=bfloat16 \ + --policy.num_flow_timesteps=8 \ + --policy.gradient_checkpointing=true \ + --policy.freeze_embedding=true \ + --policy.normalize_gripper=false \ + --policy.enable_knowledge_insulation=false \ + --policy.push_to_hub=false \ + --wandb.enable=true \ + --wandb.entity= \ + --wandb.project= \ + --job_name= \ + --output_dir=outputs/ \ + --steps=10000 \ + --batch_size=32 \ + --num_workers=4 \ + --log_freq=20 \ + --env_eval_freq=-1 \ + --save_checkpoint=true \ + --save_freq=2000 +``` + +### Training With LeRobot MolmoAct2 Weight + +Use `policy.path` when starting from a MolmoAct2 checkpoint that was saved by +LeRobot, either from a local `pretrained_model` directory or from the Hub. This +restores the saved LeRobot policy config, model weights, processor, and +normalization statistics. You can still override training-time options such as +`batch_size`, `steps`, LoRA flags, or `policy.action_mode`. + +```bash +accelerate launch \ + --num_processes=8 \ + --mixed_precision=bf16 \ + -m lerobot.scripts.lerobot_train \ + --dataset.repo_id=allenai/MolmoAct2-LIBERO-Dataset \ + --dataset.root=/path/to/lerobot/data/allenai/MolmoAct2-LIBERO-Dataset \ + --dataset.video_backend=pyav \ + --dataset.image_transforms.enable=true \ + --policy.path=/path/to/pretrained_model \ + --policy.device=cuda \ + --policy.action_mode=both \ + --policy.chunk_size=10 \ + --policy.n_action_steps=10 \ + --policy.model_dtype=bfloat16 \ + --policy.num_flow_timesteps=8 \ + --policy.gradient_checkpointing=true \ + --wandb.enable=true \ + --wandb.entity= \ + --wandb.project= \ + --job_name= \ + --output_dir=outputs/ \ + --steps=10000 \ + --batch_size=32 \ + --num_workers=4 \ + --log_freq=20 \ + --env_eval_freq=-1 \ + --save_checkpoint=true \ + --save_freq=2000 +``` + +### Common Practices + +For fine-tuning on a comparatively small dataset, such as a single LIBERO suite +or a real-world dataset with less than 200 demonstrations, a global batch size of +16 to 32 is a good starting point. In these settings, `policy.enable_lora_vlm=true` or `policy.train_action_expert_only=true` is also a practical choice. In both +cases, we intentionally keep the action expert fully trainable, which we found +to be crucial for model performance. For larger fine-tuning datasets, larger +global batch sizes and full fine-tuning are usually preferred. + +### Common Policy Options + +- `policy.checkpoint_path`: original MolmoAct2 HF checkpoint to initialize from. + Use this for released MolmoAct2 weights. +- `policy.path`: LeRobot checkpoint to initialize from. Use this for checkpoints + created by LeRobot training. +- `policy.action_mode`: training target, one of `continuous`, `discrete`, or + `both`. `both` trains the flow-matching action expert and the discrete + action-token loss. +- `policy.train_action_expert_only`: trains only parameters whose names contain + `action_expert`. It requires `policy.action_mode=continuous`. +- `policy.enable_lora_vlm`: enables LoRA on VLM linear layers. Use + `policy.enable_lora_action_expert=true` only if LoRA should also cover action + expert linear layers. When `policy.enable_lora_action_expert=false`, the + action expert base weights remain fully trainable while the VLM is trained + through LoRA adapters. When `policy.enable_lora_action_expert=true`, the + action expert is also adapter-tuned instead of fully fine-tuned. +- `policy.enable_knowledge_insulation`: when `true`, detaches action-expert + context K/V states before the action loss. The default is `false`. +- `policy.chunk_size`: action horizon used by the policy. For LIBERO we use + `10`. This LeRobot port overrides the loaded checkpoint's + `max_action_horizon` with this value. +- `policy.n_action_steps`: number of actions consumed from each predicted + chunk before querying the policy again. For LIBERO, set it to `chunk_size`. +- `policy.setup_type`: text inserted into the prompt to describe the robot and + scene, e.g. `single franka robotic arm in libero`. More examples are listed + in the `metadata_by_tag` entries of + [`norm_stats.json`](https://huggingface.co/allenai/MolmoAct2/blob/main/norm_stats.json). +- `policy.control_mode`: text inserted into the prompt to describe the action + space, e.g. `delta end-effector pose` or `absolute joint pose`. +- `policy.image_keys`: ordered LeRobot image observation keys passed to the + processor. +- `policy.model_dtype`: checkpoint/forward dtype, one of `float32`, + `bfloat16`, or `float16`. Use `bfloat16` for normal training. +- `policy.num_flow_timesteps`: number of flow-matching timesteps sampled per + example during training. We use `8` for fine-tuning. +- `policy.num_inference_steps`: optional override for continuous action + generation steps at inference time. +- `policy.gradient_checkpointing`: enables checkpointing in the VLM/action path + to reduce activation memory. +- `policy.freeze_embedding`: freezes input embeddings. The default is `true`. +- `policy.normalize_gripper`: controls whether gripper dimensions are included + in state/action quantile normalization. The default is `false`. +- `policy.normalize_language`: normalizes task strings before prompt + construction. The default is `true`. +- `policy.mask_action_dim_padding`: masks padded dimensions in the flow loss. + Released checkpoints use `policy.expected_max_action_dim=32`. +- `policy.max_sequence_length`: optional manual sequence cap. Leave unset to + infer it from images, state dimension, action dimension, action horizon, and + discrete-action mode. + +### Learning Rates + +MolmoAct2 uses parameter-group learning rates to match the original MolmoAct2 +fine-tuning experiments. + +- Full fine-tuning uses `policy.optimizer_lr=1e-5` for the VLM, + `policy.optimizer_vit_lr=5e-6` for the vision tower, + `policy.optimizer_connector_lr=5e-6` for image connector layers, and + `policy.optimizer_action_expert_lr=5e-5` for the action expert. +- LoRA VLM fine-tuning sets the VLM, vision, and connector LoRA parameter + groups to `5e-5` when `policy.enable_lora_vlm=true`. By default, + `policy.enable_lora_action_expert=false`, so the action expert is still fully + fine-tuned with `policy.optimizer_action_expert_lr`. If + `policy.enable_lora_action_expert=true`, the action expert is trained through + LoRA adapters instead. +- Action-expert-only fine-tuning trains only the action expert and uses + `policy.optimizer_action_expert_lr=5e-5`. + +You can override the full fine-tuning and action-expert learning rates with +`policy.optimizer_lr`, `policy.optimizer_vit_lr`, +`policy.optimizer_connector_lr`, and `policy.optimizer_action_expert_lr`. +Scheduler settings can be changed with `policy.scheduler_warmup_steps`, +`policy.scheduler_decay_steps`, and `policy.scheduler_decay_lr`. + +### Dataset Quantile Statistics + +MolmoAct2 defaults to quantile normalization for state and action features. If +your dataset has not been converted with quantile statistics, you can add them +with: + +```bash +python src/lerobot/scripts/augment_dataset_quantile_stats.py \ + --repo-id=your_dataset +``` + +Alternatively, train MolmoAct2 with mean/std normalization: + +```bash +--policy.normalization_mapping='{"ACTION": "MEAN_STD", "STATE": "MEAN_STD", "VISUAL": "IDENTITY"}' +``` + +## Evaluation + +Evaluation also supports both LeRobot-saved checkpoints and original MolmoAct2 +HF checkpoints. For LIBERO replication, keep the EGL rendering environment +fixed and use `policy.per_episode_seed=true`. + +**Important:** We found that `num_steps_wait=10` does not reliably let the +LIBERO scene stabilize and can degrade measured success. All LIBERO evaluation +results reported here use `num_steps_wait=50`. + +### Evaluation With LeRobot MolmoAct2 Weight + +Use `policy.path` for a checkpoint saved by LeRobot. The saved processor and +normalization statistics are restored together with the model. + +```bash +export MUJOCO_GL=egl +export PYOPENGL_PLATFORM=egl +export OMP_NUM_THREADS=1 +export MKL_NUM_THREADS=1 + +lerobot-eval \ + --policy.path=allenai/MolmoAct2-LIBERO-LeRobot \ + --policy.inference_action_mode=continuous \ + --policy.model_dtype=bfloat16 \ + --policy.use_amp=true \ + --policy.enable_inference_cuda_graph=true \ + --policy.device=cuda \ + --policy.per_episode_seed=true \ + --policy.eval_seed=1000 \ + --env.type=libero \ + --env.task=libero_10,libero_goal,libero_object,libero_spatial \ + --env.camera_name_mapping='{"agentview_image":"image","robot0_eye_in_hand_image":"wrist_image"}' \ + --eval.batch_size=1 \ + --eval.n_episodes=50 \ + --seed=1000 +``` + +### Evaluation With Original MolmoAct2 Weight + +You can evaluate a released Hugging Face checkpoint directly without first +converting it to a LeRobot checkpoint. In this case, set +`policy.checkpoint_path` to the HF model repo and provide `policy.norm_tag`. +For LIBERO, `policy.norm_tag=libero` loads the LIBERO action/state +normalization statistics, action horizon, prompt metadata, and image-key order +from the checkpoint's `norm_stats.json`. + +To fully replicate the MolmoAct2 paper results with released Hugging Face +checkpoints, we recommend using the v0.5.1-pinned +[`allenai/lerobot` `molmoact2-hf-inference`](https://github.com/allenai/lerobot/tree/molmoact2-hf-inference) +branch. That branch matches the original evaluation settings used for the +reported numbers. + +```bash +export MUJOCO_GL=egl +export PYOPENGL_PLATFORM=egl +export OMP_NUM_THREADS=1 +export MKL_NUM_THREADS=1 + +lerobot-eval \ + --policy.type=molmoact2 \ + --policy.checkpoint_path=allenai/MolmoAct2-LIBERO \ + --policy.norm_tag=libero \ + --policy.inference_action_mode=continuous \ + --policy.model_dtype=float32 \ + --policy.use_amp=false \ + --policy.enable_inference_cuda_graph=true \ + --policy.device=cuda \ + --policy.per_episode_seed=true \ + --policy.eval_seed=1000 \ + --env.type=libero \ + --env.task=libero_goal \ + --env.camera_name_mapping='{"agentview_image":"image","robot0_eye_in_hand_image":"wrist_image"}' \ + --eval.batch_size=1 \ + --eval.n_episodes=50 \ + --seed=1000 +``` + +Use `--env.task=libero_10,libero_goal,libero_object,libero_spatial` to run the +full LIBERO suite. The same command works for other released MolmoAct2 +checkpoints as long as the requested `policy.norm_tag` exists in that +checkpoint's `norm_stats.json`. + +### Common Evaluation Options + +- `policy.inference_action_mode`: required for rollout. Use `continuous` for + flow-matching inference or `discrete` for action-token inference. It must be + compatible with the training-time `policy.action_mode` saved in the + checkpoint. +- `policy.path`: LeRobot checkpoint path or Hub repo. Use this for checkpoints + saved by LeRobot. +- `policy.checkpoint_path`: original MolmoAct2 HF checkpoint path or Hub repo. + Use this with `policy.type=molmoact2` and `policy.norm_tag`. +- `policy.norm_tag`: selects normalization statistics, prompt metadata, + image-key order, and action horizon from the original checkpoint's + `norm_stats.json`. It is required for direct original-HF checkpoint + evaluation. +- `policy.model_dtype`: model load/forward dtype. Use `bfloat16` for normal + GPU evaluation. Use `float32` only when you explicitly want fp32 inference. +- `policy.use_amp`: runs the policy forward under autocast during eval. For + `model_dtype=bfloat16`, keep this enabled. +- `policy.enable_inference_cuda_graph`: enables the MolmoAct2 inference CUDA + graph path for faster repeated continuous-action rollout. +- `policy.per_episode_seed` and `policy.eval_seed`: make stochastic continuous + action generation deterministic per episode for replication. +- `env.task`: comma-separated LIBERO suites or a single suite. Use + `libero_10,libero_goal,libero_object,libero_spatial` for the full benchmark. +- `env.camera_name_mapping`: maps LIBERO camera names to the image keys expected + by the policy processor. + +## Performance Results + +### LIBERO Benchmark Results + +MolmoAct2 has demonstrated strong performance on the LIBERO benchmark suite. To +compare and test its LeRobot implementation, we fine-tuned +[`allenai/MolmoAct2-LIBERO`](https://huggingface.co/allenai/MolmoAct2-LIBERO) +for an additional 10k steps on the LIBERO dataset with per-GPU batch size 32 on +8 H100 GPUs, then compared the results to the original MolmoAct2 reference +results. + +The LeRobot fine-tuned checkpoint reported here is available at +[`allenai/MolmoAct2-LIBERO-LeRobot`](https://huggingface.co/allenai/MolmoAct2-LIBERO-LeRobot) +and was trained on +[`allenai/MolmoAct2-LIBERO-Dataset`](https://huggingface.co/datasets/allenai/MolmoAct2-LIBERO-Dataset). + +| Benchmark | LeRobot Implementation | MolmoAct2 Original | +| -------------- | ---------------------: | -----------------: | +| LIBERO Spatial | 98.4% | 97.8% | +| LIBERO Object | 100.0% | 100.0% | +| LIBERO Goal | 98.0% | 97.8% | +| LIBERO 10 | 96.6% | 93.2% | +| Average | 98.25% | 97.20% | + +These results demonstrate MolmoAct2's strong performance across diverse robotic +manipulation tasks. To reproduce them, follow the instructions in the LIBERO +evaluation section. + +## Hardware Deployment (lerobot-rollout) + +LeRobot-format checkpoints are available on the Hub for direct use with +`lerobot-rollout`. Each checkpoint uses specific camera names that must +match your robot's camera configuration. + +### Camera naming convention + +Each checkpoint expects specific `observation.images.*` keys. +If your robot cameras have different names, use `--rename_map` to map them: + +| Checkpoint | Camera keys | Description | +| ----------------------------- | ---------------------- | ------------------------ | +| MolmoAct2-LIBERO-LeRobot | `image`, `wrist_image` | LIBERO sim cameras | +| MolmoAct2-BimanualYAM-LeRobot | `top`, `left`, `right` | YAM 3-camera setup | +| MolmoAct2-DROID-LeRobot | `cam0`, `cam1` | External + wrist | +| MolmoAct2-SO100_101-LeRobot | `cam0`, `cam1` | Primary + secondary view | + +Example with an SO-100 robot using top and side cameras: + +```bash +lerobot-rollout \ + --policy.path=lerobot/MolmoAct2-SO100_101-LeRobot \ + --rename_map='{"observation.images.top": "observation.images.cam0", "observation.images.side": "observation.images.cam1"}' \ + --robot.type=so100_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.cameras='{ + top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, + side: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30} + }' \ + --task="pick up the red cube" --duration=30 +``` + +To use a wrist camera instead, just change the rename mapping: + +```bash +--rename_map='{"observation.images.top": "observation.images.cam0", "observation.images.wrist": "observation.images.cam1"}' +``` + +### Joint frame transform (SO-100/101 zero-shot) + + +The MolmoAct2-SO100_101 checkpoint was trained on data that uses a different +joint calibration convention than LeRobot >= 0.5.0. Without a frame +correction, the arm may move in the wrong direction. + +This affects both **zero-shot deployment** and **fine-tuning** from the +original checkpoint. The pretrained weights expect the old convention, so +all joint data (observations and actions) must be transformed to match. + +The converted LeRobot checkpoint (`lerobot/MolmoAct2-SO100_101-LeRobot`) +already includes this correction in its processor pipeline. If you convert +or fine-tune the checkpoint yourself, set the following in the policy config (`configuration_molmoact2.py`): + +- `joint_signs`: `[1, -1, 1, 1, 1, 1]` (flips shoulder_lift direction) +- `joint_offsets`: `[0, 90, 90, 0, 0, 0]` (shifts shoulder_lift and elbow_flex by 90°) + +See the [backward compatibility guide](./backwardcomp) for details on the +calibration change. + + + +## Differences From the Original Implementation + +This LeRobot port is intended to match MolmoAct2 behavior while using LeRobot's +dataset, training, evaluation, checkpoint, and logging infrastructure. The main +differences from the original training repository are: + +- The original paper training stack loads the model in fp32 and trains under + mixed precision. This LeRobot port usually loads the checkpoint directly in + `policy.model_dtype=bfloat16` for lower memory use. +- The original repository uses its own FSDP/model-parallel training path. The + LeRobot port uses the standard LeRobot/Accelerate training path and has not + been tested for multi-node training. +- The original repository supports sequence packing. The LeRobot port trains on + one LeRobot sample per item and pads to an inferred fixed sequence budget. +- The LeRobot port follows LeRobot's optimizer, scheduler, checkpoint saving, + dataset transforms, image augmentation, and Weights & Biases logging + conventions. +- The original training path supports mixed action horizons by padding to + `max_action_horizon` and masking padded horizon slots in the action expert + self-attention. This is useful when training across datasets with different + control frequencies. The LeRobot port currently targets single-dataset + fine-tuning, so `policy.chunk_size` overrides the checkpoint + `max_action_horizon` and horizon masking is not implemented yet. Support for + this mixed-horizon path is planned. + +## Citation + +```bibtex +@misc{fang2026molmoact2actionreasoningmodels, + title={MolmoAct2: Action Reasoning Models for Real-world Deployment}, + author={Haoquan Fang and Jiafei Duan and Donovan Clay and Sam Wang and Shuo Liu and Weikai Huang and Xiang Fan and Wei-Chuan Tsai and Shirui Chen and Yi Ru Wang and Shanli Xing and Jaemin Cho and Jae Sung Park and Ainaz Eftekhar and Peter Sushko and Karen Farley and Angad Wadhwa and Cole Harrison and Winson Han and Ying-Chun Lee and Eli VanderBilt and Rose Hendrix and Suveen Ellawela and Lucas Ngoo and Joyce Chai and Zhongzheng Ren and Ali Farhadi and Dieter Fox and Ranjay Krishna}, + year={2026}, + eprint={2605.02881}, + archivePrefix={arXiv}, + primaryClass={cs.RO}, + url={https://arxiv.org/abs/2605.02881}, +} +``` + +## License + +This model is licensed under Apache 2.0. It is intended for research and +educational use in accordance with +[Ai2's Responsible Use Guidelines](https://allenai.org/responsible-use), +consistent with [allenai/molmoact2](https://github.com/allenai/molmoact2). diff --git a/docs/source/multi_gpu_training.mdx b/docs/source/multi_gpu_training.mdx index d7369e8f8..7c212364e 100644 --- a/docs/source/multi_gpu_training.mdx +++ b/docs/source/multi_gpu_training.mdx @@ -95,7 +95,7 @@ If you want to scale your hyperparameters when using multiple GPUs, you should d accelerate launch --num_processes=2 $(which lerobot-train) \ --optimizer.lr=2e-4 \ --dataset.repo_id=lerobot/pusht \ - --policy=act + --policy.type=act ``` **Training Steps Scaling:** @@ -110,9 +110,64 @@ accelerate launch --num_processes=2 $(which lerobot-train) \ --batch_size=8 \ --steps=50000 \ --dataset.repo_id=lerobot/pusht \ - --policy=act + --policy.type=act ``` +## Training Large Models with FSDP + +DDP replicates the full model on every GPU, so a model that doesn't fit on one GPU won't fit under +DDP either. For large models, use **FSDP** (Fully Sharded Data Parallel), which shards parameters, +gradients, and optimizer state across GPUs. See the [accelerate FSDP guide](https://huggingface.co/docs/accelerate/usage_guides/fsdp) for background. + +An example on how to launch LeRobot training with FSDP across 4 GPUs (1 machine): + +```bash +accelerate launch --config_file fsdp.yaml --num_processes=4 $(which lerobot-train) \ + --dataset.repo_id=${HF_USER}/my_dataset \ + --policy.type= \ + --output_dir=outputs/train/my_policy_fsdp +``` + +A minimal `fsdp.yaml` (FSDP1; shards params/grads/optimizer — ZeRO-3-equivalent): + +```yaml +compute_environment: LOCAL_MACHINE +distributed_type: FSDP +mixed_precision: bf16 +num_machines: 1 +num_processes: 4 +fsdp_config: + fsdp_version: 1 + fsdp_sharding_strategy: FULL_SHARD # params + grads + optimizer (ZeRO-3) + fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP + fsdp_transformer_layer_cls_to_wrap: # repeated block class to shard + fsdp_use_orig_params: true # required: optimizer is built pre-prepare + fsdp_state_dict_type: FULL_STATE_DICT +``` + +Set `fsdp_transformer_layer_cls_to_wrap` to your model's repeated transformer-block class so each +block is sharded as its own unit. `fsdp_use_orig_params: true` is required because LeRobot builds the +optimizer before `accelerator.prepare()`. + +### FSDP checkpoints + +LeRobot gathers the full state dict across all ranks and the main process writes it as a single +`model.safetensors`, loadable as usual with `Policy.from_pretrained(...)`. Two things to look out for: + +- **Checkpoints store fp32 weights.** Under mixed precision (`bf16`/`fp16`) FSDP keeps an fp32 master + copy, and the checkpoint saves it (~2× the bf16 size on disk) so training can resume consistently + with the fp32 optimizer state; `from_pretrained` casts back to the policy dtype on load. FSDP-specific + caveat: an fp32 checkpoint is materialized in full precision on the target device _before_ casting, + so loading it for inference on a tight GPU can OOM even when the bf16 model would fit — load on CPU + first, or cast `model.safetensors` to the deployment dtype offline. +- The sharded optimizer state is gathered into a full (world-size-independent) state dict and saved + alongside the model in the same `optimizer_state.safetensors` / `optimizer_param_groups.json` + format as single-GPU training, so **resume-from-checkpoint is supported** with `--resume=true`. + Resume reshards both the model and the optimizer state to the _current_ FSDP topology, so you can + resume an FSDP checkpoint on a different number of GPUs. Note that the data sampler is only + sample-exact when the world size and batch size match the original run (a warning is logged + otherwise); the optimizer/model state itself is unaffected. + ## Notes - The `--policy.use_amp` flag in `lerobot-train` is only used when **not** running with accelerate. When using accelerate, mixed precision is controlled by accelerate's configuration. diff --git a/docs/source/multi_task_dit.mdx b/docs/source/multi_task_dit.mdx index 450d8a9f2..ebe46489a 100644 --- a/docs/source/multi_task_dit.mdx +++ b/docs/source/multi_task_dit.mdx @@ -314,7 +314,7 @@ lerobot-train \ --steps=30000 \ --save_freq=1000 \ --log_freq=100 \ - --eval_freq=1000 \ + --env_eval_freq=1000 \ --policy.type=multi_task_dit \ --policy.device=cuda \ --policy.horizon=32 \ diff --git a/docs/source/peft_training.mdx b/docs/source/peft_training.mdx index dd0b10075..d44a3f081 100644 --- a/docs/source/peft_training.mdx +++ b/docs/source/peft_training.mdx @@ -28,13 +28,15 @@ lerobot-train \ --steps=100000 \ --batch_size=32 \ --peft.method_type=LORA \ - --peft.r=64 + --peft.r=64 \ + --peft.lora_alpha=64 ``` Note the `--peft.method_type` parameter that let's you select which PEFT method to use. Here we use [LoRA](https://huggingface.co/docs/peft/main/en/package_reference/lora) (Low-Rank Adapter) which is probably the most popular fine-tuning method to date. Low-rank adaption means that we only fine-tune a matrix with comparably low rank -instead of the full weight matrix. This rank can be specified using the `--peft.r` parameter. The higher the rank +instead of the full weight matrix. This rank can be specified using the `--peft.r` parameter, and the LoRA scaling factor with +`--peft.lora_alpha` (where `scaling = lora_alpha / r`). The higher the rank the closer you get to full fine-tuning There are more complex methods that have more parameters. These are not yet supported, feel free to raise an issue diff --git a/docs/source/pi05.mdx b/docs/source/pi05.mdx index f99ad3286..127a2adc7 100644 --- a/docs/source/pi05.mdx +++ b/docs/source/pi05.mdx @@ -91,7 +91,7 @@ lerobot-train \ If your dataset is not converted with `quantiles`, you can convert it with the following command: ```bash -python src/lerobot/datasets/v30/augment_dataset_quantile_stats.py \ +python src/lerobot/scripts/augment_dataset_quantile_stats.py \ --repo-id=your_dataset \ ``` diff --git a/docs/source/pi0fast.mdx b/docs/source/pi0fast.mdx index f7272acc5..15dff8071 100644 --- a/docs/source/pi0fast.mdx +++ b/docs/source/pi0fast.mdx @@ -96,7 +96,7 @@ lerobot-train \ --policy.type=pi0_fast \ --output_dir=./outputs/pi0fast_training \ --job_name=pi0fast_training \ - --policy.pretrained_path=lerobot/pi0_fast_base \ + --policy.pretrained_path=lerobot/pi0fast-base \ --policy.dtype=bfloat16 \ --policy.gradient_checkpointing=true \ --policy.chunk_size=10 \ @@ -187,7 +187,7 @@ lerobot-train \ --dataset.repo_id=lerobot/libero \ --output_dir=outputs/libero_pi0fast \ --job_name=libero_pi0fast \ - --policy.path=lerobot/pi0fast_base \ + --policy.path=lerobot/pi0fast-base \ --policy.dtype=bfloat16 \ --steps=100000 \ --save_freq=20000 \ diff --git a/docs/source/policy_fastwam_README.md b/docs/source/policy_fastwam_README.md new file mode 100644 index 000000000..6af0eaa79 --- /dev/null +++ b/docs/source/policy_fastwam_README.md @@ -0,0 +1,56 @@ +## Research Paper + +Paper: https://arxiv.org/abs/2603.16666 + +## Repository + +Code: https://github.com/yuantianyuan01/FastWAM + +Project page: https://yuantianyuan01.github.io/FastWAM/ + +## Citation + +```bibtex +@article{yuan2026fastwam, + title = {Fast-WAM: Do World Action Models Need Test-time Future Imagination?}, + author = {Tianyuan Yuan and Zibin Dong and Yicheng Liu and Hang Zhao}, + journal = {arXiv preprint arXiv:2603.16666}, + year = {2026}, + url = {https://arxiv.org/abs/2603.16666} +} +``` + +## Additional Resources + +Base video model: https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B + +Released upstream checkpoints: https://huggingface.co/yuanty/fastwam + +## Results + +Evaluated on LIBERO with [`ZibinDong/fastwam_libero_uncond_2cam224`](https://huggingface.co/ZibinDong/fastwam_libero_uncond_2cam224): + +| Suite | Success rate | n_episodes | +| -------------- | -----------: | ---------: | +| libero_spatial | 97.6% | 500 | +| libero_object | 99.0% | 500 | +| libero_goal | 95.0% | 500 | +| libero_10 | 94.0% | 500 | +| **average** | **96.4%** | 2000 | + +Reproduce: `lerobot-eval --policy.path=ZibinDong/fastwam_libero_uncond_2cam224 --policy.device=cuda --policy.torch_dtype=float32 --policy.n_action_steps=10 --env.type=libero --env.task=libero_spatial --env.observation_height=256 --env.observation_width=256 --eval.batch_size=1 --eval.n_episodes=50 --seed=0 --env.episode_length=300`. + +For LIBERO-10, use `--env.task=libero_10 --env.episode_length=600`: + +```bash +lerobot-eval \ + --policy.path=ZibinDong/fastwam_libero_uncond_2cam224 \ + --policy.device=cuda \ + --policy.torch_dtype=float32 \ + --policy.n_action_steps=10 \ + --env.type=libero \ + --env.task=libero_10 --env.observation_height=256 --env.observation_width=256 \ + --eval.batch_size=1 \ + --eval.n_episodes=50 \ + --seed=0 --env.episode_length=600 +``` diff --git a/docs/source/policy_molmoact2_README.md b/docs/source/policy_molmoact2_README.md new file mode 100644 index 000000000..df3a6341e --- /dev/null +++ b/docs/source/policy_molmoact2_README.md @@ -0,0 +1,39 @@ +# MolmoAct2 + +This repository contains the LeRobot policy implementation of +[MolmoAct2](https://allenai.org/blog/molmoact2), ported into LeRobot for +training, evaluation, checkpointing, and dataset compatibility. + +This implementation currently supports training and evaluation for the regular +MolmoAct2 model. MolmoAct2-Think, which supports adaptive depth reasoning, is +not included in this LeRobot policy yet and is coming soon. + +For the original MolmoAct2 training code used for the experiments reported in +the paper, see [allenai/molmoact2](https://github.com/allenai/molmoact2). + +## LIBERO Evaluation + +Important: we found that `num_steps_wait=10` does not reliably let the LIBERO +scene stabilize and can degrade measured success. All LIBERO evaluation results +reported for this LeRobot implementation use `num_steps_wait=50`. + +## Citation + +```bibtex +@misc{fang2026molmoact2actionreasoningmodels, + title={MolmoAct2: Action Reasoning Models for Real-world Deployment}, + author={Haoquan Fang and Jiafei Duan and Donovan Clay and Sam Wang and Shuo Liu and Weikai Huang and Xiang Fan and Wei-Chuan Tsai and Shirui Chen and Yi Ru Wang and Shanli Xing and Jaemin Cho and Jae Sung Park and Ainaz Eftekhar and Peter Sushko and Karen Farley and Angad Wadhwa and Cole Harrison and Winson Han and Ying-Chun Lee and Eli VanderBilt and Rose Hendrix and Suveen Ellawela and Lucas Ngoo and Joyce Chai and Zhongzheng Ren and Ali Farhadi and Dieter Fox and Ranjay Krishna}, + year={2026}, + eprint={2605.02881}, + archivePrefix={arXiv}, + primaryClass={cs.RO}, + url={https://arxiv.org/abs/2605.02881}, +} +``` + +## License + +This model is licensed under Apache 2.0. It is intended for research and +educational use in accordance with +[Ai2's Responsible Use Guidelines](https://allenai.org/responsible-use), +consistent with [allenai/molmoact2](https://github.com/allenai/molmoact2). diff --git a/docs/source/policy_vla_jepa_README.md b/docs/source/policy_vla_jepa_README.md new file mode 100644 index 000000000..70cdbd6b5 --- /dev/null +++ b/docs/source/policy_vla_jepa_README.md @@ -0,0 +1,39 @@ +# VLA-JEPA + +This repository contains the LeRobot port of **VLA-JEPA**, a Vision-Language-Action model that combines a Qwen3-VL language backbone with a self-supervised video world model (V-JEPA2) and a flow-matching DiT action head. + +Converted from [ginwind/VLA-JEPA](https://huggingface.co/ginwind/VLA-JEPA). + +--- + +## Architecture Overview + +| Component | Module | Role | +| ----------------------- | --------------------------------- | ------------------------------------------------------- | +| **Qwen3-VL backbone** | `Qwen3VLInterface` | Fuses images + language instruction into context tokens | +| **DiT-B action head** | `VLAJEPAActionHead` | Flow-matching diffusion over the action chunk | +| **V-JEPA2 world model** | `ActionConditionedVideoPredictor` | Self-supervised video prediction loss (training only) | + +At inference time only the Qwen backbone and action head are used; the world model is not needed. + +--- + +## Citation + +```bibtex +@misc{sun2026vlajepaenhancingvisionlanguageactionmodel, + title = {VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model}, + author = {Jingwen Sun and Wenyao Zhang and Zekun Qi and Shaojie Ren and Zezhi Liu and Hanxin Zhu and Guangzhong Sun and Xin Jin and Zhibo Chen}, + year = {2026}, + eprint = {2602.10098}, + archivePrefix = {arXiv}, + primaryClass = {cs.RO}, + url = {https://arxiv.org/abs/2602.10098}, +} +``` + +--- + +## License + +Weights are distributed under the license terms of the original [ginwind/VLA-JEPA](https://huggingface.co/ginwind/VLA-JEPA) repository (**Apache 2.0 License**). The LeRobot integration code follows the **Apache 2.0 License**. diff --git a/docs/source/porting_datasets_v3.mdx b/docs/source/porting_datasets_v3.mdx index 46793265e..b2c3c15a0 100644 --- a/docs/source/porting_datasets_v3.mdx +++ b/docs/source/porting_datasets_v3.mdx @@ -300,7 +300,7 @@ This replaces the old episode-per-file structure with efficient, optimally-sized If you have existing datasets in v2.1 format, use the migration tool: ```bash -python src/lerobot/datasets/v30/convert_dataset_v21_to_v30.py \ +python src/lerobot/scripts/convert_dataset_v21_to_v30.py \ --repo-id your_id/existing_dataset ``` diff --git a/docs/source/reachy2.mdx b/docs/source/reachy2.mdx index 1b868711a..7f975af43 100644 --- a/docs/source/reachy2.mdx +++ b/docs/source/reachy2.mdx @@ -161,7 +161,7 @@ lerobot-record \ --dataset.private=true \ --dataset.streaming_encoding=true \ --dataset.encoder_threads=2 \ - # --dataset.vcodec=auto \ + # --dataset.rgb_encoder.vcodec=auto \ --display_data=true ``` @@ -203,7 +203,7 @@ lerobot-record \ --dataset.private=true \ --dataset.streaming_encoding=true \ --dataset.encoder_threads=2 \ - # --dataset.vcodec=auto \ + # --dataset.rgb_encoder.vcodec=auto \ --display_data=true ``` diff --git a/docs/source/rebot_b601.mdx b/docs/source/rebot_b601.mdx new file mode 100644 index 000000000..adb751560 --- /dev/null +++ b/docs/source/rebot_b601.mdx @@ -0,0 +1,186 @@ +# reBot B601-DM + +[reBot B601-DM](https://wiki.seeedstudio.com/rebot_arm_b601_dm_lerobot/) is an open-source, low-cost robot arm from Seeed Studio for embodied-AI and imitation learning. It comes as a **follower** arm (the `B601-DM`, a 6-DOF arm plus gripper driven by Damiao CAN motors) and a **leader** arm (the `StarArm102` / `reBot Arm 102`, driven by FashionStar UART smart servos) used to teleoperate it. + +This page covers **calibration** and **teleoperation** for both single-arm and bimanual (dual-arm) setups. + +
+ reBot B601-DM follower arm at its zero position + reBot Arm 102 leader arm at its zero position +
+ +_Left: the B601-DM follower at its zero position. Right: the reBot Arm 102 leader at its zero position. Images courtesy of [Seeed Studio](https://wiki.seeedstudio.com/rebot_arm_b601_dm_lerobot/)._ + +## Install LeRobot 🤗 + +Follow our [Installation Guide](./installation), then install the reBot support: + +```bash +pip install -e ".[rebot]" +``` + +This pulls in `motorbridge` (CAN motor control for the B601-DM follower) and `motorbridge-smart-servo` (FashionStar UART servos for the reBot Arm 102 leader). + +## Registered device types + +| Type | Kind | +| ------------------------ | -------------------------------------------- | +| `rebot_b601_follower` | single-arm B601-DM follower robot | +| `bi_rebot_b601_follower` | bimanual (dual-arm) follower robot | +| `rebot_102_leader` | single-arm reBot Arm 102 leader teleoperator | +| `bi_rebot_102_leader` | bimanual (dual-arm) leader teleoperator | + +The bimanual types compose two single-arm instances and namespace each arm's +observation/action keys with a `left_` / `right_` prefix. Per-arm settings are +passed through nested `left_arm_config.*` / `right_arm_config.*` arguments. + +## Find the USB ports + +For each device, find the USB port associated with its motor bus using: + +```bash +lerobot-find-port +``` + + + On Linux, remove `brltty` (`sudo apt remove brltty`) so it does not hold the + leader's USB serial port. You may also need to grant access to the serial + devices: `sudo chmod 666 /dev/ttyACM* /dev/ttyUSB*`. + + +## Calibration + +Neither arm stores a persistent hardware calibration: every time it connects, the motors are re-zeroed against the pose the arm is physically holding. Calibration simply records that zero pose. When prompted, **manually move the arm to its zero position** (the default sit-down pose shown above, gripper fully closed) and press ENTER. + +### Follower (B601-DM) + + + + +```bash +lerobot-calibrate \ + --robot.type=rebot_b601_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.id=follower \ + --robot.can_adapter=damiao +``` + + + + +Connect the bimanual follower; calibration runs for the left arm, then the right arm. + +```bash +lerobot-calibrate \ + --robot.type=bi_rebot_b601_follower \ + --robot.id=bi_follower \ + --robot.left_arm_config.port=/dev/ttyACM0 \ + --robot.left_arm_config.can_adapter=damiao \ + --robot.right_arm_config.port=/dev/ttyACM1 \ + --robot.right_arm_config.can_adapter=damiao +``` + +Per-arm calibration files are saved with `_left` / `_right` suffixes on the id. + + + + +### Leader (reBot Arm 102) + + + + +```bash +lerobot-calibrate \ + --teleop.type=rebot_102_leader \ + --teleop.port=/dev/ttyUSB0 \ + --teleop.id=leader +``` + + + + +```bash +lerobot-calibrate \ + --teleop.type=bi_rebot_102_leader \ + --teleop.id=bi_leader \ + --teleop.left_arm_config.port=/dev/ttyUSB0 \ + --teleop.right_arm_config.port=/dev/ttyUSB1 +``` + + + + +## Teleoperation + +Once both arms are calibrated, drive the follower with the leader. The follower talks to its CAN bus through a Damiao serial bridge (`can_adapter=damiao`, the default) or a SocketCAN adapter (`can_adapter=socketcan`). See the [OpenArm page](./openarm) for more details on the SocketCAN adapter configuration. + + + + +```bash +lerobot-teleoperate \ + --robot.type=rebot_b601_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.id=follower \ + --robot.can_adapter=damiao \ + --teleop.type=rebot_102_leader \ + --teleop.port=/dev/ttyUSB0 \ + --teleop.id=leader +``` + + + + +The bimanual leader and follower reuse the single-arm classes; each arm is +configured through nested `left_arm_config.*` / `right_arm_config.*` arguments, +so a bimanual reBot Arm 102 leader drives a bimanual B601-DM follower. + +```bash +lerobot-teleoperate \ + --robot.type=bi_rebot_b601_follower \ + --robot.id=bi_follower \ + --robot.left_arm_config.port=/dev/ttyACM0 \ + --robot.left_arm_config.can_adapter=damiao \ + --robot.right_arm_config.port=/dev/ttyACM1 \ + --robot.right_arm_config.can_adapter=damiao \ + --teleop.type=bi_rebot_102_leader \ + --teleop.id=bi_leader \ + --teleop.left_arm_config.port=/dev/ttyUSB0 \ + --teleop.right_arm_config.port=/dev/ttyUSB1 +``` + + + + + + The leader and follower share the same joint names (`shoulder_pan, + shoulder_lift, elbow_flex, wrist_flex, wrist_yaw, wrist_roll, gripper`), so + leader actions map directly onto the follower. + + +If the motion of a joint is reversed, flip its sign in the leader's `joint_directions` (the gripper also carries a scale to widen its range to the follower): + +```bash +lerobot-teleoperate \ + --robot.type=rebot_b601_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.can_adapter=damiao \ + --teleop.type=rebot_102_leader \ + --teleop.port=/dev/ttyUSB0 \ + --teleop.joint_directions='{"shoulder_pan":-1,"shoulder_lift":-1,"elbow_flex":1,"wrist_flex":1,"wrist_yaw":1,"wrist_roll":-1,"gripper":-6}' +``` + +## Recording datasets + +Swap `lerobot-teleoperate` for `lerobot-record` (with the same `--robot.*` / `--teleop.*` arguments, plus `--dataset.*`) to record demonstrations for training. See [Imitation Learning for Robots](./il_robots) for the full workflow. + +For hardware assembly and wiring, see the [Seeed Studio reBot wiki](https://wiki.seeedstudio.com/rebot_arm_b601_dm_lerobot/). diff --git a/docs/source/robocasa.mdx b/docs/source/robocasa.mdx index f6a784e72..5a335a484 100644 --- a/docs/source/robocasa.mdx +++ b/docs/source/robocasa.mdx @@ -166,7 +166,7 @@ lerobot-train \ --output_dir=./outputs/smolvla_robocasa_CloseFridge \ --steps=100000 \ --batch_size=4 \ - --eval_freq=5000 \ + --env_eval_freq=5000 \ --eval.batch_size=1 \ --eval.n_episodes=5 \ --save_freq=10000 diff --git a/docs/source/robometer.mdx b/docs/source/robometer.mdx new file mode 100644 index 000000000..5af6882d3 --- /dev/null +++ b/docs/source/robometer.mdx @@ -0,0 +1,185 @@ +# ROBOMETER + +ROBOMETER is a **general-purpose video-language robotic reward model**. It predicts dense, frame-level task progress and frame-level success from a trajectory video and a task description. + +**Paper**: [ROBOMETER: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons](https://arxiv.org/abs/2603.02115) +**Project**: [robometer.github.io](https://robometer.github.io/) +**Original code**: [github.com/robometer/robometer](https://github.com/robometer/robometer) +**Checkpoint**: [lerobot/Robometer-4B](https://huggingface.co/lerobot/Robometer-4B) + +## Overview + +ROBOMETER builds on `Qwen/Qwen3-VL-4B-Instruct` and adds three lightweight prediction heads: + +- **Progress head**: predicts per-frame task progress in `[0, 1]`. +- **Success head**: predicts per-frame task success probability. +- **Preference head**: predicts which of two trajectories better completes the task during training. + +The paper trains ROBOMETER with a composite objective: + +```text +L = L_pref + L_prog + L_succ +``` + +The LeRobot integration is currently **inference-only**. It preserves the preference head so that the published `Robometer-4B` checkpoint loads without remapping, but `compute_reward()` queries the progress or success head only. + +## What the LeRobot Integration Covers + +- Standard `reward_model.type=robometer` configuration through LeRobot. +- Qwen3-VL image and text preprocessing through `RobometerEncoderProcessorStep`. +- LeRobot reward-model save/load APIs through `PreTrainedRewardModel`. +- Dense, frame-level progress and success predictions internally. +- A scalar reward through `compute_reward()` for downstream LeRobot reward-model usage. + +This page focuses on using the published ROBOMETER checkpoint as a zero-shot reward model. Training ROBOMETER from scratch is outside the current LeRobot integration. + +## Installation Requirements + +1. Install LeRobot by following the [Installation Guide](./installation). +2. Install the ROBOMETER dependencies: + +```bash +pip install -e ".[robometer]" +``` + +If you use `uv` directly from a source checkout: + +```bash +uv sync --extra robometer +``` + +ROBOMETER uses a Qwen3-VL-4B backbone, so GPU inference is strongly recommended. + +## Model Inputs and Outputs + +ROBOMETER expects: + +- A trajectory video or sequence of frames. +- A natural-language task description. + +In LeRobot datasets, the preprocessor reads: + +| Config field | Default | Meaning | +| ------------------------- | ------------------------ | ----------------------------------------------------- | +| `reward_model.image_key` | `observation.images.top` | Camera/video observation used by ROBOMETER | +| `reward_model.task_key` | `task` | Key in complementary data that stores the task string | +| `reward_model.max_frames` | `8` | Maximum number of frames passed to ROBOMETER | + +The model predicts per-frame progress and success internally. The LeRobot reward API returns a scalar per sample: + +- `reward_output="progress"` (default): return the last-frame progress, clamped to `[0, 1]`. +- `reward_output="success"`: return `1.0` if the last-frame success probability is above `success_threshold`, otherwise `0.0`. + +## Usage + +### Load the Reward Model Directly + +```python +from lerobot.rewards.robometer import RobometerConfig, RobometerRewardModel + +cfg = RobometerConfig( + pretrained_path="lerobot/Robometer-4B", + device="cuda", + reward_output="progress", +) +reward_model = RobometerRewardModel.from_pretrained(cfg.pretrained_path, config=cfg) +``` + +### Encode Frames and Compute a Reward + +For a direct Python call, provide frames as `uint8` arrays with shape `(T, H, W, C)` and a task string: + +```python +from lerobot.rewards.robometer.modeling_robometer import ROBOMETER_FEATURE_PREFIX +from lerobot.rewards.robometer.processor_robometer import RobometerEncoderProcessorStep + +# frames: np.ndarray, shape (T, H, W, C), dtype uint8 +# task: str +encoder = RobometerEncoderProcessorStep( + base_model_id=cfg.base_model_id, + use_multi_image=cfg.use_multi_image, + use_per_frame_progress_token=cfg.use_per_frame_progress_token, + max_frames=cfg.max_frames, +) + +encoded = encoder.encode_samples([(frames, task)]) +batch = {f"{ROBOMETER_FEATURE_PREFIX}{key}": value for key, value in encoded.items()} + +reward = reward_model.compute_reward(batch) +``` + +`reward` is a tensor of shape `(batch_size,)`. + +### Use the Reward Factory + +You can also instantiate ROBOMETER through the reward factory: + +```python +from lerobot.rewards import make_reward_model, make_reward_model_config, make_reward_pre_post_processors + +cfg = make_reward_model_config( + "robometer", + pretrained_path="lerobot/Robometer-4B", + device="cuda", + image_key="observation.images.top", +) +reward_model = make_reward_model(cfg) +preprocessor, postprocessor = make_reward_pre_post_processors(cfg) +``` + +The preprocessor writes Qwen-VL tensors under the `observation.robometer.*` namespace, and `compute_reward()` reads those encoded tensors. + +## Configuration Notes + +### Backbone and Vocabulary + +The published checkpoint uses a Qwen3-VL-4B backbone. ROBOMETER adds five special tokens to the tokenizer in a fixed order: + +```text +<|split_token|> +<|reward_token|> +<|pref_token|> +<|sim_token|> +<|prog_token|> +``` + +`<|prog_token|>` is inserted after each frame and is the hidden-state position used for per-frame progress and success prediction. `<|split_token|>` and `<|pref_token|>` are used by the paper's pairwise trajectory preference objective. `<|reward_token|>` and `<|sim_token|>` are preserved for checkpoint compatibility. + +The LeRobot config stores a serialized `vlm_config` with the post-resize vocabulary so the model can reload from `config.json` without downloading the base Qwen weights first. For `Qwen/Qwen3-VL-4B-Instruct`, the tokenizer length is `151669`, and the five ROBOMETER tokens produce the checkpoint vocabulary size `151674`. + +### Progress Prediction + +In the published checkpoint, progress is discrete. The progress head outputs logits over `progress_discrete_bins=10` uniformly spaced bin centers in `[0, 1]`. LeRobot converts these logits into a continuous value by applying a softmax and taking the expectation over bin centers, matching the upstream ROBOMETER implementation. + +### Success Prediction + +The success head outputs raw logits per frame. LeRobot converts them to probabilities with `sigmoid`. When `reward_output="success"`, `compute_reward()` thresholds the last-frame success probability using `success_threshold`. + +## Limitations + +- The current LeRobot integration is inference-only; it does not implement ROBOMETER training or preference-pair training. +- `compute_reward()` returns a scalar per sample for the LeRobot reward-model API, even though ROBOMETER predicts per-frame progress and success internally. +- ROBOMETER is video-language based; it does not use privileged robot state such as contact forces or object poses. + +## References + +- [ROBOMETER project](https://robometer.github.io/) +- [ROBOMETER paper](https://arxiv.org/abs/2603.02115) +- [Original ROBOMETER code](https://github.com/robometer/robometer) +- [Published ROBOMETER-4B checkpoint](https://huggingface.co/lerobot/Robometer-4B) +- [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) + +## Citation + +```bibtex +@inproceedings{liang2026robometer, +title = {Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons}, +author={Anthony Liang and Yigit Korkmaz and Jiahui Zhang and Minyoung Hwang and Abrar Anwar and Sidhant Kaushik and Aditya Shah and Alex S. Huang and Luke Zettlemoyer and Dieter Fox and Yu Xiang and Anqi Li and Andreea Bobu and Abhishek Gupta and Stephen Tu and Erdem Biyik and Jesse Zhang}, +year={2026}, +booktitle={Robotics: Science and Systems 2026}, +} +``` + +## License + +This LeRobot integration follows the **Apache 2.0 License** used by LeRobot. Check the upstream ROBOMETER code and model pages for the licenses of the original implementation and released checkpoints. diff --git a/docs/source/smolvla.mdx b/docs/source/smolvla.mdx index bf8a0d2f0..e28270c9b 100644 --- a/docs/source/smolvla.mdx +++ b/docs/source/smolvla.mdx @@ -97,22 +97,22 @@ Similarly for when recording an episode, it is recommended that you are logged i Once you are logged in, you can run inference in your setup by doing: ```bash -lerobot-record \ +lerobot-rollout \ + --strategy.type=base \ --robot.type=so101_follower \ --robot.port=/dev/ttyACM0 \ # <- Use your port --robot.id=my_blue_follower_arm \ # <- Use your robot id --robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \ # <- Use your cameras - --dataset.single_task="Grasp a lego block and put it in the bin." \ # <- Use the same task description you used in your dataset recording - --dataset.repo_id=${HF_USER}/eval_DATASET_NAME_test \ # <- This will be the dataset name on HF Hub - --dataset.episode_time_s=50 \ - --dataset.num_episodes=10 \ - --dataset.streaming_encoding=true \ - --dataset.encoder_threads=2 \ - # --dataset.vcodec=auto \ + --task="Grasp a lego block and put it in the bin." \ # <- Use the same task description you used in your dataset recording + # <- RTC optional, use when running on low power hardware \ + # --inference.type=rtc \ + # --inference.rtc.execution_horizon=10 \ + # --inference.rtc.max_guidance_weight=10.0 \ # <- Teleop optional if you want to teleoperate in between episodes \ # --teleop.type=so100_leader \ # --teleop.port=/dev/ttyACM0 \ # --teleop.id=my_red_leader_arm \ + # --display_data=true #optional use if you want to see the camera stream \ --policy.path=HF_USER/FINETUNE_MODEL_NAME # <- Use your fine-tuned model ``` diff --git a/docs/source/so101.mdx b/docs/source/so101.mdx index 1274b8282..5b4ed0985 100644 --- a/docs/source/so101.mdx +++ b/docs/source/so101.mdx @@ -122,7 +122,7 @@ The video below shows the sequence of steps for setting the motor ids. #### Follower -Connect the usb cable from your computer and the power supply to the follower arm's controller board. Then, run the following command or run the API example with the port you got from the previous step. You'll also need to give your leader arm a name with the `id` parameter. +Connect the usb cable from your computer and the power supply to the follower arm's controller board. Then, run the following command or run the API example with the port you got from the previous step. You'll also need to give your follower arm a name with the `id` parameter. diff --git a/docs/source/streaming_video_encoding.mdx b/docs/source/streaming_video_encoding.mdx index 40004200e..0be32b717 100644 --- a/docs/source/streaming_video_encoding.mdx +++ b/docs/source/streaming_video_encoding.mdx @@ -17,9 +17,9 @@ This makes `save_episode()` near-instant (the video is already encoded by the ti | Parameter | CLI Flag | Type | Default | Description | | ----------------------- | --------------------------------- | ------------- | ------------- | ----------------------------------------------------------------- | | `streaming_encoding` | `--dataset.streaming_encoding` | `bool` | `True` | Enable real-time encoding during capture | -| `vcodec` | `--dataset.vcodec` | `str` | `"libsvtav1"` | Video codec. `"auto"` detects best HW encoder | +| `vcodec` | `--dataset.rgb_encoder.vcodec` | `str` | `"libsvtav1"` | Video codec. `"auto"` detects best HW encoder | | `encoder_threads` | `--dataset.encoder_threads` | `int \| None` | `None` (auto) | Threads per encoder instance. `None` will leave the vcoded decide | -| `encoder_queue_maxsize` | `--dataset.encoder_queue_maxsize` | `int` | `60` | Max buffered frames per camera (~2s at 30fps). Consumes RAM | +| `encoder_queue_maxsize` | `--dataset.encoder_queue_maxsize` | `int` | `30` | Max buffered frames per camera (~1s at 30fps). Consumes RAM | ## 3. Performance Considerations @@ -48,7 +48,7 @@ This parameter controls how many threads each encoder instance uses internally: ### Backpressure and Frame Dropping -Each camera has a bounded queue (`encoder_queue_maxsize`, default 60 frames). When the encoder can't keep up: +Each camera has a bounded queue (`encoder_queue_maxsize`, default 30 frames). When the encoder can't keep up: 1. The queue fills up (consuming RAM) 2. New frames are **dropped** (not blocked) — the capture loop continues uninterrupted @@ -82,15 +82,15 @@ Use HW encoding when: ### Available HW Encoders -| Encoder | Platform | Hardware | CLI Value | -| ------------------- | ------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------ | -| `h264_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.vcodec=h264_videotoolbox` | -| `hevc_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.vcodec=hevc_videotoolbox` | -| `h264_nvenc` | Linux/Windows | NVIDIA GPU | `--dataset.vcodec=h264_nvenc` | -| `hevc_nvenc` | Linux/Windows | NVIDIA GPU | `--dataset.vcodec=hevc_nvenc` | -| `h264_vaapi` | Linux | Intel/AMD GPU | `--dataset.vcodec=h264_vaapi` | -| `h264_qsv` | Linux/Windows | Intel Quick Sync | `--dataset.vcodec=h264_qsv` | -| `auto` | Any | Probes the system for available HW encoders. Falls back to `libsvtav1` if no HW encoder is found | `--dataset.vcodec=auto` | +| Encoder | Platform | Hardware | CLI Value | +| ------------------- | ------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------ | +| `h264_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.rgb_encoder.vcodec=h264_videotoolbox` | +| `hevc_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.rgb_encoder.vcodec=hevc_videotoolbox` | +| `h264_nvenc` | Linux/Windows | NVIDIA GPU | `--dataset.rgb_encoder.vcodec=h264_nvenc` | +| `hevc_nvenc` | Linux/Windows | NVIDIA GPU | `--dataset.rgb_encoder.vcodec=hevc_nvenc` | +| `h264_vaapi` | Linux | Intel/AMD GPU | `--dataset.rgb_encoder.vcodec=h264_vaapi` | +| `h264_qsv` | Linux/Windows | Intel Quick Sync | `--dataset.rgb_encoder.vcodec=h264_qsv` | +| `auto` | Any | Probes the system for available HW encoders. Falls back to `libsvtav1` if no HW encoder is found | `--dataset.rgb_encoder.vcodec=auto` | > [!NOTE] > In order to use the HW accelerated encoders you might need to upgrade your GPU drivers. @@ -100,15 +100,15 @@ Use HW encoding when: ## 5. Troubleshooting -| Symptom | Likely Cause | Fix | -| ------------------------------------------------------------------ | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| System freezes or choppy robot movement or Rerun visualization lag | CPU starved (100% load usage) | Close other apps, reduce encoding throughput, lower `encoder_threads`, use `h264`, use `display_data=False`. If the CPU continues to be at 100% then it might be insufficient for your setup, consider `--dataset.streaming_encoding=false` or HW encoding (`--dataset.vcodec=auto`) | -| "Encoder queue full" warnings or dropped frames in dataset | Encoder can't keep up (Queue overflow) | If CPU is not at 100%: Increase `encoder_threads`, increase `encoder_queue_maxsize` or use HW encoding (`--dataset.vcodec=auto`). | -| High RAM usage | Queue filling faster than encoding | `encoder_threads` too low or CPU insufficient. Reduce `encoder_queue_maxsize` or use HW encoding | -| Large video files | Using HW encoder or H.264 | Expected trade-off. Switch to `libsvtav1` if CPU allows | -| `save_episode()` still slow | `streaming_encoding` is `False` | Set `--dataset.streaming_encoding=true` | -| Encoder thread crash | Codec not available or invalid settings | Check `vcodec` is installed, try `--dataset.vcodec=auto` | -| Recorded dataset is missing frames | CPU/GPU starvation or occasional load spikes | If ~5% of frames are missing, your system is likely overloaded — follow the recommendations above. If fewer frames are missing (~2%), they are probably due to occasional transient load spikes (often at startup) and can be considered expected. | +| Symptom | Likely Cause | Fix | +| ------------------------------------------------------------------ | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| System freezes or choppy robot movement or Rerun visualization lag | CPU starved (100% load usage) | Close other apps, reduce encoding throughput, lower `encoder_threads`, use `h264`, use `display_data=False`. If the CPU continues to be at 100% then it might be insufficient for your setup, consider `--dataset.streaming_encoding=false` or HW encoding (`--dataset.rgb_encoder.vcodec=auto`) | +| "Encoder queue full" warnings or dropped frames in dataset | Encoder can't keep up (Queue overflow) | If CPU is not at 100%: Increase `encoder_threads`, increase `encoder_queue_maxsize` or use HW encoding (`--dataset.rgb_encoder.vcodec=auto`). | +| High RAM usage | Queue filling faster than encoding | `encoder_threads` too low or CPU insufficient. Reduce `encoder_queue_maxsize` or use HW encoding | +| Large video files | Using HW encoder or H.264 | Expected trade-off. Switch to `libsvtav1` if CPU allows | +| `save_episode()` still slow | `streaming_encoding` is `False` | Set `--dataset.streaming_encoding=true` | +| Encoder thread crash | Codec not available or invalid settings | Check `vcodec` is installed, try `--dataset.rgb_encoder.vcodec=auto` | +| Recorded dataset is missing frames | CPU/GPU starvation or occasional load spikes | If ~5% of frames are missing, your system is likely overloaded — follow the recommendations above. If fewer frames are missing (~2%), they are probably due to occasional transient load spikes (often at startup) and can be considered expected. | ## 6. Recommended Configurations @@ -146,7 +146,7 @@ On very constrained systems, streaming encoding may compete too heavily with the # 2camsx 640x480x3 @30fps: Requires some tuning. # Use H.264, disable streaming, consider batching encoding -lerobot-record --dataset.vcodec=h264 --dataset.streaming_encoding=false ... +lerobot-record --dataset.rgb_encoder.vcodec=h264 --dataset.streaming_encoding=false ... ``` ## 7. Closing note diff --git a/docs/source/tools.mdx b/docs/source/tools.mdx new file mode 100644 index 000000000..d88881184 --- /dev/null +++ b/docs/source/tools.mdx @@ -0,0 +1,210 @@ +# Tools + +LeRobot v3.1 supports **tool calls** in policies — assistant messages can +emit structured invocations like `say(text="OK, starting now")` that the +runtime dispatches to a real implementation (TTS, controller, logger, …). + +This page covers: + +1. Where the tool catalog lives. +2. How the annotation pipeline produces tool-call atoms. +3. How to add your own tool. + +## Where tools are declared + +Two layers. + +**The catalog** — a list of OpenAI-style function schemas — lives at +`meta/info.json["tools"]` on each dataset. Example: + +```json +{ + "features": { "...": "..." }, + "tools": [ + { + "type": "function", + "function": { + "name": "say", + "description": "Speak a short utterance to the user via the TTS executor.", + "parameters": { + "type": "object", + "properties": { + "text": { + "type": "string", + "description": "The verbatim text to speak." + } + }, + "required": ["text"] + } + } + } + ] +} +``` + +Read it via the dataset metadata accessor: + +```python +from lerobot.datasets.dataset_metadata import LeRobotDatasetMetadata + +meta = LeRobotDatasetMetadata(repo_id="pepijn/super_poulain_final_annotations") +tools = meta.tools # list[dict] — OpenAI tool schemas +``` + +If the dataset's `info.json` doesn't declare any tools, `meta.tools` +returns `DEFAULT_TOOLS` from `lerobot.datasets.language` — currently a +single-entry list with the canonical `say` schema. So unannotated +datasets and chat-template consumers keep working without any +configuration: + +```python +prompt_str = tokenizer.apply_chat_template( + sample["messages"], + tools=meta.tools, # works either way + add_generation_prompt=False, + tokenize=False, +) +``` + +**The implementations** — runnable Python — will live under +`src/lerobot/tools/`, one file per tool. The runtime dispatcher and +the canonical `say` implementation (wrapping Kyutai's pocket-tts) are +not part of the catalog layer described here; today this layer ships +only the schema storage and the `DEFAULT_TOOLS` fallback constant. + +## Per-row tool _invocations_ + +The catalog above describes _what can be called_. The actual _call_ — the +function name plus the argument values — is stored per-row, on the +assistant atoms in `language_events`: + +```python +{ + "role": "assistant", + "content": null, + "style": null, + "timestamp": 12.4, + "camera": null, + "tool_calls": [ + { "type": "function", + "function": { "name": "say", "arguments": { "text": "On it." } } } + ] +} +``` + +Recipes splice these into rendered messages via `tool_calls_from`: + +```yaml +user_interjection_response: + bindings: + speech: "emitted_at(t, role=assistant, tool_name=say)" + messages: + - { role: user, content: "${task}", stream: high_level } + - { + role: assistant, + content: "${current_plan}", + stream: high_level, + target: true, + tool_calls_from: speech, + } +``` + +The model's training target is one assistant turn that carries both the +plan text _and_ the `say` tool call. At inference, the runtime parses +the generated text back into structured `tool_calls` and dispatches to +the matching implementation. + +## How to add your own tool + +> **Note:** Steps 2 and 3 below describe the runtime layer +> (`src/lerobot/tools/`, the `Tool` protocol, `TOOL_REGISTRY`, +> `get_tools(meta)`) which is not part of the catalog layer shipped +> today — those modules don't yet exist in the tree. Step 1 alone is +> enough to make the tool visible to the chat template via +> `meta.tools` so the model can learn to _generate_ the call; +> executing the call at inference requires the runtime layer. + +Three steps. Concrete example: a `record_observation` tool the policy +can call to capture an extra observation outside the regular control +loop. + +### Step 1 — declare the schema + +Add an entry under `meta/info.json["tools"]`. Either edit the file +directly on disk _before_ running the annotation pipeline (it'll be +preserved) or hand it to `lerobot-annotate` via a config flag. + +```json +{ + "tools": [ + { "type": "function", "function": { "name": "say", "...": "..." } }, + { + "type": "function", + "function": { + "name": "record_observation", + "description": "Capture a high-resolution still image for the user.", + "parameters": { + "type": "object", + "properties": { + "label": { + "type": "string", + "description": "Short label for the saved image." + } + }, + "required": ["label"] + } + } + } + ] +} +``` + +The schema follows OpenAI's function-calling convention exactly, so the +chat template can render it natively. + +### Step 2 — implement the call + +Create `src/lerobot/tools/record_observation.py`: + +```python +from .base import Tool +from typing import Any + +RECORD_OBSERVATION_SCHEMA: dict[str, Any] = { "...": "..." } # mirrors the JSON above + + +class RecordObservationTool: + name = "record_observation" + schema = RECORD_OBSERVATION_SCHEMA + + def __init__(self, schema: dict | None = None, output_dir: str = "."): + self.output_dir = output_dir + + def call(self, arguments: dict) -> str: + label = arguments["label"] + # ... save the latest camera frame to /