Merge branch 'main' into feature/add-multitask-dit

This commit is contained in:
Bryson Jones
2026-01-12 08:22:31 -08:00
committed by GitHub
52 changed files with 336 additions and 395 deletions
+5 -5
View File
@@ -100,11 +100,11 @@ lerobot-train \
--dataset.repo_id=lerobot/aloha_mobile_cabinet --dataset.repo_id=lerobot/aloha_mobile_cabinet
``` ```
| Category | Models | | Category | Models |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Imitation Learning** | [ACT](./docs/source/policy_act_README.md), [Diffusion](./docs/source/policy_diffusion_README.md), [VQ-BeT](./docs/source/policy_vqbet_README.md) | | **Imitation Learning** | [ACT](./docs/source/policy_act_README.md), [Diffusion](./docs/source/policy_diffusion_README.md), [VQ-BeT](./docs/source/policy_vqbet_README.md) |
| **Reinforcement Learning** | [HIL-SERL](./docs/source/hilserl.mdx), [TDMPC](./docs/source/policy_tdmpc_README.md) & QC-FQL (coming soon) | | **Reinforcement Learning** | [HIL-SERL](./docs/source/hilserl.mdx), [TDMPC](./docs/source/policy_tdmpc_README.md) & QC-FQL (coming soon) |
| **VLAs Models** | [Pi0.5](./docs/source/pi05.mdx), [GR00T N1.5](./docs/source/policy_groot_README.md), [SmolVLA](./docs/source/policy_smolvla_README.md), [XVLA](./docs/source/xvla.mdx) | | **VLAs Models** | [Pi0Fast](./docs/source/pi0fast.mdx), [Pi0.5](./docs/source/pi05.mdx), [GR00T N1.5](./docs/source/policy_groot_README.md), [SmolVLA](./docs/source/policy_smolvla_README.md), [XVLA](./docs/source/xvla.mdx) |
Similarly to the hardware, you can easily implement your own policy & leverage LeRobot's data collection, training, and visualization tools, and share your model to the HF Hub Similarly to the hardware, you can easily implement your own policy & leverage LeRobot's data collection, training, and visualization tools, and share your model to the HF Hub
+1
View File
@@ -138,6 +138,7 @@ from lerobot.teleoperators import ( # noqa: F401
TeleoperatorConfig, TeleoperatorConfig,
make_teleoperator_from_config, make_teleoperator_from_config,
so_leader, so_leader,
bi_so_leader,
) )
from lerobot.utils.robot_utils import precise_sleep from lerobot.utils.robot_utils import precise_sleep
from lerobot.utils.utils import init_logging from lerobot.utils.utils import init_logging
+7 -1
View File
@@ -12,6 +12,12 @@ Developers and researchers can post-train GR00T N1.5 with their own real or synt
GR00T N1.5 (specifically the GR00T-N1.5-3B model) is built using pre-trained vision and language encoders. It utilizes a flow matching action transformer to model a chunk of actions, conditioned on vision, language, and proprioception. GR00T N1.5 (specifically the GR00T-N1.5-3B model) is built using pre-trained vision and language encoders. It utilizes a flow matching action transformer to model a chunk of actions, conditioned on vision, language, and proprioception.
<img
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/lerobot-groot-paper1%20(1).png"
alt="An overview of GR00T"
width="80%"
/>
Its strong performance comes from being trained on an expansive and diverse humanoid dataset, which includes: Its strong performance comes from being trained on an expansive and diverse humanoid dataset, which includes:
- Real captured data from robots. - Real captured data from robots.
@@ -103,7 +109,7 @@ Once you have trained your model using your parameters you can run inference in
```bash ```bash
lerobot-record \ lerobot-record \
--robot.type=bi_so100_follower \ --robot.type=bi_so_follower \
--robot.left_arm_port=/dev/ttyACM1 \ --robot.left_arm_port=/dev/ttyACM1 \
--robot.right_arm_port=/dev/ttyACM0 \ --robot.right_arm_port=/dev/ttyACM0 \
--robot.id=bimanual_follower \ --robot.id=bimanual_follower \
+6
View File
@@ -6,6 +6,12 @@
π₀ represents a breakthrough in robotics as the first general-purpose robot foundation model developed by [Physical Intelligence](https://www.physicalintelligence.company/blog/pi0). Unlike traditional robot programs that are narrow specialists programmed for repetitive motions, π₀ is designed to be a generalist policy that can understand visual inputs, interpret natural language instructions, and control a variety of different robots across diverse tasks. π₀ represents a breakthrough in robotics as the first general-purpose robot foundation model developed by [Physical Intelligence](https://www.physicalintelligence.company/blog/pi0). Unlike traditional robot programs that are narrow specialists programmed for repetitive motions, π₀ is designed to be a generalist policy that can understand visual inputs, interpret natural language instructions, and control a variety of different robots across diverse tasks.
<img
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/lerobot-pi0%20(1).png"
alt="An overview of Pi0"
width="85%"
/>
### The Vision for Physical Intelligence ### The Vision for Physical Intelligence
As described by Physical Intelligence, while AI has achieved remarkable success in digital domains, from chess-playing to drug discovery, human intelligence still dramatically outpaces AI in the physical world. To paraphrase Moravec's paradox, winning a game of chess represents an "easy" problem for AI, but folding a shirt or cleaning up a table requires solving some of the most difficult engineering problems ever conceived. π₀ represents a first step toward developing artificial physical intelligence that enables users to simply ask robots to perform any task they want, just like they can with large language models. As described by Physical Intelligence, while AI has achieved remarkable success in digital domains, from chess-playing to drug discovery, human intelligence still dramatically outpaces AI in the physical world. To paraphrase Moravec's paradox, winning a game of chess represents an "easy" problem for AI, but folding a shirt or cleaning up a table requires solving some of the most difficult engineering problems ever conceived. π₀ represents a first step toward developing artificial physical intelligence that enables users to simply ask robots to perform any task they want, just like they can with large language models.
+66 -2
View File
@@ -6,6 +6,12 @@
π₀-FAST combines the power of Vision-Language Models with a novel action tokenization approach called **FAST (Frequency-space Action Sequence Tokenization)**. This enables training autoregressive VLAs on highly dexterous tasks that are impossible with standard binning-based discretization, while training **up to 5x faster** than diffusion-based approaches like π₀. π₀-FAST combines the power of Vision-Language Models with a novel action tokenization approach called **FAST (Frequency-space Action Sequence Tokenization)**. This enables training autoregressive VLAs on highly dexterous tasks that are impossible with standard binning-based discretization, while training **up to 5x faster** than diffusion-based approaches like π₀.
<img
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/lerobot-pifast.png"
alt="An overview of Pi0-FAST"
width="85%"
/>
### Why FAST? ### Why FAST?
Standard approaches for robot action tokenization use simple per-dimension, per-timestep binning schemes. While passable for simple behaviors, this rapidly breaks down for complex and dexterous skills that require precision and high-frequency control. Standard approaches for robot action tokenization use simple per-dimension, per-timestep binning schemes. While passable for simple behaviors, this rapidly breaks down for complex and dexterous skills that require precision and high-frequency control.
@@ -53,7 +59,7 @@ You have two options for the FAST tokenizer:
### Training Your Own Tokenizer ### Training Your Own Tokenizer
```bash ```bash
python src/lerobot/policies/pi0_fast/train_fast_tokenizer.py \ lerobot-train-tokenizer \
--repo_id "user/my-lerobot-dataset" \ --repo_id "user/my-lerobot-dataset" \
--action_horizon 10 \ --action_horizon 10 \
--encoded_dims "0:6" \ --encoded_dims "0:6" \
@@ -90,7 +96,7 @@ policy.type=pi0_fast
For training π₀-FAST, you can use the LeRobot training script: For training π₀-FAST, you can use the LeRobot training script:
```bash ```bash
python src/lerobot/scripts/lerobot_train.py \ lerobot-train \
--dataset.repo_id=your_dataset \ --dataset.repo_id=your_dataset \
--policy.type=pi0_fast \ --policy.type=pi0_fast \
--output_dir=./outputs/pi0fast_training \ --output_dir=./outputs/pi0fast_training \
@@ -171,6 +177,64 @@ The model takes images, text instructions, and robot state as input, and outputs
| Inference Method | Iterative Denoising | Autoregressive Decoding | | Inference Method | Iterative Denoising | Autoregressive Decoding |
| KV-Caching | N/A | Supported | | KV-Caching | N/A | Supported |
## Reproducing π₀Fast results
We reproduce the results of π₀Fast on the LIBERO benchmark using the LeRobot implementation. We take the LeRobot PiFast base model [lerobot/pi0fast-base](https://huggingface.co/lerobot/pi0fast-base) and finetune for an additional 40kk steps in bfloat16, with batch size of 256 on 8 H100 GPUs using the [HuggingFace LIBERO dataset](https://huggingface.co/datasets/HuggingFaceVLA/libero).
The finetuned model can be found here:
- **π₀Fast LIBERO**: [lerobot/pi0fast-libero](https://huggingface.co/lerobot/pi0fast-libero)
With the following training command:
```bash
lerobot-train \
--dataset.repo_id=lerobot/libero \
--output_dir=outputs/libero_pi0fast \
--job_name=libero_pi0fast \
--policy.path=lerobot/pi0fast_base \
--policy.dtype=bfloat16 \
--steps=100000 \
--save_freq=20000 \
--batch_size=4 \
--policy.device=cuda \
--policy.scheduler_warmup_steps=4000 \
--policy.scheduler_decay_steps=100000 \
--policy.scheduler_decay_lr=1e-5 \
--policy.gradient_checkpointing=true \
--policy.chunk_size=10 \
--policy.n_action_steps=10 \
--policy.max_action_tokens=256 \
--policy.empty_cameras=1 \
```
We then evaluate the finetuned model using the LeRobot LIBERO implementation, by running the following command:
```bash
tasks="libero_object,libero_spatial,libero_goal,libero_10"
lerobot-eval \
--policy.path=lerobot/pi0fast-libero \
--policy.max_action_tokens=256 \
--env.type=libero \
--policy.gradient_checkpointing=false \
--env.task=${tasks} \
--eval.batch_size=1 \
--eval.n_episodes=1 \
--rename_map='{"observation.images.image":"observation.images.base_0_rgb","observation.images.image2":"observation.images.left_wrist_0_rgb"}'
```
**Note:** We set `n_action_steps=10`, similar to the original OpenPI implementation.
### Results
We obtain the following results on the LIBERO benchmark:
| Model | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average |
| ----------- | -------------- | ------------- | ----------- | --------- | -------- |
| **π₀-fast** | 70.0 | 100.0 | 100.0 | 60.0 | **82.5** |
The full evaluation output folder, including videos, is available [here](https://drive.google.com/drive/folders/1HXpwPTRm4hx6g1sF2P7OOqGG0TwPU7LQ?usp=sharing)
## License ## License
This model follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi). This model follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi).
+6
View File
@@ -4,6 +4,12 @@ SARM (Stage-Aware Reward Modeling) is a video-based reward modeling framework fo
**Paper**: [SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation](https://arxiv.org/abs/2509.25358) **Paper**: [SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation](https://arxiv.org/abs/2509.25358)
<img
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/lerobot-sarm.png"
alt="An overview of SARM"
width="80%"
/>
## Why Reward Models? ## Why Reward Models?
Standard behavior cloning treats all demonstration frames equally, but real-world robot datasets are messy. They contain hesitations, corrections, and variable-quality trajectories. Reward models solve this by learning a generalizable notion of **task progress** from demonstrations: given video frames and a task description, they predict how close the robot is to completing the task (0→1). This learned "progress signal" can be used in multiple ways, two promising applications are: (1) **weighted imitation learning** (RA-BC), where high-progress frames receive more weight during policy training, and (2) **reinforcement learning**, where the reward model provides dense rewards for online or offline policy improvement. Standard behavior cloning treats all demonstration frames equally, but real-world robot datasets are messy. They contain hesitations, corrections, and variable-quality trajectories. Reward models solve this by learning a generalizable notion of **task progress** from demonstrations: given video frames and a task description, they predict how close the robot is to completing the task (0→1). This learned "progress signal" can be used in multiple ways, two promising applications are: (1) **weighted imitation learning** (RA-BC), where high-progress frames receive more weight during policy training, and (2) **reinforcement learning**, where the reward model provides dense rewards for online or offline policy improvement.
+6
View File
@@ -8,6 +8,12 @@ X Square Robots WALL-OSS is now integrated into Hugging Faces LeRobot ecos
The WALL-OSS team is building the embodied foundation model to capture and compress the world's most valuable data: the continuous, high-fidelity stream of physical interaction. By creating a direct feedback loop between the model's decisions and the body's lived experience, the emergence of a truly generalizable intelligence is enabled—one that understands not just how the world works, but how to act effectively within it. The WALL-OSS team is building the embodied foundation model to capture and compress the world's most valuable data: the continuous, high-fidelity stream of physical interaction. By creating a direct feedback loop between the model's decisions and the body's lived experience, the emergence of a truly generalizable intelligence is enabled—one that understands not just how the world works, but how to act effectively within it.
<img
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/walloss-lerobot-paper.png"
alt="An overview of WALL-OSS"
width="85%"
/>
Technically, WALL-OSS introduces a tightly coupled multimodal architecture (tightly-coupled MoE structure) that integrates both discrete and continuous action modeling strategies. Through a two-stage training pipeline (Inspiration → Integration), the model gradually unifies semantic reasoning and high-frequency action generation. Its core innovations include: Technically, WALL-OSS introduces a tightly coupled multimodal architecture (tightly-coupled MoE structure) that integrates both discrete and continuous action modeling strategies. Through a two-stage training pipeline (Inspiration → Integration), the model gradually unifies semantic reasoning and high-frequency action generation. Its core innovations include:
- **Embodied perceptionenhanced multimodal pretraining**: Large-scale training on unified visionlanguageaction data to strengthen spatial, causal, and manipulation understanding. - **Embodied perceptionenhanced multimodal pretraining**: Large-scale training on unified visionlanguageaction data to strengthen spatial, causal, and manipulation understanding.
+1
View File
@@ -94,6 +94,7 @@ from lerobot.rl.process import ProcessSignalHandler
from lerobot.robots import ( # noqa: F401 from lerobot.robots import ( # noqa: F401
Robot, Robot,
RobotConfig, RobotConfig,
bi_so_follower,
koch_follower, koch_follower,
so_follower, so_follower,
) )
+1 -2
View File
@@ -34,8 +34,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
InverseKinematicsEEToJoints, InverseKinematicsEEToJoints,
) )
from lerobot.scripts.lerobot_record import record_loop from lerobot.scripts.lerobot_record import record_loop
from lerobot.teleoperators.so_leader import SO100LeaderConfig from lerobot.teleoperators.so_leader import SO100Leader, SO100LeaderConfig
from lerobot.teleoperators.so_leader.so100_leader import SO100Leader
from lerobot.utils.control_utils import init_keyboard_listener from lerobot.utils.control_utils import init_keyboard_listener
from lerobot.utils.utils import log_say from lerobot.utils.utils import log_say
from lerobot.utils.visualization_utils import init_rerun from lerobot.utils.visualization_utils import init_rerun
+1 -2
View File
@@ -29,8 +29,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
ForwardKinematicsJointsToEE, ForwardKinematicsJointsToEE,
InverseKinematicsEEToJoints, InverseKinematicsEEToJoints,
) )
from lerobot.teleoperators.so_leader import SO100LeaderConfig from lerobot.teleoperators.so_leader import SO100Leader, SO100LeaderConfig
from lerobot.teleoperators.so_leader.so100_leader import SO100Leader
from lerobot.utils.robot_utils import precise_sleep from lerobot.utils.robot_utils import precise_sleep
from lerobot.utils.visualization_utils import init_rerun, log_rerun_data from lerobot.utils.visualization_utils import init_rerun, log_rerun_data
+1
View File
@@ -198,6 +198,7 @@ lerobot-setup-motors="lerobot.scripts.lerobot_setup_motors:main"
lerobot-teleoperate="lerobot.scripts.lerobot_teleoperate:main" lerobot-teleoperate="lerobot.scripts.lerobot_teleoperate:main"
lerobot-eval="lerobot.scripts.lerobot_eval:main" lerobot-eval="lerobot.scripts.lerobot_eval:main"
lerobot-train="lerobot.scripts.lerobot_train:main" lerobot-train="lerobot.scripts.lerobot_train:main"
lerobot-train-tokenizer="lerobot.scripts.lerobot_train_tokenizer:main"
lerobot-dataset-viz="lerobot.scripts.lerobot_dataset_viz:main" lerobot-dataset-viz="lerobot.scripts.lerobot_dataset_viz:main"
lerobot-info="lerobot.scripts.lerobot_info:main" lerobot-info="lerobot.scripts.lerobot_info:main"
lerobot-find-joint-limits="lerobot.scripts.lerobot_find_joint_limits:main" lerobot-find-joint-limits="lerobot.scripts.lerobot_find_joint_limits:main"
+1 -1
View File
@@ -26,4 +26,4 @@ DEFAULT_OBS_QUEUE_TIMEOUT = 2
SUPPORTED_POLICIES = ["act", "smolvla", "diffusion", "tdmpc", "vqbet", "pi0", "pi05"] SUPPORTED_POLICIES = ["act", "smolvla", "diffusion", "tdmpc", "vqbet", "pi0", "pi05"]
# TODO: Add all other robots # TODO: Add all other robots
SUPPORTED_ROBOTS = ["so100_follower", "so101_follower", "bi_so100_follower", "omx_follower"] SUPPORTED_ROBOTS = ["so100_follower", "so101_follower", "bi_so_follower", "omx_follower"]
+1 -1
View File
@@ -51,7 +51,7 @@ from lerobot.cameras.realsense.configuration_realsense import RealSenseCameraCon
from lerobot.robots import ( # noqa: F401 from lerobot.robots import ( # noqa: F401
Robot, Robot,
RobotConfig, RobotConfig,
bi_so100_follower, bi_so_follower,
koch_follower, koch_follower,
make_robot_from_config, make_robot_from_config,
omx_follower, omx_follower,
@@ -1,39 +0,0 @@
#!/usr/bin/env python
# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from dataclasses import dataclass, field
from lerobot.cameras import CameraConfig
from ..config import RobotConfig
@RobotConfig.register_subclass("bi_so100_follower")
@dataclass
class BiSO100FollowerConfig(RobotConfig):
left_arm_port: str
right_arm_port: str
# Optional
left_arm_disable_torque_on_disconnect: bool = True
left_arm_max_relative_target: float | dict[str, float] | None = None
left_arm_use_degrees: bool = False
right_arm_disable_torque_on_disconnect: bool = True
right_arm_max_relative_target: float | dict[str, float] | None = None
right_arm_use_degrees: bool = False
# cameras (shared between both arms)
cameras: dict[str, CameraConfig] = field(default_factory=dict)
@@ -1,6 +1,6 @@
#!/usr/bin/env python #!/usr/bin/env python
# Copyright 2025 The HuggingFace Inc. team. All rights reserved. # Copyright 2026 The HuggingFace Inc. team. All rights reserved.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@@ -14,5 +14,5 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from .bi_so100_leader import BiSO100Leader from .bi_so_follower import BiSOFollower
from .config_bi_so100_leader import BiSO100LeaderConfig from .config_bi_so_follower import BiSOFollowerConfig
@@ -15,66 +15,73 @@
# limitations under the License. # limitations under the License.
import logging import logging
import time
from functools import cached_property from functools import cached_property
from typing import Any from typing import Any
from lerobot.cameras.utils import make_cameras_from_configs from lerobot.robots.so_follower import SOFollower, SOFollowerRobotConfig
from lerobot.robots.so_follower import SO100Follower, SO100FollowerConfig
from ..robot import Robot from ..robot import Robot
from .config_bi_so100_follower import BiSO100FollowerConfig from .config_bi_so_follower import BiSOFollowerConfig
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class BiSO100Follower(Robot): class BiSOFollower(Robot):
""" """
[Bimanual SO-100 Follower Arms](https://github.com/TheRobotStudio/SO-ARM100) designed by TheRobotStudio [Bimanual SO Follower Arms](https://github.com/TheRobotStudio/SO-ARM100) designed by TheRobotStudio
This bimanual robot can also be easily adapted to use SO-101 follower arms, just replace the SO100Follower class with SO101Follower and SO100FollowerConfig with SO101FollowerConfig.
""" """
config_class = BiSO100FollowerConfig config_class = BiSOFollowerConfig
name = "bi_so100_follower" name = "bi_so_follower"
def __init__(self, config: BiSO100FollowerConfig): def __init__(self, config: BiSOFollowerConfig):
super().__init__(config) super().__init__(config)
self.config = config self.config = config
left_arm_config = SO100FollowerConfig( left_arm_config = SOFollowerRobotConfig(
id=f"{config.id}_left" if config.id else None, id=f"{config.id}_left" if config.id else None,
calibration_dir=config.calibration_dir, calibration_dir=config.calibration_dir,
port=config.left_arm_port, port=config.left_arm_config.port,
disable_torque_on_disconnect=config.left_arm_disable_torque_on_disconnect, disable_torque_on_disconnect=config.left_arm_config.disable_torque_on_disconnect,
max_relative_target=config.left_arm_max_relative_target, max_relative_target=config.left_arm_config.max_relative_target,
use_degrees=config.left_arm_use_degrees, use_degrees=config.left_arm_config.use_degrees,
cameras={}, cameras=config.left_arm_config.cameras,
) )
right_arm_config = SO100FollowerConfig( right_arm_config = SOFollowerRobotConfig(
id=f"{config.id}_right" if config.id else None, id=f"{config.id}_right" if config.id else None,
calibration_dir=config.calibration_dir, calibration_dir=config.calibration_dir,
port=config.right_arm_port, port=config.right_arm_config.port,
disable_torque_on_disconnect=config.right_arm_disable_torque_on_disconnect, disable_torque_on_disconnect=config.right_arm_config.disable_torque_on_disconnect,
max_relative_target=config.right_arm_max_relative_target, max_relative_target=config.right_arm_config.max_relative_target,
use_degrees=config.right_arm_use_degrees, use_degrees=config.right_arm_config.use_degrees,
cameras={}, cameras=config.right_arm_config.cameras,
) )
self.left_arm = SO100Follower(left_arm_config) self.left_arm = SOFollower(left_arm_config)
self.right_arm = SO100Follower(right_arm_config) self.right_arm = SOFollower(right_arm_config)
self.cameras = make_cameras_from_configs(config.cameras)
# Only for compatibility with other parts of the codebase that expect a `robot.cameras` attribute
self.cameras = {**self.left_arm.cameras, **self.right_arm.cameras}
@property @property
def _motors_ft(self) -> dict[str, type]: def _motors_ft(self) -> dict[str, type]:
return {f"left_{motor}.pos": float for motor in self.left_arm.bus.motors} | { left_arm_motors_ft = self.left_arm._motors_ft
f"right_{motor}.pos": float for motor in self.right_arm.bus.motors right_arm_motors_ft = self.right_arm._motors_ft
return {
**{f"left_{k}": v for k, v in left_arm_motors_ft.items()},
**{f"right_{k}": v for k, v in right_arm_motors_ft.items()},
} }
@property @property
def _cameras_ft(self) -> dict[str, tuple]: def _cameras_ft(self) -> dict[str, tuple]:
left_arm_cameras_ft = self.left_arm._cameras_ft
right_arm_cameras_ft = self.right_arm._cameras_ft
return { return {
cam: (self.config.cameras[cam].height, self.config.cameras[cam].width, 3) for cam in self.cameras **{f"left_{k}": v for k, v in left_arm_cameras_ft.items()},
**{f"right_{k}": v for k, v in right_arm_cameras_ft.items()},
} }
@cached_property @cached_property
@@ -87,19 +94,12 @@ class BiSO100Follower(Robot):
@property @property
def is_connected(self) -> bool: def is_connected(self) -> bool:
return ( return self.left_arm.is_connected and self.right_arm.is_connected
self.left_arm.bus.is_connected
and self.right_arm.bus.is_connected
and all(cam.is_connected for cam in self.cameras.values())
)
def connect(self, calibrate: bool = True) -> None: def connect(self, calibrate: bool = True) -> None:
self.left_arm.connect(calibrate) self.left_arm.connect(calibrate)
self.right_arm.connect(calibrate) self.right_arm.connect(calibrate)
for cam in self.cameras.values():
cam.connect()
@property @property
def is_calibrated(self) -> bool: def is_calibrated(self) -> bool:
return self.left_arm.is_calibrated and self.right_arm.is_calibrated return self.left_arm.is_calibrated and self.right_arm.is_calibrated
@@ -127,12 +127,6 @@ class BiSO100Follower(Robot):
right_obs = self.right_arm.get_observation() right_obs = self.right_arm.get_observation()
obs_dict.update({f"right_{key}": value for key, value in right_obs.items()}) obs_dict.update({f"right_{key}": value for key, value in right_obs.items()})
for cam_key, cam in self.cameras.items():
start = time.perf_counter()
obs_dict[cam_key] = cam.async_read()
dt_ms = (time.perf_counter() - start) * 1e3
logger.debug(f"{self} read {cam_key}: {dt_ms:.1f}ms")
return obs_dict return obs_dict
def send_action(self, action: dict[str, Any]) -> dict[str, Any]: def send_action(self, action: dict[str, Any]) -> dict[str, Any]:
@@ -145,18 +139,15 @@ class BiSO100Follower(Robot):
key.removeprefix("right_"): value for key, value in action.items() if key.startswith("right_") key.removeprefix("right_"): value for key, value in action.items() if key.startswith("right_")
} }
send_action_left = self.left_arm.send_action(left_action) sent_action_left = self.left_arm.send_action(left_action)
send_action_right = self.right_arm.send_action(right_action) sent_action_right = self.right_arm.send_action(right_action)
# Add prefixes back # Add prefixes back
prefixed_send_action_left = {f"left_{key}": value for key, value in send_action_left.items()} prefixed_sent_action_left = {f"left_{key}": value for key, value in sent_action_left.items()}
prefixed_send_action_right = {f"right_{key}": value for key, value in send_action_right.items()} prefixed_sent_action_right = {f"right_{key}": value for key, value in sent_action_right.items()}
return {**prefixed_send_action_left, **prefixed_send_action_right} return {**prefixed_sent_action_left, **prefixed_sent_action_right}
def disconnect(self): def disconnect(self):
self.left_arm.disconnect() self.left_arm.disconnect()
self.right_arm.disconnect() self.right_arm.disconnect()
for cam in self.cameras.values():
cam.disconnect()
@@ -16,11 +16,15 @@
from dataclasses import dataclass from dataclasses import dataclass
from ...config import TeleoperatorConfig from lerobot.robots.so_follower import SOFollowerConfig
from ..so_leader_config_base import SOLeaderConfigBase
from ..config import RobotConfig
@TeleoperatorConfig.register_subclass("so101_leader") @RobotConfig.register_subclass("bi_so_follower")
@dataclass @dataclass
class SO101LeaderConfig(SOLeaderConfigBase): class BiSOFollowerConfig(RobotConfig):
pass """Configuration class for Bi SO Follower robots."""
left_arm_config: SOFollowerConfig
right_arm_config: SOFollowerConfig
+7 -7
View File
@@ -14,10 +14,10 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from .config_so_follower import (
from .so100_follower.config_so100_follower import SO100FollowerConfig SO100FollowerConfig,
from .so100_follower.so100_follower import SO100Follower SO101FollowerConfig,
from .so101_follower.config_so101_follower import SO101FollowerConfig SOFollowerConfig,
from .so101_follower.so101_follower import SO101Follower SOFollowerRobotConfig,
from .so_follower_base import SOFollowerBase )
from .so_follower_config_base import SOFollowerConfigBase from .so_follower import SO100Follower, SO101Follower, SOFollower
@@ -15,6 +15,7 @@
# limitations under the License. # limitations under the License.
from dataclasses import dataclass, field from dataclasses import dataclass, field
from typing import TypeAlias
from lerobot.cameras import CameraConfig from lerobot.cameras import CameraConfig
@@ -22,7 +23,7 @@ from ..config import RobotConfig
@dataclass @dataclass
class SOFollowerConfigBase(RobotConfig): class SOFollowerConfig:
"""Base configuration class for SO Follower robots.""" """Base configuration class for SO Follower robots."""
# Port to connect to the arm # Port to connect to the arm
@@ -40,3 +41,14 @@ class SOFollowerConfigBase(RobotConfig):
# Set to `True` for backward compatibility with previous policies/dataset # Set to `True` for backward compatibility with previous policies/dataset
use_degrees: bool = False use_degrees: bool = False
@RobotConfig.register_subclass("so101_follower")
@RobotConfig.register_subclass("so100_follower")
@dataclass
class SOFollowerRobotConfig(RobotConfig, SOFollowerConfig):
pass
SO100FollowerConfig: TypeAlias = SOFollowerRobotConfig
SO101FollowerConfig: TypeAlias = SOFollowerRobotConfig
+1
View File
@@ -0,0 +1 @@
../../../../docs/source/so100.mdx
@@ -1,26 +0,0 @@
#!/usr/bin/env python
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from dataclasses import dataclass
from ...config import RobotConfig
from ..so_follower_config_base import SOFollowerConfigBase
@RobotConfig.register_subclass("so100_follower")
@dataclass
class SO100FollowerConfig(SOFollowerConfigBase):
pass
@@ -1 +0,0 @@
../../../../../docs/source/so100.mdx
@@ -1,27 +0,0 @@
#!/usr/bin/env python
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from ..so_follower_base import SOFollowerBase
from .config_so100_follower import SO100FollowerConfig
class SO100Follower(SOFollowerBase):
"""
SO-101 follower robot class. [SO-101 Follower Arm](https://github.com/TheRobotStudio/SO-ARM100) designed by TheRobotStudio
"""
config_class = SO100FollowerConfig
name = "so100_follower"
+1
View File
@@ -0,0 +1 @@
../../../../docs/source/so101.mdx
@@ -1,26 +0,0 @@
#!/usr/bin/env python
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from dataclasses import dataclass
from ...config import RobotConfig
from ..so_follower_config_base import SOFollowerConfigBase
@RobotConfig.register_subclass("so101_follower")
@dataclass
class SO101FollowerConfig(SOFollowerConfigBase):
pass
@@ -1 +0,0 @@
../../../../../docs/source/so101.mdx
@@ -1,27 +0,0 @@
#!/usr/bin/env python
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from ..so_follower_base import SOFollowerBase
from .config_so101_follower import SO101FollowerConfig
class SO101Follower(SOFollowerBase):
"""
SO-101 follower robot class. [SO-101 Follower Arm](https://github.com/TheRobotStudio/SO-ARM100) designed by TheRobotStudio
"""
config_class = SO101FollowerConfig
name = "so101_follower"
@@ -17,7 +17,7 @@
import logging import logging
import time import time
from functools import cached_property from functools import cached_property
from typing import Any from typing import Any, TypeAlias
from lerobot.cameras.utils import make_cameras_from_configs from lerobot.cameras.utils import make_cameras_from_configs
from lerobot.motors import Motor, MotorCalibration, MotorNormMode from lerobot.motors import Motor, MotorCalibration, MotorNormMode
@@ -29,20 +29,21 @@ from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnected
from ..robot import Robot from ..robot import Robot
from ..utils import ensure_safe_goal_position from ..utils import ensure_safe_goal_position
from .so_follower_config_base import SOFollowerConfigBase from .config_so_follower import SOFollowerRobotConfig
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class SOFollowerBase(Robot): class SOFollower(Robot):
""" """
Generic SO follower base implementing common functionality for SO-100/101/10X. Generic SO follower base implementing common functionality for SO-100/101/10X.
Designed to be subclassed with a per-hardware-model `config_class` and `name`. Designed to be subclassed with a per-hardware-model `config_class` and `name`.
""" """
# `config_class` and `name` should be set by subclasses config_class = SOFollowerRobotConfig
name = "so_follower"
def __init__(self, config: SOFollowerConfigBase): def __init__(self, config: SOFollowerRobotConfig):
super().__init__(config) super().__init__(config)
self.config = config self.config = config
# choose normalization mode depending on config if available # choose normalization mode depending on config if available
@@ -232,3 +233,7 @@ class SOFollowerBase(Robot):
cam.disconnect() cam.disconnect()
logger.info(f"{self} disconnected.") logger.info(f"{self} disconnected.")
SO100Follower: TypeAlias = SOFollower
SO101Follower: TypeAlias = SOFollower
+3 -3
View File
@@ -52,10 +52,10 @@ def make_robot_from_config(config: RobotConfig) -> Robot:
from .hope_jr import HopeJrArm from .hope_jr import HopeJrArm
return HopeJrArm(config) return HopeJrArm(config)
elif config.type == "bi_so100_follower": elif config.type == "bi_so_follower":
from .bi_so100_follower import BiSO100Follower from .bi_so_follower import BiSOFollower
return BiSO100Follower(config) return BiSOFollower(config)
elif config.type == "reachy2": elif config.type == "reachy2":
from .reachy2 import Reachy2Robot from .reachy2 import Reachy2Robot
+2
View File
@@ -36,6 +36,7 @@ from lerobot.cameras.realsense.configuration_realsense import RealSenseCameraCon
from lerobot.robots import ( # noqa: F401 from lerobot.robots import ( # noqa: F401
Robot, Robot,
RobotConfig, RobotConfig,
bi_so_follower,
hope_jr, hope_jr,
koch_follower, koch_follower,
lekiwi, lekiwi,
@@ -46,6 +47,7 @@ from lerobot.robots import ( # noqa: F401
from lerobot.teleoperators import ( # noqa: F401 from lerobot.teleoperators import ( # noqa: F401
Teleoperator, Teleoperator,
TeleoperatorConfig, TeleoperatorConfig,
bi_so_leader,
homunculus, homunculus,
koch_leader, koch_leader,
make_teleoperator_from_config, make_teleoperator_from_config,
@@ -44,6 +44,7 @@ import numpy as np
from lerobot.model.kinematics import RobotKinematics from lerobot.model.kinematics import RobotKinematics
from lerobot.robots import ( # noqa: F401 from lerobot.robots import ( # noqa: F401
RobotConfig, RobotConfig,
bi_so_follower,
koch_follower, koch_follower,
make_robot_from_config, make_robot_from_config,
omx_follower, omx_follower,
@@ -51,6 +52,7 @@ from lerobot.robots import ( # noqa: F401
) )
from lerobot.teleoperators import ( # noqa: F401 from lerobot.teleoperators import ( # noqa: F401
TeleoperatorConfig, TeleoperatorConfig,
bi_so_leader,
gamepad, gamepad,
koch_leader, koch_leader,
make_teleoperator_from_config, make_teleoperator_from_config,
+15 -13
View File
@@ -40,21 +40,23 @@ lerobot-record \
Example recording with bimanual so100: Example recording with bimanual so100:
```shell ```shell
lerobot-record \ lerobot-record \
--robot.type=bi_so100_follower \ --robot.type=bi_so_follower \
--robot.left_arm_port=/dev/tty.usbmodem5A460851411 \ --robot.left_arm_config.port=/dev/tty.usbmodem5A460822851 \
--robot.right_arm_port=/dev/tty.usbmodem5A460812391 \ --robot.right_arm_config.port=/dev/tty.usbmodem5A460814411 \
--robot.id=bimanual_follower \ --robot.id=bimanual_follower \
--robot.cameras='{ --robot.left_arm_config.cameras='{
left: {"type": "opencv", "index_or_path": 0, "width": 640, "height": 480, "fps": 30}, wrist: {"type": "opencv", "index_or_path": 1, "width": 640, "height": 480, "fps": 30},
top: {"type": "opencv", "index_or_path": 1, "width": 640, "height": 480, "fps": 30}, top: {"type": "opencv", "index_or_path": 3, "width": 640, "height": 480, "fps": 30},
right: {"type": "opencv", "index_or_path": 2, "width": 640, "height": 480, "fps": 30} }' --robot.right_arm_config.cameras='{
wrist: {"type": "opencv", "index_or_path": 2, "width": 640, "height": 480, "fps": 30},
front: {"type": "opencv", "index_or_path": 4, "width": 640, "height": 480, "fps": 30},
}' \ }' \
--teleop.type=bi_so100_leader \ --teleop.type=bi_so_leader \
--teleop.left_arm_port=/dev/tty.usbmodem5A460828611 \ --teleop.left_arm_config.port=/dev/tty.usbmodem5A460852721 \
--teleop.right_arm_port=/dev/tty.usbmodem5A460826981 \ --teleop.right_arm_config.port=/dev/tty.usbmodem5A460819811 \
--teleop.id=bimanual_leader \ --teleop.id=bimanual_leader \
--display_data=true \ --display_data=true \
--dataset.repo_id=${HF_USER}/bimanual-so100-handover-cube \ --dataset.repo_id=${HF_USER}/bimanual-so-handover-cube \
--dataset.num_episodes=25 \ --dataset.num_episodes=25 \
--dataset.single_task="Grab and handover the red cube to the other arm" --dataset.single_task="Grab and handover the red cube to the other arm"
``` ```
@@ -94,7 +96,7 @@ from lerobot.processor.rename_processor import rename_stats
from lerobot.robots import ( # noqa: F401 from lerobot.robots import ( # noqa: F401
Robot, Robot,
RobotConfig, RobotConfig,
bi_so100_follower, bi_so_follower,
earthrover_mini_plus, earthrover_mini_plus,
hope_jr, hope_jr,
koch_follower, koch_follower,
@@ -105,7 +107,7 @@ from lerobot.robots import ( # noqa: F401
from lerobot.teleoperators import ( # noqa: F401 from lerobot.teleoperators import ( # noqa: F401
Teleoperator, Teleoperator,
TeleoperatorConfig, TeleoperatorConfig,
bi_so100_leader, bi_so_leader,
homunculus, homunculus,
koch_leader, koch_leader,
make_teleoperator_from_config, make_teleoperator_from_config,
+2 -2
View File
@@ -29,7 +29,7 @@ lerobot-replay \
Example replay with bimanual so100: Example replay with bimanual so100:
```shell ```shell
lerobot-replay \ lerobot-replay \
--robot.type=bi_so100_follower \ --robot.type=bi_so_follower \
--robot.left_arm_port=/dev/tty.usbmodem5A460851411 \ --robot.left_arm_port=/dev/tty.usbmodem5A460851411 \
--robot.right_arm_port=/dev/tty.usbmodem5A460812391 \ --robot.right_arm_port=/dev/tty.usbmodem5A460812391 \
--robot.id=bimanual_follower \ --robot.id=bimanual_follower \
@@ -53,7 +53,7 @@ from lerobot.processor import (
from lerobot.robots import ( # noqa: F401 from lerobot.robots import ( # noqa: F401
Robot, Robot,
RobotConfig, RobotConfig,
bi_so100_follower, bi_so_follower,
earthrover_mini_plus, earthrover_mini_plus,
hope_jr, hope_jr,
koch_follower, koch_follower,
@@ -30,6 +30,7 @@ import draccus
from lerobot.robots import ( # noqa: F401 from lerobot.robots import ( # noqa: F401
RobotConfig, RobotConfig,
bi_so_follower,
koch_follower, koch_follower,
lekiwi, lekiwi,
make_robot_from_config, make_robot_from_config,
@@ -38,6 +39,7 @@ from lerobot.robots import ( # noqa: F401
) )
from lerobot.teleoperators import ( # noqa: F401 from lerobot.teleoperators import ( # noqa: F401
TeleoperatorConfig, TeleoperatorConfig,
bi_so_leader,
koch_leader, koch_leader,
make_teleoperator_from_config, make_teleoperator_from_config,
omx_leader, omx_leader,
+12 -12
View File
@@ -33,18 +33,18 @@ Example teleoperation with bimanual so100:
```shell ```shell
lerobot-teleoperate \ lerobot-teleoperate \
--robot.type=bi_so100_follower \ --robot.type=bi_so_follower \
--robot.left_arm_port=/dev/tty.usbmodem5A460851411 \ --robot.left_arm_config.port=/dev/tty.usbmodem5A460822851 \
--robot.right_arm_port=/dev/tty.usbmodem5A460812391 \ --robot.right_arm_config.port=/dev/tty.usbmodem5A460814411 \
--robot.id=bimanual_follower \ --robot.id=bimanual_follower \
--robot.cameras='{ --robot.left_arm_config.cameras='{
left: {"type": "opencv", "index_or_path": 0, "width": 1920, "height": 1080, "fps": 30}, wrist: {"type": "opencv", "index_or_path": 1, "width": 640, "height": 480, "fps": 30},
top: {"type": "opencv", "index_or_path": 1, "width": 1920, "height": 1080, "fps": 30}, }' --robot.right_arm_config.cameras='{
right: {"type": "opencv", "index_or_path": 2, "width": 1920, "height": 1080, "fps": 30} wrist: {"type": "opencv", "index_or_path": 2, "width": 640, "height": 480, "fps": 30},
}' \ }' \
--teleop.type=bi_so100_leader \ --teleop.type=bi_so_leader \
--teleop.left_arm_port=/dev/tty.usbmodem5A460828611 \ --teleop.left_arm_config.port=/dev/tty.usbmodem5A460852721 \
--teleop.right_arm_port=/dev/tty.usbmodem5A460826981 \ --teleop.right_arm_config.port=/dev/tty.usbmodem5A460819811 \
--teleop.id=bimanual_leader \ --teleop.id=bimanual_leader \
--display_data=true --display_data=true
``` ```
@@ -70,7 +70,7 @@ from lerobot.processor import (
from lerobot.robots import ( # noqa: F401 from lerobot.robots import ( # noqa: F401
Robot, Robot,
RobotConfig, RobotConfig,
bi_so100_follower, bi_so_follower,
earthrover_mini_plus, earthrover_mini_plus,
hope_jr, hope_jr,
koch_follower, koch_follower,
@@ -81,7 +81,7 @@ from lerobot.robots import ( # noqa: F401
from lerobot.teleoperators import ( # noqa: F401 from lerobot.teleoperators import ( # noqa: F401
Teleoperator, Teleoperator,
TeleoperatorConfig, TeleoperatorConfig,
bi_so100_leader, bi_so_leader,
gamepad, gamepad,
homunculus, homunculus,
keyboard, keyboard,
+8 -1
View File
@@ -259,7 +259,14 @@ def train(cfg: TrainPipelineConfig, accelerator: Accelerator | None = None):
from accelerate.utils import DistributedDataParallelKwargs from accelerate.utils import DistributedDataParallelKwargs
ddp_kwargs = DistributedDataParallelKwargs(find_unused_parameters=True) ddp_kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)
accelerator = Accelerator(step_scheduler_with_optimizer=False, kwargs_handlers=[ddp_kwargs]) # Accelerate auto-detects the device based on the available hardware and ignores the policy.device setting.
# Force the device to be CPU when policy.device is set to CPU.
force_cpu = cfg.policy.device == "cpu"
accelerator = Accelerator(
step_scheduler_with_optimizer=False,
kwargs_handlers=[ddp_kwargs],
cpu=force_cpu,
)
init_logging(accelerator=accelerator) init_logging(accelerator=accelerator)
@@ -1,3 +1,16 @@
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Train FAST tokenizer for action encoding. """Train FAST tokenizer for action encoding.
This script: This script:
@@ -6,6 +19,26 @@ This script:
3. Trains FAST tokenizer on specified action dimensions 3. Trains FAST tokenizer on specified action dimensions
4. Saves tokenizer to assets directory 4. Saves tokenizer to assets directory
5. Reports compression statistics 5. Reports compression statistics
Example:
```shell
lerobot-train-tokenizer \
--repo_id=user/dataset_name \
--action_horizon=10 \
--max_episodes=100 \
--sample_fraction=0.1 \
--encoded_dims="0:6" \
--delta_dims="0,1,2,3,4,5" \
--use_delta_transform=true \
--state_key="observation.state" \
--normalization_mode="QUANTILES" \
--vocab_size=1024 \
--scale=10.0 \
--output_dir="./fast_tokenizer_dataset_name" \
--push_to_hub=true \
--hub_repo_id="user/fast_tokenizer_dataset_name" \
--hub_private=false
""" """
import json import json
@@ -1,6 +1,6 @@
#!/usr/bin/env python #!/usr/bin/env python
# Copyright 2025 The HuggingFace Inc. team. All rights reserved. # Copyright 2026 The HuggingFace Inc. team. All rights reserved.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@@ -14,5 +14,4 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from .bi_so100_follower import BiSO100Follower from .bi_so_leader import BiSOLeader, BiSOLeaderConfig
from .config_bi_so100_follower import BiSO100FollowerConfig
@@ -17,46 +17,50 @@
import logging import logging
from functools import cached_property from functools import cached_property
from lerobot.teleoperators.so_leader import SO100Leader, SO100LeaderConfig from lerobot.teleoperators.so_leader import SOLeaderTeleopConfig
from ..so_leader import SOLeader
from ..teleoperator import Teleoperator from ..teleoperator import Teleoperator
from .config_bi_so100_leader import BiSO100LeaderConfig from .config_bi_so_leader import BiSOLeaderConfig
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class BiSO100Leader(Teleoperator): class BiSOLeader(Teleoperator):
""" """
[Bimanual SO-100 Leader Arms](https://github.com/TheRobotStudio/SO-ARM100) designed by TheRobotStudio [Bimanual SO Leader Arms](https://github.com/TheRobotStudio/SO-ARM100) designed by TheRobotStudio
This bimanual leader arm can also be easily adapted to use SO-101 leader arms, just replace the SO100Leader class with SO101Leader and SO100LeaderConfig with SO101LeaderConfig.
""" """
config_class = BiSO100LeaderConfig config_class = BiSOLeaderConfig
name = "bi_so100_leader" name = "bi_so_leader"
def __init__(self, config: BiSO100LeaderConfig): def __init__(self, config: BiSOLeaderConfig):
super().__init__(config) super().__init__(config)
self.config = config self.config = config
left_arm_config = SO100LeaderConfig( left_arm_config = SOLeaderTeleopConfig(
id=f"{config.id}_left" if config.id else None, id=f"{config.id}_left" if config.id else None,
calibration_dir=config.calibration_dir, calibration_dir=config.calibration_dir,
port=config.left_arm_port, port=config.left_arm_config.port,
) )
right_arm_config = SO100LeaderConfig( right_arm_config = SOLeaderTeleopConfig(
id=f"{config.id}_right" if config.id else None, id=f"{config.id}_right" if config.id else None,
calibration_dir=config.calibration_dir, calibration_dir=config.calibration_dir,
port=config.right_arm_port, port=config.right_arm_config.port,
) )
self.left_arm = SO100Leader(left_arm_config) self.left_arm = SOLeader(left_arm_config)
self.right_arm = SO100Leader(right_arm_config) self.right_arm = SOLeader(right_arm_config)
@cached_property @cached_property
def action_features(self) -> dict[str, type]: def action_features(self) -> dict[str, type]:
return {f"left_{motor}.pos": float for motor in self.left_arm.bus.motors} | { left_arm_features = self.left_arm.action_features
f"right_{motor}.pos": float for motor in self.right_arm.bus.motors right_arm_features = self.right_arm.action_features
return {
**{f"left_{k}": v for k, v in left_arm_features.items()},
**{f"right_{k}": v for k, v in right_arm_features.items()},
} }
@cached_property @cached_property
@@ -101,19 +105,8 @@ class BiSO100Leader(Teleoperator):
return action_dict return action_dict
def send_feedback(self, feedback: dict[str, float]) -> None: def send_feedback(self, feedback: dict[str, float]) -> None:
# Remove "left_" prefix # TODO: Implement force feedback
left_feedback = { raise NotImplementedError
key.removeprefix("left_"): value for key, value in feedback.items() if key.startswith("left_")
}
# Remove "right_" prefix
right_feedback = {
key.removeprefix("right_"): value for key, value in feedback.items() if key.startswith("right_")
}
if left_feedback:
self.left_arm.send_feedback(left_feedback)
if right_feedback:
self.right_arm.send_feedback(right_feedback)
def disconnect(self) -> None: def disconnect(self) -> None:
self.left_arm.disconnect() self.left_arm.disconnect()
@@ -16,11 +16,15 @@
from dataclasses import dataclass from dataclasses import dataclass
from lerobot.teleoperators.so_leader import SOLeaderConfig
from ..config import TeleoperatorConfig from ..config import TeleoperatorConfig
@TeleoperatorConfig.register_subclass("bi_so100_leader") @TeleoperatorConfig.register_subclass("bi_so_leader")
@dataclass @dataclass
class BiSO100LeaderConfig(TeleoperatorConfig): class BiSOLeaderConfig(TeleoperatorConfig):
left_arm_port: str """Configuration class for Bi SO Leader teleoperators."""
right_arm_port: str
left_arm_config: SOLeaderConfig
right_arm_config: SOLeaderConfig
@@ -14,9 +14,10 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from .so100_leader.config_so100_leader import SO100LeaderConfig from .config_so_leader import (
from .so100_leader.so100_leader import SO100Leader SO100LeaderConfig,
from .so101_leader.config_so101_leader import SO101LeaderConfig SO101LeaderConfig,
from .so101_leader.so101_leader import SO101Leader SOLeaderConfig,
from .so_leader_base import SOLeaderBase SOLeaderTeleopConfig,
from .so_leader_config_base import SOLeaderConfigBase )
from .so_leader import SO100Leader, SO101Leader, SOLeader
@@ -15,12 +15,13 @@
# limitations under the License. # limitations under the License.
from dataclasses import dataclass from dataclasses import dataclass
from typing import TypeAlias
from ..config import TeleoperatorConfig from ..config import TeleoperatorConfig
@dataclass @dataclass
class SOLeaderConfigBase(TeleoperatorConfig): class SOLeaderConfig:
"""Base configuration class for SO Leader teleoperators.""" """Base configuration class for SO Leader teleoperators."""
# Port to connect to the arm # Port to connect to the arm
@@ -28,3 +29,14 @@ class SOLeaderConfigBase(TeleoperatorConfig):
# Whether to use degrees for angles # Whether to use degrees for angles
use_degrees: bool = False use_degrees: bool = False
@TeleoperatorConfig.register_subclass("so101_leader")
@TeleoperatorConfig.register_subclass("so100_leader")
@dataclass
class SOLeaderTeleopConfig(TeleoperatorConfig, SOLeaderConfig):
pass
SO100LeaderConfig: TypeAlias = SOLeaderTeleopConfig
SO101LeaderConfig: TypeAlias = SOLeaderTeleopConfig
+1
View File
@@ -0,0 +1 @@
../../../../docs/source/so100.mdx
@@ -1,26 +0,0 @@
#!/usr/bin/env python
# Copyright 2024 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from dataclasses import dataclass
from ...config import TeleoperatorConfig
from ..so_leader_config_base import SOLeaderConfigBase
@TeleoperatorConfig.register_subclass("so100_leader")
@dataclass
class SO100LeaderConfig(SOLeaderConfigBase):
pass
@@ -1 +0,0 @@
../../../../../docs/source/so100.mdx
@@ -1,27 +0,0 @@
# !/usr/bin/env python
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from ..so_leader_base import SOLeaderBase
from .config_so100_leader import SO100LeaderConfig
class SO100Leader(SOLeaderBase):
"""
SO-101 leader robot class. [SO-101 Leader Arm](https://github.com/TheRobotStudio/SO-ARM100) designed by TheRobotStudio
"""
config_class = SO100LeaderConfig
name = "so100_leader"
+1
View File
@@ -0,0 +1 @@
../../../../docs/source/so101.mdx
@@ -1 +0,0 @@
../../../../../docs/source/so101.mdx
@@ -1,27 +0,0 @@
# !/usr/bin/env python
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from ..so_leader_base import SOLeaderBase
from .config_so101_leader import SO101LeaderConfig
class SO101Leader(SOLeaderBase):
"""
SO-101 leader robot class. [SO-101 Leader Arm](https://github.com/TheRobotStudio/SO-ARM100) designed by TheRobotStudio
"""
config_class = SO101LeaderConfig
name = "so101_leader"
@@ -16,6 +16,7 @@
import logging import logging
import time import time
from typing import TypeAlias
from lerobot.motors import Motor, MotorCalibration, MotorNormMode from lerobot.motors import Motor, MotorCalibration, MotorNormMode
from lerobot.motors.feetech import ( from lerobot.motors.feetech import (
@@ -25,15 +26,18 @@ from lerobot.motors.feetech import (
from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError
from ..teleoperator import Teleoperator from ..teleoperator import Teleoperator
from .so_leader_config_base import SOLeaderConfigBase from .config_so_leader import SOLeaderTeleopConfig
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class SOLeaderBase(Teleoperator): class SOLeader(Teleoperator):
"""Generic SO leader base for SO-100/101/10X teleoperators.""" """Generic SO leader base for SO-100/101/10X teleoperators."""
def __init__(self, config: SOLeaderConfigBase): config_class = SOLeaderTeleopConfig
name = "so_leader"
def __init__(self, config: SOLeaderTeleopConfig):
super().__init__(config) super().__init__(config)
self.config = config self.config = config
norm_mode_body = MotorNormMode.DEGREES if config.use_degrees else MotorNormMode.RANGE_M100_100 norm_mode_body = MotorNormMode.DEGREES if config.use_degrees else MotorNormMode.RANGE_M100_100
@@ -153,3 +157,7 @@ class SOLeaderBase(Teleoperator):
self.bus.disconnect() self.bus.disconnect()
logger.info(f"{self} disconnected.") logger.info(f"{self} disconnected.")
SO100Leader: TypeAlias = SOLeader
SO101Leader: TypeAlias = SOLeader
+3 -3
View File
@@ -73,10 +73,10 @@ def make_teleoperator_from_config(config: TeleoperatorConfig) -> Teleoperator:
from .homunculus import HomunculusArm from .homunculus import HomunculusArm
return HomunculusArm(config) return HomunculusArm(config)
elif config.type == "bi_so100_leader": elif config.type == "bi_so_leader":
from .bi_so100_leader import BiSO100Leader from .bi_so_leader import BiSOLeader
return BiSO100Leader(config) return BiSOLeader(config)
elif config.type == "reachy2_teleoperator": elif config.type == "reachy2_teleoperator":
from .reachy2_teleoperator import Reachy2Teleoperator from .reachy2_teleoperator import Reachy2Teleoperator
+1 -1
View File
@@ -66,7 +66,7 @@ def follower():
with ( with (
patch( patch(
"lerobot.robots.so_follower.so_follower_base.FeetechMotorsBus", "lerobot.robots.so_follower.so_follower.FeetechMotorsBus",
side_effect=_bus_side_effect, side_effect=_bus_side_effect,
), ),
patch.object(SO100Follower, "configure", lambda self: None), patch.object(SO100Follower, "configure", lambda self: None),