From 5ebbdf3d0573e66bce7cbb28fb43757fc04a802f Mon Sep 17 00:00:00 2001 From: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com> Date: Mon, 18 May 2026 14:51:26 +0200 Subject: [PATCH 01/17] Mention the new Lance LeRobotDataset implementation in the docs (#3609) * Enhance documentation with Lance format details Added information about Lance format and `lerobot-lancedb` package for multimodal AI datasets. Signed-off-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com> --- docs/source/lerobot-dataset-v3.mdx | 37 ++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/source/lerobot-dataset-v3.mdx b/docs/source/lerobot-dataset-v3.mdx index 6f3e6d948..c23677d8c 100644 --- a/docs/source/lerobot-dataset-v3.mdx +++ b/docs/source/lerobot-dataset-v3.mdx @@ -10,6 +10,7 @@ This docs will guide you to: - Stream datasets without downloading using `StreamingLeRobotDataset` - Apply image transforms for data augmentation during training - Migrate existing `v2.1` datasets to `v3.0` +- Experiment with other `LeRobotDataset` formats and implementations like Lance ## What’s new in `v3` @@ -315,3 +316,39 @@ Dataset v3.0 uses incremental parquet writing with buffered metadata for efficie - Ensures the dataset is valid for loading Without calling `finalize()`, your parquet files will be incomplete and the dataset won't load properly. + +## Other formats and implementations + +### Lance + +Lance is a useful format for multimodal AI datasets, especially for large-scale training requiring high performance IO and random access. + +The `lerobot-lancedb` package implements `LeRobotLanceDataset` (for JPEG images) and `LeRobotLanceVideoDataset` (for mp4 videos). +Those two storage layouts both subclass LeRobotDataset and can provide data loading speed ups. + +`LeRobotLanceDataset` is a drop-in replacement for `LeRobotDataset`: + +```python +from lerobot.datasets import LeRobotDatasetMetadata +from lerobot.policies.diffusion.configuration_diffusion import DiffusionConfig +from lerobot_lancedb import LeRobotLanceDataset, LeRobotLanceVideoDataset + +cfg = DiffusionConfig(...) +meta = LeRobotDatasetMetadata(root=local_dataset_path) # or use repo_id=... to load metadata from the Hub +delta_timestamps = {...} + +# Use LeRobotLanceDataset for image datasets +dataset = LeRobotLanceDataset( + root=local_dataset_path, # or use repo_id=... to stream from the Hub + delta_timestamps=delta_timestamps, + return_uint8=True, +) +# Or use LeRobotLanceVideoDataset for video datasets: +dataset = LeRobotLanceVideoDataset( + root=local_dataset_path, # or use repo_id=... to stream from the Hub + delta_timestamps=delta_timestamps, + return_uint8=True, +) +``` + +Join the discussion on [Github](https://github.com/huggingface/lerobot/issues/3608) and explore the `lerobot-lancedb` documentation [here](https://lancedb.github.io/lerobot-lancedb/). From 3c15fd8537c7e84c0465d07d4aa8c6de2912f640 Mon Sep 17 00:00:00 2001 From: Pepijn <138571049+pkooij@users.noreply.github.com> Date: Mon, 18 May 2026 19:49:21 +0200 Subject: [PATCH 02/17] feat(robots): natively integrate Seeed Studio reBot B601-DM arm (#3624) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(robots): natively integrate Seeed Studio reBot B601-DM arm Add first-class LeRobot support for the Seeed Studio reBot arm, replacing the out-of-tree `lerobot-robot-seeed-b601` / `lerobot-teleoperator-rebot-arm-102` plugin packages. New devices: - robot `rebot_b601_follower` — single-arm B601-DM follower (6-DOF + gripper, Damiao CAN motors via `motorbridge`) - robot `bi_rebot_b601_follower` — bimanual follower composing two single arms - teleoperator `rebot_102_leader` — single-arm StarArm102 / reBot Arm 102 leader (FashionStar UART servos via `motorbridge-smart-servo`) - teleoperator `bi_rebot_102_leader` — bimanual leader composing two single arms The bimanual variants reuse the single-arm classes and namespace each arm's observation/action keys with `left_` / `right_` prefixes, so a bimanual StarArm102 leader can teleoperate a bimanual reBot B601 follower. Optional SDK imports are guarded; a `rebot` extra installs `motorbridge` and `motorbridge-smart-servo`. Co-Authored-By: Claude Opus 4.7 (1M context) * docs: add reBot B601-DM calibration & dual-arm teleoperation guide Add docs/source/rebot_b601.mdx covering single-arm and bimanual calibration and teleoperation for the reBot B601-DM follower and reBot Arm 102 leader, with zero-position reference images from the Seeed Studio wiki. Register the page in the docs toctree. Co-Authored-By: Claude Opus 4.7 (1M context) * docs: fix reBot B601 MDX build (move JSON example out of ) The doc-builder parses `{...}` inside MDX component children as a Svelte expression, so the joint_directions JSON example broke the build. Move it into a top-level fenced code block. Co-Authored-By: Claude Opus 4.7 (1M context) * docs: apply prettier formatting to reBot B601 page Co-Authored-By: Claude Opus 4.7 (1M context) * docs: remove duplicate colocated reBot B601 page docs/source/rebot_b601.mdx is the canonical, toctree-registered page; the colocated rebot_b601.md was a redundant thinner copy. Co-Authored-By: Claude Opus 4.7 (1M context) * docs: clarify 6-DOF leader fallback comment in reBot B601 follower Explain that holding wrist_yaw at zero is what lets a 6-DOF leader (e.g. so100_leader / so101_leader) teleoperate the 7-DOF follower. Co-Authored-By: Claude Opus 4.7 (1M context) * refactor: address Caroline's PR review on reBot B601 integration - leader: remove _validate_config (no other lerobot device validates its config; a key mismatch now surfaces as a plain KeyError) - leader: simplify _round_to_valid_range to direct modular arithmetic instead of a bidirectional search loop - leader: inline the single-use _clamp helper - follower & leader: write MotorCalibration range_min/range_max from the configured joint_limits / joint_ranges instead of a fixed [-90, 90] - docs: add a "Find the USB ports" section (lerobot-find-port) and move the brltty/permissions tip there; link the OpenArm page for SocketCAN adapter configuration Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Claude Opus 4.7 (1M context) --- docs/source/_toctree.yml | 2 + docs/source/rebot_b601.mdx | 186 +++++++++++ pyproject.toml | 6 + .../robots/bi_rebot_b601_follower/__init__.py | 20 ++ .../bi_rebot_b601_follower.py | 150 +++++++++ .../config_bi_rebot_b601_follower.py | 29 ++ .../robots/rebot_b601_follower/__init__.py | 20 ++ .../config_rebot_b601_follower.py | 94 ++++++ .../rebot_b601_follower.py | 289 ++++++++++++++++++ src/lerobot/robots/utils.py | 8 + src/lerobot/scripts/lerobot_calibrate.py | 4 + .../scripts/lerobot_find_joint_limits.py | 4 + src/lerobot/scripts/lerobot_record.py | 4 + src/lerobot/scripts/lerobot_replay.py | 2 + src/lerobot/scripts/lerobot_rollout.py | 4 + src/lerobot/scripts/lerobot_setup_motors.py | 4 + src/lerobot/scripts/lerobot_teleoperate.py | 4 + .../bi_rebot_102_leader/__init__.py | 20 ++ .../bi_rebot_102_leader.py | 113 +++++++ .../config_bi_rebot_102_leader.py | 29 ++ .../rebot_102_leader/__init__.py | 20 ++ .../config_rebot_102_leader.py | 83 +++++ .../rebot_102_leader/rebot_102_leader.py | 207 +++++++++++++ src/lerobot/teleoperators/utils.py | 8 + src/lerobot/utils/import_utils.py | 4 + tests/robots/test_rebot_b601_follower.py | 116 +++++++ tests/teleoperators/test_rebot_102_leader.py | 102 +++++++ uv.lock | 52 +++- 28 files changed, 1581 insertions(+), 3 deletions(-) create mode 100644 docs/source/rebot_b601.mdx create mode 100644 src/lerobot/robots/bi_rebot_b601_follower/__init__.py create mode 100644 src/lerobot/robots/bi_rebot_b601_follower/bi_rebot_b601_follower.py create mode 100644 src/lerobot/robots/bi_rebot_b601_follower/config_bi_rebot_b601_follower.py create mode 100644 src/lerobot/robots/rebot_b601_follower/__init__.py create mode 100644 src/lerobot/robots/rebot_b601_follower/config_rebot_b601_follower.py create mode 100644 src/lerobot/robots/rebot_b601_follower/rebot_b601_follower.py create mode 100644 src/lerobot/teleoperators/bi_rebot_102_leader/__init__.py create mode 100644 src/lerobot/teleoperators/bi_rebot_102_leader/bi_rebot_102_leader.py create mode 100644 src/lerobot/teleoperators/bi_rebot_102_leader/config_bi_rebot_102_leader.py create mode 100644 src/lerobot/teleoperators/rebot_102_leader/__init__.py create mode 100644 src/lerobot/teleoperators/rebot_102_leader/config_rebot_102_leader.py create mode 100644 src/lerobot/teleoperators/rebot_102_leader/rebot_102_leader.py create mode 100644 tests/robots/test_rebot_b601_follower.py create mode 100644 tests/teleoperators/test_rebot_102_leader.py diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index f1dfe9aae..470319c48 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -143,6 +143,8 @@ title: OMX - local: openarm title: OpenArm + - local: rebot_b601 + title: reBot B601-DM title: "Robots" - sections: - local: phone_teleop diff --git a/docs/source/rebot_b601.mdx b/docs/source/rebot_b601.mdx new file mode 100644 index 000000000..adb751560 --- /dev/null +++ b/docs/source/rebot_b601.mdx @@ -0,0 +1,186 @@ +# reBot B601-DM + +[reBot B601-DM](https://wiki.seeedstudio.com/rebot_arm_b601_dm_lerobot/) is an open-source, low-cost robot arm from Seeed Studio for embodied-AI and imitation learning. It comes as a **follower** arm (the `B601-DM`, a 6-DOF arm plus gripper driven by Damiao CAN motors) and a **leader** arm (the `StarArm102` / `reBot Arm 102`, driven by FashionStar UART smart servos) used to teleoperate it. + +This page covers **calibration** and **teleoperation** for both single-arm and bimanual (dual-arm) setups. + +
+ reBot B601-DM follower arm at its zero position + reBot Arm 102 leader arm at its zero position +
+ +_Left: the B601-DM follower at its zero position. Right: the reBot Arm 102 leader at its zero position. Images courtesy of [Seeed Studio](https://wiki.seeedstudio.com/rebot_arm_b601_dm_lerobot/)._ + +## Install LeRobot 🤗 + +Follow our [Installation Guide](./installation), then install the reBot support: + +```bash +pip install -e ".[rebot]" +``` + +This pulls in `motorbridge` (CAN motor control for the B601-DM follower) and `motorbridge-smart-servo` (FashionStar UART servos for the reBot Arm 102 leader). + +## Registered device types + +| Type | Kind | +| ------------------------ | -------------------------------------------- | +| `rebot_b601_follower` | single-arm B601-DM follower robot | +| `bi_rebot_b601_follower` | bimanual (dual-arm) follower robot | +| `rebot_102_leader` | single-arm reBot Arm 102 leader teleoperator | +| `bi_rebot_102_leader` | bimanual (dual-arm) leader teleoperator | + +The bimanual types compose two single-arm instances and namespace each arm's +observation/action keys with a `left_` / `right_` prefix. Per-arm settings are +passed through nested `left_arm_config.*` / `right_arm_config.*` arguments. + +## Find the USB ports + +For each device, find the USB port associated with its motor bus using: + +```bash +lerobot-find-port +``` + + + On Linux, remove `brltty` (`sudo apt remove brltty`) so it does not hold the + leader's USB serial port. You may also need to grant access to the serial + devices: `sudo chmod 666 /dev/ttyACM* /dev/ttyUSB*`. + + +## Calibration + +Neither arm stores a persistent hardware calibration: every time it connects, the motors are re-zeroed against the pose the arm is physically holding. Calibration simply records that zero pose. When prompted, **manually move the arm to its zero position** (the default sit-down pose shown above, gripper fully closed) and press ENTER. + +### Follower (B601-DM) + + + + +```bash +lerobot-calibrate \ + --robot.type=rebot_b601_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.id=follower \ + --robot.can_adapter=damiao +``` + + + + +Connect the bimanual follower; calibration runs for the left arm, then the right arm. + +```bash +lerobot-calibrate \ + --robot.type=bi_rebot_b601_follower \ + --robot.id=bi_follower \ + --robot.left_arm_config.port=/dev/ttyACM0 \ + --robot.left_arm_config.can_adapter=damiao \ + --robot.right_arm_config.port=/dev/ttyACM1 \ + --robot.right_arm_config.can_adapter=damiao +``` + +Per-arm calibration files are saved with `_left` / `_right` suffixes on the id. + + + + +### Leader (reBot Arm 102) + + + + +```bash +lerobot-calibrate \ + --teleop.type=rebot_102_leader \ + --teleop.port=/dev/ttyUSB0 \ + --teleop.id=leader +``` + + + + +```bash +lerobot-calibrate \ + --teleop.type=bi_rebot_102_leader \ + --teleop.id=bi_leader \ + --teleop.left_arm_config.port=/dev/ttyUSB0 \ + --teleop.right_arm_config.port=/dev/ttyUSB1 +``` + + + + +## Teleoperation + +Once both arms are calibrated, drive the follower with the leader. The follower talks to its CAN bus through a Damiao serial bridge (`can_adapter=damiao`, the default) or a SocketCAN adapter (`can_adapter=socketcan`). See the [OpenArm page](./openarm) for more details on the SocketCAN adapter configuration. + + + + +```bash +lerobot-teleoperate \ + --robot.type=rebot_b601_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.id=follower \ + --robot.can_adapter=damiao \ + --teleop.type=rebot_102_leader \ + --teleop.port=/dev/ttyUSB0 \ + --teleop.id=leader +``` + + + + +The bimanual leader and follower reuse the single-arm classes; each arm is +configured through nested `left_arm_config.*` / `right_arm_config.*` arguments, +so a bimanual reBot Arm 102 leader drives a bimanual B601-DM follower. + +```bash +lerobot-teleoperate \ + --robot.type=bi_rebot_b601_follower \ + --robot.id=bi_follower \ + --robot.left_arm_config.port=/dev/ttyACM0 \ + --robot.left_arm_config.can_adapter=damiao \ + --robot.right_arm_config.port=/dev/ttyACM1 \ + --robot.right_arm_config.can_adapter=damiao \ + --teleop.type=bi_rebot_102_leader \ + --teleop.id=bi_leader \ + --teleop.left_arm_config.port=/dev/ttyUSB0 \ + --teleop.right_arm_config.port=/dev/ttyUSB1 +``` + + + + + + The leader and follower share the same joint names (`shoulder_pan, + shoulder_lift, elbow_flex, wrist_flex, wrist_yaw, wrist_roll, gripper`), so + leader actions map directly onto the follower. + + +If the motion of a joint is reversed, flip its sign in the leader's `joint_directions` (the gripper also carries a scale to widen its range to the follower): + +```bash +lerobot-teleoperate \ + --robot.type=rebot_b601_follower \ + --robot.port=/dev/ttyACM0 \ + --robot.can_adapter=damiao \ + --teleop.type=rebot_102_leader \ + --teleop.port=/dev/ttyUSB0 \ + --teleop.joint_directions='{"shoulder_pan":-1,"shoulder_lift":-1,"elbow_flex":1,"wrist_flex":1,"wrist_yaw":1,"wrist_roll":-1,"gripper":-6}' +``` + +## Recording datasets + +Swap `lerobot-teleoperate` for `lerobot-record` (with the same `--robot.*` / `--teleop.*` arguments, plus `--dataset.*`) to record demonstrations for training. See [Imitation Learning for Robots](./il_robots) for the full workflow. + +For hardware assembly and wiring, see the [Seeed Studio reBot wiki](https://wiki.seeedstudio.com/rebot_arm_b601_dm_lerobot/). diff --git a/pyproject.toml b/pyproject.toml index f983134ab..93953cd57 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -151,6 +151,8 @@ pyserial-dep = ["pyserial>=3.5,<4.0"] deepdiff-dep = ["deepdiff>=7.0.1,<9.0.0"] pynput-dep = ["pynput>=1.7.8,<1.9.0"] pyzmq-dep = ["pyzmq>=26.2.1,<28.0.0"] +motorbridge-dep = ["motorbridge>=0.3.2,<0.4.0"] +motorbridge-smart-servo-dep = ["motorbridge-smart-servo>=0.0.4,<0.1.0"] # Motors feetech = ["feetech-servo-sdk>=1.0.0,<2.0.0", "lerobot[pyserial-dep]", "lerobot[deepdiff-dep]"] @@ -174,6 +176,9 @@ unitree_g1 = [ "lerobot[pygame-dep]", ] reachy2 = ["reachy2_sdk>=1.0.15,<1.1.0"] +# Seeed Studio reBot B601-DM follower (motorbridge / CAN) + StarArm102 / reBot Arm 102 +# leader (motorbridge-smart-servo / FashionStar UART servos). +rebot = ["lerobot[motorbridge-dep]", "lerobot[motorbridge-smart-servo-dep]"] kinematics = ["lerobot[placo-dep]"] intelrealsense = [ "pyrealsense2>=2.55.1.6486,<2.57.0 ; sys_platform != 'darwin'", @@ -260,6 +265,7 @@ all = [ "lerobot[lekiwi]", "lerobot[openarms]", "lerobot[reachy2]", + "lerobot[rebot]", "lerobot[kinematics]", "lerobot[intelrealsense]", "lerobot[diffusion]", diff --git a/src/lerobot/robots/bi_rebot_b601_follower/__init__.py b/src/lerobot/robots/bi_rebot_b601_follower/__init__.py new file mode 100644 index 000000000..8ef454f45 --- /dev/null +++ b/src/lerobot/robots/bi_rebot_b601_follower/__init__.py @@ -0,0 +1,20 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .bi_rebot_b601_follower import BiRebotB601Follower +from .config_bi_rebot_b601_follower import BiRebotB601FollowerConfig + +__all__ = ["BiRebotB601Follower", "BiRebotB601FollowerConfig"] diff --git a/src/lerobot/robots/bi_rebot_b601_follower/bi_rebot_b601_follower.py b/src/lerobot/robots/bi_rebot_b601_follower/bi_rebot_b601_follower.py new file mode 100644 index 000000000..bd19f1b62 --- /dev/null +++ b/src/lerobot/robots/bi_rebot_b601_follower/bi_rebot_b601_follower.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from functools import cached_property + +from lerobot.types import RobotAction, RobotObservation +from lerobot.utils.decorators import check_if_already_connected, check_if_not_connected + +from ..rebot_b601_follower import RebotB601Follower, RebotB601FollowerRobotConfig +from ..robot import Robot +from .config_bi_rebot_b601_follower import BiRebotB601FollowerConfig + +logger = logging.getLogger(__name__) + + +class BiRebotB601Follower(Robot): + """Bimanual Seeed Studio reBot B601-DM follower. + + Composes two single-arm :class:`RebotB601Follower` instances. Observation and + action keys of each arm are namespaced with a ``left_`` / ``right_`` prefix. + """ + + config_class = BiRebotB601FollowerConfig + name = "bi_rebot_b601_follower" + + def __init__(self, config: BiRebotB601FollowerConfig): + super().__init__(config) + self.config = config + + left_arm_config = RebotB601FollowerRobotConfig( + id=f"{config.id}_left" if config.id else None, + calibration_dir=config.calibration_dir, + port=config.left_arm_config.port, + can_adapter=config.left_arm_config.can_adapter, + dm_serial_baud=config.left_arm_config.dm_serial_baud, + disable_torque_on_disconnect=config.left_arm_config.disable_torque_on_disconnect, + max_relative_target=config.left_arm_config.max_relative_target, + cameras=config.left_arm_config.cameras, + motor_can_ids=config.left_arm_config.motor_can_ids, + pos_vel_velocity=config.left_arm_config.pos_vel_velocity, + gripper_torque_ratio=config.left_arm_config.gripper_torque_ratio, + joint_limits=config.left_arm_config.joint_limits, + ) + + right_arm_config = RebotB601FollowerRobotConfig( + id=f"{config.id}_right" if config.id else None, + calibration_dir=config.calibration_dir, + port=config.right_arm_config.port, + can_adapter=config.right_arm_config.can_adapter, + dm_serial_baud=config.right_arm_config.dm_serial_baud, + disable_torque_on_disconnect=config.right_arm_config.disable_torque_on_disconnect, + max_relative_target=config.right_arm_config.max_relative_target, + cameras=config.right_arm_config.cameras, + motor_can_ids=config.right_arm_config.motor_can_ids, + pos_vel_velocity=config.right_arm_config.pos_vel_velocity, + gripper_torque_ratio=config.right_arm_config.gripper_torque_ratio, + joint_limits=config.right_arm_config.joint_limits, + ) + + self.left_arm = RebotB601Follower(left_arm_config) + self.right_arm = RebotB601Follower(right_arm_config) + + # Only for compatibility with parts of the codebase that expect `robot.cameras`. + self.cameras = {**self.left_arm.cameras, **self.right_arm.cameras} + + @property + def _motors_ft(self) -> dict[str, type]: + return { + **{f"left_{k}": v for k, v in self.left_arm._motors_ft.items()}, + **{f"right_{k}": v for k, v in self.right_arm._motors_ft.items()}, + } + + @property + def _cameras_ft(self) -> dict[str, tuple]: + return { + **{f"left_{k}": v for k, v in self.left_arm._cameras_ft.items()}, + **{f"right_{k}": v for k, v in self.right_arm._cameras_ft.items()}, + } + + @cached_property + def observation_features(self) -> dict[str, type | tuple]: + return {**self._motors_ft, **self._cameras_ft} + + @cached_property + def action_features(self) -> dict[str, type]: + return self._motors_ft + + @property + def is_connected(self) -> bool: + return self.left_arm.is_connected and self.right_arm.is_connected + + @check_if_already_connected + def connect(self, calibrate: bool = True) -> None: + self.left_arm.connect(calibrate) + self.right_arm.connect(calibrate) + + @property + def is_calibrated(self) -> bool: + return self.left_arm.is_calibrated and self.right_arm.is_calibrated + + def calibrate(self) -> None: + self.left_arm.calibrate() + self.right_arm.calibrate() + + def configure(self) -> None: + self.left_arm.configure() + self.right_arm.configure() + + @check_if_not_connected + def get_observation(self) -> RobotObservation: + obs_dict = {} + obs_dict.update({f"left_{k}": v for k, v in self.left_arm.get_observation().items()}) + obs_dict.update({f"right_{k}": v for k, v in self.right_arm.get_observation().items()}) + return obs_dict + + @check_if_not_connected + def send_action(self, action: RobotAction) -> RobotAction: + left_action = { + key.removeprefix("left_"): value for key, value in action.items() if key.startswith("left_") + } + right_action = { + key.removeprefix("right_"): value for key, value in action.items() if key.startswith("right_") + } + + sent_action_left = self.left_arm.send_action(left_action) + sent_action_right = self.right_arm.send_action(right_action) + + return { + **{f"left_{k}": v for k, v in sent_action_left.items()}, + **{f"right_{k}": v for k, v in sent_action_right.items()}, + } + + @check_if_not_connected + def disconnect(self) -> None: + self.left_arm.disconnect() + self.right_arm.disconnect() diff --git a/src/lerobot/robots/bi_rebot_b601_follower/config_bi_rebot_b601_follower.py b/src/lerobot/robots/bi_rebot_b601_follower/config_bi_rebot_b601_follower.py new file mode 100644 index 000000000..079b7a355 --- /dev/null +++ b/src/lerobot/robots/bi_rebot_b601_follower/config_bi_rebot_b601_follower.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from dataclasses import dataclass + +from ..config import RobotConfig +from ..rebot_b601_follower import RebotB601FollowerConfig + + +@RobotConfig.register_subclass("bi_rebot_b601_follower") +@dataclass +class BiRebotB601FollowerConfig(RobotConfig): + """Configuration class for the bimanual reBot B601-DM follower robot.""" + + left_arm_config: RebotB601FollowerConfig + right_arm_config: RebotB601FollowerConfig diff --git a/src/lerobot/robots/rebot_b601_follower/__init__.py b/src/lerobot/robots/rebot_b601_follower/__init__.py new file mode 100644 index 000000000..43fcbb769 --- /dev/null +++ b/src/lerobot/robots/rebot_b601_follower/__init__.py @@ -0,0 +1,20 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .config_rebot_b601_follower import RebotB601FollowerConfig, RebotB601FollowerRobotConfig +from .rebot_b601_follower import RebotB601Follower + +__all__ = ["RebotB601Follower", "RebotB601FollowerConfig", "RebotB601FollowerRobotConfig"] diff --git a/src/lerobot/robots/rebot_b601_follower/config_rebot_b601_follower.py b/src/lerobot/robots/rebot_b601_follower/config_rebot_b601_follower.py new file mode 100644 index 000000000..096548afb --- /dev/null +++ b/src/lerobot/robots/rebot_b601_follower/config_rebot_b601_follower.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from dataclasses import dataclass, field + +from lerobot.cameras import CameraConfig + +from ..config import RobotConfig + + +@dataclass +class RebotB601FollowerConfig: + """Base configuration class for the Seeed Studio reBot B601-DM follower arm. + + The B601-DM is a 6-DOF arm plus gripper driven by Damiao CAN motors. Motor + communication goes through the ``motorbridge`` package. + """ + + # Communication port. For ``can_adapter="damiao"`` this is the Damiao serial + # bridge device (e.g. "/dev/ttyACM0"); for ``can_adapter="socketcan"`` it is + # the CAN channel name (e.g. "can0"). + port: str + + # CAN adapter type: + # "damiao" - Damiao dedicated serial bridge (default) + # "socketcan" - SocketCAN based adapters (PCAN, slcan, embedded controllers, ...) + can_adapter: str = "damiao" + + # Baud rate for the Damiao serial bridge (only used when can_adapter="damiao"). + dm_serial_baud: int = 921600 + + disable_torque_on_disconnect: bool = True + + # `max_relative_target` limits the magnitude of the relative positional target + # vector for safety purposes (in degrees). Set to a positive scalar to apply the + # same value to all motors, or to a dict mapping motor names to per-motor values. + max_relative_target: float | dict[str, float] | None = None + + # cameras + cameras: dict[str, CameraConfig] = field(default_factory=dict) + + # Maps motor names to their (send_can_id, recv_can_id) pair. + motor_can_ids: dict[str, tuple[int, int]] = field( + default_factory=lambda: { + "shoulder_pan": (0x01, 0x11), + "shoulder_lift": (0x02, 0x12), + "elbow_flex": (0x03, 0x13), + "wrist_flex": (0x04, 0x14), + "wrist_yaw": (0x05, 0x15), + "wrist_roll": (0x06, 0x16), + "gripper": (0x07, 0x17), + } + ) + + # Target velocity for joints running in POS_VEL mode, in degrees/s. A scalar is + # applied to every joint; a list provides one value per joint (in motor order). + pos_vel_velocity: float | list[float] = field(default_factory=lambda: [150.0] * 7) + + # Torque/current ratio for the gripper's FORCE_POS mode, in range [0, 1]. + gripper_torque_ratio: float = 0.1 + + # Soft joint limits (degrees). These are clipped against on every action. + joint_limits: dict[str, tuple[float, float]] = field( + default_factory=lambda: { + "shoulder_pan": (-145.0, 145.0), + "shoulder_lift": (-170.0, 1.0), + "elbow_flex": (-200.0, 1.0), + "wrist_flex": (-80.0, 90.0), + "wrist_yaw": (-90.0, 90.0), + "wrist_roll": (-90.0, 90.0), + "gripper": (-270.0, 0.0), + } + ) + + +@RobotConfig.register_subclass("rebot_b601_follower") +@dataclass +class RebotB601FollowerRobotConfig(RobotConfig, RebotB601FollowerConfig): + """Registered configuration for the reBot B601-DM follower robot.""" + + pass diff --git a/src/lerobot/robots/rebot_b601_follower/rebot_b601_follower.py b/src/lerobot/robots/rebot_b601_follower/rebot_b601_follower.py new file mode 100644 index 000000000..ec00f4aa9 --- /dev/null +++ b/src/lerobot/robots/rebot_b601_follower/rebot_b601_follower.py @@ -0,0 +1,289 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import math +import time +from functools import cached_property +from typing import TYPE_CHECKING + +from lerobot.cameras import make_cameras_from_configs +from lerobot.motors import MotorCalibration +from lerobot.types import RobotAction, RobotObservation +from lerobot.utils.decorators import check_if_already_connected, check_if_not_connected +from lerobot.utils.import_utils import _motorbridge_available, require_package + +from ..robot import Robot +from ..utils import ensure_safe_goal_position +from .config_rebot_b601_follower import RebotB601FollowerRobotConfig + +if TYPE_CHECKING or _motorbridge_available: + from motorbridge import Controller as MotorBridgeController, Mode as MotorBridgeMode +else: + MotorBridgeController = None + MotorBridgeMode = None + +logger = logging.getLogger(__name__) + +# Joint controlled in FORCE_POS mode; every other joint runs in POS_VEL mode. +GRIPPER_MOTOR = "gripper" +# Per-joint Damiao motor models for the B601-DM (passed to motorbridge). +MOTOR_MODELS = { + "shoulder_pan": "4340P", + "shoulder_lift": "4340P", + "elbow_flex": "4340P", + "wrist_flex": "4310", + "wrist_yaw": "4310", + "wrist_roll": "4310", + "gripper": "4310", +} +_ENSURE_MODE_RETRIES = 9 +_SETTLE_SEC = 0.01 +_ZERO_SETTLE_SEC = 0.1 + + +class RebotB601Follower(Robot): + """Seeed Studio reBot B601-DM follower arm (6-DOF + gripper, Damiao CAN motors). + + Motor communication is handled by the ``motorbridge`` package over a CAN bus, + reached either through a Damiao serial bridge or a SocketCAN adapter. + """ + + config_class = RebotB601FollowerRobotConfig + name = "rebot_b601_follower" + + def __init__(self, config: RebotB601FollowerRobotConfig): + require_package("motorbridge", extra="rebot") + super().__init__(config) + self.config = config + self.bus: MotorBridgeController | None = None + self.motors: dict = {} + self.motor_names = list(config.motor_can_ids.keys()) + self.cameras = make_cameras_from_configs(config.cameras) + + @property + def _motors_ft(self) -> dict[str, type]: + return {f"{motor}.pos": float for motor in self.motor_names} + + @property + def _cameras_ft(self) -> dict[str, tuple]: + return { + cam: (self.config.cameras[cam].height, self.config.cameras[cam].width, 3) for cam in self.cameras + } + + @cached_property + def observation_features(self) -> dict[str, type | tuple]: + return {**self._motors_ft, **self._cameras_ft} + + @cached_property + def action_features(self) -> dict[str, type]: + return self._motors_ft + + @property + def is_connected(self) -> bool: + return self.bus is not None and all(cam.is_connected for cam in self.cameras.values()) + + @check_if_already_connected + def connect(self, calibrate: bool = True) -> None: + logger.info(f"Connecting {self} on {self.config.port} (adapter={self.config.can_adapter})...") + if self.config.can_adapter == "damiao": + self.bus = MotorBridgeController.from_dm_serial( + serial_port=self.config.port, + baud=self.config.dm_serial_baud, + ) + elif self.config.can_adapter == "socketcan": + self.bus = MotorBridgeController(channel=self.config.port) + else: + raise ValueError( + f"Unsupported can_adapter '{self.config.can_adapter}'. Use 'damiao' or 'socketcan'." + ) + + for motor_name, (send_id, recv_id) in self.config.motor_can_ids.items(): + self.motors[motor_name] = self.bus.add_damiao_motor(send_id, recv_id, MOTOR_MODELS[motor_name]) + + if not self.is_calibrated and calibrate: + logger.info( + "Mismatch between calibration values in the motor and the calibration file or no calibration file found" + ) + self.calibrate() + + for cam in self.cameras.values(): + cam.connect() + + self.configure() + logger.info(f"{self} connected.") + + @property + def is_calibrated(self) -> bool: + return bool(self.calibration) + + def calibrate(self) -> None: + if self.calibration: + user_input = input( + f"Press ENTER to use provided calibration file associated with the id {self.id}, " + "or type 'c' and press ENTER to run calibration: " + ) + if user_input.strip().lower() != "c": + logger.info(f"Using calibration file associated with the id {self.id}") + return + + logger.info(f"\nRunning calibration of {self}") + self.bus.disable_all() + print( + "\nCalibration: set zero position.\n" + "Manually move the reBot B601 to its ZERO POSITION and close the gripper.\n" + "See the B601 manual for the zero pose (the default sit-down position).\n" + ) + input("Press ENTER when ready...") + + for motor in self.motors.values(): + motor.set_zero_position() + time.sleep(_ZERO_SETTLE_SEC) + logger.info("Arm zero position set.") + + self.calibration = {} + for motor_name, (send_id, _recv_id) in self.config.motor_can_ids.items(): + range_min, range_max = self.config.joint_limits[motor_name] + self.calibration[motor_name] = MotorCalibration( + id=send_id, + drive_mode=0, + homing_offset=0, + range_min=int(range_min), + range_max=int(range_max), + ) + + self._save_calibration() + print(f"Calibration saved to {self.calibration_fpath}") + + def configure(self) -> None: + self.bus.enable_all() + for motor_name, motor in self.motors.items(): + target_mode = ( + MotorBridgeMode.FORCE_POS if motor_name == GRIPPER_MOTOR else MotorBridgeMode.POS_VEL + ) + for attempt in range(_ENSURE_MODE_RETRIES + 1): + try: + motor.ensure_mode(target_mode) + break + except Exception: + if attempt == _ENSURE_MODE_RETRIES: + raise + time.sleep(_SETTLE_SEC) + logger.debug(f"{motor_name} mode set to {target_mode}") + + @check_if_not_connected + def disable_torque(self) -> None: + """Disable motor torque so the arm can be moved by hand (read-only debugging).""" + self.bus.disable_all() + logger.info(f"{self} torque disabled.") + + def _present_pos(self) -> dict[str, float]: + """Read present joint positions in degrees.""" + for motor in self.motors.values(): + motor.request_feedback() + try: + self.bus.poll_feedback_once() + except Exception: + logger.warning("CAN bus poll feedback failed.") + + present_pos = {} + for motor_name, motor in self.motors.items(): + state = motor.get_state() + present_pos[motor_name] = math.degrees(state.pos) if state is not None else 0.0 + return present_pos + + @check_if_not_connected + def get_observation(self) -> RobotObservation: + start = time.perf_counter() + obs_dict = {f"{motor}.pos": pos for motor, pos in self._present_pos().items()} + dt_ms = (time.perf_counter() - start) * 1e3 + logger.debug(f"{self} read state: {dt_ms:.1f}ms") + + for cam_key, cam in self.cameras.items(): + start = time.perf_counter() + obs_dict[cam_key] = cam.read_latest() + dt_ms = (time.perf_counter() - start) * 1e3 + logger.debug(f"{self} read {cam_key}: {dt_ms:.1f}ms") + + return obs_dict + + @check_if_not_connected + def send_action(self, action: RobotAction) -> RobotAction: + """Command the arm to a target joint configuration. + + Positions are expressed in degrees. The relative action magnitude may be + clipped depending on `max_relative_target`, so the action actually sent is + always returned. + """ + goal_pos = {key.removesuffix(".pos"): val for key, val in action.items() if key.endswith(".pos")} + + # Clip against soft joint limits. + for motor_name in list(goal_pos): + if motor_name in self.config.joint_limits: + min_limit, max_limit = self.config.joint_limits[motor_name] + clipped = max(min_limit, min(max_limit, goal_pos[motor_name])) + if clipped != goal_pos[motor_name]: + logger.debug(f"Clipped {motor_name} from {goal_pos[motor_name]:.2f} to {clipped:.2f}") + goal_pos[motor_name] = clipped + + # Tolerate 6-DOF leaders that have no wrist_yaw joint by holding it at zero. + # This is intentional: it lets a 6-DOF leader such as the SO-100 / SO-101 + # (so100_leader / so101_leader) teleoperate this 7-DOF follower — the missing + # wrist_yaw command is simply treated as 0.0 instead of raising. + if "wrist_yaw" not in goal_pos: + goal_pos["wrist_yaw"] = 0.0 + + # Cap relative target when too far from the present position. + if self.config.max_relative_target is not None: + present_pos = self._present_pos() + goal_present_pos = {key: (g, present_pos.get(key, g)) for key, g in goal_pos.items()} + goal_pos = ensure_safe_goal_position(goal_present_pos, self.config.max_relative_target) + + for motor_name, position_deg in goal_pos.items(): + motor = self.motors.get(motor_name) + if motor is None: + continue + idx = self.motor_names.index(motor_name) + vel_deg_s = ( + self.config.pos_vel_velocity[idx] + if isinstance(self.config.pos_vel_velocity, list) + else self.config.pos_vel_velocity + ) + pos_rad = math.radians(position_deg) + vel_rad = math.radians(vel_deg_s) + if motor_name == GRIPPER_MOTOR: + motor.send_force_pos(pos_rad, vel_rad, self.config.gripper_torque_ratio) + else: + motor.send_pos_vel(pos_rad, vel_rad) + + return {f"{motor}.pos": val for motor, val in goal_pos.items()} + + @check_if_not_connected + def disconnect(self) -> None: + for motor in self.motors.values(): + if self.config.disable_torque_on_disconnect: + motor.disable() + motor.clear_error() + motor.close() + + self.bus.close() + self.bus = None + self.motors = {} + + for cam in self.cameras.values(): + cam.disconnect() + + logger.info(f"{self} disconnected.") diff --git a/src/lerobot/robots/utils.py b/src/lerobot/robots/utils.py index 92da597f1..f897a560e 100644 --- a/src/lerobot/robots/utils.py +++ b/src/lerobot/robots/utils.py @@ -68,6 +68,14 @@ def make_robot_from_config(config: RobotConfig) -> Robot: from .bi_openarm_follower import BiOpenArmFollower return BiOpenArmFollower(config) + elif config.type == "rebot_b601_follower": + from .rebot_b601_follower import RebotB601Follower + + return RebotB601Follower(config) + elif config.type == "bi_rebot_b601_follower": + from .bi_rebot_b601_follower import BiRebotB601Follower + + return BiRebotB601Follower(config) elif config.type == "mock_robot": from tests.mocks.mock_robot import MockRobot diff --git a/src/lerobot/scripts/lerobot_calibrate.py b/src/lerobot/scripts/lerobot_calibrate.py index e68d7438b..e43736954 100644 --- a/src/lerobot/scripts/lerobot_calibrate.py +++ b/src/lerobot/scripts/lerobot_calibrate.py @@ -39,6 +39,7 @@ from lerobot.robots import ( # noqa: F401 Robot, RobotConfig, bi_openarm_follower, + bi_rebot_b601_follower, bi_so_follower, hope_jr, koch_follower, @@ -46,12 +47,14 @@ from lerobot.robots import ( # noqa: F401 make_robot_from_config, omx_follower, openarm_follower, + rebot_b601_follower, so_follower, ) from lerobot.teleoperators import ( # noqa: F401 Teleoperator, TeleoperatorConfig, bi_openarm_leader, + bi_rebot_102_leader, bi_so_leader, homunculus, koch_leader, @@ -59,6 +62,7 @@ from lerobot.teleoperators import ( # noqa: F401 omx_leader, openarm_leader, openarm_mini, + rebot_102_leader, so_leader, unitree_g1, ) diff --git a/src/lerobot/scripts/lerobot_find_joint_limits.py b/src/lerobot/scripts/lerobot_find_joint_limits.py index c4f867631..5b9166a2e 100644 --- a/src/lerobot/scripts/lerobot_find_joint_limits.py +++ b/src/lerobot/scripts/lerobot_find_joint_limits.py @@ -45,16 +45,19 @@ from lerobot.model import RobotKinematics from lerobot.robots import ( # noqa: F401 RobotConfig, bi_openarm_follower, + bi_rebot_b601_follower, bi_so_follower, koch_follower, make_robot_from_config, omx_follower, openarm_follower, + rebot_b601_follower, so_follower, ) from lerobot.teleoperators import ( # noqa: F401 TeleoperatorConfig, bi_openarm_leader, + bi_rebot_102_leader, bi_so_leader, gamepad, koch_leader, @@ -62,6 +65,7 @@ from lerobot.teleoperators import ( # noqa: F401 omx_leader, openarm_leader, openarm_mini, + rebot_102_leader, so_leader, ) from lerobot.utils.robot_utils import precise_sleep diff --git a/src/lerobot/scripts/lerobot_record.py b/src/lerobot/scripts/lerobot_record.py index c8419cb14..c411ebf9e 100644 --- a/src/lerobot/scripts/lerobot_record.py +++ b/src/lerobot/scripts/lerobot_record.py @@ -120,6 +120,7 @@ from lerobot.robots import ( # noqa: F401 Robot, RobotConfig, bi_openarm_follower, + bi_rebot_b601_follower, bi_so_follower, earthrover_mini_plus, hope_jr, @@ -128,6 +129,7 @@ from lerobot.robots import ( # noqa: F401 omx_follower, openarm_follower, reachy2, + rebot_b601_follower, so_follower, unitree_g1 as unitree_g1_robot, ) @@ -135,6 +137,7 @@ from lerobot.teleoperators import ( # noqa: F401 Teleoperator, TeleoperatorConfig, bi_openarm_leader, + bi_rebot_102_leader, bi_so_leader, homunculus, koch_leader, @@ -143,6 +146,7 @@ from lerobot.teleoperators import ( # noqa: F401 openarm_leader, openarm_mini, reachy2_teleoperator, + rebot_102_leader, so_leader, unitree_g1, ) diff --git a/src/lerobot/scripts/lerobot_replay.py b/src/lerobot/scripts/lerobot_replay.py index 41d2926cc..1851f7c2b 100644 --- a/src/lerobot/scripts/lerobot_replay.py +++ b/src/lerobot/scripts/lerobot_replay.py @@ -56,6 +56,7 @@ from lerobot.robots import ( # noqa: F401 Robot, RobotConfig, bi_openarm_follower, + bi_rebot_b601_follower, bi_so_follower, earthrover_mini_plus, hope_jr, @@ -64,6 +65,7 @@ from lerobot.robots import ( # noqa: F401 omx_follower, openarm_follower, reachy2, + rebot_b601_follower, so_follower, unitree_g1, ) diff --git a/src/lerobot/scripts/lerobot_rollout.py b/src/lerobot/scripts/lerobot_rollout.py index 7015e707c..3378b6de4 100644 --- a/src/lerobot/scripts/lerobot_rollout.py +++ b/src/lerobot/scripts/lerobot_rollout.py @@ -144,6 +144,7 @@ from lerobot.robots import ( # noqa: F401 Robot, RobotConfig, bi_openarm_follower, + bi_rebot_b601_follower, bi_so_follower, earthrover_mini_plus, hope_jr, @@ -151,6 +152,7 @@ from lerobot.robots import ( # noqa: F401 omx_follower, openarm_follower, reachy2, + rebot_b601_follower, so_follower, unitree_g1 as unitree_g1_robot, ) @@ -159,6 +161,7 @@ from lerobot.teleoperators import ( # noqa: F401 Teleoperator, TeleoperatorConfig, bi_openarm_leader, + bi_rebot_102_leader, bi_so_leader, homunculus, koch_leader, @@ -166,6 +169,7 @@ from lerobot.teleoperators import ( # noqa: F401 openarm_leader, openarm_mini, reachy2_teleoperator, + rebot_102_leader, so_leader, unitree_g1, ) diff --git a/src/lerobot/scripts/lerobot_setup_motors.py b/src/lerobot/scripts/lerobot_setup_motors.py index 2c962a6e2..69ebcf5fa 100644 --- a/src/lerobot/scripts/lerobot_setup_motors.py +++ b/src/lerobot/scripts/lerobot_setup_motors.py @@ -30,20 +30,24 @@ import draccus from lerobot.robots import ( # noqa: F401 RobotConfig, + bi_rebot_b601_follower, bi_so_follower, koch_follower, lekiwi, make_robot_from_config, omx_follower, + rebot_b601_follower, so_follower, ) from lerobot.teleoperators import ( # noqa: F401 TeleoperatorConfig, + bi_rebot_102_leader, bi_so_leader, koch_leader, make_teleoperator_from_config, omx_leader, openarm_mini, + rebot_102_leader, so_leader, ) diff --git a/src/lerobot/scripts/lerobot_teleoperate.py b/src/lerobot/scripts/lerobot_teleoperate.py index 76157595e..2ff02bda0 100644 --- a/src/lerobot/scripts/lerobot_teleoperate.py +++ b/src/lerobot/scripts/lerobot_teleoperate.py @@ -72,6 +72,7 @@ from lerobot.robots import ( # noqa: F401 Robot, RobotConfig, bi_openarm_follower, + bi_rebot_b601_follower, bi_so_follower, earthrover_mini_plus, hope_jr, @@ -80,6 +81,7 @@ from lerobot.robots import ( # noqa: F401 omx_follower, openarm_follower, reachy2, + rebot_b601_follower, so_follower, unitree_g1 as unitree_g1_robot, ) @@ -87,6 +89,7 @@ from lerobot.teleoperators import ( # noqa: F401 Teleoperator, TeleoperatorConfig, bi_openarm_leader, + bi_rebot_102_leader, bi_so_leader, gamepad, homunculus, @@ -97,6 +100,7 @@ from lerobot.teleoperators import ( # noqa: F401 openarm_leader, openarm_mini, reachy2_teleoperator, + rebot_102_leader, so_leader, unitree_g1, ) diff --git a/src/lerobot/teleoperators/bi_rebot_102_leader/__init__.py b/src/lerobot/teleoperators/bi_rebot_102_leader/__init__.py new file mode 100644 index 000000000..c15cf76d8 --- /dev/null +++ b/src/lerobot/teleoperators/bi_rebot_102_leader/__init__.py @@ -0,0 +1,20 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .bi_rebot_102_leader import BiRebotArm102Leader +from .config_bi_rebot_102_leader import BiRebotArm102LeaderConfig + +__all__ = ["BiRebotArm102Leader", "BiRebotArm102LeaderConfig"] diff --git a/src/lerobot/teleoperators/bi_rebot_102_leader/bi_rebot_102_leader.py b/src/lerobot/teleoperators/bi_rebot_102_leader/bi_rebot_102_leader.py new file mode 100644 index 000000000..a4e5fd8c6 --- /dev/null +++ b/src/lerobot/teleoperators/bi_rebot_102_leader/bi_rebot_102_leader.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from functools import cached_property + +from lerobot.types import RobotAction +from lerobot.utils.decorators import check_if_already_connected, check_if_not_connected + +from ..rebot_102_leader import RebotArm102Leader, RebotArm102LeaderTeleopConfig +from ..teleoperator import Teleoperator +from .config_bi_rebot_102_leader import BiRebotArm102LeaderConfig + +logger = logging.getLogger(__name__) + + +class BiRebotArm102Leader(Teleoperator): + """Bimanual Seeed Studio StarArm102 / reBot Arm 102 leader. + + Composes two single-arm :class:`RebotArm102Leader` instances. Action keys of + each arm are namespaced with a ``left_`` / ``right_`` prefix, so a bimanual + leader can teleoperate a bimanual reBot B601 follower. + """ + + config_class = BiRebotArm102LeaderConfig + name = "bi_rebot_102_leader" + + def __init__(self, config: BiRebotArm102LeaderConfig): + super().__init__(config) + self.config = config + + left_arm_config = RebotArm102LeaderTeleopConfig( + id=f"{config.id}_left" if config.id else None, + calibration_dir=config.calibration_dir, + port=config.left_arm_config.port, + baudrate=config.left_arm_config.baudrate, + joint_ids=config.left_arm_config.joint_ids, + joint_directions=config.left_arm_config.joint_directions, + joint_ranges=config.left_arm_config.joint_ranges, + ) + + right_arm_config = RebotArm102LeaderTeleopConfig( + id=f"{config.id}_right" if config.id else None, + calibration_dir=config.calibration_dir, + port=config.right_arm_config.port, + baudrate=config.right_arm_config.baudrate, + joint_ids=config.right_arm_config.joint_ids, + joint_directions=config.right_arm_config.joint_directions, + joint_ranges=config.right_arm_config.joint_ranges, + ) + + self.left_arm = RebotArm102Leader(left_arm_config) + self.right_arm = RebotArm102Leader(right_arm_config) + + @cached_property + def action_features(self) -> dict[str, type]: + return { + **{f"left_{k}": v for k, v in self.left_arm.action_features.items()}, + **{f"right_{k}": v for k, v in self.right_arm.action_features.items()}, + } + + @cached_property + def feedback_features(self) -> dict[str, type]: + return {} + + @property + def is_connected(self) -> bool: + return self.left_arm.is_connected and self.right_arm.is_connected + + @check_if_already_connected + def connect(self, calibrate: bool = True) -> None: + self.left_arm.connect(calibrate) + self.right_arm.connect(calibrate) + + @property + def is_calibrated(self) -> bool: + return self.left_arm.is_calibrated and self.right_arm.is_calibrated + + def calibrate(self) -> None: + self.left_arm.calibrate() + self.right_arm.calibrate() + + def configure(self) -> None: + self.left_arm.configure() + self.right_arm.configure() + + @check_if_not_connected + def get_action(self) -> RobotAction: + action_dict = {} + action_dict.update({f"left_{k}": v for k, v in self.left_arm.get_action().items()}) + action_dict.update({f"right_{k}": v for k, v in self.right_arm.get_action().items()}) + return action_dict + + def send_feedback(self, feedback: dict[str, float]) -> None: + raise NotImplementedError("Feedback is not implemented for the reBot Arm 102 leader.") + + @check_if_not_connected + def disconnect(self) -> None: + self.left_arm.disconnect() + self.right_arm.disconnect() diff --git a/src/lerobot/teleoperators/bi_rebot_102_leader/config_bi_rebot_102_leader.py b/src/lerobot/teleoperators/bi_rebot_102_leader/config_bi_rebot_102_leader.py new file mode 100644 index 000000000..265ae26c1 --- /dev/null +++ b/src/lerobot/teleoperators/bi_rebot_102_leader/config_bi_rebot_102_leader.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from dataclasses import dataclass + +from ..config import TeleoperatorConfig +from ..rebot_102_leader import RebotArm102LeaderConfig + + +@TeleoperatorConfig.register_subclass("bi_rebot_102_leader") +@dataclass +class BiRebotArm102LeaderConfig(TeleoperatorConfig): + """Configuration class for the bimanual reBot Arm 102 leader teleoperator.""" + + left_arm_config: RebotArm102LeaderConfig + right_arm_config: RebotArm102LeaderConfig diff --git a/src/lerobot/teleoperators/rebot_102_leader/__init__.py b/src/lerobot/teleoperators/rebot_102_leader/__init__.py new file mode 100644 index 000000000..a13524707 --- /dev/null +++ b/src/lerobot/teleoperators/rebot_102_leader/__init__.py @@ -0,0 +1,20 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .config_rebot_102_leader import RebotArm102LeaderConfig, RebotArm102LeaderTeleopConfig +from .rebot_102_leader import RebotArm102Leader + +__all__ = ["RebotArm102Leader", "RebotArm102LeaderConfig", "RebotArm102LeaderTeleopConfig"] diff --git a/src/lerobot/teleoperators/rebot_102_leader/config_rebot_102_leader.py b/src/lerobot/teleoperators/rebot_102_leader/config_rebot_102_leader.py new file mode 100644 index 000000000..d1beea2ed --- /dev/null +++ b/src/lerobot/teleoperators/rebot_102_leader/config_rebot_102_leader.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from dataclasses import dataclass, field + +from ..config import TeleoperatorConfig + + +@dataclass +class RebotArm102LeaderConfig: + """Base configuration class for the Seeed Studio StarArm102 / reBot Arm 102 leader. + + The reBot Arm 102 is a 7-joint (incl. gripper) leader arm driven by FashionStar + UART smart servos. Servo communication goes through ``motorbridge-smart-servo``. + """ + + # USB-to-UART device the leader arm is connected to (e.g. "/dev/ttyUSB0"). + port: str + + baudrate: int = 1_000_000 + + # Servo id of each joint on the UART bus. + joint_ids: dict[str, int] = field( + default_factory=lambda: { + "shoulder_pan": 0, + "shoulder_lift": 1, + "elbow_flex": 2, + "wrist_flex": 3, + "wrist_yaw": 4, + "wrist_roll": 5, + "gripper": 6, + } + ) + + # Per-joint sign applied to raw servo angles so the leader matches the follower + # convention. The gripper additionally carries a scale (e.g. -6) to widen its + # range to the reBot B601 follower's gripper travel. + joint_directions: dict[str, int] = field( + default_factory=lambda: { + "shoulder_pan": -1, + "shoulder_lift": -1, + "elbow_flex": 1, + "wrist_flex": 1, + "wrist_yaw": 1, + "wrist_roll": -1, + "gripper": -6, + } + ) + + # Per-joint [min, max] output range in degrees. Matches the reBot B601 follower + # joint limits so leader actions can drive the follower key-for-key. + joint_ranges: dict[str, list[int]] = field( + default_factory=lambda: { + "shoulder_pan": [-150, 150], + "shoulder_lift": [-170, 1], + "elbow_flex": [-200, 1], + "wrist_flex": [-80, 90], + "wrist_yaw": [-90, 90], + "wrist_roll": [-90, 90], + "gripper": [-270, 0], + } + ) + + +@TeleoperatorConfig.register_subclass("rebot_102_leader") +@dataclass +class RebotArm102LeaderTeleopConfig(TeleoperatorConfig, RebotArm102LeaderConfig): + """Registered configuration for the reBot Arm 102 leader teleoperator.""" + + pass diff --git a/src/lerobot/teleoperators/rebot_102_leader/rebot_102_leader.py b/src/lerobot/teleoperators/rebot_102_leader/rebot_102_leader.py new file mode 100644 index 000000000..f9f10ed69 --- /dev/null +++ b/src/lerobot/teleoperators/rebot_102_leader/rebot_102_leader.py @@ -0,0 +1,207 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import time +from typing import TYPE_CHECKING + +from lerobot.motors import MotorCalibration +from lerobot.types import RobotAction +from lerobot.utils.decorators import check_if_already_connected, check_if_not_connected +from lerobot.utils.import_utils import _motorbridge_smart_servo_available, require_package + +from ..teleoperator import Teleoperator +from .config_rebot_102_leader import RebotArm102LeaderTeleopConfig + +if TYPE_CHECKING or _motorbridge_smart_servo_available: + from motorbridge_smart_servo import FashionStarServo, ServoMonitor +else: + FashionStarServo = None + ServoMonitor = None + +logger = logging.getLogger(__name__) + +_SETTLE_SEC = 0.01 + + +class RebotArm102Leader(Teleoperator): + """Seeed Studio StarArm102 / reBot Arm 102 leader arm. + + A 7-joint (incl. gripper) leader built on FashionStar UART smart servos. Servo + communication is handled by the ``motorbridge-smart-servo`` package; this class + only reads joint angles, so it produces actions but accepts no feedback. + """ + + config_class = RebotArm102LeaderTeleopConfig + name = "rebot_102_leader" + + def __init__(self, config: RebotArm102LeaderTeleopConfig): + require_package("motorbridge-smart-servo", extra="rebot", import_name="motorbridge_smart_servo") + super().__init__(config) + self.config = config + self.bus: FashionStarServo | None = None + self.motor_names = list(config.joint_ids.keys()) + self._last_raw_positions: dict[str, float] = {} + + @property + def action_features(self) -> dict[str, type]: + return {f"{motor}.pos": float for motor in self.motor_names} + + @property + def feedback_features(self) -> dict[str, type]: + return {} + + @property + def is_connected(self) -> bool: + return self.bus is not None + + @check_if_already_connected + def connect(self, calibrate: bool = True) -> None: + logger.info(f"Connecting {self} on {self.config.port}...") + bus = FashionStarServo(self.config.port, baudrate=self.config.baudrate) + try: + for motor_name, motor_id in self.config.joint_ids.items(): + if not bus.ping(motor_id): + raise RuntimeError(f"Servo not found for {motor_name} (id={motor_id}).") + self._last_raw_positions[motor_name] = 0.0 + self.bus = bus + + if not self.is_calibrated and calibrate: + logger.info( + "Mismatch between calibration values in the motor and the calibration file or no calibration file found" + ) + self.calibrate() + + self.configure() + except Exception: + bus.close() + self.bus = None + raise + + logger.info(f"{self} connected.") + + @property + def is_calibrated(self) -> bool: + return bool(self.calibration) and set(self.calibration) == set(self.motor_names) + + def calibrate(self) -> None: + if self.calibration: + user_input = input( + f"Press ENTER to use provided calibration file associated with the id {self.id}, " + "or type 'c' and press ENTER to run calibration: " + ) + if user_input.strip().lower() != "c": + logger.info(f"Using calibration file associated with the id {self.id}") + return + + logger.info(f"\nRunning calibration of {self}") + input( + "\nCalibration: set zero position.\n" + "Manually move the reBot Arm 102 to its zero pose and close the gripper.\n" + "Press ENTER when ready..." + ) + + self.calibration = {} + for motor_name, motor_id in self.config.joint_ids.items(): + self.bus.unlock(motor_id) + time.sleep(_SETTLE_SEC) + self.bus.set_origin_point(motor_id) + range_min, range_max = self.config.joint_ranges[motor_name] + self.calibration[motor_name] = MotorCalibration( + id=motor_id, + drive_mode=0, + homing_offset=0, + range_min=int(range_min), + range_max=int(range_max), + ) + + self._save_calibration() + logger.info(f"Calibration saved to {self.calibration_fpath}") + + def configure(self) -> None: + for motor_id in self.config.joint_ids.values(): + self.bus.unlock(motor_id) + time.sleep(_SETTLE_SEC) + # Reset the multi-turn counter of each servo individually. + for motor_id in self.config.joint_ids.values(): + self.bus.reset_multi_turn(motor_id) + + def _read_raw_positions(self) -> dict[str, float]: + result: dict[int, ServoMonitor | None] = self.bus.sync_monitor(list(self.config.joint_ids.values())) + id_to_name = {v: k for k, v in self.config.joint_ids.items()} + raw_positions: dict[str, float] = {} + for motor_id, monitor in result.items(): + motor_name = id_to_name[motor_id] + if monitor is None: + raise RuntimeError(f"Servo {motor_name} (id={motor_id}) has never responded.") + raw_positions[motor_name] = monitor.angle_deg + return raw_positions + + @staticmethod + def _round_to_valid_range(value: float, min_value: float, max_value: float) -> tuple[float, int]: + """Unwrap a multi-turn angle into the ±180° window centred on (min+max)/2. + + The servo may report an angle that has accumulated extra full rotations + (value = true_angle + N*360). Subtract the nearest whole number of turns + to bring it back into [center-180, center+180]. Returns the unwrapped + angle and the number of turns removed. + """ + center = (min_value + max_value) / 2.0 + turns = round((value - center) / 360.0) + return value - turns * 360.0, abs(turns) + + @check_if_not_connected + def get_action(self) -> RobotAction: + start = time.perf_counter() + try: + raw_positions = self._read_raw_positions() + self._last_raw_positions = raw_positions + except Exception as e: + logger.error(f"Failed to read raw positions: {e}") + logger.warning("[EMERGENCY STOP] Hold the follower arm and cut off the main power to the arms.") + logger.warning( + "[EMERGENCY STOP] Break the teleoperation session and check the leader USB connection or power." + ) + raw_positions = self._last_raw_positions + + action_dict: dict[str, float] = {} + for motor_name in self.motor_names: + range_min, range_max = self.config.joint_ranges[motor_name] + direction = self.config.joint_directions[motor_name] + sign = 1.0 if direction >= 0 else -1.0 + unwrapped, k = self._round_to_valid_range( + raw_positions[motor_name], range_min * sign, range_max * sign + ) + position = unwrapped * direction + if k > 0: + logger.debug( + f"Servo {motor_name} (id={self.config.joint_ids[motor_name]}) wrapped {k} * 360°. " + f"Unwrapped pos: {unwrapped:.1f}° (raw: {raw_positions[motor_name]:.1f}°)" + ) + action_dict[f"{motor_name}.pos"] = max(float(range_min), min(float(range_max), position)) + + dt_ms = (time.perf_counter() - start) * 1e3 + logger.debug(f"{self} read action: {dt_ms:.1f}ms") + return action_dict + + def send_feedback(self, feedback: dict[str, float]) -> None: + raise NotImplementedError("Feedback is not implemented for the reBot Arm 102 leader.") + + @check_if_not_connected + def disconnect(self) -> None: + self.bus.close() + self.bus = None + logger.info(f"{self} disconnected.") diff --git a/src/lerobot/teleoperators/utils.py b/src/lerobot/teleoperators/utils.py index db685f396..5a6d4ecde 100644 --- a/src/lerobot/teleoperators/utils.py +++ b/src/lerobot/teleoperators/utils.py @@ -99,6 +99,14 @@ def make_teleoperator_from_config(config: TeleoperatorConfig) -> "Teleoperator": from .openarm_mini import OpenArmMini return OpenArmMini(config) + elif config.type == "rebot_102_leader": + from .rebot_102_leader import RebotArm102Leader + + return RebotArm102Leader(config) + elif config.type == "bi_rebot_102_leader": + from .bi_rebot_102_leader import BiRebotArm102Leader + + return BiRebotArm102Leader(config) else: try: return cast("Teleoperator", make_device_from_device_class(config)) diff --git a/src/lerobot/utils/import_utils.py b/src/lerobot/utils/import_utils.py index ef03367eb..5dbce2c5b 100644 --- a/src/lerobot/utils/import_utils.py +++ b/src/lerobot/utils/import_utils.py @@ -114,6 +114,10 @@ _dynamixel_sdk_available = is_package_available("dynamixel-sdk", import_name="dy _feetech_sdk_available = is_package_available("feetech-servo-sdk", import_name="scservo_sdk") _reachy2_sdk_available = is_package_available("reachy2_sdk") _can_available = is_package_available("python-can", "can") +_motorbridge_available = is_package_available("motorbridge") +_motorbridge_smart_servo_available = is_package_available( + "motorbridge-smart-servo", import_name="motorbridge_smart_servo" +) _unitree_sdk_available = is_package_available("unitree-sdk2py", "unitree_sdk2py") _pyrealsense2_available = is_package_available("pyrealsense2") or is_package_available( "pyrealsense2-macosx", import_name="pyrealsense2" diff --git a/tests/robots/test_rebot_b601_follower.py b/tests/robots/test_rebot_b601_follower.py new file mode 100644 index 000000000..553675be0 --- /dev/null +++ b/tests/robots/test_rebot_b601_follower.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math +from unittest.mock import MagicMock, patch + +import pytest + +from lerobot.robots.bi_rebot_b601_follower import BiRebotB601Follower, BiRebotB601FollowerConfig +from lerobot.robots.rebot_b601_follower import ( + RebotB601Follower, + RebotB601FollowerConfig, + RebotB601FollowerRobotConfig, +) + +_MODULE = "lerobot.robots.rebot_b601_follower.rebot_b601_follower" + + +def _make_motor_mock(position_rad: float = 0.0) -> MagicMock: + motor = MagicMock(name="MotorMock") + state = MagicMock() + state.pos = position_rad + motor.get_state.return_value = state + return motor + + +def _make_bus_mock() -> MagicMock: + bus = MagicMock(name="MotorBridgeControllerMock") + # add_damiao_motor returns a fresh motor mock; position encodes the call order. + bus._motor_count = 0 + + def _add_motor(_send_id, _recv_id, _model): + bus._motor_count += 1 + return _make_motor_mock(position_rad=math.radians(bus._motor_count)) + + bus.add_damiao_motor.side_effect = _add_motor + return bus + + +@pytest.fixture +def follower(): + bus_mock = _make_bus_mock() + with ( + patch(f"{_MODULE}.require_package", lambda *a, **kw: None), + patch(f"{_MODULE}.MotorBridgeController") as controller_cls, + patch(f"{_MODULE}.MotorBridgeMode", MagicMock()), + ): + controller_cls.from_dm_serial.return_value = bus_mock + cfg = RebotB601FollowerRobotConfig(port="/dev/null") + robot = RebotB601Follower(cfg) + robot.connect(calibrate=False) + yield robot + if robot.is_connected: + robot.disconnect() + + +def test_features_match_joints(): + with patch(f"{_MODULE}.require_package", lambda *a, **kw: None): + robot = RebotB601Follower(RebotB601FollowerRobotConfig(port="/dev/null")) + expected = {f"{m}.pos" for m in robot.motor_names} + assert set(robot.action_features) == expected + assert set(robot.observation_features) == expected + assert "gripper.pos" in expected + + +def test_connect_disconnect(follower): + assert follower.is_connected + follower.disconnect() + assert not follower.is_connected + + +def test_get_observation_converts_to_degrees(follower): + obs = follower.get_observation() + assert set(obs) == {f"{m}.pos" for m in follower.motor_names} + # The bus mock seeds each motor's position with its 1-indexed creation order (radians). + for idx, motor in enumerate(follower.motor_names, 1): + assert obs[f"{motor}.pos"] == pytest.approx(math.degrees(math.radians(idx))) + + +def test_send_action_clips_to_joint_limits(follower): + # shoulder_pan limit is (-145, 145); request beyond the upper bound. + returned = follower.send_action({"shoulder_pan.pos": 999.0}) + assert returned["shoulder_pan.pos"] == 145.0 + follower.motors["shoulder_pan"].send_pos_vel.assert_called_once() + + +def test_send_action_routes_gripper_to_force_pos(follower): + follower.send_action({"gripper.pos": -10.0}) + follower.motors["gripper"].send_force_pos.assert_called_once() + follower.motors["gripper"].send_pos_vel.assert_not_called() + + +def test_bimanual_prefixes_features(): + with patch(f"{_MODULE}.require_package", lambda *a, **kw: None): + cfg = BiRebotB601FollowerConfig( + left_arm_config=RebotB601FollowerConfig(port="/dev/null0"), + right_arm_config=RebotB601FollowerConfig(port="/dev/null1"), + ) + robot = BiRebotB601Follower(cfg) + assert any(k.startswith("left_") for k in robot.action_features) + assert any(k.startswith("right_") for k in robot.action_features) + assert "left_gripper.pos" in robot.action_features + assert "right_gripper.pos" in robot.action_features diff --git a/tests/teleoperators/test_rebot_102_leader.py b/tests/teleoperators/test_rebot_102_leader.py new file mode 100644 index 000000000..bea10e131 --- /dev/null +++ b/tests/teleoperators/test_rebot_102_leader.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python + +# Copyright 2026 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from unittest.mock import MagicMock, patch + +import pytest + +from lerobot.teleoperators.bi_rebot_102_leader import BiRebotArm102Leader, BiRebotArm102LeaderConfig +from lerobot.teleoperators.rebot_102_leader import ( + RebotArm102Leader, + RebotArm102LeaderConfig, + RebotArm102LeaderTeleopConfig, +) + +_MODULE = "lerobot.teleoperators.rebot_102_leader.rebot_102_leader" + + +def _make_bus_mock(joint_ids: dict[str, int]) -> MagicMock: + bus = MagicMock(name="FashionStarServoMock") + bus.ping.return_value = True + + def _sync_monitor(ids): + # Report each servo at 5 degrees raw. + monitors = {} + for servo_id in ids: + monitor = MagicMock() + monitor.angle_deg = 5.0 + monitors[servo_id] = monitor + return monitors + + bus.sync_monitor.side_effect = _sync_monitor + return bus + + +@pytest.fixture +def leader(): + cfg = RebotArm102LeaderTeleopConfig(port="/dev/null") + bus_mock = _make_bus_mock(cfg.joint_ids) + with ( + patch(f"{_MODULE}.require_package", lambda *a, **kw: None), + patch(f"{_MODULE}.FashionStarServo", return_value=bus_mock), + ): + teleop = RebotArm102Leader(cfg) + teleop.connect(calibrate=False) + yield teleop + if teleop.is_connected: + teleop.disconnect() + + +def test_action_features_match_joints(): + with patch(f"{_MODULE}.require_package", lambda *a, **kw: None): + teleop = RebotArm102Leader(RebotArm102LeaderTeleopConfig(port="/dev/null")) + assert set(teleop.action_features) == {f"{m}.pos" for m in teleop.motor_names} + assert teleop.feedback_features == {} + + +def test_connect_disconnect(leader): + assert leader.is_connected + leader.disconnect() + assert not leader.is_connected + + +def test_get_action_applies_direction_and_clamp(leader): + action = leader.get_action() + assert set(action) == {f"{m}.pos" for m in leader.motor_names} + # shoulder_pan has direction -1, so a +5deg raw reading flips to -5deg. + assert action["shoulder_pan.pos"] == pytest.approx(-5.0) + # Every joint stays within its configured range. + for motor, value in action.items(): + lo, hi = leader.config.joint_ranges[motor.removesuffix(".pos")] + assert lo <= value <= hi + + +def test_send_feedback_not_implemented(leader): + with pytest.raises(NotImplementedError): + leader.send_feedback({}) + + +def test_bimanual_prefixes_features(): + with patch(f"{_MODULE}.require_package", lambda *a, **kw: None): + cfg = BiRebotArm102LeaderConfig( + left_arm_config=RebotArm102LeaderConfig(port="/dev/null0"), + right_arm_config=RebotArm102LeaderConfig(port="/dev/null1"), + ) + teleop = BiRebotArm102Leader(cfg) + assert any(k.startswith("left_") for k in teleop.action_features) + assert any(k.startswith("right_") for k in teleop.action_features) + assert "left_gripper.pos" in teleop.action_features + assert "right_gripper.pos" in teleop.action_features diff --git a/uv.lock b/uv.lock index 408a9a351..692029986 100644 --- a/uv.lock +++ b/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 2 +revision = 3 requires-python = ">=3.12" resolution-markers = [ "(python_full_version >= '3.15' and platform_machine == 'AMD64' and sys_platform == 'linux') or (python_full_version >= '3.15' and platform_machine == 'x86_64' and sys_platform == 'linux')", @@ -1142,7 +1142,7 @@ name = "decord" version = "0.6.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numpy", marker = "(platform_machine != 'arm64' and sys_platform == 'darwin') or (platform_machine == 'AMD64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'darwin' and sys_platform != 'linux')" }, + { name = "numpy", marker = "(platform_machine != 'arm64' and platform_machine != 's390x' and sys_platform == 'darwin') or (platform_machine == 'AMD64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine != 's390x' and sys_platform != 'darwin' and sys_platform != 'linux')" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/11/79/936af42edf90a7bd4e41a6cac89c913d4b47fa48a26b042d5129a9242ee3/decord-0.6.0-py3-none-manylinux2010_x86_64.whl", hash = "sha256:51997f20be8958e23b7c4061ba45d0efcd86bffd5fe81c695d0befee0d442976", size = 13602299, upload-time = "2021-06-14T21:30:55.486Z" }, @@ -2710,6 +2710,8 @@ all = [ { name = "matplotlib" }, { name = "metaworld" }, { name = "mock-serial", marker = "sys_platform != 'win32'" }, + { name = "motorbridge" }, + { name = "motorbridge-smart-servo" }, { name = "mypy" }, { name = "num2words" }, { name = "pandas" }, @@ -2913,6 +2915,12 @@ metaworld = [ { name = "scipy" }, { name = "torchcodec", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'AMD64' and sys_platform == 'linux') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'arm64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or sys_platform == 'win32'" }, ] +motorbridge-dep = [ + { name = "motorbridge" }, +] +motorbridge-smart-servo-dep = [ + { name = "motorbridge-smart-servo" }, +] multi-task-dit = [ { name = "diffusers" }, { name = "transformers" }, @@ -2972,6 +2980,10 @@ qwen-vl-utils-dep = [ reachy2 = [ { name = "reachy2-sdk" }, ] +rebot = [ + { name = "motorbridge" }, + { name = "motorbridge-smart-servo" }, +] robstride = [ { name = "python-can" }, ] @@ -3116,6 +3128,8 @@ requires-dist = [ { name = "lerobot", extras = ["matplotlib-dep"], marker = "extra == 'sarm'" }, { name = "lerobot", extras = ["matplotlib-dep"], marker = "extra == 'unitree-g1'" }, { name = "lerobot", extras = ["metaworld"], marker = "extra == 'all'" }, + { name = "lerobot", extras = ["motorbridge-dep"], marker = "extra == 'rebot'" }, + { name = "lerobot", extras = ["motorbridge-smart-servo-dep"], marker = "extra == 'rebot'" }, { name = "lerobot", extras = ["multi-task-dit"], marker = "extra == 'all'" }, { name = "lerobot", extras = ["notebook"], marker = "extra == 'dev'" }, { name = "lerobot", extras = ["openarms"], marker = "extra == 'all'" }, @@ -3142,6 +3156,7 @@ requires-dist = [ { name = "lerobot", extras = ["qwen-vl-utils-dep"], marker = "extra == 'sarm'" }, { name = "lerobot", extras = ["qwen-vl-utils-dep"], marker = "extra == 'wallx'" }, { name = "lerobot", extras = ["reachy2"], marker = "extra == 'all'" }, + { name = "lerobot", extras = ["rebot"], marker = "extra == 'all'" }, { name = "lerobot", extras = ["robstride"], marker = "extra == 'all'" }, { name = "lerobot", extras = ["sarm"], marker = "extra == 'all'" }, { name = "lerobot", extras = ["scipy-dep"], marker = "extra == 'aloha'" }, @@ -3174,6 +3189,8 @@ requires-dist = [ { name = "meshcat", marker = "extra == 'unitree-g1'", specifier = ">=0.3.0,<0.4.0" }, { name = "metaworld", marker = "extra == 'metaworld'", specifier = "==3.0.0" }, { name = "mock-serial", marker = "sys_platform != 'win32' and extra == 'test'", specifier = ">=0.0.1,<0.1.0" }, + { name = "motorbridge", marker = "extra == 'motorbridge-dep'", specifier = ">=0.3.2,<0.4.0" }, + { name = "motorbridge-smart-servo", marker = "extra == 'motorbridge-smart-servo-dep'", specifier = ">=0.0.4,<0.1.0" }, { name = "mypy", marker = "extra == 'dev'", specifier = ">=1.19.1" }, { name = "ninja", marker = "extra == 'groot'", specifier = ">=1.11.1,<2.0.0" }, { name = "num2words", marker = "extra == 'smolvla'", specifier = ">=0.5.14,<0.6.0" }, @@ -3227,7 +3244,7 @@ requires-dist = [ { name = "transformers", marker = "extra == 'transformers-dep'", specifier = ">=5.4.0,<5.6.0" }, { name = "wandb", marker = "extra == 'training'", specifier = ">=0.24.0,<0.25.0" }, ] -provides-extras = ["dataset", "training", "hardware", "viz", "core-scripts", "evaluation", "dataset-viz", "av-dep", "pygame-dep", "placo-dep", "transformers-dep", "grpcio-dep", "can-dep", "peft-dep", "scipy-dep", "diffusers-dep", "qwen-vl-utils-dep", "matplotlib-dep", "pyserial-dep", "deepdiff-dep", "pynput-dep", "pyzmq-dep", "feetech", "dynamixel", "damiao", "robstride", "openarms", "gamepad", "hopejr", "lekiwi", "unitree-g1", "reachy2", "kinematics", "intelrealsense", "phone", "diffusion", "wallx", "pi", "smolvla", "multi-task-dit", "groot", "sarm", "xvla", "eo1", "hilserl", "async", "peft", "dev", "notebook", "test", "video-benchmark", "aloha", "pusht", "libero", "metaworld", "all"] +provides-extras = ["dataset", "training", "hardware", "viz", "core-scripts", "evaluation", "dataset-viz", "av-dep", "pygame-dep", "placo-dep", "transformers-dep", "grpcio-dep", "can-dep", "peft-dep", "scipy-dep", "diffusers-dep", "qwen-vl-utils-dep", "matplotlib-dep", "pyserial-dep", "deepdiff-dep", "pynput-dep", "pyzmq-dep", "motorbridge-dep", "motorbridge-smart-servo-dep", "feetech", "dynamixel", "damiao", "robstride", "openarms", "gamepad", "hopejr", "lekiwi", "unitree-g1", "reachy2", "rebot", "kinematics", "intelrealsense", "phone", "diffusion", "wallx", "pi", "smolvla", "multi-task-dit", "groot", "sarm", "xvla", "eo1", "hilserl", "async", "peft", "dev", "notebook", "test", "video-benchmark", "aloha", "pusht", "libero", "metaworld", "all"] [[package]] name = "librt" @@ -3653,6 +3670,35 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/98/c2/8c1e6bf77cf62a10203a107179e34e0965fc5369386e0b7034a247ed054d/mock_serial-0.0.1-py3-none-any.whl", hash = "sha256:b6b8cc10c302354bf3ca270a3d4d6bf199c4bbe41478c65046db8f30ea967675", size = 6080, upload-time = "2021-11-23T09:34:51.108Z" }, ] +[[package]] +name = "motorbridge" +version = "0.3.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/58/f2/b824ac4d611c71020dccdb72fc50606e543c77c68455ea824b26d9a6de03/motorbridge-0.3.2.tar.gz", hash = "sha256:5cf85dd22c46c7f3c5e6981e90b1034af2deb1bc4e7d74c13074d1d4a7b75ceb", size = 30158, upload-time = "2026-05-18T07:13:17.239Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/1a/7d367039a8325c0e2796c14a1503dfc563e7b244c815b26e079114244b4b/motorbridge-0.3.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8ad158928e93fafd2a7814eaffe8e6ecbec4686f64c2df85f80d7979dfc82532", size = 1108065, upload-time = "2026-05-18T07:13:04.669Z" }, + { url = "https://files.pythonhosted.org/packages/fe/d6/fafa2b8a3635a6fe7f6e8129e140a68d30f4d6438350a86e51b8198b7834/motorbridge-0.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2adde5f26ea4e37d05da6b41b03b637efa6c80db4676bc6dbdb91ac6e811e54a", size = 1184657, upload-time = "2026-05-18T07:13:06.081Z" }, + { url = "https://files.pythonhosted.org/packages/d8/30/aca01e81ec523d37b98a1ce6e41688d31827625eb15ecf0cf0485d91d62c/motorbridge-0.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a03b6dc0be80db7b47d3f190f8c6f4fc43b0b4089235283f53763153a6d4e58c", size = 1201394, upload-time = "2026-05-18T07:13:07.476Z" }, + { url = "https://files.pythonhosted.org/packages/70/eb/97b2f93682a1ce67bad50e9b598af889be4a3156ebcec129ebb41fa44e5b/motorbridge-0.3.2-cp312-cp312-win_amd64.whl", hash = "sha256:b0657d47aa94f8535d0663538be4a86c46e314303fba513122d17612b584c6e6", size = 839087, upload-time = "2026-05-18T07:13:08.664Z" }, + { url = "https://files.pythonhosted.org/packages/6e/b0/03246c25ae67c2b33bd19b5d11bae668bb8baa7d9cbd75b035a8bef61d62/motorbridge-0.3.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f305a69c7c3c91dca19c43084beb4cd30a93fd85ff35c712cc3fb0ae33a5c7d3", size = 1108065, upload-time = "2026-05-18T07:13:10.032Z" }, + { url = "https://files.pythonhosted.org/packages/a9/40/b82d86fbfcc6b18946567f15a7d76d1c673d43bc0c8d268b668506811981/motorbridge-0.3.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:778fdde2b12df20184fb8c8f4c7665919d969bd582589a267c7956d4c57336ad", size = 1184657, upload-time = "2026-05-18T07:13:11.812Z" }, + { url = "https://files.pythonhosted.org/packages/f2/3e/90e41d798814db89605d9a021e0c182608aec3d40eef2be211427e2bb863/motorbridge-0.3.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eac3a2d27ca387e8d537ec148bea0c28b9517ff4fb9ea0b12f6e78c1e9a7faa4", size = 1201393, upload-time = "2026-05-18T07:13:13.396Z" }, + { url = "https://files.pythonhosted.org/packages/34/75/3c9ba7514fd0ec330c1fe0b4d76dedfd221abc1b750fe063b6e3f9a88075/motorbridge-0.3.2-cp313-cp313-win_amd64.whl", hash = "sha256:d7d1eb76ae29e8673a320fd1a86b944fb0869129fd4114f0983e43cd48f67372", size = 839087, upload-time = "2026-05-18T07:13:14.555Z" }, + { url = "https://files.pythonhosted.org/packages/87/33/6787dd22914291a640c2821f175abc7cbb9a1e0fe6c1143f92d7ac362903/motorbridge-0.3.2-cp314-cp314-win_amd64.whl", hash = "sha256:c5f05e36c6607d2145f38fb6f1f11090bb01dbd1012e8251b0d2ae4d60fa4f50", size = 870167, upload-time = "2026-05-18T07:13:15.898Z" }, +] + +[[package]] +name = "motorbridge-smart-servo" +version = "0.0.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e6/56/45af87189dc49abbe46157b792b7c71f502a5f819f04e7485de0cfa52d9b/motorbridge_smart_servo-0.0.4.tar.gz", hash = "sha256:fb65f3f6e765e6b1915071c255caaf112fad3796fa1761aeee0132d15b8a0989", size = 20415, upload-time = "2026-05-08T09:24:57.563Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e9/ee/bec4b3acf55cd18e7db83a6d951caccf699533dbd038c1f0b5f2d16d5208/motorbridge_smart_servo-0.0.4-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:8bc1f034fa9f96e23229a834db6e7cfe1368dba7b9a2a6f6dbd316448c4390dc", size = 304384, upload-time = "2026-05-08T09:24:52.619Z" }, + { url = "https://files.pythonhosted.org/packages/3f/d2/71c87063b826433553ce8869b99df3e4f191b107710dd5c905e637512b10/motorbridge_smart_servo-0.0.4-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:348cef6a647e5c7f9cc8e8ce1f3c806af4522e1087172bac2f8a1a0daa3592b6", size = 345668, upload-time = "2026-05-08T09:24:53.735Z" }, + { url = "https://files.pythonhosted.org/packages/9b/6b/e65e7227a510236c6334cf054c501d3de2cbd463f4c594e42c6e965d5143/motorbridge_smart_servo-0.0.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8c1982643c496c9f425fa9238f9a92ba601d77f4f2279df68c6868e7b997cbe1", size = 348123, upload-time = "2026-05-08T09:24:55.191Z" }, + { url = "https://files.pythonhosted.org/packages/2d/fa/539ea123a5660c22c5e5cdad62d7bc5e931c816a0ffd402ae6e4623ab45b/motorbridge_smart_servo-0.0.4-cp39-abi3-win_amd64.whl", hash = "sha256:ea3baa9ba25bcec5541f3d86d73a3406ba2fcffe5dbf900c22e058638fc31ab0", size = 194130, upload-time = "2026-05-08T09:24:56.369Z" }, +] + [[package]] name = "mpmath" version = "1.3.0" From ca8c60a0ed0778fc6af7bc511f0b63e28a3a4612 Mon Sep 17 00:00:00 2001 From: von Neumann 101 <2961978672@qq.com> Date: Tue, 19 May 2026 20:06:41 +0800 Subject: [PATCH 03/17] Set OpenCV fourcc after size and fps (#3620) * Set OpenCV fourcc after size and fps * Set OpenCV fourcc last on Windows * Add comment explaining DSHOW fourcc ordering --- src/lerobot/cameras/opencv/camera_opencv.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lerobot/cameras/opencv/camera_opencv.py b/src/lerobot/cameras/opencv/camera_opencv.py index f3289ddc7..3e92eaf06 100644 --- a/src/lerobot/cameras/opencv/camera_opencv.py +++ b/src/lerobot/cameras/opencv/camera_opencv.py @@ -199,12 +199,13 @@ class OpenCVCamera(Camera): DeviceNotConnectedError: If the camera is not connected. """ - # Set FOURCC first (if specified) as it can affect available FPS/resolution options - if self.config.fourcc is not None: - self._validate_fourcc() if self.videocapture is None: raise DeviceNotConnectedError(f"{self} videocapture is not initialized") + set_fourcc_after_size_and_fps = platform.system() == "Windows" + if self.config.fourcc is not None and not set_fourcc_after_size_and_fps: + self._validate_fourcc() + default_width = int(round(self.videocapture.get(cv2.CAP_PROP_FRAME_WIDTH))) default_height = int(round(self.videocapture.get(cv2.CAP_PROP_FRAME_HEIGHT))) @@ -222,6 +223,11 @@ class OpenCVCamera(Camera): else: self._validate_fps() + if self.config.fourcc is not None and set_fourcc_after_size_and_fps: + # On Windows with DSHOW, changing the resolution can silently override the FOURCC setting. + # Set FOURCC last to make sure the requested pixel format is actually enforced. + self._validate_fourcc() + def _validate_fps(self) -> None: """Validates and sets the camera's frames per second (FPS).""" From 7ab4936b1bca9e349250f6a7bb4e24fcd9667cca Mon Sep 17 00:00:00 2001 From: Pepijn <138571049+pkooij@users.noreply.github.com> Date: Tue, 19 May 2026 14:46:11 +0200 Subject: [PATCH 04/17] Add extensive language support (#3467) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add extensive language support * Address review: split persistent/event schemas, drop event timestamps - recipe.py: derive _VALID_ROLES/_VALID_STREAMS from MessageRole/MessageStream Literals - dataset_metadata.py: keep CODEBASE_VERSION at v3.0 - language.py: remove RESERVED_STYLES; split arrow/feature schemas into persistent (with timestamp) and event (without timestamp); add docstrings - language_render.py: events use frame-row timestamp implicitly; no per-event timestamp filtering or sorting - converters.py: drop unused subtask_key passthrough - add docstrings to new public APIs (recipe, render_messages_processor, collate) - update tests for split schemas; revert uv.lock Co-Authored-By: Claude Opus 4.7 (1M context) * Add docstrings to all new helpers; revert uv.lock Covers private helpers in recipe.py, language.py, language_render.py, and render_messages_processor.py. Also reverts uv.lock to main (it was re-generated by `uv run` during local checks). Co-Authored-By: Claude Opus 4.7 (1M context) * feat(language): add motion (persistent) and trace (event-only) styles Promote the previously-reserved motion/trace styles to first-class core styles. motion routes to language_persistent (it tracks robot state over time); trace routes to language_events (single-moment annotations). Co-Authored-By: Claude Opus 4.7 (1M context) * feat(language): per-camera tagging on view-dependent styles Adds a nullable `camera` field to the language row struct (both persistent and event variants) so view-dependent styles like `vqa` can carry which `observation.images.*` view they were grounded against. Without this, multi-camera datasets ended up with multiple `(vqa, role)` rows at the same timestamp that the resolver could not disambiguate. - `language.py`: add `camera` to PERSISTENT_ROW_FIELDS / EVENT_ROW_FIELDS, to both Arrow struct types and the HF datasets feature mappings; introduce VIEW_DEPENDENT_STYLES = {vqa, motion, trace} plus `is_view_dependent_style` and `validate_camera_field` helpers (camera required iff style is view-dependent). - `language_render.py`: thread an optional `camera=` kwarg through every resolver (`active_at`, `emitted_at`, `nth_prev`, `nth_next`) and through `_matching_rows` / `_select_*`, so recipes can disambiguate per-camera VQA with `emitted_at(t, style=vqa, role=assistant, camera=...)`. Without a `camera` filter, multi-row matches keep raising the existing ambiguity error — which is the desired behaviour on multi-camera data. - `recipes/pi05_hirobot.yaml`: replace the single `ask_vqa` branch with `ask_vqa_top` and `ask_vqa_wrist` per-camera sub-recipes (each carrying the matching image block), keeping the original 0.20 budget and documenting the customization point for datasets with different cameras. - Tests: schema test asserts the new field order; new tests cover `is_view_dependent_style`, `validate_camera_field` (both required and forbidden directions), per-camera `emitted_at` filtering, and the ambiguity error when two cameras emit `(vqa, assistant)` at the same timestamp without a `camera=` filter. RenderMessagesStep + dataset passthrough fixtures updated to include the new field. - `docs/source/language_and_recipes.mdx`: document the `camera` field, the per-camera resolver pattern, and the canonical recipe convention. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(language): drop motion from VIEW_DEPENDENT_STYLES Motion primitives are described in robot-frame (joint / Cartesian) terms, not pixel space, so they are camera-agnostic. Only `vqa` (event) and `trace` (event, pixel-trajectory) are view-dependent. The `camera` field stays on PERSISTENT_ROW_FIELDS for schema symmetry — the validator, resolver, and HF feature mapping behave identically across the two columns regardless of which styles populate `camera` today — but persistent rows now always have `camera=None` in practice. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(language): task_aug style + automatic ${task} rephrasing rotation Adds task-prompt diversity (Xiao 2022 / CAST) without touching ``meta/tasks.parquet`` or forcing recipes to opt in. The plan reserved ``task_aug`` as a future style; this lands it now. - ``language.py``: add ``task_aug`` to ``CORE_STYLES`` and ``PERSISTENT_STYLES``. ``column_for_style("task_aug")`` returns ``language_persistent`` so PR 2 writers route it correctly. - ``language_render.py``: ``_resolve_task`` now consults the persistent slice for rows of ``style="task_aug", role="user"``. When any exist it picks one deterministically by ``sample_idx`` (blake2b-keyed, not Python's randomized hash) so an epoch sees every rephrasing of every episode while the same sample still resolves identically across reruns. Falls back to the canonical ``meta/tasks.parquet`` task when no rephrasings are present, so existing datasets and unannotated runs keep their behaviour. Explicit ``task=`` overrides still win. - Tests: rephrasing coverage across samples, determinism on repeat ``sample_idx``, fallback when persistent has no ``task_aug`` rows, and explicit override priority. Recipes get this for free: any ``${task}`` placeholder rotates through the available rephrasings. Recipes that want the literal canonical task can override the binding. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(language): tool catalog in meta/info.json + LeRobotDatasetMetadata.tools Stores OpenAI-style function schemas at ``meta/info.json["tools"]`` so datasets can declare which tools are available (today: just ``say``; tomorrow: per-dataset extensions). The ``DEFAULT_TOOLS`` constant fills in for unannotated datasets so chat-template consumers don't have to special-case anything. Three pieces: - ``language.py``: ``SAY_TOOL_SCHEMA`` and ``DEFAULT_TOOLS`` constants. Single source of truth — PR 2's writer and PR 3's runtime tool registry will both import from here instead of duplicating the dict. - ``dataset_metadata.py``: ``LeRobotDatasetMetadata.tools`` property reads ``info.json["tools"]`` and falls back to ``DEFAULT_TOOLS``. Returns deep-copied dicts so callers can mutate the result safely. - ``docs/source/tools.mdx``: spec page covering the catalog, per-row invocations, and the three-step "how to add a new tool" workflow (declare schema, implement, register). Linked from the docs toctree under the Datasets section. This lays the groundwork for PR 2's pipeline writing the catalog out during annotation, and PR 3's ``src/lerobot/tools/`` package shipping runnable implementations (one file per tool — first up: ``say.py`` wrapping Kyutai's pocket-tts). Co-Authored-By: Claude Opus 4.7 (1M context) * Apply ruff and prettier formatting after merge Co-Authored-By: Claude Opus 4.7 (1M context) * refactor(language): unify resolver dispatch and prune redundant test scaffolding * Drop the unused `events` kwarg from `active_at`/`nth_prev`/`nth_next`; only `emitted_at` actually consults events. The dispatcher in `_resolve_spec` now passes events conditionally. * Replace the dual `_persistent_sort_key`/`_event_sort_key` pair with a single `_row_sort_key` and drop the `sort_key` parameter from `_select_one`. Event rows lack `timestamp` (it is implicit in the frame) and now default to `0.0` for sort purposes — the `(style, role)` tiebreaker is unchanged. * Inline `_select_latest` into `active_at` (its only caller). * Collapse `emitted_at`'s dual-branch into one `_select_one` call. * Tighten `_validate_persistent_resolver` to a single `column_for_style(style) != LANGUAGE_PERSISTENT` check. * Parameterize `test_per_camera_blend_renders_both_views` over the two cameras and factor the sub-recipe builder into `_vqa_subrecipe` so the test no longer hand-rolls two near-identical recipe blocks. Net -98 LOC; behavior, public resolver names, and test expectations unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(language): always raise on ambiguous resolver matches `_select_one` previously skipped its ambiguity check whenever any of `role`/`tool_name`/`camera` was set, on the assumption that the caller had already pinned down a unique row. That left a real ambiguity hole for VQA: with two cameras emitting `(vqa, assistant)` at the same frame, `emitted_at(..., role="assistant")` silently picked the first sorted row instead of telling the recipe to add `camera=...`. The existing `test_emitted_at_raises_on_ambiguous_per_camera_vqa` test already encoded the desired behavior. Tighten the check: any time `len(rows) > 1` we now raise with the selectors echoed back, so users see exactly which fields they passed and that more is needed to disambiguate. Co-Authored-By: Claude Opus 4.7 (1M context) * chore: fix CI — collapse short ValueError to one line, refresh uv.lock * `ruff format` on CI (newer version) wants the short `camera=None` ValueError on a single line. * `uv.lock` was stale relative to `pyproject.toml`'s `datasets>=4.7.0` pin (and picked up upstream `s390x` marker fixes for cuda packages). CI runs `uv sync --locked` which rejected the divergence. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(language): keep base install green — drop processor re-export, gate dataset-extra tests `lerobot.processor` re-exported `RenderMessagesStep` at the package level, so importing anything from `lerobot.processor` pulled in `lerobot.datasets.language` → `lerobot.datasets/__init__.py` → `require_package("datasets")`, which fails in the Tier 1 base install that intentionally omits the `[dataset]` extra. The chain bricked collection for unrelated suites (`tests/policies/pi0_pi05/...`, `tests/envs/...`, etc.). * Stop re-exporting `RenderMessagesStep` from `lerobot.processor`. The only consumer (the test) already imports from the submodule. Document the deliberate omission in the module docstring. * Add `pytest.importorskip("datasets", ...)` (and `pandas` where needed) at the top of the four PR-added tests that exercise the language stack: - tests/datasets/test_language.py - tests/datasets/test_language_render.py - tests/processor/test_render_messages_processor.py - tests/utils/test_collate.py Co-Authored-By: Claude Opus 4.7 (1M context) * fix(language): address review — tools accessor, motion docs, conditional collate * **`meta.tools` actually reads `info.json["tools"]`.** `DatasetInfo` had no `tools` field, so `from_dict` silently dropped the key (it warned about unknown fields then discarded them) and the property always returned `DEFAULT_TOOLS`. Added `tools: list[dict] | None` to the dataclass; `to_dict()` drops it when unset so existing datasets keep a clean `info.json`. Fixed the accessor to read `self.info.tools` (the previous `.get(...)` would have raised AttributeError on the dataclass anyway). Added regression tests: fallback when absent, round-trip from disk, and round-trip through `DatasetInfo.from_dict` / `to_dict`. * **`motion` is not view-dependent — fix the docs.** The mdx claimed rows of style `motion` must carry `camera`, but `VIEW_DEPENDENT_STYLES = {"vqa", "trace"}` and the validator agrees: motion primitives are joint/Cartesian-frame, not pixel-space. Updated both call-out paragraphs in `language_and_recipes.mdx`. * **Conditional `collate_fn` swap.** Added `meta.has_language_columns` and gate the `lerobot_collate_fn` swap in `lerobot_train.py` on it, so non-language datasets keep PyTorch's `default_collate`. Also added a pass-through test in `test_collate.py` that asserts on a plain tensor batch the custom collate matches `default_collate` key-for-key, plus a test for the `None`-sample drop path. Co-Authored-By: Claude Opus 4.7 (1M context) * review: dedupe regex, centralize column names, harden collate, more tests * **#2 — dedupe `_PLACEHOLDER_RE`.** The same regex was compiled in `recipe.py` and `language_render.py`. Promote to module-level `PLACEHOLDER_RE` in `recipe.py` (its primary owner — declares template syntax) and import from `language_render.py`. * **#3 — centralize language column names.** `io_utils.py` had hardcoded `{"language_persistent", "language_events"}` literals at two sites. Replace with `LANGUAGE_COLUMNS` import so a future column rename can't silently desync. * **#4 — defensive collate preserved-keys.** `lerobot_collate_fn` silently filtered language fields from samples that didn't have them, which would hand downstream consumers a preserved list shorter than the tensor batch. Now: if any sample carries a key, every sample in the batch must carry it; otherwise raise a `ValueError` so the upstream rendering bug surfaces at the boundary. * **#5 — `_scalar` rejects non-singleton lists.** Previously a zero- or multi-element list fell through and triggered confusing `float([])` errors downstream. Now raises `ValueError` with the actual length. * **#6 — refactor `_extract_complementary_data`.** Replace 11 lines of `key = {... if ... else {}}` plus an 11-line splat dict with a single `_COMPLEMENTARY_KEYS` tuple iterated once. * **#7 — document `EXTENDED_STYLES`.** Was an empty `set()` with no comment. Add a docstring explaining it's an intentional extension point: downstream modules append project-local styles before `column_for_style` is called. * **#9 — `tools.mdx` notes the runtime layer is future work.** The page referenced `src/lerobot/tools/`, `registry.py`, and `get_tools(meta)` — none exist in this PR. Added a callout at the start of "How to add your own tool" plus a note on the implementations paragraph. * **#10 — tests for YAML round-trip, malformed rows, blend validation.** `test_recipe.py` grew from 1 case to 12 covering: blend-or-messages exclusivity, target-turn requirement, blend emptiness, weight presence/positivity, nested-blend rejection, `from_dict` with nested blends, `from_yaml` / `load_recipe` agreement, top-level non-mapping rejection. Added a malformed-row test for `_normalize_rows` that asserts non-dict entries raise `TypeError`. Co-Authored-By: Claude Opus 4.7 (1M context) * review: emitted_at uses 0.1s tolerance; MessageTurn requires stream at construction * **Float tolerance in `emitted_at` for persistent styles.** The ``_timestamp(row) == t`` exact-equality check silently missed any caller that derived ``t`` arithmetically (e.g. ``frame_idx / fps``) even though the parquet timestamp would only differ by ULPs. Added ``EMITTED_AT_TOLERANCE_S = 0.1`` and check ``abs(...) <= tolerance`` instead, with a docstring explaining why exact equality wasn't enough and why 0.1 s is safe at typical 30–100 Hz control rates. Test asserts the new behavior at half-window (matches) and double-window (no match) using the constant so it stays in sync. * **`MessageTurn.stream` is required at construction.** It was typed ``MessageStream | None = None`` so YAML could omit ``stream:`` and pass the dataclass invariant — but ``_validate_rendered`` rejected ``None`` streams later, surfacing the error at the first sample instead of at recipe load. Now ``__post_init__`` raises ``ValueError`` if ``stream`` is ``None``, with the list of valid streams in the message. The redundant late-stage check in ``_validate_rendered`` is replaced with a one-line comment that cites the upstream invariant. Test pins the new construction-time rejection. Co-Authored-By: Claude Opus 4.7 (1M context) * docs(tools): drop follow-up-PR references Reword the two callouts in `tools.mdx` to describe the runtime layer in present tense ("not part of the catalog layer shipped today", "those modules don't yet exist in the tree") instead of pointing at a specific follow-up PR. Keeps the doc honest about what works now without coupling it to a particular release order. Co-Authored-By: Claude Opus 4.7 (1M context) * review: address CarolinePascal feedback - language timestamps: float64 -> float32 to match LeRobotDataset frame timestamps (Arrow struct + HF feature) - dataset_metadata: hoist `.language` imports to module top — language.py has no lerobot imports, so there is no circular-import risk - dataset_metadata: add a `meta.tools` setter that persists the catalog to info.json and reloads `meta.info` - feature_utils: validate the `language` dtype instead of returning "" — warn (non-fatal) when a non-empty value is written at record time - centralize the scalar-unwrap helper as `lerobot.utils.utils.unwrap_scalar`, shared by render_messages_processor and language_render - docs: move `## Layer 2 — recipe anatomy` ahead of the resolver sections, which describe recipe bindings rather than dataset layout - language_render: note in EMITTED_AT_TOLERANCE_S that persistent rows change on a human-action timescale, not the camera frame rate Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Claude Opus 4.7 (1M context) --- docs/source/_toctree.yml | 6 +- docs/source/dataset_subtask.mdx | 277 --------- docs/source/language_and_recipes.mdx | 147 +++++ docs/source/tools.mdx | 210 +++++++ pyproject.toml | 2 +- src/lerobot/configs/__init__.py | 4 + src/lerobot/configs/recipe.py | 206 +++++++ src/lerobot/datasets/__init__.py | 14 + src/lerobot/datasets/compute_stats.py | 2 +- src/lerobot/datasets/dataset_metadata.py | 47 +- src/lerobot/datasets/dataset_reader.py | 5 - src/lerobot/datasets/feature_utils.py | 41 +- src/lerobot/datasets/io_utils.py | 17 +- src/lerobot/datasets/language.py | 242 ++++++++ src/lerobot/datasets/language_render.py | 545 ++++++++++++++++++ src/lerobot/datasets/utils.py | 8 +- src/lerobot/processor/__init__.py | 7 + src/lerobot/processor/batch_processor.py | 18 + src/lerobot/processor/converters.py | 36 +- .../processor/render_messages_processor.py | 84 +++ src/lerobot/scripts/lerobot_train.py | 6 + src/lerobot/utils/collate.py | 65 +++ src/lerobot/utils/utils.py | 19 + tests/configs/test_recipe.py | 168 ++++++ tests/datasets/test_dataset_metadata.py | 137 +++++ tests/datasets/test_language.py | 173 ++++++ tests/datasets/test_language_render.py | 417 ++++++++++++++ tests/datasets/test_subtask_dataset.py | 193 ------- .../test_render_messages_processor.py | 60 ++ tests/utils/test_collate.py | 84 +++ uv.lock | 2 +- 31 files changed, 2730 insertions(+), 512 deletions(-) delete mode 100644 docs/source/dataset_subtask.mdx create mode 100644 docs/source/language_and_recipes.mdx create mode 100644 docs/source/tools.mdx create mode 100644 src/lerobot/configs/recipe.py create mode 100644 src/lerobot/datasets/language.py create mode 100644 src/lerobot/datasets/language_render.py create mode 100644 src/lerobot/processor/render_messages_processor.py create mode 100644 src/lerobot/utils/collate.py create mode 100644 tests/configs/test_recipe.py create mode 100644 tests/datasets/test_language.py create mode 100644 tests/datasets/test_language_render.py delete mode 100644 tests/datasets/test_subtask_dataset.py create mode 100644 tests/processor/test_render_messages_processor.py create mode 100644 tests/utils/test_collate.py diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index 470319c48..412386e2d 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -39,8 +39,10 @@ title: Porting Large Datasets - local: using_dataset_tools title: Using the Dataset Tools - - local: dataset_subtask - title: Using Subtasks in the Dataset + - local: language_and_recipes + title: Language Columns and Recipes + - local: tools + title: Tools - local: video_encoding_parameters title: Video encoding parameters - local: streaming_video_encoding diff --git a/docs/source/dataset_subtask.mdx b/docs/source/dataset_subtask.mdx deleted file mode 100644 index 6264aca22..000000000 --- a/docs/source/dataset_subtask.mdx +++ /dev/null @@ -1,277 +0,0 @@ -# Using Subtasks in LeRobot Datasets - -Subtask support in robotics datasets has proven effective in improving robot reasoning and understanding. Subtasks are particularly useful for: - -- **Hierarchical policies**: Building policies that include subtask predictions to visualize robot reasoning in real time -- **Reward modeling**: Helping reward models understand task progression (e.g., SARM-style stage-aware reward models) -- **Task decomposition**: Breaking down complex manipulation tasks into atomic, interpretable steps - -LeRobotDataset now supports subtasks as part of its dataset structure, alongside tasks. - -## What are Subtasks? - -While a **task** describes the overall goal (e.g., "Pick up the apple and place it in the basket"), **subtasks** break down the execution into finer-grained steps: - -1. "Approach the apple" -2. "Grasp the apple" -3. "Lift the apple" -4. "Move to basket" -5. "Release the apple" - -Each frame in the dataset can be annotated with its corresponding subtask, enabling models to learn and predict these intermediate stages. - -An overview of subtask annotation showing how frames are labeled with intermediate subtask stages - -

- Figure: Overview of subtask annotation. -

- -**Reference:** _Subtask-learning based for robot self-assembly in flexible collaborative assembly in manufacturing_, Original Article, Published: 19 April 2022. - -## Dataset Structure - -Subtask information is stored in the dataset metadata: - -``` -my-dataset/ -├── data/ -│ └── ... -├── meta/ -│ ├── info.json -│ ├── stats.json -│ ├── tasks.parquet -│ ├── subtasks.parquet # Subtask index → subtask string mapping -│ └── episodes/ -│ └── ... -└── videos/ - └── ... -``` - -### Subtasks Parquet File - -The `meta/subtasks.parquet` file maps subtask indices to their natural language descriptions: - -| subtask_index | subtask (index column) | -| ------------- | ---------------------- | -| 0 | "Approach the apple" | -| 1 | "Grasp the apple" | -| 2 | "Lift the apple" | -| ... | ... | - -### Frame-Level Annotations - -Each frame in the dataset can include a `subtask_index` field that references the subtasks parquet file: - -```python -# Example frame data in the parquet file -{ - "index": 42, - "timestamp": 1.4, - "episode_index": 0, - "task_index": 0, - "subtask_index": 2, # References "Lift the apple" - "observation.state": [...], - "action": [...], -} -``` - -## Annotating Datasets with Subtasks - -We provide a HuggingFace Space for easily annotating any LeRobotDataset with subtasks: - -**[https://huggingface.co/spaces/lerobot/annotate](https://huggingface.co/spaces/lerobot/annotate)** - -After completing your annotation: - -1. Click "Push to Hub" to upload your annotated dataset -2. You can also run the annotation space locally by following the instructions at [github.com/huggingface/lerobot-annotate](https://github.com/huggingface/lerobot-annotate) - -## Loading Datasets with Subtasks - -When you load a dataset with subtask annotations, the subtask information is automatically available: - -```python -from lerobot.datasets import LeRobotDataset - -# Load a dataset with subtask annotations -dataset = LeRobotDataset("jadechoghari/collect-fruit-annotated") - -# Access a sample -sample = dataset[100] - -# The sample includes both task and subtask information -print(sample["task"]) # "Collect the fruit" -print(sample["subtask"]) # "Grasp the apple" -print(sample["task_index"]) # tensor(0) -print(sample["subtask_index"]) # tensor(2) -``` - -### Checking for Subtask Support - -You can check if a dataset has subtask annotations: - -```python -# Check if subtasks are available -has_subtasks = ( - "subtask_index" in dataset.features - and dataset.meta.subtasks is not None -) - -if has_subtasks: - print(f"Dataset has {len(dataset.meta.subtasks)} unique subtasks") - print("Subtasks:", list(dataset.meta.subtasks.index)) -``` - -## Using Subtasks for Training - -### With the Tokenizer Processor - -The `TokenizerProcessor` automatically handles subtask tokenization for Vision-Language Action (VLA) models: - -```python -from lerobot.processor import TokenizerProcessorStep - -# Create a tokenizer processor step -tokenizer_processor = TokenizerProcessorStep( - tokenizer_name_or_path="google/paligemma-3b-pt-224", - padding="max_length", - max_length=64, -) - -# The processor will automatically tokenize subtasks if present in the batch -# and add them to the observation under: -# - "observation.subtask.tokens" -# - "observation.subtask.attention_mask" -``` - -When subtasks are available in the batch, the tokenizer processor adds: - -- `observation.subtask.tokens`: Tokenized subtask text -- `observation.subtask.attention_mask`: Attention mask for the subtask tokens - -### DataLoader with Subtasks - -```python -import torch -from lerobot.datasets import LeRobotDataset - -dataset = LeRobotDataset("jadechoghari/collect-fruit-annotated") - -dataloader = torch.utils.data.DataLoader( - dataset, - batch_size=16, - shuffle=True, -) - -for batch in dataloader: - # Access subtask information in the batch - subtasks = batch["subtask"] # List of subtask strings - subtask_indices = batch["subtask_index"] # Tensor of subtask indices - - # Use for training hierarchical policies or reward models - print(f"Batch subtasks: {set(subtasks)}") -``` - -## Example Datasets with Subtask Annotations - -Try loading a dataset with subtask annotations: - -```python -from lerobot.datasets import LeRobotDataset - -# Example dataset with subtask annotations -dataset = LeRobotDataset("jadechoghari/collect-fruit-annotated") - -# Explore the subtasks -print("Available subtasks:") -for subtask_name in dataset.meta.subtasks.index: - print(f" - {subtask_name}") - -# Get subtask distribution -subtask_counts = {} -for i in range(len(dataset)): - sample = dataset[i] - subtask = sample["subtask"] - subtask_counts[subtask] = subtask_counts.get(subtask, 0) + 1 - -print("\nSubtask distribution:") -for subtask, count in sorted(subtask_counts.items(), key=lambda x: -x[1]): - print(f" {subtask}: {count} frames") -``` - -## Use Cases - -### 1. Hierarchical Policy Training - -Train policies that predict both actions and current subtask: - -```python -class HierarchicalPolicy(nn.Module): - def __init__(self, num_subtasks): - super().__init__() - self.action_head = nn.Linear(hidden_dim, action_dim) - self.subtask_head = nn.Linear(hidden_dim, num_subtasks) - - def forward(self, observations): - features = self.encoder(observations) - actions = self.action_head(features) - subtask_logits = self.subtask_head(features) - return actions, subtask_logits -``` - -### 2. Stage-Aware Reward Modeling (SARM) - -Build reward models that understand task progression: - -```python -# SARM predicts: -# - Stage: Which subtask is being executed (discrete) -# - Progress: How far along the subtask (continuous 0-1) - -class SARMRewardModel(nn.Module): - def forward(self, observations): - features = self.encoder(observations) - stage_logits = self.stage_classifier(features) - progress = self.progress_regressor(features) - return stage_logits, progress -``` - -### 3. Progress Visualization - -Monitor robot execution by tracking subtask progression: - -```python -def visualize_execution(model, observations): - for t, obs in enumerate(observations): - action, subtask_logits = model(obs) - predicted_subtask = subtask_names[subtask_logits.argmax()] - print(f"t={t}: Executing '{predicted_subtask}'") -``` - -## API Reference - -### LeRobotDataset Properties - -| Property | Type | Description | -| --------------------------- | ---------------------- | ------------------------------------------ | -| `meta.subtasks` | `pd.DataFrame \| None` | DataFrame mapping subtask names to indices | -| `features["subtask_index"]` | `dict` | Feature spec for subtask_index if present | - -### Sample Keys - -When subtasks are available, each sample includes: - -| Key | Type | Description | -| --------------- | -------------- | ------------------------------------ | -| `subtask_index` | `torch.Tensor` | Integer index of the current subtask | -| `subtask` | `str` | Natural language subtask description | - -## Related Resources - -- [SARM Paper](https://arxiv.org/pdf/2509.25358) - Stage-Aware Reward Modeling for Long Horizon Robot Manipulation -- [LeRobot Annotate Space](https://huggingface.co/spaces/lerobot/annotate) - Interactive annotation tool -- [LeRobotDataset v3.0](./lerobot-dataset-v3) - Dataset format documentation diff --git a/docs/source/language_and_recipes.mdx b/docs/source/language_and_recipes.mdx new file mode 100644 index 000000000..4181dbe34 --- /dev/null +++ b/docs/source/language_and_recipes.mdx @@ -0,0 +1,147 @@ +# Language columns and recipes + +Most LeRobot datasets ship with a single `task` string per episode — fine for +short, single-instruction skills, but not enough for the longer-horizon, +multi-modal robot policies the field is moving toward (high-level planning, +memory, interjections, VQA, tool use). To support those policies without +forking the dataset format, LeRobot extends `LeRobotDataset` with two optional +language columns and a small recipe layer that turns those rows into +chat-style training samples on the fly. + +The design splits cleanly into three layers: + +1. **Data in the dataset** — language annotations stored next to frames in + `data/chunk-*/file-*.parquet` as two optional columns (`language_persistent` + and `language_events`). Datasets without these columns keep their existing + behavior. +2. **Recipe** — a YAML file that declares which annotation rows to bind and + how to lay them out as chat turns (`role`, `content`, optional images, + optional tool calls). Recipes are pure config; no Python required to add a + new one. +3. **Training format** — at sample time, `RenderMessagesStep` resolves the + recipe against the per-frame annotations and emits HF-style `messages` plus + LeRobot-specific sidecars (`message_streams`, `target_message_indices`) + that policy processors consume. + +This page describes each layer in turn. + +## Layer 1 — language columns in the dataset + +The two optional columns live next to frame data in +`data/chunk-*/file-*.parquet`: + +- `language_persistent`: a list of rows broadcast across every frame in an episode for state that remains active, such as `subtask`, `plan`, and `memory`. +- `language_events`: a list of rows only on the exact frame where an event was emitted, such as `interjection`, `vqa`, and speech tool calls. + +Both columns share the same row shape (event rows omit `timestamp` because the +frame the row sits on already provides it): + +```text +role: string +content: string | null +style: string | null +timestamp: float32 # persistent rows only +camera: string | null # observation.images.* feature key, view-dependent rows only +tool_calls: list[Json] | null +``` + +The `camera` field tags rows whose `content` is grounded in a specific camera +view. Rows of view-dependent styles (`vqa` and `trace`) MUST set `camera` to +the matching `observation.images.*` feature key. Rows of every other style — +including `motion`, which describes robot-frame primitives in joint / Cartesian +terms — MUST leave `camera` as `null`. Pipeline writers and the validator +enforce this via `validate_camera_field(style, camera)`. + +`meta/tasks.parquet` remains the canonical source for the task. The special `${task}` recipe binding always reads that task string and does not depend on language annotations. + +### Architecture + +The language stack itself has three internal modules backing layer 1: + +1. `lerobot.datasets.language` defines the schema, style registry, and `column_for_style`. +2. `lerobot.datasets.language_render` resolves rows and renders messages. +3. `RenderMessagesStep` turns dataset samples into `messages`, `message_streams`, and `target_message_indices`. + +`LeRobotDataset` stays recipe-agnostic. It passes `language_persistent` and `language_events` through when present, and unannotated datasets keep their existing behavior. + +## Layer 2 — recipe anatomy + +Recipes are YAML files backed by `TrainingRecipe` and `MessageTurn`. They +declare which annotation rows to pull (via `bindings`) and how to compose them +into chat turns (`messages`). + +```yaml +messages: + - { role: user, content: "${task}", stream: high_level } + - { role: assistant, content: "${subtask}", stream: low_level, target: true } +``` + +A recipe can also branch into a weighted **blend** of sub-recipes. At sample +time, exactly one branch is selected deterministically from the sample index, +so different frames train different objectives (e.g. memory updates vs. +low-level execution vs. VQA) without any Python wiring. + +### Temporal semantics + +Persistent styles are active after emission until replaced: + +- `active_at(t, style=subtask)` +- `nth_prev(style=memory, offset=1)` +- `nth_next(style=subtask, offset=1)` + +Event styles only exist on their exact timestamp: + +- `emitted_at(t, style=interjection)` +- `emitted_at(t, style=vqa, role=user, camera=observation.images.top)` +- `emitted_at(t, role=assistant, tool_name=say)` + +Exact event matching has no tolerance window, so writers must stamp event rows with frame timestamps from the parquet data. + +### View-dependent resolution + +For view-dependent styles (`vqa` and `trace`), the resolver gains a +`camera=` filter parallel to `role=` and `tool_name=`. Datasets with multiple +cameras typically emit one (`vqa`, `user`) + (`vqa`, `assistant`) pair per +camera at the same timestamp; without `camera=`, those resolvers see two +matches and raise an ambiguity error. Recipes consume each camera through its +own binding plus a matching image block, e.g. + +```yaml +ask_vqa_top: + bindings: + vqa_query: "emitted_at(t, style=vqa, role=user, camera=observation.images.top)" + vqa: "emitted_at(t, style=vqa, role=assistant, camera=observation.images.top)" + messages: + - role: user + stream: high_level + if_present: vqa_query + content: + - { type: image, feature: observation.images.top } + - { type: text, text: "${vqa_query}" } + - { + role: assistant, + content: "${vqa}", + stream: high_level, + target: true, + if_present: vqa, + } +``` + +Add one such sub-recipe per camera the dataset records. + +## Layer 3 — training format + +Rendered samples use HF-style chat messages plus LeRobot sidecars: + +```python +sample["messages"] +sample["message_streams"] +sample["target_message_indices"] +``` + +The renderer does not apply a tokenizer chat template. Policy processors decide how to serialize the messages for their backbone, which keeps the same dataset usable across SmolVLA, Pi0.5, and any future VLM that expects OpenAI-style chat messages. + +## Graceful absence + +If both language columns are missing, `None`, or empty, `RenderMessagesStep` is a no-op. +If an event-scoped branch is selected on a frame without the required event row, rendering returns `None`, allowing a loader to retry another sample. diff --git a/docs/source/tools.mdx b/docs/source/tools.mdx new file mode 100644 index 000000000..d88881184 --- /dev/null +++ b/docs/source/tools.mdx @@ -0,0 +1,210 @@ +# Tools + +LeRobot v3.1 supports **tool calls** in policies — assistant messages can +emit structured invocations like `say(text="OK, starting now")` that the +runtime dispatches to a real implementation (TTS, controller, logger, …). + +This page covers: + +1. Where the tool catalog lives. +2. How the annotation pipeline produces tool-call atoms. +3. How to add your own tool. + +## Where tools are declared + +Two layers. + +**The catalog** — a list of OpenAI-style function schemas — lives at +`meta/info.json["tools"]` on each dataset. Example: + +```json +{ + "features": { "...": "..." }, + "tools": [ + { + "type": "function", + "function": { + "name": "say", + "description": "Speak a short utterance to the user via the TTS executor.", + "parameters": { + "type": "object", + "properties": { + "text": { + "type": "string", + "description": "The verbatim text to speak." + } + }, + "required": ["text"] + } + } + } + ] +} +``` + +Read it via the dataset metadata accessor: + +```python +from lerobot.datasets.dataset_metadata import LeRobotDatasetMetadata + +meta = LeRobotDatasetMetadata(repo_id="pepijn/super_poulain_final_annotations") +tools = meta.tools # list[dict] — OpenAI tool schemas +``` + +If the dataset's `info.json` doesn't declare any tools, `meta.tools` +returns `DEFAULT_TOOLS` from `lerobot.datasets.language` — currently a +single-entry list with the canonical `say` schema. So unannotated +datasets and chat-template consumers keep working without any +configuration: + +```python +prompt_str = tokenizer.apply_chat_template( + sample["messages"], + tools=meta.tools, # works either way + add_generation_prompt=False, + tokenize=False, +) +``` + +**The implementations** — runnable Python — will live under +`src/lerobot/tools/`, one file per tool. The runtime dispatcher and +the canonical `say` implementation (wrapping Kyutai's pocket-tts) are +not part of the catalog layer described here; today this layer ships +only the schema storage and the `DEFAULT_TOOLS` fallback constant. + +## Per-row tool _invocations_ + +The catalog above describes _what can be called_. The actual _call_ — the +function name plus the argument values — is stored per-row, on the +assistant atoms in `language_events`: + +```python +{ + "role": "assistant", + "content": null, + "style": null, + "timestamp": 12.4, + "camera": null, + "tool_calls": [ + { "type": "function", + "function": { "name": "say", "arguments": { "text": "On it." } } } + ] +} +``` + +Recipes splice these into rendered messages via `tool_calls_from`: + +```yaml +user_interjection_response: + bindings: + speech: "emitted_at(t, role=assistant, tool_name=say)" + messages: + - { role: user, content: "${task}", stream: high_level } + - { + role: assistant, + content: "${current_plan}", + stream: high_level, + target: true, + tool_calls_from: speech, + } +``` + +The model's training target is one assistant turn that carries both the +plan text _and_ the `say` tool call. At inference, the runtime parses +the generated text back into structured `tool_calls` and dispatches to +the matching implementation. + +## How to add your own tool + +> **Note:** Steps 2 and 3 below describe the runtime layer +> (`src/lerobot/tools/`, the `Tool` protocol, `TOOL_REGISTRY`, +> `get_tools(meta)`) which is not part of the catalog layer shipped +> today — those modules don't yet exist in the tree. Step 1 alone is +> enough to make the tool visible to the chat template via +> `meta.tools` so the model can learn to _generate_ the call; +> executing the call at inference requires the runtime layer. + +Three steps. Concrete example: a `record_observation` tool the policy +can call to capture an extra observation outside the regular control +loop. + +### Step 1 — declare the schema + +Add an entry under `meta/info.json["tools"]`. Either edit the file +directly on disk _before_ running the annotation pipeline (it'll be +preserved) or hand it to `lerobot-annotate` via a config flag. + +```json +{ + "tools": [ + { "type": "function", "function": { "name": "say", "...": "..." } }, + { + "type": "function", + "function": { + "name": "record_observation", + "description": "Capture a high-resolution still image for the user.", + "parameters": { + "type": "object", + "properties": { + "label": { + "type": "string", + "description": "Short label for the saved image." + } + }, + "required": ["label"] + } + } + } + ] +} +``` + +The schema follows OpenAI's function-calling convention exactly, so the +chat template can render it natively. + +### Step 2 — implement the call + +Create `src/lerobot/tools/record_observation.py`: + +```python +from .base import Tool +from typing import Any + +RECORD_OBSERVATION_SCHEMA: dict[str, Any] = { "...": "..." } # mirrors the JSON above + + +class RecordObservationTool: + name = "record_observation" + schema = RECORD_OBSERVATION_SCHEMA + + def __init__(self, schema: dict | None = None, output_dir: str = "."): + self.output_dir = output_dir + + def call(self, arguments: dict) -> str: + label = arguments["label"] + # ... save the latest camera frame to /