* Add Isaac Teleop SO-101 leader-arm teleoperator
Add the NVIDIA Isaac Teleop teleoperator scaffolding and its first device:
SO101LeaderArm, a back-drivable SO-101 leader arm on Isaac Teleop's generic
joint-space device path. It reads the leader's joints from the so101_leader
plugin via JointStateSource and emits follower-ready {joint}.pos (rad2deg arm,
gripper -> RANGE_0_100) for direct 1:1 joint drive.
- IsaacTeleopTeleoperator base + IsaacTeleopConfig (shared session/CloudXR config)
- SO101LeaderArm / SO101LeaderArmConfig and leader_joints_to_robot_action
- examples/isaac_teleop_to_so101/teleoperate_leader.py example
- pure-numpy conversion tests
- isaac-teleop optional extra + NVIDIA PyPI index in pyproject
* Add Isaac Teleop XR controller teleoperator for SO-101
Add end-to-end XR (VR) controller teleoperation of an SO-101 follower arm via
the NVIDIA Isaac Teleop stack, layered on the Isaac Teleop scaffolding.
Teleoperator (src/lerobot/teleoperators/isaac_teleop/):
- XRController / XRControllerConfig: connect to the CloudXR runtime, auto-launch
the Isaac Teleop session, and expose get_action() emitting the raw base-frame
grip pose, squeeze, and trigger.
- MapXRControllerActionToRobotAction: stateless per-frame mapper from the XR
action to the IK input contract (absolute ee.x/y/z, ee.gripper_pos, wrist_roll).
- OverwriteWristRollFromAngle: post-IK step writing the operator wrist-roll [rad]
onto wrist_roll.pos [deg], recovering the under-determined roll DOF.
Example (examples/isaac_teleop_to_so101/):
- teleoperate.py: thin absolute-pose IK pipeline with an in-loop clutch (engage
latch + 1:1 delta rebase of position and orientation), EEBoundsAndSafety, and
InverseKinematicsEEToJoints; slews to a recorded home on startup.
- record_reset_pose.py / download_assets.py / webxr.env / .gitignore.
Also:
- Extend robot_kinematic_processor.py with EEBoundsAndSafety and
InverseKinematicsEEToJoints.
- Add XRControllerConfig + base_T_anchor to the Isaac Teleop config.
- Add docs/source/isaac_teleop.mdx and the _toctree entry.
- Add unit tests for the CloudXR launcher and the XR controller processor.
* Unify Isaac Teleop SO-101 scripts behind a mandatory device selector
Merge teleoperate.py (XR controller: clutch + soft-orientation IK) and
teleoperate_leader.py (SO-101 leader arm: 1:1 joint mirror) into a single
teleoperate.py driven by a `lerobot-teleoperate`-style draccus CLI: a follower
`--robot.*` and an input `--teleop.*`, where `--teleop.type` (xr_controller |
so101_leader) selects the Isaac device.
Uses a "dispatch, don't merge" shape: per-device setup_xr/setup_leader build a
Device bundle (compute / startup / cleanup / command); a shared slew() takes a
per-step target callable (XR a fixed reset pose, leader a live re-read so the
1:1 handoff stays continuous); one device-branchless outer loop runs both, with
compute() -> None meaning "hold at the measured pose" (XR disengaged or leader
stale). The entrypoint is @parser.wrap()'d over a TeleoperateConfig dataclass and
dispatches on the parsed config type; device knobs ride on --teleop.* (the leader
serial port is --teleop.port, forwarded to the plugin) and loop/launch knobs are
top-level (--launch_plugin=<path> collapses the old --launch-plugin/--plugin-bin
pair; --reset_to_origin/--align/--dry_run).
To let the Isaac devices claim the natural --teleop.type names without colliding
with the serial so101_leader of lerobot-teleoperate, give IsaacTeleopConfig its
own draccus choice registry (own _choice_registry, decoupled from the global
TeleoperatorConfig one) and register XRControllerConfig as "xr_controller" and
SO101LeaderArmConfig as "so101_leader" there; the example types its teleop field
as IsaacTeleopConfig so the choices resolve against that scoped registry. These
devices drive the bespoke clutch/IK/align loop and are not routed through
make_teleoperator_from_config, so dropping them from the global registry is inert.
YAGNI sweep of the commit train: delete the orphaned OverwriteWristRollFromAngle
(wrist_roll_processor.py) plus its export and tests -- no producer emits
wrist_roll; the live XR path uses orientation-weight IK on the 5-DOF arm by
design. Kept the load-bearing knobs (orientation_weight, raise_on_jump,
base_T_anchor) and the optional reset-pose recorder. Updated isaac_teleop.mdx
for the unified entrypoint and excised the stale roll-retargeter prose.
Net LOC down (two scripts 714 lines -> one), in-loop device branches reduced to
zero. Planned and reviewed via a 6-persona multi-agent panel (3-round planning
convergence + 2-round review). Verification (isaacteleop/placo not installable
here, so the device classes cannot connect, but their config dataclasses and the
script import fine via deferred imports): the teleoperators test suite passes
(45 passed, 2 skipped), draccus parsing of both target command lines yields the
right config subclass with scoped --teleop.type, --help renders the scoped
choices, the serial so101_leader stays in the global registry, and ruff
check/format are green.
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
* Add Isaac Teleop SO-101 dataset recording script
record.py records a LeRobot dataset while driving the SO-101 follower
with either Isaac Teleop device (--teleop.type=xr_controller |
so101_leader), mirroring teleoperate.py's device dispatch.
* Extract shared Isaac Teleop SO-101 example infra into common.py
teleoperate.py and record.py both built the per-device pipeline and ran the
same read -> compute -> hold-when-idle -> sleep loop, with record.py importing
internals from teleoperate.py. Move the shared device/loop infrastructure
(Device, slew, Clutch, setup_xr/setup_leader + leader helpers, reset infra and
constants) into a new common.py, and add build_device() + hold_action() to
collapse the connect/dispatch/startup and idle-hold glue duplicated in both
entry points. The setup functions now type their config against a LoopConfig
Protocol, so common.py is decoupled from either CLI; both import from it.
Also rename record_reset_pose.py -> override_reset_pose.py so it is not confused
with record.py, and update the doc references.
* Add stdin keyboard backend so recording shortcuts work over SSH/headless
lerobot's init_keyboard_listener() uses pynput, which hooks GLOBAL key events
from the display server. Over SSH, under Wayland, or on a headless box with only a
TTY, keystrokes go to the terminal's stdin instead, so the listener never fires and
the Right/Left/Esc recording shortcuts silently do nothing.
Add a stdin (termios) keyboard backend to the example's common.py and an
init_keyboard_listener() that prefers it whenever stdin is an interactive TTY
(works over SSH / Wayland / headless-with-tty), falling back to lerobot's
pynput/headless listener for GUI launches with no controlling terminal. Selectable
via LEROBOT_KEYBOARD_BACKEND={auto,stdin,pynput,none}. The backend keeps ISIG so
Ctrl-C still works and always restores the terminal (on stop() and via atexit).
record.py now sources init_keyboard_listener from common; the Right/Left/Esc -> flag
mapping and the (listener, events) contract are unchanged.
Also convert record.py's loop_kwargs to a dict literal (ruff C408).
* Wait for the XR headset to connect before driving the arm
On the xr_controller path the example connected CloudXR and immediately ran the
reset slew + control loop, even if no headset was connected — the arm moved before
the operator was in VR, and get_action() just returned zeros so the clutch never
engaged.
Add an is_tracking property to XRController (set from the controller stream's
optional group, mirroring SO101LeaderArm) and a _wait_for_xr_controller() helper in
common.py that prints connection instructions (CloudXR web client URL + this
workstation's candidate IPv4s, with loopback/link-local and virtual/bridge/USB-gadget
interfaces filtered out) and polls until the controllers stream (indefinite, 15s
reminder, Ctrl-C to abort). setup_xr.startup() now connects, waits for the headset,
THEN runs the reset slew and seeds the clutch — so the arm only moves once the
operator is connected and watching. Mirrors the leader path's _wait_for_leader; both
record.py and teleoperate.py inherit it via the shared setup_xr.
* Address review feedback on the Isaac Teleop -> SO-101 example
Review-response and CI fixes for the Isaac Teleop -> SO-101 example.
- Move the XR Clutch into src/lerobot/teleoperators/isaac_teleop/clutch.py
(pure numpy + Rotation, no isaacteleop import), export it, and add
tests/teleoperators/test_clutch.py.
- Drop the vendored stdin keyboard listener; record.py uses a small terminal-
first wrapper over upstream's TerminalKeyListener (works over SSH even with a
local X display), falling back to upstream init_keyboard_listener otherwise.
- record.py: pass rgb_encoder/depth_encoder to LeRobotDataset create()/resume()
(upstream removed camera_encoder), fixing the AttributeError at record time.
- build_device: derive motor names from robot.action_features instead of
robot.bus (supports non-bus robots), and disconnect the follower if any step
after connect() fails so a failed setup never leaks the connection.
- Read leader joints by the group's declared names (_joints_group_to_rad)
instead of positionally, so a layout mismatch can't silently mirror the wrong
DOF onto the follower; add tests including a reversed-layout group.
- base.py: hoist `from pathlib import Path` to module scope; only the
isaacteleop CloudXRLauncher import stays lazy (optional dep).
- Trim the common.py module docstring and point to docs/source/isaac_teleop.mdx.
- default.env: correct the NV_DEVICE_PROFILE comment (auto-webrtc is the default;
this file overrides to Quest3, which works for both Quest 3 and Pico 4).
- download_assets.py: correct the RAW_BASE comment (tracks main, not pinned) and
add `# nosec B310` next to the existing `# noqa: S310` for the bandit hook.
- uv.lock: add the isaac-teleop extra's deps so `uv sync --locked` matches
pyproject; regenerated with uv 0.8.0 to keep lockfile revision 2 (CI's uv).
- isaac_teleop.mdx: prettier formatting.
* fix(.gitignore): removing .gitignore and using lerobot cache folder instead to store local user files
* chore(docstrings): reducing docstrings in default.env
* feat(URDF): cleaning up and simplifying the URDF download procedure
* feat(robot guard): adding a guard in case an unsupported robot type is provided (so-arms only)
* fix(imports): enforcing a python module structure to simplify imports
* feat(safe read): extending the motor bus safe read rationale to reset pose setting
* chore(trim): trimming lenghty comments and docstrings
* fix(deps): use isaacteleop [retargeters-lite] extra to unblock aarch64 (DGX Spark) (#3933)
* fix(deps): drop isaacteleop [retargeters] extra to unblock aarch64
The [retargeters] extra pulls dex-retargeting (pins numpy<2.0, conflicting
with lerobot's numpy>=2.0) and nlopt>=2.8 (no aarch64 wheels), making
lerobot[isaac-teleop] unresolvable on ARM (DGX Spark, Jetson Thor, GH200)
and over-constrained on numpy everywhere else.
The LeRobot teleoperators only import isaacteleop.retargeting_engine,
isaacteleop.cloudxr and isaacteleop.teleop_session_manager, all shipped in
the base wheel (requires only numpy>=1.23), so the extra is unused.
Verified on DGX Spark (aarch64, Python 3.12): resolves and installs with
isaacteleop 1.3.131 + numpy 2.2.6; all imported symbols load.
* fix(deps): use isaacteleop [retargeters-lite] extra for aarch64 support
Pin to isaacteleop ~=1.3.131 (the release that added ARM64/aarch64 support)
and swap the full [retargeters] extra for the new [retargeters-lite] one
(scipy-only). The full extra drags in dex-retargeting (pins numpy<2,
conflicting with lerobot's numpy>=2.0) and nlopt>=2.8 (no aarch64 wheels),
making lerobot[isaac-teleop] unresolvable on ARM hosts (DGX Spark, Jetson
Thor, GH200) and over-constrained on numpy everywhere else.
The LeRobot teleoperators only import isaacteleop.retargeting_engine,
isaacteleop.cloudxr and isaacteleop.teleop_session_manager — all covered
by the base wheel + retargeters-lite.
Verified on DGX Spark (aarch64, Python 3.12/3.13): resolves and installs
with isaacteleop 1.3.131 + numpy 2.2.6 + scipy 1.18.
* feat(deps): re-add full [retargeters] extra gated to x86_64
Keep the dex-retargeting/nlopt-based retargeters available on x86_64 (where
their wheels exist) via an environment marker, while ARM hosts (DGX Spark,
Jetson Thor, GH200) resolve with base + [retargeters-lite] only.
Verified: uv lock resolves on both platforms; on aarch64 the compile
excludes nlopt/dex-retargeting, on x86_64 they are included.
---------
Co-authored-by: Johnny Nunez <22727137+johnnynunez@users.noreply.github.com>
* chore(docstrings): trimming latest docstrings
* chore(teleop): move isaac-teleop to examples + update docs + add readme with installation notes
* chore(deps): restore uv.lock
* fix(example: isaac teleop parsing config
* fix(examples): isaac atomic-gripper controller
* feat(Examples): isaac-teleop holdlatch
* chore(examples): some other minor improvements for isaac-teleop
* chore(examples): top-level imports isaac-teleop
* chore(Examples): address ai review isaac-teleop
---------
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
Co-authored-by: Jiwen Cai <jiwenc@nvidia.com>
Co-authored-by: Johnny <johnnync13@gmail.com>
Co-authored-by: Johnny Nunez <22727137+johnnynunez@users.noreply.github.com>
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
LeRobot aims to provide models, datasets, and tools for real-world robotics in PyTorch. The goal is to lower the barrier to entry so that everyone can contribute to and benefit from shared datasets and pretrained models.
🤗 A hardware-agnostic, Python-native interface that standardizes control across diverse platforms, from low-cost arms (SO-100) to humanoids.
🤗 A standardized, scalable LeRobotDataset format (Parquet + MP4 or images) hosted on the Hugging Face Hub, enabling efficient storage, streaming and visualization of massive robotic datasets.
🤗 State-of-the-art policies that have been shown to transfer to the real-world ready for training and deployment.
🤗 Comprehensive support for the open-source ecosystem to democratize physical AI.
Quick Start
LeRobot can be installed directly from PyPI.
pip install lerobot
lerobot-info
Important
For detailed installation guide, please see the Installation Documentation.
Robots & Control
LeRobot provides a unified Robot class interface that decouples control logic from hardware specifics. It supports a wide range of robots and teleoperation devices.
from lerobot.robots.myrobot import MyRobot
# Connect to a robot
robot = MyRobot(config=...)
robot.connect()
# Read observation and send action
obs = robot.get_observation()
action = model.select_action(obs)
robot.send_action(action)
Supported Hardware: SO100, LeKiwi, Koch, HopeJR, OMX, EarthRover, Reachy2, Gamepads, Keyboards, Phones, OpenARM, Unitree G1, reBot B601.
While these devices are natively integrated into the LeRobot codebase, the library is designed to be extensible. You can easily implement the Robot interface to utilize LeRobot's data collection, training, and visualization tools for your own custom robot.
For detailed hardware setup guides, see the Hardware Documentation.
LeRobot Dataset
To solve the data fragmentation problem in robotics, we utilize the LeRobotDataset format.
- Structure: Synchronized MP4 videos (or images) for vision and Parquet files for state/action data.
- HF Hub Integration: Explore thousands of robotics datasets on the Hugging Face Hub.
- Tools: Seamlessly delete episodes, split by indices/fractions, add/remove features, and merge multiple datasets.
from lerobot.datasets.lerobot_dataset import LeRobotDataset
# Load a dataset from the Hub
dataset = LeRobotDataset("lerobot/aloha_mobile_cabinet")
# Access data (automatically handles video decoding)
episode_index=0
print(f"{dataset[episode_index]['action'].shape=}\n")
Learn more about it in the LeRobotDataset Documentation
SoTA Models
LeRobot implements state-of-the-art policies in pure PyTorch, covering Imitation Learning, Reinforcement Learning, Vision-Language-Action (VLA) models, World Models, and Reward Models, with more coming soon. It also provides you with the tools to instrument and inspect your training process.
Training a policy is as simple as running a script configuration:
lerobot-train \
--policy.type=act \
--dataset.repo_id=lerobot/aloha_mobile_cabinet
| Category | Models |
|---|---|
| Imitation Learning | ACT, Diffusion, VQ-BeT, Multitask DiT Policy |
| Reinforcement Learning | HIL-SERL, TDMPC & QC-FQL (coming soon) |
| VLAs Models | Pi0, Pi0Fast, Pi0.5, GR00T N1.7, SmolVLA, XVLA, EO-1, MolmoAct2, WALL-OSS, EVO1 |
| World Models | VLA-JEPA, LingBot-VA, FastWAM |
| Reward Models | SARM, TOPReward, Robometer |
Similarly to the hardware, you can easily implement your own policy & leverage LeRobot's data collection, training, and visualization tools, and share your model to the HF Hub
For detailed policy setup guides, see the Policy Documentation. For GPU/RAM requirements and expected training time per policy, see the Compute Hardware Guide.
Inference & Evaluation
Evaluate your policies in simulation or on real hardware using the unified evaluation script. LeRobot supports standard benchmarks like LIBERO, MetaWorld and more to come.
# Evaluate a policy on the LIBERO benchmark
lerobot-eval \
--policy.path=lerobot/pi0_libero_finetuned \
--env.type=libero \
--env.task=libero_object \
--eval.n_episodes=10
Learn how to implement your own simulation environment or benchmark and distribute it from the HF Hub by following the EnvHub Documentation
Resources
- Documentation: The complete guide to tutorials & API.
- Chinese Tutorials: LeRobot+SO-ARM101中文教程-同济子豪兄 Detailed doc for assembling, teleoperate, dataset, train, deploy. Verified by Seed Studio and 5 global hackathon players.
- Discord: Join the
LeRobotserver to discuss with the community. - X: Follow us on X to stay up-to-date with the latest developments.
- Robot Learning Tutorial: A free, hands-on course to learn robot learning using LeRobot.
- T-Shirt Folding Experiment: An end-to-end demonstration of folding t-shirts with LeRobot.
- LeLab: A web interface for LeRobot — teleoperate, calibrate, record datasets, replay, and train your SO arm from the browser, no CLI required.
Citation
If you use LeRobot in your project, please cite the GitHub repository to acknowledge the ongoing development and contributors:
@misc{cadene2024lerobot,
author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Meftah, Khalil and Ellerbach, Maxime and Moss, Jess and Wolf, Thomas},
title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
howpublished = "\url{https://github.com/huggingface/lerobot}",
year = {2024}
}
If you are referencing our research or the academic paper, please also cite our ICLR publication:
ICLR 2026 Paper
@inproceedings{cadenelerobot,
title={LeRobot: An Open-Source Library for End-to-End Robot Learning},
author={Cadene, Remi and Alibert, Simon and Capuano, Francesco and Aractingi, Michel and Zouitine, Adil and Kooijmans, Pepijn and Choghari, Jade and Russi, Martino and Pascal, Caroline and Palma, Steven and Shukor, Mustafa and Moss, Jess and Soare, Alexander and Aubakirova, Dana and Lhoest, Quentin and Gallou\'edec, Quentin and Wolf, Thomas},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://arxiv.org/abs/2602.22818}
}
Contribute
We welcome contributions from everyone in the community! To get started, please read our CONTRIBUTING.md guide. Whether you're adding a new feature, improving documentation, or fixing a bug, your help and feedback are invaluable. We're incredibly excited about the future of open-source robotics and can't wait to work with you on what's next—thank you for your support!


