admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-09 11:01:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(examples): add Isaac Teleop → SO-101 teleoperation and dataset recording example (#3927)

Browse Source 
* Add Isaac Teleop SO-101 leader-arm teleoperator

Add the NVIDIA Isaac Teleop teleoperator scaffolding and its first device:
SO101LeaderArm, a back-drivable SO-101 leader arm on Isaac Teleop's generic
joint-space device path. It reads the leader's joints from the so101_leader
plugin via JointStateSource and emits follower-ready {joint}.pos (rad2deg arm,
gripper -> RANGE_0_100) for direct 1:1 joint drive.

- IsaacTeleopTeleoperator base + IsaacTeleopConfig (shared session/CloudXR config)
- SO101LeaderArm / SO101LeaderArmConfig and leader_joints_to_robot_action
- examples/isaac_teleop_to_so101/teleoperate_leader.py example
- pure-numpy conversion tests
- isaac-teleop optional extra + NVIDIA PyPI index in pyproject

* Add Isaac Teleop XR controller teleoperator for SO-101

Add end-to-end XR (VR) controller teleoperation of an SO-101 follower arm via
the NVIDIA Isaac Teleop stack, layered on the Isaac Teleop scaffolding.

Teleoperator (src/lerobot/teleoperators/isaac_teleop/):
- XRController / XRControllerConfig: connect to the CloudXR runtime, auto-launch
  the Isaac Teleop session, and expose get_action() emitting the raw base-frame
  grip pose, squeeze, and trigger.
- MapXRControllerActionToRobotAction: stateless per-frame mapper from the XR
  action to the IK input contract (absolute ee.x/y/z, ee.gripper_pos, wrist_roll).
- OverwriteWristRollFromAngle: post-IK step writing the operator wrist-roll [rad]
  onto wrist_roll.pos [deg], recovering the under-determined roll DOF.

Example (examples/isaac_teleop_to_so101/):
- teleoperate.py: thin absolute-pose IK pipeline with an in-loop clutch (engage
  latch + 1:1 delta rebase of position and orientation), EEBoundsAndSafety, and
  InverseKinematicsEEToJoints; slews to a recorded home on startup.
- record_reset_pose.py / download_assets.py / webxr.env / .gitignore.

Also:
- Extend robot_kinematic_processor.py with EEBoundsAndSafety and
  InverseKinematicsEEToJoints.
- Add XRControllerConfig + base_T_anchor to the Isaac Teleop config.
- Add docs/source/isaac_teleop.mdx and the _toctree entry.
- Add unit tests for the CloudXR launcher and the XR controller processor.

* Unify Isaac Teleop SO-101 scripts behind a mandatory device selector

Merge teleoperate.py (XR controller: clutch + soft-orientation IK) and
teleoperate_leader.py (SO-101 leader arm: 1:1 joint mirror) into a single
teleoperate.py driven by a `lerobot-teleoperate`-style draccus CLI: a follower
`--robot.*` and an input `--teleop.*`, where `--teleop.type` (xr_controller |
so101_leader) selects the Isaac device.

Uses a "dispatch, don't merge" shape: per-device setup_xr/setup_leader build a
Device bundle (compute / startup / cleanup / command); a shared slew() takes a
per-step target callable (XR a fixed reset pose, leader a live re-read so the
1:1 handoff stays continuous); one device-branchless outer loop runs both, with
compute() -> None meaning "hold at the measured pose" (XR disengaged or leader
stale). The entrypoint is @parser.wrap()'d over a TeleoperateConfig dataclass and
dispatches on the parsed config type; device knobs ride on --teleop.* (the leader
serial port is --teleop.port, forwarded to the plugin) and loop/launch knobs are
top-level (--launch_plugin=<path> collapses the old --launch-plugin/--plugin-bin
pair; --reset_to_origin/--align/--dry_run).

To let the Isaac devices claim the natural --teleop.type names without colliding
with the serial so101_leader of lerobot-teleoperate, give IsaacTeleopConfig its
own draccus choice registry (own _choice_registry, decoupled from the global
TeleoperatorConfig one) and register XRControllerConfig as "xr_controller" and
SO101LeaderArmConfig as "so101_leader" there; the example types its teleop field
as IsaacTeleopConfig so the choices resolve against that scoped registry. These
devices drive the bespoke clutch/IK/align loop and are not routed through
make_teleoperator_from_config, so dropping them from the global registry is inert.

YAGNI sweep of the commit train: delete the orphaned OverwriteWristRollFromAngle
(wrist_roll_processor.py) plus its export and tests -- no producer emits
wrist_roll; the live XR path uses orientation-weight IK on the 5-DOF arm by
design. Kept the load-bearing knobs (orientation_weight, raise_on_jump,
base_T_anchor) and the optional reset-pose recorder. Updated isaac_teleop.mdx
for the unified entrypoint and excised the stale roll-retargeter prose.

Net LOC down (two scripts 714 lines -> one), in-loop device branches reduced to
zero. Planned and reviewed via a 6-persona multi-agent panel (3-round planning
convergence + 2-round review). Verification (isaacteleop/placo not installable
here, so the device classes cannot connect, but their config dataclasses and the
script import fine via deferred imports): the teleoperators test suite passes
(45 passed, 2 skipped), draccus parsing of both target command lines yields the
right config subclass with scoped --teleop.type, --help renders the scoped
choices, the serial so101_leader stays in the global registry, and ruff
check/format are green.

Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>

* Add Isaac Teleop SO-101 dataset recording script

record.py records a LeRobot dataset while driving the SO-101 follower
with either Isaac Teleop device (--teleop.type=xr_controller |
so101_leader), mirroring teleoperate.py's device dispatch.

* Extract shared Isaac Teleop SO-101 example infra into common.py

teleoperate.py and record.py both built the per-device pipeline and ran the
same read -> compute -> hold-when-idle -> sleep loop, with record.py importing
internals from teleoperate.py. Move the shared device/loop infrastructure
(Device, slew, Clutch, setup_xr/setup_leader + leader helpers, reset infra and
constants) into a new common.py, and add build_device() + hold_action() to
collapse the connect/dispatch/startup and idle-hold glue duplicated in both
entry points. The setup functions now type their config against a LoopConfig
Protocol, so common.py is decoupled from either CLI; both import from it.

Also rename record_reset_pose.py -> override_reset_pose.py so it is not confused
with record.py, and update the doc references.

* Add stdin keyboard backend so recording shortcuts work over SSH/headless

lerobot's init_keyboard_listener() uses pynput, which hooks GLOBAL key events
from the display server. Over SSH, under Wayland, or on a headless box with only a
TTY, keystrokes go to the terminal's stdin instead, so the listener never fires and
the Right/Left/Esc recording shortcuts silently do nothing.

Add a stdin (termios) keyboard backend to the example's common.py and an
init_keyboard_listener() that prefers it whenever stdin is an interactive TTY
(works over SSH / Wayland / headless-with-tty), falling back to lerobot's
pynput/headless listener for GUI launches with no controlling terminal. Selectable
via LEROBOT_KEYBOARD_BACKEND={auto,stdin,pynput,none}. The backend keeps ISIG so
Ctrl-C still works and always restores the terminal (on stop() and via atexit).
record.py now sources init_keyboard_listener from common; the Right/Left/Esc -> flag
mapping and the (listener, events) contract are unchanged.

Also convert record.py's loop_kwargs to a dict literal (ruff C408).

* Wait for the XR headset to connect before driving the arm

On the xr_controller path the example connected CloudXR and immediately ran the
reset slew + control loop, even if no headset was connected — the arm moved before
the operator was in VR, and get_action() just returned zeros so the clutch never
engaged.

Add an is_tracking property to XRController (set from the controller stream's
optional group, mirroring SO101LeaderArm) and a _wait_for_xr_controller() helper in
common.py that prints connection instructions (CloudXR web client URL + this
workstation's candidate IPv4s, with loopback/link-local and virtual/bridge/USB-gadget
interfaces filtered out) and polls until the controllers stream (indefinite, 15s
reminder, Ctrl-C to abort). setup_xr.startup() now connects, waits for the headset,
THEN runs the reset slew and seeds the clutch — so the arm only moves once the
operator is connected and watching. Mirrors the leader path's _wait_for_leader; both
record.py and teleoperate.py inherit it via the shared setup_xr.

* Address review feedback on the Isaac Teleop -> SO-101 example

Review-response and CI fixes for the Isaac Teleop -> SO-101 example.

- Move the XR Clutch into src/lerobot/teleoperators/isaac_teleop/clutch.py
  (pure numpy + Rotation, no isaacteleop import), export it, and add
  tests/teleoperators/test_clutch.py.
- Drop the vendored stdin keyboard listener; record.py uses a small terminal-
  first wrapper over upstream's TerminalKeyListener (works over SSH even with a
  local X display), falling back to upstream init_keyboard_listener otherwise.
- record.py: pass rgb_encoder/depth_encoder to LeRobotDataset create()/resume()
  (upstream removed camera_encoder), fixing the AttributeError at record time.
- build_device: derive motor names from robot.action_features instead of
  robot.bus (supports non-bus robots), and disconnect the follower if any step
  after connect() fails so a failed setup never leaks the connection.
- Read leader joints by the group's declared names (_joints_group_to_rad)
  instead of positionally, so a layout mismatch can't silently mirror the wrong
  DOF onto the follower; add tests including a reversed-layout group.
- base.py: hoist `from pathlib import Path` to module scope; only the
  isaacteleop CloudXRLauncher import stays lazy (optional dep).
- Trim the common.py module docstring and point to docs/source/isaac_teleop.mdx.
- default.env: correct the NV_DEVICE_PROFILE comment (auto-webrtc is the default;
  this file overrides to Quest3, which works for both Quest 3 and Pico 4).
- download_assets.py: correct the RAW_BASE comment (tracks main, not pinned) and
  add `# nosec B310` next to the existing `# noqa: S310` for the bandit hook.
- uv.lock: add the isaac-teleop extra's deps so `uv sync --locked` matches
  pyproject; regenerated with uv 0.8.0 to keep lockfile revision 2 (CI's uv).
- isaac_teleop.mdx: prettier formatting.

* fix(.gitignore): removing .gitignore and using lerobot cache folder instead to store local user files

* chore(docstrings): reducing docstrings in default.env

* feat(URDF): cleaning up and simplifying the URDF download procedure

* feat(robot guard): adding a guard in case an unsupported robot type is provided (so-arms only)

* fix(imports): enforcing a python module structure to simplify imports

* feat(safe read): extending the motor bus safe read rationale to reset pose setting

* chore(trim): trimming lenghty comments and docstrings

* fix(deps): use isaacteleop [retargeters-lite] extra to unblock aarch64 (DGX Spark) (#3933)

* fix(deps): drop isaacteleop [retargeters] extra to unblock aarch64

The [retargeters] extra pulls dex-retargeting (pins numpy<2.0, conflicting
with lerobot's numpy>=2.0) and nlopt>=2.8 (no aarch64 wheels), making
lerobot[isaac-teleop] unresolvable on ARM (DGX Spark, Jetson Thor, GH200)
and over-constrained on numpy everywhere else.

The LeRobot teleoperators only import isaacteleop.retargeting_engine,
isaacteleop.cloudxr and isaacteleop.teleop_session_manager, all shipped in
the base wheel (requires only numpy>=1.23), so the extra is unused.

Verified on DGX Spark (aarch64, Python 3.12): resolves and installs with
isaacteleop 1.3.131 + numpy 2.2.6; all imported symbols load.

* fix(deps): use isaacteleop [retargeters-lite] extra for aarch64 support

Pin to isaacteleop ~=1.3.131 (the release that added ARM64/aarch64 support)
and swap the full [retargeters] extra for the new [retargeters-lite] one
(scipy-only). The full extra drags in dex-retargeting (pins numpy<2,
conflicting with lerobot's numpy>=2.0) and nlopt>=2.8 (no aarch64 wheels),
making lerobot[isaac-teleop] unresolvable on ARM hosts (DGX Spark, Jetson
Thor, GH200) and over-constrained on numpy everywhere else.

The LeRobot teleoperators only import isaacteleop.retargeting_engine,
isaacteleop.cloudxr and isaacteleop.teleop_session_manager — all covered
by the base wheel + retargeters-lite.

Verified on DGX Spark (aarch64, Python 3.12/3.13): resolves and installs
with isaacteleop 1.3.131 + numpy 2.2.6 + scipy 1.18.

* feat(deps): re-add full [retargeters] extra gated to x86_64

Keep the dex-retargeting/nlopt-based retargeters available on x86_64 (where
their wheels exist) via an environment marker, while ARM hosts (DGX Spark,
Jetson Thor, GH200) resolve with base + [retargeters-lite] only.

Verified: uv lock resolves on both platforms; on aarch64 the compile
excludes nlopt/dex-retargeting, on x86_64 they are included.

---------

Co-authored-by: Johnny Nunez <22727137+johnnynunez@users.noreply.github.com>

* chore(docstrings): trimming latest docstrings

* chore(teleop): move isaac-teleop to examples + update docs + add readme with installation notes

* chore(deps): restore uv.lock

* fix(example: isaac teleop parsing config

* fix(examples): isaac atomic-gripper controller

* feat(Examples): isaac-teleop holdlatch

* chore(examples): some other minor improvements for isaac-teleop

* chore(examples): top-level imports isaac-teleop

* chore(Examples): address ai review isaac-teleop

---------

Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
Co-authored-by: Jiwen Cai <jiwenc@nvidia.com>
Co-authored-by: Johnny <johnnync13@gmail.com>
Co-authored-by: Johnny Nunez <22727137+johnnynunez@users.noreply.github.com>
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
This commit is contained in:
Caroline Pascal
2026-07-05 20:56:26 +02:00
committed by GitHub
parent 7957d4e2dc
commit 293a8d9a77
17 changed files with 2795 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/_toctree.yml
+2
View File
@@ -169,6 +169,8 @@
- sections:
- local: phone_teleop
title: Phone
- local: isaac_teleop
title: Isaac Teleop
title: "Teleoperators"
- sections:
- local: cameras
docs/source/isaac_teleop.mdx
+397
View File
@@ -0,0 +1,397 @@
# Isaac Teleop
Control your robot with NVIDIA [Isaac Teleop](https://github.com/NVIDIA/IsaacTeleop), a
multi-modal teleoperation framework. Isaac Teleop drives a single `TeleopSession` from a range
of input devices — XR (VR) controllers, hand tracking, full-body tracking, Manus gloves, foot
pedals, and more.
In LeRobot, Isaac Teleop ships as a self-contained example under
[`examples/isaac_teleop_to_so101/`](https://github.com/huggingface/lerobot/tree/main/examples/isaac_teleop_to_so101).
Each Isaac Teleop input device is its own `Teleoperator` subclass in the example's
`isaac_teleop` package, sharing one session lifecycle (see `IsaacTeleopTeleoperator`). The
devices available today are the **XR controller** (`XRController`) and a back-drivable
**SO-101 leader arm** (`SO101LeaderArm`); Manus gloves and hand/full-body tracking are the
natural next devices. This guide focuses on the XR controller; the SO-101 leader is summarized
under [Run the example](#step-3-run-the-example).
**In this guide you'll learn:**
- How an Isaac Teleop device drives a robot end‑effector (EE) target
- How the _clutch_ (squeeze/grip on the XR controller) engages teleoperation without jerking the arm
- How to run the SO‑101 teleoperation example and tune motion / gripper / IK
## Installation
The example lives in the LeRobot repository (it is not part of the `lerobot` pip package), so
clone the repo and install from source. The canonical, always-up-to-date install and usage
reference is the example's
[`README.md`](https://github.com/huggingface/lerobot/tree/main/examples/isaac_teleop_to_so101/README.md);
in short:
```bash
git clone https://github.com/huggingface/lerobot.git
cd lerobot
uv pip install -e ".[feetech,kinematics,dataset]" "huggingface_hub>=1.5"
uv pip install "isaacteleop[cloudxr,retargeters-lite]~=1.3.131" "scipy>=1.14"
```
`isaacteleop` is published on public PyPI (Linux only). The `cloudxr` extra brings the CloudXR
runtime bindings; `retargeters-lite` is the scipy-based retargeter path that resolves on both
x86_64 and ARM (on aarch64 — e.g. a DGX Spark — the full `retargeters` extra does not resolve
because of its `dex-retargeting`/`nlopt` pins, which is why it is not the default here). On
x86_64 you can additionally install the full retargeter stack:
```bash
uv pip install "isaacteleop[retargeters]~=1.3.131"
```
### Set up CloudXR and connect a headset
Isaac Teleop streams the headset to your machine over **NVIDIA CloudXR**, which provides the
OpenXR runtime the session connects to. By default LeTeleop **auto-launches the CloudXR runtime
for you** when you call `teleop_device.connect()` — you no longer have to run `python -m
isaacteleop.cloudxr` and `source cloudxr.env` in a separate shell. All you need is a supported
headset connected and the CloudXR firewall ports open. Follow the Isaac Teleop
[Quick Start](https://nvidia.github.io/IsaacTeleop/main/getting_started/quick_start.html) for the
headset-pairing and firewall details.
**First run (EULA).** The very first launch must accept the NVIDIA CloudXR EULA. The auto-launch
prompts for it **on stdin**, so on a headless machine it will hang waiting for input. Bootstrap
the EULA once, interactively, with:
```bash
python -m isaacteleop.cloudxr --accept-eula # one-time: accept the CloudXR EULA
```
After that, `connect()` launches the runtime non-interactively. The launch **blocks for ~30s**
while the runtime comes up.
**Configuration.** Two fields on `IsaacTeleopConfig` (shared by every device) control this:
- `auto_launch_cloudxr` (default `True`) — whether `connect()` starts the runtime. Set `False`
when CloudXR is already running externally.
- `cloudxr_env_file` (default `None`) — an optional CloudXR device-profile `.env` selecting the
headset transport (e.g. an Apple Vision Pro profile). This is launcher **input**; it is not the
`~/.cloudxr/run/cloudxr.env` **output** file the old manual flow told you to `source`. `None`
keeps the default auto-WebRTC profile — though the SO-101 example overrides it to the
`default.env` shipped next to `teleoperate.py` unless you pass `--teleop.cloudxr_env_file`.
**Opting out.** To skip the auto-launch (CloudXR already running), either set
`auto_launch_cloudxr=False` or export:
```bash
export LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH=1
```
The **env var takes precedence over the config field**: if `LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH=1` is
set, the auto-launch is skipped even when `auto_launch_cloudxr=True`. This variable is
**independent** of Isaac Lab's `ISAACLAB_CXR_SKIP_AUTOLAUNCH` — setting one does not affect the
other.
**One teleoperator per process.** The CloudXR runtime configures the environment process-wide (a
singleton), so run a single Isaac Teleop teleoperator per process.
**Shutting down.** Always call `teleop_device.disconnect()` on exit — including on Ctrl-C. Wrap
your teleoperation loop in `try/finally` and call `disconnect()` in the `finally`. This tears down
the OpenXR session **before** the CloudXR runtime, which is the required order; the launcher's
`atexit` hook only reaps the runtime and does not run the session's `__exit__`, so without an
explicit `disconnect()` an interrupted run shuts down in the wrong order.
```python
teleop_device.connect()
try:
while True:
action = teleop_device.get_action()
# ... drive the robot ...
finally:
teleop_device.disconnect()
```
See [System Requirements](https://nvidia.github.io/IsaacTeleop/main/references/requirements.html)
for supported OS / GPU / CloudXR versions and headsets.
## How it works
The XR controller is one Isaac Teleop **input** device. `XRController` is a deliberately thin
reader: it exposes the **raw** controller grip pose — already statically rebased into the robot
base frame — plus the squeeze and trigger analog values. It has **no** retargeters and **no**
clutch logic of its own. The clutch (engage latch + delta rebasing onto the EE) and the gripper
mapping live downstream in the example loop, which then feeds LeRobot's existing closed‑loop
Cartesian IK pipeline — the same one the phone teleoperator uses. The device‑specific pieces are
`XRController`, the loop's `Clutch`, and `MapXRControllerActionToRobotAction`; everything downstream
(`EEBoundsAndSafety`, `InverseKinematicsEEToJoints`) is shared, and a future device (e.g. Manus
gloves) would swap in its own `teleop_<device>.py` + processor while reusing the rest.
`XRController._build_pipeline` wires Isaac Teleop's `ControllersSource` — statically rebased into
the robot base frame by the native `ControllerTransform` (`base_T_anchor`) — and exposes the
transformed controller stream verbatim. `get_action()` reads the grip pose, squeeze, and trigger
straight off it; the session is always stepped `RUNNING` (there is no clutch retargeter to gate).
The `Clutch` class (in `examples/isaac_teleop_to_so101/isaac_teleop/clutch.py`, driven by the
loop in `common.py`) mirrors Isaac Teleop's `SO101ClutchRetargeter`, but lives in-loop so the
device can stay a thin reader:
- It latches its engage origin on the squeeze **engage edge** (the frame the squeeze first crosses
`clutch_threshold`) and rebases both position and orientation around it, so engaging does not
teleport the arm. `Clutch.rebase` returns the absolute base-frame target as a `(pos, quat)`
pair, which the loop concatenates into the 7D `ee_pose` fed to the processor.
- The analog trigger becomes a gripper `closedness` in `[0, 1]` (0 = open, 1 = closed),
proportional to the trigger pull, which `MapXRControllerActionToRobotAction` maps to a jaw target.
See the Isaac Teleop
[Retargeting interface](https://nvidia.github.io/IsaacTeleop/main/references/retargeting/index.html)
and [architecture overview](https://nvidia.github.io/IsaacTeleop/main/overview/architecture.html)
for how source nodes and retargeters compose.
```text
VR controller (OpenXR)
│
▼
XRController.get_action() ── raw base-frame grip_pos / grip_quat + squeeze + trigger
│ (TeleopSession always stepped RUNNING; clutch lives downstream)
▼
Clutch.rebase(grip_pos, grip_quat) ── engage-relative delta applied to the EE home (pos + orient)
│ ee_pose (7) / closedness → absolute ee_pose; closedness = trigger
▼
MapXRControllerActionToRobotAction ── absolute ee.x/y/z; ee.w* = orientation rotvec target;
│ ee.x/y/z / ee.w* / ee.gripper_pos ee.gripper_pos = (1 - closedness) * 100
▼
EEBoundsAndSafety ── workspace clip + per-frame step clamp (clamp+warn)
│
▼
InverseKinematicsEEToJoints ── closed-loop Placo IK; position + soft-orientation
│ (orientation_weight=0.01) (passes ee.gripper_pos → gripper.pos)
▼
SO-101 follower joint targets
```
### The clutch: owned by the example loop
Unlike the phone pipeline (which splits the clutch across `MapPhoneActionToRobotAction` and
`EEReferenceAndDelta`), the XR clutch lives entirely in the example loop's `Clutch` class. It emits
an **absolute** EE pose, so there is no `EEReferenceAndDelta` stage and no delta accumulation in the
processor — `MapXRControllerActionToRobotAction` is a pure, stateless per‑frame mapping.
The clutch latches its engage origin on the squeeze **engage edge** (the moment the squeeze crosses
`clutch_threshold`) and drives the EE from the motion _relative_ to that origin, so the arm does not
teleport on engage. On **every** engage — startup and mid‑task re‑clutch alike — the home
_position_ is latched from forward kinematics on the arm's **measured joints**, so the home equals
where the arm physically is even if it moved while disengaged, and the engage is jump‑free. The
home _orientation_ keeps the last commanded rotation: the 5‑DOF arm tracks orientation only
softly, so latching the measured wrist orientation would inject its tracking offset into the
command on every re‑clutch.
## Controls
- **Squeeze / grip** — the **clutch** (deadman). Hold it past `clutch_threshold` to engage
teleoperation; release to pause. Each engage re‑captures the origin, so you can reposition
your hand while paused and re‑engage without the arm jumping (index/clutch style).
- **Trigger** — the **gripper**, controlled **analog**. The jaw tracks the trigger
proportionally — a half‑pressed trigger leaves the jaw half‑closed — via a closedness in
`[0, 1]` (0 = open, 1 = closed) that maps to an absolute gripper joint target.
- **Controller orientation** — the **wrist**. The clutch rebases the controller orientation
(engage‑relative, base‑frame) into a soft IK orientation target the wrist tracks alongside
position. On the 5‑DOF SO‑101 the wrist follows the hand only partially by design — see
`orientation_weight` below.
## Get started
### Step 1: Create the teleoperator
```python
# Run from the repo root so the `examples` package is importable.
from examples.isaac_teleop_to_so101.isaac_teleop import XRController, XRControllerConfig
teleop_config = XRControllerConfig(
hand_side="right", # "left" or "right" controller
clutch_threshold=0.5, # squeeze value above which the clutch engages
)
teleop_device = XRController(teleop_config)
```
`XRController.get_action()` returns the **raw** base‑frame controller pose, not a clutch‑rebased
target: `grip_pos` (3,) `[x, y, z]` [m] and `grip_quat` (4,) `[qx, qy, qz, qw]` in the robot base
frame, plus scalar `squeeze` and `trigger` analog values in `[0, 1]`. The example loop's `Clutch`
turns these into the absolute `ee_pose`, and the squeeze is thresholded by the loop against
`clutch_threshold` to engage.
### Step 2: Connect
Calling `teleop_device.connect()` first auto-launches the CloudXR runtime (unless you opted out —
see [Set up CloudXR and connect a headset](#set-up-cloudxr-and-connect-a-headset); this blocks for
~30s and on the first run prompts for the EULA on stdin), then starts the Isaac Teleop
[`TeleopSession`](https://nvidia.github.io/IsaacTeleop/main/getting_started/teleop_session.html)
(opens the OpenXR session and discovers the controllers). XR controllers are self‑calibrating, so
there is no manual calibration step — the clutch handles re‑centering each time you engage. Pair
`connect()` with a `try/finally` that calls `disconnect()` so the session tears down before the
runtime on exit/Ctrl-C.
### Step 3: Run the example
The example assumes you configured your robot (SO‑101 follower) and set the correct serial port.
The **robot URDF and its meshes are fetched automatically** on first run: the XR device downloads
the SO-101 URDF from the
[`lerobot/robot-urdfs` Hugging Face bucket](https://huggingface.co/buckets/lerobot/robot-urdfs/tree/so101)
into the LeRobot cache (`HF_LEROBOT_HOME/robot-urdfs/so101/`) and reuses it after, so there is no
separate download step :
```bash
python -m examples.isaac_teleop_to_so101.teleoperate --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
--robot.id=so101_follower_arm --teleop.type=xr_controller
```
The CLI is `lerobot-teleoperate`-style (draccus): `--robot.*` configures the SO-101 follower and
`--teleop.type` selects the Isaac input device (`xr_controller` | `so101_leader`), with
`--teleop.*` its device knobs. `--teleop.type=xr_controller` runs the XR-controller path described
above. The startup safety contract: by default it slews all joints to a default reset pose over
`--reset_duration` seconds (`--reset_to_origin=false` keeps the arm where it is), then seeds the
clutch home from the arm's measured pose so the first engage is jump-free; the follower is
commanded only while the clutch is engaged.
**Customizing the reset pose.** The reset pose ships as a built-in default (a comfortable mid-range
pose) and works out of the box — you do **not** need to record anything. To tailor it to your setup,
back-drive the arm to the pose you want and run
`python -m examples.isaac_teleop_to_so101.override_reset_pose --id <robot.id>`; it writes the
current joints to a per-arm file in the LeRobot cache
(`HF_LEROBOT_HOME/reset_poses/<robot.name>/<robot.id>.json`, keyed like calibration), which then takes
priority over the built-in default on the next run. Because it lives in the user-local cache (not
the repo), your override stays on your machine, and both `teleoperate` and `record` honor it
when launched with the same `--robot.id`.
The other device, `--teleop.type=so101_leader`, mirrors the follower 1:1 from a back-drivable
SO-101 _leader arm_ whose joints are streamed by Isaac Teleop's native `so101_leader` plugin (no
clutch, no IK — the leader and follower share the SO-101 kinematics).
The `so101_leader_plugin` binary is a C++ plugin that is **not** part of the `isaacteleop` pip
package — you build it from the Isaac Teleop source tree. Follow
[Build Isaac Teleop from source](https://nvidia.github.io/IsaacTeleop/main/getting_started/build_from_source/index.html)
(in short, from your Isaac Teleop checkout: `cmake -B build && cmake --build build --parallel &&
cmake --install build`); the build installs the plugins under `<IsaacTeleop>/install/plugins/`, so
the binary lands at `install/plugins/so101_leader/so101_leader_plugin` — the `--launch_plugin` path
below. See the plugin's own `README.md` (next to the binary) for its serial/calibration details.
Point `--teleop.port` at the physical leader's serial port and `--launch_plugin` at that plugin
binary to have the script spawn it after CloudXR is up:
```bash
python -m examples.isaac_teleop_to_so101.teleoperate --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
--robot.id=so101_follower_arm --teleop.type=so101_leader \
--teleop.port=/dev/ttyACM1 --teleop.id=so101_leader_arm \
--launch_plugin=/code/Teleop/install/plugins/so101_leader/so101_leader_plugin
```
(Note `so101_leader` here is the _Isaac_ leader, resolved against the Isaac Teleop device
registry, distinct from `lerobot-teleoperate`'s serial `so101_leader`.) When a `--teleop.port` is
set, the plugin's tick→radian calibration is inferred from `--teleop.id` and passed to the plugin
as its third positional arg — the LeRobot-format JSON at
`HF_LEROBOT_CALIBRATION/teleoperators/so_leader/<id>.json`, the same file the serial SO-101 leader
uses (`lerobot-calibrate --teleop.type=so101_leader --teleop.id=<id>`). If it is missing the script
warns and the plugin uses built-in defaults. Run `python -m examples.isaac_teleop_to_so101.teleoperate --help` for all flags. Its
startup safety contract: by default the follower is
slewed to the leader's first reading over `--align_duration` seconds (`--align=false` to skip) so
the arm does not snap when the mirror begins, and while the leader stream is stale the follower is
held at its measured pose.
The URDF fetch uses `huggingface_hub` (already a LeRobot dependency) against the public
`lerobot/robot-urdfs` bucket, so it needs no login. It is cached under
`HF_LEROBOT_HOME/robot-urdfs/so101/`; delete that folder to force a re‑download.
Then, in your headset: squeeze and hold the grip to engage, move the controller to drive the
arm, twist/tilt it to orient the wrist, and press the trigger to close the gripper
(proportionally — release to open).
To record a dataset (not just teleoperate), use `record.py` in the same folder. It dispatches on
`--teleop.type` (`xr_controller` | `so101_leader`) exactly like `teleoperate.py`, so either device
can drive the follower, and it saves the commanded joints to a LeRobot dataset (`lerobot-record`-style
`--dataset.*` flags). See its module docstring for the full CLI and the keyboard recording shortcuts.
## Important pipeline steps and options
The clutch already produces an absolute base‑frame pose, so the processor side is a thin
**absolute‑pose** path — there is no frame remap, no delta accumulation, and no
`EEReferenceAndDelta` stage.
- `MapXRControllerActionToRobotAction` is a stateless per‑frame mapping from the device output to
the IK input contract. It writes the absolute base‑frame position, encodes the absolute
orientation as a rotvec target, and inverts the closedness into a motor gripper target:
```python
action["ee.x"], action["ee.y"], action["ee.z"] = ee_pose[:3] # absolute, base frame [m]
action["ee.wx"], action["ee.wy"], action["ee.wz"] = orient_rotvec # orientation target (rotvec)
action["ee.gripper_pos"] = (1 - closedness) * 100 # motor units; SO-101 calibrates 100 = open
```
The gripper polarity (`100 = open, 0 = closed`) is a hardware‑calibration convention in the source — flip it there if the jaw opens when it should close.
- `EEBoundsAndSafety` clamps the EE to a workspace and rate‑limits per‑frame jumps. The clutch's
no‑teleport keeps frames small, so `max_ee_step_m` mostly catches transient controller tracking
glitches. The z floor is `0.0` (the table plane) so a stray target cannot drive the EE below the
table; x/y stay at the loose `[-1, 1]` m box. Set `raise_on_jump=False` so an over‑limit frame is
**clamped and warned** instead of raising — a crash mid‑loop would leave the arm uncontrolled:
```python
EEBoundsAndSafety(
end_effector_bounds={"min": [-1.0, -1.0, 0.0], "max": [1.0, 1.0, 1.0]},
max_ee_step_m=0.10,
raise_on_jump=False,
)
```
- `InverseKinematicsEEToJoints(initial_guess_current_joints=False, orientation_weight=0.01)` solves
closed‑loop Placo IK. SO‑101 is a 5‑DOF arm, so the IK is position‑dominant; the small
`orientation_weight` lets it softly track the orientation target carried in `ee.w*` so the wrist
follows the hand, while the under‑determined roll stays partial by design. There is **no**
`GripperVelocityToJoint`: the absolute `ee.gripper_pos` is passed straight to `gripper.pos`.
`initial_guess_current_joints=False` warm‑starts each solve from the **previous IK solution**
rather than re‑seeding from the measured joints, so the joint trajectory stays continuous
frame‑to‑frame. Tune `orientation_weight` on hardware — too high fights position tracking, too
low ignores the orientation command.
The example also gates safety at the loop level: after the startup reset slew (on by default —
pass `--reset_to_origin=false` to keep the arm where it is), it commands the robot **only while
the clutch is engaged**, and re‑sends the measured joints while disengaged, so releasing the
clutch freezes the arm in place.
See the [Processors for Robots and Teleoperators](./processors_robots_teleop) guide for more on
adapting the pipeline to other robots.
## Troubleshooting
- **`ModuleNotFoundError: isaacteleop`** — the `isaacteleop` package is not installed in the
active environment. Re-run the install command at the top of this guide:
`uv pip install "isaacteleop[cloudxr,retargeters-lite]~=1.3.131"`.
- **No controllers found** — make sure the CloudXR runtime is running, the firewall ports are
whitelisted, and the headset is connected (see
[Set up CloudXR and connect a headset](#set-up-cloudxr-and-connect-a-headset) and the Isaac
Teleop [Quick Start](https://nvidia.github.io/IsaacTeleop/main/getting_started/quick_start.html)).
- **CloudXR auto-launch failed** — `connect()` raises a `RuntimeError` if the runtime does not
come up within its startup timeout. Check the launcher logs under `~/.cloudxr/logs`. Common
causes: the EULA was never accepted (run `python -m isaacteleop.cloudxr --accept-eula` once,
interactively — the auto-launch prompts on stdin and hangs headless), or the runtime is already
running externally (set `LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH=1` or `auto_launch_cloudxr=False` to
skip the auto-launch).
- **Arm does not move** — the clutch is a deadman: you must hold the squeeze/grip past
`clutch_threshold`. Lower the threshold if your controller's squeeze is reported softly.
- **Motion feels misaligned** — confirm the headset/play space orientation. The controller stream
is rebased into the robot base frame by the `base_T_anchor` transform on `XRControllerConfig`
(default: standard OpenXR → robot axis convention); adjust it if your anchor frame differs.
## Learn more
NVIDIA Isaac Teleop documentation ([docs home](https://nvidia.github.io/IsaacTeleop/),
[GitHub](https://github.com/NVIDIA/IsaacTeleop)):
- [Quick Start](https://nvidia.github.io/IsaacTeleop/main/getting_started/quick_start.html) —
install, run the CloudXR server, connect a headset, run a teleop example.
- [TeleopSession](https://nvidia.github.io/IsaacTeleop/main/getting_started/teleop_session.html) —
the session API `XRController` wraps.
- [Retargeting interface](https://nvidia.github.io/IsaacTeleop/main/references/retargeting/index.html)
and [architecture overview](https://nvidia.github.io/IsaacTeleop/main/overview/architecture.html) —
how source nodes and retargeters compose into a pipeline.
- [Build from source](https://nvidia.github.io/IsaacTeleop/main/getting_started/build_from_source/index.html) —
build `isaacteleop` (and its C++ plugins, including the `so101_leader` plugin used above) from a
local checkout.
- [System Requirements](https://nvidia.github.io/IsaacTeleop/main/references/requirements.html) and
the [CloudXR SDK docs](https://docs.nvidia.com/cloudxr-sdk) — supported platforms, GPUs,
CloudXR/OpenXR runtime versions, and headsets.