lerobot/examples/isaac_teleop_to_so101

Isaac Teleop → SO-101

Teleoperate an SO-101/SO-100 follower arm — and record LeRobot datasets — with NVIDIA Isaac Teleop. Two input devices ship today:

• XR (VR) controller (--teleop.type=xr_controller) — the controller's grip pose drives the end-effector through a squeeze-to-engage clutch and LeRobot's Cartesian IK pipeline; the analog trigger drives the gripper.
• SO-101 leader arm (--teleop.type=so101_leader) — a back-drivable leader arm mirrored 1:1 onto the follower via Isaac Teleop's native so101_leader plugin (no clutch, no IK).

The full narrative guide (how the clutch works, CloudXR setup, headset pairing, tuning, and troubleshooting) is in the LeRobot docs (source: docs/source/isaac_teleop.mdx). This README is the canonical install and usage reference.

Requirements

• Linux workstation (see NVIDIA's system requirements for supported OS/GPU/headset combinations; isaacteleop publishes Linux wheels only).
• An SO-101 (or SO-100) follower arm, calibrated with lerobot-calibrate.
• For the XR device: a CloudXR-capable headset (e.g. Quest 3, Pico 4, Apple Vision Pro) on the same network.
• For the leader device: a second, back-drivable SO-101 leader arm and the so101_leader plugin binary built from the Isaac Teleop source tree (see Build from source).

Installation

This example lives in the LeRobot repository and is not part of the lerobot pip package, so work from a source checkout. From the repo root:

# LeRobot with the extras this example uses:
#   feetech    - SO-101 serial motor bus
#   kinematics - Placo IK solver (XR controller path)
#   dataset    - dataset recording (record.py)
# huggingface_hub >= 1.5 is needed by the automatic URDF fetch (Buckets API).
uv pip install -e ".[feetech,kinematics,dataset]" "huggingface_hub>=1.5"

# Isaac Teleop from public PyPI. `cloudxr` brings the CloudXR runtime bindings;
# `retargeters-lite` is the scipy-based retargeter path that resolves on both
# x86_64 and ARM (the full `retargeters` extra does not resolve on aarch64).
uv pip install "isaacteleop[cloudxr,retargeters-lite]~=1.3.131" "scipy>=1.14"

# Optional, x86_64 only: the full retargeter stack.
uv pip install "isaacteleop[retargeters]~=1.3.131"

One-time CloudXR EULA (the auto-launch prompts on stdin and would hang on a headless machine):

python -m isaacteleop.cloudxr --accept-eula

Usage

Run everything from the repo root with python -m so the examples package resolves.

Teleoperate — XR controller

python -m examples.isaac_teleop_to_so101.teleoperate \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=so101_follower_arm \
    --teleop.type=xr_controller

On startup the script launches the CloudXR runtime (~30 s), prints the workstation IP to enter in the headset's CloudXR web client, waits for the controllers to stream, slews the arm to a reset pose (--reset_to_origin=false to skip), and then: hold the squeeze/grip to engage, move the controller to drive the arm, pull the trigger to close the gripper. Releasing the squeeze freezes the arm. The SO-101 URDF is fetched automatically from the lerobot/robot-urdfs Hugging Face bucket into the LeRobot cache on first run.

To customize the reset pose: back-drive the arm to the pose you want, then

python -m examples.isaac_teleop_to_so101.override_reset_pose --port /dev/ttyACM0 --id so101_follower_arm

which writes it to HF_LEROBOT_HOME/reset_poses/<robot.name>/<robot.id>.json; runs with the same --robot.id use it automatically.

Teleoperate — SO-101 leader arm

python -m examples.isaac_teleop_to_so101.teleoperate \
    --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm \
    --teleop.type=so101_leader --teleop.port=/dev/ttyACM1 --teleop.id=so101_leader_arm \
    --launch_plugin=/path/to/IsaacTeleop/install/plugins/so101_leader/so101_leader_plugin

The follower is first slewed to the leader's pose over --align_duration seconds (--align=false to skip), then mirrors it 1:1. The plugin reuses the serial leader's calibration (HF_LEROBOT_CALIBRATION/teleoperators/so_leader/<teleop.id>.json).

Record a dataset

record.py takes the same --robot.*/--teleop.*/loop flags plus lerobot-record-style --dataset.* flags:

python -m examples.isaac_teleop_to_so101.record \
    --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm \
    --teleop.type=xr_controller \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --dataset.repo_id=<hf_user>/<dataset_name> \
    --dataset.single_task="Pick up the cube" \
    --dataset.num_episodes=3 --dataset.episode_time_s=20 --dataset.reset_time_s=5

Keyboard shortcuts (terminal-first, so they work over SSH): Right/n end episode early, Left/r re-record, Esc/q stop after the current episode.

Run either script with --help for all flags.

Layout

isaac_teleop/            device library: session lifecycle (base.py), XRController,
                         SO101LeaderArm, Clutch, configs, and the XR→IK processor step
common.py                shared loop infra: device bundles, clutch/IK pipeline wiring,
                         reset/align slews, URDF fetch, keyboard listener
teleoperate.py           teleoperation CLI (device selected via --teleop.type)
record.py                dataset-recording CLI (same device selection + --dataset.*)
override_reset_pose.py   save the current joints as the per-arm reset pose
default.env              CloudXR device-profile overrides passed to the launcher