Feeding the per-frame SMPL root quaternion into the mode-2 anchor produced
root-acceleration spikes (NaN QACC at DOF 0) mid-episode during replay. Keep the
anchor self-driven until the reference root trajectory is smoothed/rate-matched
(30 Hz dataset -> 50 Hz control).
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): WGO-tuned subtask prompt (atomic completed-events + duration prior)
Rework the plan-module subtask segmentation prompt toward the WGO-Bench
atomic annotation protocol: segment by completed world-state changes
(grasp/place/open/close/pour/insert), fold approach+retreat into their
event, keep separate events separate, and add a 2-10s duration prior.
Drops the pi0.7 "fewer larger composites preferred" bias that drove
under-segmentation on the benchmark. Output JSON shape unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): seeded-relabeling second pass for subtasks
Add an opt-in relabel pass (plan.subtask_seeded_relabel) that, after
segmentation, re-labels each span using previous/current/next segment
contact sheets and the seed label as a strong prior, minimally correcting
it. Mirrors macrodata's best end-to-end labeling step. Boundaries are
untouched; one extra VLM call per span. Off by default.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): robust OpenAI-compat client for hosted VLMs
Guard against a choice with no message (safety filter or a thinking model
that spends its whole budget before emitting content) so one empty reply
no longer crashes the whole annotation run; treat it as an empty response
and let the existing JSON-retry path handle it.
Add an optional `reasoning_effort` knob on VlmConfig, forwarded to the
server when set, to cap a thinking model's reasoning (needed for Gemini
via its OpenAI-compatible endpoint).
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): legible tile-scaled timestamp on contact sheets
The burned-in timestamp used the ~10px bitmap default font, which blurs
once the model downsamples a full contact sheet into 768px tiles, so the
VLM can no longer read the exact source time a boundary depends on. Scale
the timestamp to the tile height (with a graceful fallback on older
Pillow) so the visual time cue stays readable at sheet resolution.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): lean GEPA-aligned subtask segmentation prompt
Replace the verbose, label-heavy segmentation prompt with a lean
adaptation of the blog's GEPA-found completed_events_duration_prior
recipe: focus on completed manipulation events, explicit no-split /
no-merge rules, a 2-10s duration prior, and an instruction to prioritize
temporally correct boundaries over label wording. The previous prompt
over-weighted label guidance, which traded away boundary precision.
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert: restore original subtask segmentation prompt
The lean GEPA-aligned paraphrase (dd4b0110d) regressed Gemini on the
30-ep subset: Seg F1 0.259 -> 0.189 and E2E 0.184 -> 0.135, driven by
worse under-segmentation (224 -> 188 preds). The blog's 0.306 came from
the actual GEPA-search artifact, which a hand paraphrase does not
reproduce. Restore the original prompt, which remains our best config.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(annotate): env-var override for prompt templates
Allow LEROBOT_PROMPT_OVERRIDE_<name> to supersede the packaged prompt
file at load time. Enables prompt search (GEPA) to inject candidate
segmentation prompts into a remote annotate job via an env secret,
without committing a branch per candidate.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(annotate): genericize hosted-VLM comments (no model name)
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(annotate): document seeded-relabel and reasoning_effort flags
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(annotate): update subtask-prompt marker to match WGO-tuned prompt
The three plan-module tests keyed the canned VLM responder on the
literal 'atomic subtasks', which the WGO-tuned segmentation prompt no
longer contains (it now segments 'COMPLETED manipulation events'). Point
the fixture markers at the current wording so the subtask call is matched
again.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
- Remove unused RawKeyboard/drain_keyboard/process_keyboard from sonic_pipeline
(dead code duplicating lerobot.utils.keyboard_input); the G1 integration uses
the joystick path. Drop now-unused sys/select/termios/tty imports.
- Add a comment explaining the smpl.0 presence check is a sentinel for a full
SMPL window (review question).
Co-authored-by: Cursor <cursoragent@cursor.com>
Address review feedback that sonic_pipeline.py was dense and hard to read.
Adds a module-level architecture overview plus class and key-function
docstrings (planner subprocess, encoder/decoder, movement state, input
helpers). No behavior change.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a self-contained rt/smpl publisher in the pico_headset teleoperator
(pico_publisher.py + numpy SMPL FK in smpl_fk.py + vendored skeleton table)
so headset whole-body teleop no longer depends on gear_sonic/torch; only
xrobotoolkit_sdk is needed at the headset.
Also: share lowstate_to_obs/get_gravity_orientation via g1_utils (dedup
sonic_pipeline and UnitreeG1.get_observation), and fix dataset-replay joint
ordering (Unitree -> IsaacLab) for sonic.py --replay-dataset.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add Isaac Teleop SO-101 leader-arm teleoperator
Add the NVIDIA Isaac Teleop teleoperator scaffolding and its first device:
SO101LeaderArm, a back-drivable SO-101 leader arm on Isaac Teleop's generic
joint-space device path. It reads the leader's joints from the so101_leader
plugin via JointStateSource and emits follower-ready {joint}.pos (rad2deg arm,
gripper -> RANGE_0_100) for direct 1:1 joint drive.
- IsaacTeleopTeleoperator base + IsaacTeleopConfig (shared session/CloudXR config)
- SO101LeaderArm / SO101LeaderArmConfig and leader_joints_to_robot_action
- examples/isaac_teleop_to_so101/teleoperate_leader.py example
- pure-numpy conversion tests
- isaac-teleop optional extra + NVIDIA PyPI index in pyproject
* Add Isaac Teleop XR controller teleoperator for SO-101
Add end-to-end XR (VR) controller teleoperation of an SO-101 follower arm via
the NVIDIA Isaac Teleop stack, layered on the Isaac Teleop scaffolding.
Teleoperator (src/lerobot/teleoperators/isaac_teleop/):
- XRController / XRControllerConfig: connect to the CloudXR runtime, auto-launch
the Isaac Teleop session, and expose get_action() emitting the raw base-frame
grip pose, squeeze, and trigger.
- MapXRControllerActionToRobotAction: stateless per-frame mapper from the XR
action to the IK input contract (absolute ee.x/y/z, ee.gripper_pos, wrist_roll).
- OverwriteWristRollFromAngle: post-IK step writing the operator wrist-roll [rad]
onto wrist_roll.pos [deg], recovering the under-determined roll DOF.
Example (examples/isaac_teleop_to_so101/):
- teleoperate.py: thin absolute-pose IK pipeline with an in-loop clutch (engage
latch + 1:1 delta rebase of position and orientation), EEBoundsAndSafety, and
InverseKinematicsEEToJoints; slews to a recorded home on startup.
- record_reset_pose.py / download_assets.py / webxr.env / .gitignore.
Also:
- Extend robot_kinematic_processor.py with EEBoundsAndSafety and
InverseKinematicsEEToJoints.
- Add XRControllerConfig + base_T_anchor to the Isaac Teleop config.
- Add docs/source/isaac_teleop.mdx and the _toctree entry.
- Add unit tests for the CloudXR launcher and the XR controller processor.
* Unify Isaac Teleop SO-101 scripts behind a mandatory device selector
Merge teleoperate.py (XR controller: clutch + soft-orientation IK) and
teleoperate_leader.py (SO-101 leader arm: 1:1 joint mirror) into a single
teleoperate.py driven by a `lerobot-teleoperate`-style draccus CLI: a follower
`--robot.*` and an input `--teleop.*`, where `--teleop.type` (xr_controller |
so101_leader) selects the Isaac device.
Uses a "dispatch, don't merge" shape: per-device setup_xr/setup_leader build a
Device bundle (compute / startup / cleanup / command); a shared slew() takes a
per-step target callable (XR a fixed reset pose, leader a live re-read so the
1:1 handoff stays continuous); one device-branchless outer loop runs both, with
compute() -> None meaning "hold at the measured pose" (XR disengaged or leader
stale). The entrypoint is @parser.wrap()'d over a TeleoperateConfig dataclass and
dispatches on the parsed config type; device knobs ride on --teleop.* (the leader
serial port is --teleop.port, forwarded to the plugin) and loop/launch knobs are
top-level (--launch_plugin=<path> collapses the old --launch-plugin/--plugin-bin
pair; --reset_to_origin/--align/--dry_run).
To let the Isaac devices claim the natural --teleop.type names without colliding
with the serial so101_leader of lerobot-teleoperate, give IsaacTeleopConfig its
own draccus choice registry (own _choice_registry, decoupled from the global
TeleoperatorConfig one) and register XRControllerConfig as "xr_controller" and
SO101LeaderArmConfig as "so101_leader" there; the example types its teleop field
as IsaacTeleopConfig so the choices resolve against that scoped registry. These
devices drive the bespoke clutch/IK/align loop and are not routed through
make_teleoperator_from_config, so dropping them from the global registry is inert.
YAGNI sweep of the commit train: delete the orphaned OverwriteWristRollFromAngle
(wrist_roll_processor.py) plus its export and tests -- no producer emits
wrist_roll; the live XR path uses orientation-weight IK on the 5-DOF arm by
design. Kept the load-bearing knobs (orientation_weight, raise_on_jump,
base_T_anchor) and the optional reset-pose recorder. Updated isaac_teleop.mdx
for the unified entrypoint and excised the stale roll-retargeter prose.
Net LOC down (two scripts 714 lines -> one), in-loop device branches reduced to
zero. Planned and reviewed via a 6-persona multi-agent panel (3-round planning
convergence + 2-round review). Verification (isaacteleop/placo not installable
here, so the device classes cannot connect, but their config dataclasses and the
script import fine via deferred imports): the teleoperators test suite passes
(45 passed, 2 skipped), draccus parsing of both target command lines yields the
right config subclass with scoped --teleop.type, --help renders the scoped
choices, the serial so101_leader stays in the global registry, and ruff
check/format are green.
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
* Add Isaac Teleop SO-101 dataset recording script
record.py records a LeRobot dataset while driving the SO-101 follower
with either Isaac Teleop device (--teleop.type=xr_controller |
so101_leader), mirroring teleoperate.py's device dispatch.
* Extract shared Isaac Teleop SO-101 example infra into common.py
teleoperate.py and record.py both built the per-device pipeline and ran the
same read -> compute -> hold-when-idle -> sleep loop, with record.py importing
internals from teleoperate.py. Move the shared device/loop infrastructure
(Device, slew, Clutch, setup_xr/setup_leader + leader helpers, reset infra and
constants) into a new common.py, and add build_device() + hold_action() to
collapse the connect/dispatch/startup and idle-hold glue duplicated in both
entry points. The setup functions now type their config against a LoopConfig
Protocol, so common.py is decoupled from either CLI; both import from it.
Also rename record_reset_pose.py -> override_reset_pose.py so it is not confused
with record.py, and update the doc references.
* Add stdin keyboard backend so recording shortcuts work over SSH/headless
lerobot's init_keyboard_listener() uses pynput, which hooks GLOBAL key events
from the display server. Over SSH, under Wayland, or on a headless box with only a
TTY, keystrokes go to the terminal's stdin instead, so the listener never fires and
the Right/Left/Esc recording shortcuts silently do nothing.
Add a stdin (termios) keyboard backend to the example's common.py and an
init_keyboard_listener() that prefers it whenever stdin is an interactive TTY
(works over SSH / Wayland / headless-with-tty), falling back to lerobot's
pynput/headless listener for GUI launches with no controlling terminal. Selectable
via LEROBOT_KEYBOARD_BACKEND={auto,stdin,pynput,none}. The backend keeps ISIG so
Ctrl-C still works and always restores the terminal (on stop() and via atexit).
record.py now sources init_keyboard_listener from common; the Right/Left/Esc -> flag
mapping and the (listener, events) contract are unchanged.
Also convert record.py's loop_kwargs to a dict literal (ruff C408).
* Wait for the XR headset to connect before driving the arm
On the xr_controller path the example connected CloudXR and immediately ran the
reset slew + control loop, even if no headset was connected — the arm moved before
the operator was in VR, and get_action() just returned zeros so the clutch never
engaged.
Add an is_tracking property to XRController (set from the controller stream's
optional group, mirroring SO101LeaderArm) and a _wait_for_xr_controller() helper in
common.py that prints connection instructions (CloudXR web client URL + this
workstation's candidate IPv4s, with loopback/link-local and virtual/bridge/USB-gadget
interfaces filtered out) and polls until the controllers stream (indefinite, 15s
reminder, Ctrl-C to abort). setup_xr.startup() now connects, waits for the headset,
THEN runs the reset slew and seeds the clutch — so the arm only moves once the
operator is connected and watching. Mirrors the leader path's _wait_for_leader; both
record.py and teleoperate.py inherit it via the shared setup_xr.
* Address review feedback on the Isaac Teleop -> SO-101 example
Review-response and CI fixes for the Isaac Teleop -> SO-101 example.
- Move the XR Clutch into src/lerobot/teleoperators/isaac_teleop/clutch.py
(pure numpy + Rotation, no isaacteleop import), export it, and add
tests/teleoperators/test_clutch.py.
- Drop the vendored stdin keyboard listener; record.py uses a small terminal-
first wrapper over upstream's TerminalKeyListener (works over SSH even with a
local X display), falling back to upstream init_keyboard_listener otherwise.
- record.py: pass rgb_encoder/depth_encoder to LeRobotDataset create()/resume()
(upstream removed camera_encoder), fixing the AttributeError at record time.
- build_device: derive motor names from robot.action_features instead of
robot.bus (supports non-bus robots), and disconnect the follower if any step
after connect() fails so a failed setup never leaks the connection.
- Read leader joints by the group's declared names (_joints_group_to_rad)
instead of positionally, so a layout mismatch can't silently mirror the wrong
DOF onto the follower; add tests including a reversed-layout group.
- base.py: hoist `from pathlib import Path` to module scope; only the
isaacteleop CloudXRLauncher import stays lazy (optional dep).
- Trim the common.py module docstring and point to docs/source/isaac_teleop.mdx.
- default.env: correct the NV_DEVICE_PROFILE comment (auto-webrtc is the default;
this file overrides to Quest3, which works for both Quest 3 and Pico 4).
- download_assets.py: correct the RAW_BASE comment (tracks main, not pinned) and
add `# nosec B310` next to the existing `# noqa: S310` for the bandit hook.
- uv.lock: add the isaac-teleop extra's deps so `uv sync --locked` matches
pyproject; regenerated with uv 0.8.0 to keep lockfile revision 2 (CI's uv).
- isaac_teleop.mdx: prettier formatting.
* fix(.gitignore): removing .gitignore and using lerobot cache folder instead to store local user files
* chore(docstrings): reducing docstrings in default.env
* feat(URDF): cleaning up and simplifying the URDF download procedure
* feat(robot guard): adding a guard in case an unsupported robot type is provided (so-arms only)
* fix(imports): enforcing a python module structure to simplify imports
* feat(safe read): extending the motor bus safe read rationale to reset pose setting
* chore(trim): trimming lenghty comments and docstrings
* fix(deps): use isaacteleop [retargeters-lite] extra to unblock aarch64 (DGX Spark) (#3933)
* fix(deps): drop isaacteleop [retargeters] extra to unblock aarch64
The [retargeters] extra pulls dex-retargeting (pins numpy<2.0, conflicting
with lerobot's numpy>=2.0) and nlopt>=2.8 (no aarch64 wheels), making
lerobot[isaac-teleop] unresolvable on ARM (DGX Spark, Jetson Thor, GH200)
and over-constrained on numpy everywhere else.
The LeRobot teleoperators only import isaacteleop.retargeting_engine,
isaacteleop.cloudxr and isaacteleop.teleop_session_manager, all shipped in
the base wheel (requires only numpy>=1.23), so the extra is unused.
Verified on DGX Spark (aarch64, Python 3.12): resolves and installs with
isaacteleop 1.3.131 + numpy 2.2.6; all imported symbols load.
* fix(deps): use isaacteleop [retargeters-lite] extra for aarch64 support
Pin to isaacteleop ~=1.3.131 (the release that added ARM64/aarch64 support)
and swap the full [retargeters] extra for the new [retargeters-lite] one
(scipy-only). The full extra drags in dex-retargeting (pins numpy<2,
conflicting with lerobot's numpy>=2.0) and nlopt>=2.8 (no aarch64 wheels),
making lerobot[isaac-teleop] unresolvable on ARM hosts (DGX Spark, Jetson
Thor, GH200) and over-constrained on numpy everywhere else.
The LeRobot teleoperators only import isaacteleop.retargeting_engine,
isaacteleop.cloudxr and isaacteleop.teleop_session_manager — all covered
by the base wheel + retargeters-lite.
Verified on DGX Spark (aarch64, Python 3.12/3.13): resolves and installs
with isaacteleop 1.3.131 + numpy 2.2.6 + scipy 1.18.
* feat(deps): re-add full [retargeters] extra gated to x86_64
Keep the dex-retargeting/nlopt-based retargeters available on x86_64 (where
their wheels exist) via an environment marker, while ARM hosts (DGX Spark,
Jetson Thor, GH200) resolve with base + [retargeters-lite] only.
Verified: uv lock resolves on both platforms; on aarch64 the compile
excludes nlopt/dex-retargeting, on x86_64 they are included.
---------
Co-authored-by: Johnny Nunez <22727137+johnnynunez@users.noreply.github.com>
* chore(docstrings): trimming latest docstrings
* chore(teleop): move isaac-teleop to examples + update docs + add readme with installation notes
* chore(deps): restore uv.lock
* fix(example: isaac teleop parsing config
* fix(examples): isaac atomic-gripper controller
* feat(Examples): isaac-teleop holdlatch
* chore(examples): some other minor improvements for isaac-teleop
* chore(examples): top-level imports isaac-teleop
* chore(Examples): address ai review isaac-teleop
---------
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
Co-authored-by: Jiwen Cai <jiwenc@nvidia.com>
Co-authored-by: Johnny <johnnync13@gmail.com>
Co-authored-by: Johnny Nunez <22727137+johnnynunez@users.noreply.github.com>
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Martino Russi