Some checkpoints ship config.device=cpu (e.g. MolmoAct2 SO100/101). The
language runtime had no device override, so it always ran on the config
device. --policy.device=cuda (or cpu) now overrides cfg.device at load.
Co-authored-by: Cursor <cursoragent@cursor.com>
Load normalization stats from the checkpoint (norm_tag) and derive the
observation/action feature schema from the connected robot when no dataset
is given, mirroring lerobot-rollout. A dataset is still honoured when
supplied and its stats take precedence.
Co-authored-by: Cursor <cursoragent@cursor.com>
Enable running MolmoAct2 policies (e.g. on an SO101) in the interactive
language runtime with direct-subtask prompting.
- policies/molmoact2/molmoact2_adapter.py: MolmoAct2PolicyAdapter — flat VLA
bridge; select_action predicts an action chunk from the packed observation,
generate_text is a no-op (no text head; use --direct_subtask).
- runtime/registry.py: register "molmoact2" -> MolmoAct2PolicyAdapter.
- runtime/cli.py:
- Preserve model-input keys emitted outside observation.* (MolmoAct2 packs
the prompt+images into input_ids/pixel_values/...) through the robot
observation filter; no-op for PI0-family policies.
- Robot observation provider now reads the live task/subtask each frame via a
get_task callback, so a typed command re-packs the instruction (also fixes
stale-task for other flat VLAs). Bound to runtime state after creation.
Co-authored-by: Cursor <cursoragent@cursor.com>
Newer PI052 training runs serialize use_flex_attention into config.json.
This branch's attention path is SDPA/eager (mathematically equivalent), so
the field is accepted as an inert no-op (mirrors the existing use_hf_kernels
compat field) — otherwise loading those checkpoints raises DecodingError.
Co-authored-by: Cursor <cursoragent@cursor.com>
Wire the existing (but previously unreachable from the runtime) PI052
FlashRT FP8 MLP swap into the language runtime. --fp8 sets
config.use_flashrt_fp8_mlp before load; the policy calibrates and swaps
every Gemma + SigLIP MLP to fused FP8 on its first predict_action_chunk.
Ignored with a warning for policies without the flag (PI052 only).
Measured ~1.12x faster action-chunk inference (124 -> 111 ms) on an
RTX 5090; needs the `kernels` package (pin <0.13 for transformers) and
CUDA SM>=8.9, else it degrades to BF16.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add physical-robot support to the language runtime, plus a live rerun viewer.
- runtime/rerun_viz.py: headless rerun gRPC + web viewer; logs camera frames
(every control tick) and joint state. Prints an auto-connect ?url= view URL.
- runtime/cli.py:
- _run_robot_interactive: real-time control loop (background thread) with a
clean chat prompt — a typed command switches task/subtask immediately and
regenerates. Starts running as soon as a task is set (via --task or the
picker); otherwise paused until the first command. No flag needed.
- --rerun (+ --rerun.web_port / --rerun.grpc_port): live camera view; the
robot obs provider and action executor log frames to rerun.
- --direct_subtask (general, sim or robot): the typed text is the subtask fed
to the action expert; the LM subtask generator is disabled.
- Inference overrides: force compile_model=False and gradient_checkpointing
=False (torch.compile recompiled on every prompt-length change -> >1min per
chunk; grad checkpointing only slows the forward pass).
Co-authored-by: Cursor <cursoragent@cursor.com>
Drive a persistent RoboCasa kitchen with open-ended prompts and watch it live.
- runtime/sim_robocasa.py: single-scene RoboCasa backend (n_envs=2 for stable
EGL rendering — single-worker rendering is broken), high-res multi-view
compositing incl. wrist cam, annotated MP4 + rolling latest.png + MJPEG live
viewer, and /reset scene re-roll.
- runtime/cli.py: --sim mode with a main-thread control loop (background-thread
rendering corrupts EGL), clean chat-style prompt (a new command switches the
task and regenerates the subtask immediately), plus --sim.render_size,
--sim.views, --sim.stream_port, --sim.direct_subtask and --disable_memory.
- runtime/adapter.py: GenerationConfig.enable_memory / enable_subtask toggles.
- runtime/registry.py + policies/pi05/pi05_adapter.py: register pi05 (flat VLA,
direct task-text conditioning; no subtask/memory head).
- policies/pi052/inference/pi052_adapter.py: condition the action expert on
"{subtask}, State: {..}" to match eval/training.
- envs/robocasa.py + envs/configs.py: terminate_on_success + horizon options so
the interactive kitchen persists across tasks (defaults preserve eval).
Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the opt-in EMA-shadow feature entirely: EMAConfig, the `ema` field on
TrainPipelineConfig, all EMA logic in lerobot_train.py (setup/resume, per-step
update, W&B observability, checkpoint save, EMA-model eval, and the sibling
`<repo_id>-ema` hub push), and the ema-pytorch dependency.
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep PI052Policy.forward's loss components as detached tensors and only
materialize loss/grad_norm/update_s to python floats on logging steps
(1-in-log_freq) via a new update_policy(log_metrics=...) gate. Also dedupe
the predict_actions .any().item() control-flow sync (2 -> 1 per step).
Keeps the training step fully async on non-logging steps so the next batch's
dataloading/enqueue overlaps GPU compute instead of stalling on a per-step
CUDA sync.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add Isaac Teleop SO-101 leader-arm teleoperator
Add the NVIDIA Isaac Teleop teleoperator scaffolding and its first device:
SO101LeaderArm, a back-drivable SO-101 leader arm on Isaac Teleop's generic
joint-space device path. It reads the leader's joints from the so101_leader
plugin via JointStateSource and emits follower-ready {joint}.pos (rad2deg arm,
gripper -> RANGE_0_100) for direct 1:1 joint drive.
- IsaacTeleopTeleoperator base + IsaacTeleopConfig (shared session/CloudXR config)
- SO101LeaderArm / SO101LeaderArmConfig and leader_joints_to_robot_action
- examples/isaac_teleop_to_so101/teleoperate_leader.py example
- pure-numpy conversion tests
- isaac-teleop optional extra + NVIDIA PyPI index in pyproject
* Add Isaac Teleop XR controller teleoperator for SO-101
Add end-to-end XR (VR) controller teleoperation of an SO-101 follower arm via
the NVIDIA Isaac Teleop stack, layered on the Isaac Teleop scaffolding.
Teleoperator (src/lerobot/teleoperators/isaac_teleop/):
- XRController / XRControllerConfig: connect to the CloudXR runtime, auto-launch
the Isaac Teleop session, and expose get_action() emitting the raw base-frame
grip pose, squeeze, and trigger.
- MapXRControllerActionToRobotAction: stateless per-frame mapper from the XR
action to the IK input contract (absolute ee.x/y/z, ee.gripper_pos, wrist_roll).
- OverwriteWristRollFromAngle: post-IK step writing the operator wrist-roll [rad]
onto wrist_roll.pos [deg], recovering the under-determined roll DOF.
Example (examples/isaac_teleop_to_so101/):
- teleoperate.py: thin absolute-pose IK pipeline with an in-loop clutch (engage
latch + 1:1 delta rebase of position and orientation), EEBoundsAndSafety, and
InverseKinematicsEEToJoints; slews to a recorded home on startup.
- record_reset_pose.py / download_assets.py / webxr.env / .gitignore.
Also:
- Extend robot_kinematic_processor.py with EEBoundsAndSafety and
InverseKinematicsEEToJoints.
- Add XRControllerConfig + base_T_anchor to the Isaac Teleop config.
- Add docs/source/isaac_teleop.mdx and the _toctree entry.
- Add unit tests for the CloudXR launcher and the XR controller processor.
* Unify Isaac Teleop SO-101 scripts behind a mandatory device selector
Merge teleoperate.py (XR controller: clutch + soft-orientation IK) and
teleoperate_leader.py (SO-101 leader arm: 1:1 joint mirror) into a single
teleoperate.py driven by a `lerobot-teleoperate`-style draccus CLI: a follower
`--robot.*` and an input `--teleop.*`, where `--teleop.type` (xr_controller |
so101_leader) selects the Isaac device.
Uses a "dispatch, don't merge" shape: per-device setup_xr/setup_leader build a
Device bundle (compute / startup / cleanup / command); a shared slew() takes a
per-step target callable (XR a fixed reset pose, leader a live re-read so the
1:1 handoff stays continuous); one device-branchless outer loop runs both, with
compute() -> None meaning "hold at the measured pose" (XR disengaged or leader
stale). The entrypoint is @parser.wrap()'d over a TeleoperateConfig dataclass and
dispatches on the parsed config type; device knobs ride on --teleop.* (the leader
serial port is --teleop.port, forwarded to the plugin) and loop/launch knobs are
top-level (--launch_plugin=<path> collapses the old --launch-plugin/--plugin-bin
pair; --reset_to_origin/--align/--dry_run).
To let the Isaac devices claim the natural --teleop.type names without colliding
with the serial so101_leader of lerobot-teleoperate, give IsaacTeleopConfig its
own draccus choice registry (own _choice_registry, decoupled from the global
TeleoperatorConfig one) and register XRControllerConfig as "xr_controller" and
SO101LeaderArmConfig as "so101_leader" there; the example types its teleop field
as IsaacTeleopConfig so the choices resolve against that scoped registry. These
devices drive the bespoke clutch/IK/align loop and are not routed through
make_teleoperator_from_config, so dropping them from the global registry is inert.
YAGNI sweep of the commit train: delete the orphaned OverwriteWristRollFromAngle
(wrist_roll_processor.py) plus its export and tests -- no producer emits
wrist_roll; the live XR path uses orientation-weight IK on the 5-DOF arm by
design. Kept the load-bearing knobs (orientation_weight, raise_on_jump,
base_T_anchor) and the optional reset-pose recorder. Updated isaac_teleop.mdx
for the unified entrypoint and excised the stale roll-retargeter prose.
Net LOC down (two scripts 714 lines -> one), in-loop device branches reduced to
zero. Planned and reviewed via a 6-persona multi-agent panel (3-round planning
convergence + 2-round review). Verification (isaacteleop/placo not installable
here, so the device classes cannot connect, but their config dataclasses and the
script import fine via deferred imports): the teleoperators test suite passes
(45 passed, 2 skipped), draccus parsing of both target command lines yields the
right config subclass with scoped --teleop.type, --help renders the scoped
choices, the serial so101_leader stays in the global registry, and ruff
check/format are green.
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
* Add Isaac Teleop SO-101 dataset recording script
record.py records a LeRobot dataset while driving the SO-101 follower
with either Isaac Teleop device (--teleop.type=xr_controller |
so101_leader), mirroring teleoperate.py's device dispatch.
* Extract shared Isaac Teleop SO-101 example infra into common.py
teleoperate.py and record.py both built the per-device pipeline and ran the
same read -> compute -> hold-when-idle -> sleep loop, with record.py importing
internals from teleoperate.py. Move the shared device/loop infrastructure
(Device, slew, Clutch, setup_xr/setup_leader + leader helpers, reset infra and
constants) into a new common.py, and add build_device() + hold_action() to
collapse the connect/dispatch/startup and idle-hold glue duplicated in both
entry points. The setup functions now type their config against a LoopConfig
Protocol, so common.py is decoupled from either CLI; both import from it.
Also rename record_reset_pose.py -> override_reset_pose.py so it is not confused
with record.py, and update the doc references.
* Add stdin keyboard backend so recording shortcuts work over SSH/headless
lerobot's init_keyboard_listener() uses pynput, which hooks GLOBAL key events
from the display server. Over SSH, under Wayland, or on a headless box with only a
TTY, keystrokes go to the terminal's stdin instead, so the listener never fires and
the Right/Left/Esc recording shortcuts silently do nothing.
Add a stdin (termios) keyboard backend to the example's common.py and an
init_keyboard_listener() that prefers it whenever stdin is an interactive TTY
(works over SSH / Wayland / headless-with-tty), falling back to lerobot's
pynput/headless listener for GUI launches with no controlling terminal. Selectable
via LEROBOT_KEYBOARD_BACKEND={auto,stdin,pynput,none}. The backend keeps ISIG so
Ctrl-C still works and always restores the terminal (on stop() and via atexit).
record.py now sources init_keyboard_listener from common; the Right/Left/Esc -> flag
mapping and the (listener, events) contract are unchanged.
Also convert record.py's loop_kwargs to a dict literal (ruff C408).
* Wait for the XR headset to connect before driving the arm
On the xr_controller path the example connected CloudXR and immediately ran the
reset slew + control loop, even if no headset was connected — the arm moved before
the operator was in VR, and get_action() just returned zeros so the clutch never
engaged.
Add an is_tracking property to XRController (set from the controller stream's
optional group, mirroring SO101LeaderArm) and a _wait_for_xr_controller() helper in
common.py that prints connection instructions (CloudXR web client URL + this
workstation's candidate IPv4s, with loopback/link-local and virtual/bridge/USB-gadget
interfaces filtered out) and polls until the controllers stream (indefinite, 15s
reminder, Ctrl-C to abort). setup_xr.startup() now connects, waits for the headset,
THEN runs the reset slew and seeds the clutch — so the arm only moves once the
operator is connected and watching. Mirrors the leader path's _wait_for_leader; both
record.py and teleoperate.py inherit it via the shared setup_xr.
* Address review feedback on the Isaac Teleop -> SO-101 example
Review-response and CI fixes for the Isaac Teleop -> SO-101 example.
- Move the XR Clutch into src/lerobot/teleoperators/isaac_teleop/clutch.py
(pure numpy + Rotation, no isaacteleop import), export it, and add
tests/teleoperators/test_clutch.py.
- Drop the vendored stdin keyboard listener; record.py uses a small terminal-
first wrapper over upstream's TerminalKeyListener (works over SSH even with a
local X display), falling back to upstream init_keyboard_listener otherwise.
- record.py: pass rgb_encoder/depth_encoder to LeRobotDataset create()/resume()
(upstream removed camera_encoder), fixing the AttributeError at record time.
- build_device: derive motor names from robot.action_features instead of
robot.bus (supports non-bus robots), and disconnect the follower if any step
after connect() fails so a failed setup never leaks the connection.
- Read leader joints by the group's declared names (_joints_group_to_rad)
instead of positionally, so a layout mismatch can't silently mirror the wrong
DOF onto the follower; add tests including a reversed-layout group.
- base.py: hoist `from pathlib import Path` to module scope; only the
isaacteleop CloudXRLauncher import stays lazy (optional dep).
- Trim the common.py module docstring and point to docs/source/isaac_teleop.mdx.
- default.env: correct the NV_DEVICE_PROFILE comment (auto-webrtc is the default;
this file overrides to Quest3, which works for both Quest 3 and Pico 4).
- download_assets.py: correct the RAW_BASE comment (tracks main, not pinned) and
add `# nosec B310` next to the existing `# noqa: S310` for the bandit hook.
- uv.lock: add the isaac-teleop extra's deps so `uv sync --locked` matches
pyproject; regenerated with uv 0.8.0 to keep lockfile revision 2 (CI's uv).
- isaac_teleop.mdx: prettier formatting.
* fix(.gitignore): removing .gitignore and using lerobot cache folder instead to store local user files
* chore(docstrings): reducing docstrings in default.env
* feat(URDF): cleaning up and simplifying the URDF download procedure
* feat(robot guard): adding a guard in case an unsupported robot type is provided (so-arms only)
* fix(imports): enforcing a python module structure to simplify imports
* feat(safe read): extending the motor bus safe read rationale to reset pose setting
* chore(trim): trimming lenghty comments and docstrings
* fix(deps): use isaacteleop [retargeters-lite] extra to unblock aarch64 (DGX Spark) (#3933)
* fix(deps): drop isaacteleop [retargeters] extra to unblock aarch64
The [retargeters] extra pulls dex-retargeting (pins numpy<2.0, conflicting
with lerobot's numpy>=2.0) and nlopt>=2.8 (no aarch64 wheels), making
lerobot[isaac-teleop] unresolvable on ARM (DGX Spark, Jetson Thor, GH200)
and over-constrained on numpy everywhere else.
The LeRobot teleoperators only import isaacteleop.retargeting_engine,
isaacteleop.cloudxr and isaacteleop.teleop_session_manager, all shipped in
the base wheel (requires only numpy>=1.23), so the extra is unused.
Verified on DGX Spark (aarch64, Python 3.12): resolves and installs with
isaacteleop 1.3.131 + numpy 2.2.6; all imported symbols load.
* fix(deps): use isaacteleop [retargeters-lite] extra for aarch64 support
Pin to isaacteleop ~=1.3.131 (the release that added ARM64/aarch64 support)
and swap the full [retargeters] extra for the new [retargeters-lite] one
(scipy-only). The full extra drags in dex-retargeting (pins numpy<2,
conflicting with lerobot's numpy>=2.0) and nlopt>=2.8 (no aarch64 wheels),
making lerobot[isaac-teleop] unresolvable on ARM hosts (DGX Spark, Jetson
Thor, GH200) and over-constrained on numpy everywhere else.
The LeRobot teleoperators only import isaacteleop.retargeting_engine,
isaacteleop.cloudxr and isaacteleop.teleop_session_manager — all covered
by the base wheel + retargeters-lite.
Verified on DGX Spark (aarch64, Python 3.12/3.13): resolves and installs
with isaacteleop 1.3.131 + numpy 2.2.6 + scipy 1.18.
* feat(deps): re-add full [retargeters] extra gated to x86_64
Keep the dex-retargeting/nlopt-based retargeters available on x86_64 (where
their wheels exist) via an environment marker, while ARM hosts (DGX Spark,
Jetson Thor, GH200) resolve with base + [retargeters-lite] only.
Verified: uv lock resolves on both platforms; on aarch64 the compile
excludes nlopt/dex-retargeting, on x86_64 they are included.
---------
Co-authored-by: Johnny Nunez <22727137+johnnynunez@users.noreply.github.com>
* chore(docstrings): trimming latest docstrings
* chore(teleop): move isaac-teleop to examples + update docs + add readme with installation notes
* chore(deps): restore uv.lock
* fix(example: isaac teleop parsing config
* fix(examples): isaac atomic-gripper controller
* feat(Examples): isaac-teleop holdlatch
* chore(examples): some other minor improvements for isaac-teleop
* chore(examples): top-level imports isaac-teleop
* chore(Examples): address ai review isaac-teleop
---------
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
Co-authored-by: Jiwen Cai <jiwenc@nvidia.com>
Co-authored-by: Johnny <johnnync13@gmail.com>
Co-authored-by: Johnny Nunez <22727137+johnnynunez@users.noreply.github.com>
Co-authored-by: Steven Palma <steven.palma@huggingface.co>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* feat(policies): add EVO1 policy
* fix(evo1): infer batch size after normalizing image dims
`_collect_image_batches` read `batch_size = batch[camera_keys[0]].shape[0]`
before normalizing per-camera tensors to `(B, C, H, W)`. For an unbatched
`(C, H, W)` input (which the function tries to support via the `image.dim() == 3`
branch), this picked up the channel count `C` instead of the real batch size,
making the subsequent per-sample loop iterate `C` times and indexing go
out of bounds.
Normalize each camera tensor up-front, then read `batch_size` from the
normalized batch dim. Adds `test_collect_image_batches_handles_unbatched_chw`
covering the regression.
Reported by Copilot review on huggingface/lerobot#3545.
* chore(lock): regenerate uv.lock for evo1 extra
Adds the `evo1` entry to `[package.metadata.requires-dist]` and the
`provides-extras` list so that `uv sync --locked --extra test` (used by
fast_tests.yml) no longer reports the lockfile as stale.
Generated with `uv 0.8.0` (matching `UV_VERSION` in fast_tests.yml).
The non-evo1 marker tweaks are produced by `uv lock` re-resolving the
existing dep graph and are not introduced by this PR.
* chore(evo1): align with policy contribution guide conventions
- Add `src/lerobot/policies/evo1/README.md` symlink into `docs/source/evo1.mdx`
to match the in-tree README convention (mirroring the EO-1 layout).
- Convert `transformers` import in `internvl3_embedder.py` to the standard
`TYPE_CHECKING + _transformers_available` two-step gating used by other
optional-backbone policies (e.g. diffusion). The previous lazy-in-`__init__`
import was functionally equivalent for runtime gating but didn't expose the
real symbols to type checkers.
- Add `lerobot[evo1]` to the `all` extra in `pyproject.toml` so
`pip install 'lerobot[all]'` keeps installing every optional policy.
Per the guidance in https://moon-ci-docs.huggingface.co/docs/lerobot/pr_3534/en/contributing_a_policy.
* fix(evo1): finalize policy guide alignment
* docs(evo1): format results table
* Fix EVO1 LIBERO rollout processors
* Fix EVO1 LIBERO eval action postprocessing
* Fix eval action conversion for bf16 policies
* fix(evo1): move LIBERO padding into policy processors
* refactor(evo1): use native HF InternVL3-1B-hf, drop trust_remote_code
- Switch from OpenGVLab/InternVL3-1B (requires trust_remote_code=True)
to OpenGVLab/InternVL3-1B-hf (native transformers implementation).
- Replace manual _extract_feature + _prepare_and_fuse_embeddings with
a single model.forward() call — verified bit-for-bit identical output.
- Remove ~170 lines of manual ViT/pixel-shuffle/projection logic.
- Symlink README.md to docs/source/ following repo convention.
Weights are byte-identical between both model variants; only the module
naming differs. All 12 existing unit tests pass. Local training (10 steps)
on maximellerbach/omx_pickandplace confirmed working.
* refactor(policy): evo1 GPU-batched preprocessing + vectorized attention masking + remove dead code
* fix(style): pre-commit
oops
* chore(evo1): delete added test + reduce diff
* refactor(policies): use config for evo1 + local imports
* refactor(policies): multiple improvements
* chore: update docs + remove legacy codepaths
* feat(policies): implement RTC to EVO1
---------
Co-authored-by: javadcc_mac <javadcc1@sjtu.edu.cn>
Co-authored-by: Yiming Wang <145452074+JAVAdcc@users.noreply.github.com>
Co-authored-by: Martino Russi <nopyeps@gmail.com>
* Add GR00T N1.7 support
Add GR00T N1.7 policy configuration, checkpoint compatibility, processor parity, LIBERO documentation, and focused tests.
Co-authored-by: Ryan Halabi <ryhalabi@nvidia.com>
* Move Groot processor compatibility into Groot loader
* Restore GR00T Flash Attention install guidance
* Allow Groot fake RTC chunk prefetch
* Fix GR00T N1.7 RTC action decoding
* Trim GR00T N1.7 RTC chunks to valid horizon
* Ignore padded GR00T N1.7 RTC prefix rows
* removed n1.5 dependency
* removed remaining N1.5 traces
* groot: auto-enable LIBERO gripper action transform for libero_sim
GR00T N1.7 emits gripper in [0,1] but LIBERO expects [-1,1]. The decode
transform existed but was never auto-enabled for embodiment_tag=libero_sim,
so the policy scored 0% on LIBERO eval. Auto-set it in __post_init__ (still
overridable). LIBERO Spatial eval: 0% -> 98%.
* Reconnect GR00T relative action processors
* groot: remove dead N1.5 code (eagle2_hg_model, flow_matching_action_head, action_encoder)
N1.7 backbone is nvidia/Cosmos-Reason2-2B via Qwen3VLForConditionalGeneration,
not Eagle2 — eagle2_hg_model/ had zero refs outside its own dir.
GR00TN17ActionHead (groot_n1_7.py) re-implements MultiEmbodimentActionEncoder +
CategorySpecificLinear + swish + SinusoidalPositionalEncoding locally, so
flow_matching_action_head.py (N1.5 FlowmatchingActionHead) and its sole
dependency action_encoder.py are dead. Verified: no src/ or tests/ reference.
Removed (~2037 LOC):
- eagle2_hg_model/ (4 files, ~1575 LOC)
- action_head/flow_matching_action_head.py (408 LOC)
- action_head/action_encoder.py (54 LOC)
cross_attention_dit.py KEPT (DiT/AlternateVLDiT/SelfAttentionTransformer live in N1.7).
* groot: reuse lerobot get_device_from_parameters instead of inline lookup
modeling_groot.py duplicated next(self.parameters()).device twice. LeRobot
ships get_device_from_parameters in policies/utils.py (used by diffusion,
vqbet, tdmpc, gaussian_actor). Reuse it for consistency with the framework.
* groot: fix stale Eagle VLM docstring in processor (N1.7 uses Qwen3-VL backbone)
Addresses checker nit: processor_groot.py docstring still described the N1.5
Eagle VLM path with eagle_content/eagle_* keys that no longer exist in the code.
* test(groot): add N1.7 original-vs-LeRobot output parity test
Verifies the LeRobot GR00T N1.7 integration produces equivalent raw
action_pred to NVIDIA Isaac-GR00T for the same checkpoint, inputs, seed,
precision (fp32) and attention kernel (SDPA): max|diff|=8.9e-7 on the
libero_sim embodiment (GR00T-N1.7-LIBERO/libero_10).
The two impls pin incompatible transformers majors (orig 4.57.3 vs
LeRobot 5.x) and cannot share a process, so the original outputs + exact
collated inputs are produced out-of-process and loaded from an .npz. The
test skips on CI / when the checkpoint or artifact are absent.
* test(groot): parametrize N1.7 parity across all checkpoint embodiments
Generalize the original-vs-LeRobot N1.7 output-parity test from a single
libero_sim case to every embodiment tag in the checkpoint (libero_sim, oxe_droid,
real_g1, the real_r1_pro_sharpa family, and the xdof family). Inputs are built
generically from checkpoint metadata; the test discovers per-tag .npz artifacts
and runs one parametrized case each, loading the LeRobot model once via a fixture.
All 9 embodiments match the original to fp32 epsilon (max|diff| < 3e-6), confirming
the integration is correct across the model's full embodiment space and not overfit
to libero_sim.
* test(groot): self-contained parity test + in-repo producer + docs
- Rename test_groot_n1_7_vs_original.py -> test_groot_vs_original.py
- Make the test self-contained: producer script (dump_original_n1_7.py) now lives
next to the test; default artifact dir is repo-relative
(tests/policies/groot/artifacts/), overridable via GROOT_N1_7_PARITY_DIR. The
test only reads artifacts and skips if absent -- it never creates external dirs.
- Heavy .npz artifacts (~6-9MB each) are gitignored and regenerated by the producer;
never committed.
- Drop the verbose 'MULTIPLE EMBODIMENTS' docstring block (kept a one-line note).
- Document the parity procedure in the groot policy README (docs/source/policy_groot_README.md).
- Rename test fn test_groot_n1_7_get_action_parity -> test_groot_get_action_parity.
9/9 embodiments still pass (max|diff| < 3e-6, fp32 eps).
* docs(groot): drop WHY TWO ENVIRONMENTS block from parity test docstring
* test(groot): move parity producer into utils/ package
Mirror the tests/policies/pi0_pi05/utils convention: move dump_original_n1_7.py into
a tests/policies/groot/utils/ package (with __init__.py) and update all path
references in the test docstring/skip-message and the policy README.
* test(groot): adopt test_groot_lerobot for GR00T N1.7, drop N1.5
The test loaded MODEL_PATH='aractingi/bimanual-handover-groot-10k', an N1.5
checkpoint (config base_model_path=nvidia/GR00T-N1.5-3B, no model_version). On
load, model_version defaults to n1.7 while the base path infers n1.5, so the
version-consistency guard in GrootConfig.__post_init__ raised ValueError and both
test_lerobot_groot_inference and test_lerobot_groot_forward_pass failed. N1.5 is no
longer a supported model_version.
Adopt the test for N1.7:
- MODEL_PATH -> nvidia/GR00T-N1.7-3B (root-level sharded safetensors; loads via
GrootPolicy.from_pretrained as a base N1.7 model).
- Embodiment tag 'gr1' (N1.5) -> 'gr1_unified' (valid N1.7 tag from the checkpoint
embodiment_id.json), via a single EMBODIMENT_TAG constant.
- DUMMY_ACTION_HORIZON 16 -> 40 to match N1.7's native action-chunk size.
- Docstrings/labels updated to 'GR00T N1.7'.
Both tests run and pass on CUDA; full tests/policies/groot/ suite is
73 passed / 0 failed / 0 skipped.
* docs(groot): document the N1.5 removal and the N1.7 parity test
- groot.mdx: breaking-change warning and migration path (pin lerobot==0.5.1 to
keep N1.5, or move to N1.7); the dead `huggingface-cli download` is replaced
with `hf download`.
- policy_groot_README.md: N1.5 removal note, updated paper / model-card links,
and the two-comparison (model parity + preprocessor parity) description of
the original-vs-LeRobot test, including the raw-observation artifacts and
recorded seed.
* fix(groot): N1.7 backbone loading and DiT parameter-count logging
- select_layer default tracks the N1.7-3B checkpoint value (16); real
checkpoint loads still override it from config.json.
- get_backbone_cls recognizes Cosmos-Reason2 / Qwen3-VL backbones by name and
warns (instead of silently assuming) when an unrecognized backbone is loaded
only on the strength of backbone_model_type='qwen'.
- 'revision' pins the GR00T checkpoint repo only and is no longer forwarded
into the unrelated backbone repo load; pin the backbone via
transformers_loading_kwargs instead.
- DiT / SelfAttentionTransformer parameter counts go through logging.debug
instead of print().
* fix(groot): N1.7 config defaults, N1.5 rejection, and processor/model runtime fixes
Covers the GR00T N1.7 source trio (configuration, processor, model wrapper).
Config:
- GrootConfig defaults are the N1.7 values; explicitly passed legacy N1.5-era
values (chunk_size=50, max_state_dim=64, ...) are remapped with a warning
instead of silently.
- action_decode_transform gains an 'auto' sentinel so an explicit 'none'
opt-out wins over the libero_sim default and survives save/load round-trips.
- action_delta_indices is cached on the inputs that determine it.
- Legacy N1.5 checkpoints/configs (tokenizer_assets_repo, model_type/
architectures/eagle backbone markers) are rejected with a single clear
error pointing to lerobot==0.5.1.
Processor:
- GrootN17ActionDecodeStep handles the 2-D (B, D) actions delivered by sync
select_action (relative eef/non-eef decode in eval/record flows).
- Postprocessor falls back to dataset stats when a raw checkpoint lacks the
configured embodiment tag; raw-state cache is per-instance, not
process-global; caller overrides (device, rename_map) are honored on the
raw-checkpoint branch.
- Camera/modality-key mismatches warn (including the zero-match fallback);
deprecated Qwen2VLImageProcessorFast replaced with Qwen2VLImageProcessor;
removed N1.5 processor steps are stubbed to raise the removal guidance and
the action-unpack step is re-registered as _v2.
Model:
- Flash-attention probe is diagnostic-only; forward raises on a missing loss;
print() replaced with logging; N1.5 base-path mismatch includes the
removal guidance.
* fix(groot): skip normalization overrides for training
* fix(groot): GPU/tensor N1.7 image preprocessing + resize to trained resolution
GR00T training was dataloader-bound (0->100->0 GPU-utilization sawtooth).
GrootN17VLMEncodeStep ran the Qwen3-VL image processor per frame on PIL images
on the single CPU main-loop thread, and that cost is timed inside dataloading_s
(preprocessor(batch) runs in the main process, not the dataloader workers), so
adding workers cannot hide it.
- Feed the torchvision-backed Qwen3-VL processor (C,H,W) uint8 tensors instead
of a per-frame Image.fromarray PIL roundtrip, and run resize/normalize/patchify
on config.device (GPU) when available. Bit-identical on CPU when no resize is
configured; with a resize only the PIL->torchvision bicubic backend differs
(<2/255 per pixel). The use_albumentations path stays PIL/cv2; reload on a box
without the saved device falls back to CPU.
- Default image_target_size/crop to the N1.7 backbone's training geometry
(256x256 / 230x230) when a checkpoint ships no image sizing (checkpoint_assets
is None, e.g. finetuning nvidia/GR00T-N1.7-3B via repo-id with a new
embodiment). Previously image_target_size=None disabled the resize, so
full-resolution frames were patchified into ~4.7x more vision tokens than the
model was trained on -- inflating dataloading_s (patchify) and update_s (VLM
sequence) and skewing the input distribution. Checkpoints that pin their own
sizing are honored; the default constants are shared with GR00T_N1_7_DEFAULTS.
Net: preprocessing leaves the CPU critical path and the VLM sees the resolution
it was trained on -- faster training/inference and a correct train/serve
distribution. Affects inference too (shared preprocessor); existing checkpoints
still load (backward compatible) but must be retrained to gain the benefits.
* refactor(groot): N1.7 style cleanup (utils, imports, flash-attn, config)
Mechanical refactor of the GR00T N1.7 policy to match the repo's architecture and
style standards. No change to policy algorithm/numerics; only UX/CLI and packaging
changes. Tests are intentionally left untouched (out of scope) and need updating
for the removed `model_version` field.
Cleanup & consolidation:
- Add `groot/utils.py` holding the pure, side-effect-free helpers (JSON I/O, value
coercion, stat flattening, rot6d/SE3 math, language/batch prep) shared by the
config and processor layers.
- Remove dead code: the unused `resolve_groot_n1_7_backbone_model` cache-resolver
cluster, `GR00TN17Config.to_filtered_dict/json`, and the `_copy_default` wrapper.
Imports & execution guards:
- Hoist nested imports to module top; relative imports within the package, absolute
for external modules. The version-gated Qwen3-VL classes import under the single
`_transformers_available` guard (transformers is pinned >=5.4, which ships them).
- No import-time side effects: `_register_with_transformers()` now runs in
`GR00TN17.__init__` (idempotent via `register(exist_ok=True)`), and the N1.5 step
stubs register lazily before pipeline deserialization (idempotent via the
registry, no run-once globals).
- Gate optional deps at the point of use with `require_package(..., extra="groot")`.
Dependencies & docs:
- Drop `flash-attn` (and its build-only dep `ninja`) from the `groot` extra; default
to SDPA (numerically equivalent) with opt-in via `--policy.use_flash_attention`.
Un-comment `lerobot[groot]` in the `all` extra and regenerate `uv.lock`.
- Rewrite the `groot.mdx` install section: flash-attn is a purely optional,
user-managed optimization that LeRobot neither installs nor requires.
Config & CLI:
- Surface previously-frozen knobs on `GrootConfig` (plumbed into `GR00TN17Config`;
no-ops at their defaults): inference — `num_inference_timesteps`, `rtc_ramp_rate`,
`use_flash_attention`; fine-tuning — `tune_top_llm_layers` (partial-LLM tuning)
and `tune_vlln` (previously hardwired to True).
- Convert the single-valued `model_version` and `n1_7_backbone_model` fields to
internal constants.
- Keep `base_model_path`: it is NOT equivalent to `pretrained_path` (raw NVIDIA
checkpoints have no LeRobot `type` field and load only via `base_model_path`) and
is genuinely user-tunable.
- Keep the deprecated Isaac-GR00T/N1.5 fields (and the dead LoRA fields) as a
back-compat block so a v0.5.1 N1.5 `config.json` still parses under draccus and is
rejected with the friendly N1.5 removal message instead of an opaque decode error.
* Optimize GR00T N1.7 image preprocessing
* Remove PIL fallback from GR00T preprocessing
* Fix GROOT relative action training stats
* Address GROOT relative action review feedback
* Fix GROOT N1.7 relative action stats
* Fix GROOT relative action training stats
* Fix GROOT relative action padding and RTC leftovers
* Reset rollout state after robot episode end
* Revert "Reset rollout state after robot episode end"
This reverts commit 1322f45aec.
* Move GROOT relative stats out of train script
* Guard GR00T relative action stepwise decode
* Match GR00T N1.7 OSS preprocessing and relative actions
* Apply LIBERO action decode override after loading
* Format GR00T OSS parity changes
* chore(policies): add guards, warnings and comments + recover tests n1.5 check
* fix(style): pre-commit
* fix(ci): guard dependecy checks
* chore(groot): move cv2 to the top as its in the default install tag
* chore(policies): add explicit dataset dependecy to gr00t implementation
* fix(test): add guard
* fix(groot): make N1.7 letterbox opt-in
* feat(groot): activate checkpoint-configured N1.7 raw-state dropout during training
Isaac-GR00T applies dual state regularization during fine-tuning: raw-state
zeroing driven by the processor sidecar's state_dropout_prob (0.2 for the
inspected N1.7 checkpoint) plus encoded-feature dropout. Baseline LeRobot kept
the processor in deterministic mode, so the raw-state dropout never activated
(RCA Tier-2 contributor to the LeRobot-trained SO-101 failures).
- GrootN17PackInputsStep: runtime-only 'training' flag + state_dropout_prob;
whole-sample state zeroing gated on torch.is_grad_enabled() so eval and
no_grad validation paths are unaffected
- sidecar loader reads state_dropout_prob from processor_config.json
- state_dropout_prob serializes with the step; the training flag intentionally
does not (reloaded pipelines default to eval, re-enabled only when processors
are rebuilt with dataset_meta)
- _set_groot_preprocessor_training toggles any dataclass step exposing a
'training' field on serialized-pipeline reloads
Verification: tests/policies/groot/test_groot_state_dropout.py (4 passed) on
RTX PRO 6000 / CUDA 13.3.
* fix(groot): align N1.7 fine-tuning optimizer/scheduler/precision with Isaac-GR00T
Evidence from the LeRobot-vs-OSS checkpoint comparison: the LeRobot/HF 8k
checkpoint's DiT moved only ~19% as far from base as the OSS-trained one
(0.0547 vs 0.285 relative L2) - undertrained because the scheduler decayed over
a hardcoded 10k steps regardless of --steps, on top of beta1/clip mismatches.
- AdamW betas (0.95, 0.999) -> (0.9, 0.999) and grad_clip_norm 10.0 -> 1.0
(Isaac defaults)
- scheduler: hardcoded CosineDecayWithWarmup(10k decay, floor 10% peak) ->
DiffuserSchedulerConfig HF cosine with ceil(max_steps * warmup_ratio) warmup,
deriving num_training_steps from the outer --steps at runtime
- model_params_fp32 (default true): keep master weights in FP32 and compute
under BF16 autocast like the native N1.7 recipe (fixes optimizer-update
numerics vs pure-BF16 params)
- weight-decay grouping via transformers get_parameter_names: biases and norm
parameters excluded from decay
- restore the TF4 lm_head/embedding weight tie so the unused Qwen LM head stays
frozen and deduplicated in checkpoints
- action_mask kept in native dtype for the masked flow-matching loss
- drop_n_last_frames: exclude episode tails that cannot supply a complete
action chunk (Isaac sampler behavior)
Verification: tests/policies/groot/test_groot_training_optim_contract.py
(7 passed) + remaining groot suite 11 passed/5 skipped on RTX PRO 6000 /
CUDA 13.3. Note: tests/policies/groot/test_groot_n1_7.py does not collect on
the base branch (pre-existing ImportError, fixed in PR #37).
* feat(groot): train-time random crop for N1.7 (eval keeps center crop)
Isaac-GR00T crops a random crop_fraction window during training and the
deterministic center window at eval, replaying the sampled window across all
camera views of a sample. This contract is unchanged since the N1.5 release
(gr00t/data/transform/video.py: "If mode is 'train', return a random crop
transform. If mode is 'eval', return a center crop transform.") and mirrors
LeRobot's own Diffusion/VQBeT crop_is_random pattern. The LeRobot N1.7 port
used the eval center crop for training too, so the fine-tuned projector/DiT
never sees frame borders and trains on a single fixed appearance point.
Scope: crop geometry ONLY - no color jitter, no new dependencies. The random
window is plain numpy slicing inside the existing cv2 eval transform:
- _transform_n1_7_image_for_vlm_albumentations gains crop_position=(y, x)
fractions; None keeps the center crop byte-identical to before (verified
by test)
- GrootN17VLMEncodeStep gains a runtime-only 'training' flag (never
serialized; reloaded pipelines default to eval); training samples ONE
window per sample and reuses it across (timestep, view) frames - Isaac's
cross-view consistency
- gated on torch.is_grad_enabled() so no_grad validation and frozen-eval
paths are unaffected
- wired via dataset_meta is not None in make_groot_pre_post_processors and
the existing _set_groot_preprocessor_training on serialized reloads
Verification: tests/policies/groot/test_groot_train_random_crop.py (8 passed:
center-crop bit-exactness with crop_position=None, corner/center windows,
cross-view replay, train!=eval, no_grad gating, seed reproducibility,
serialization contract) + groot suite 23 passed / 5 skipped on RTX PRO 6000 /
CUDA 13.3.
* docs(groot): update Training & hardware Evaluation commands
Replace the multi-GPU accelerate-launch Training snippet with the current
single-command 'uv run lerobot-train' N1.7 recipe (relative actions excluding
gripper, bf16, flash attention, chunk/n_action_steps=16, bs64/20k steps).
Replace the bimanual 'Evaluate in your hardware setup' rollout example with the
SO-101 follower RTC 'uv run lerobot-rollout' command (strategy.type=base,
inference.type=rtc, wrist+front cameras, place-the-vial task).
Docs-only; no source/test changes.
* docs(groot): parameterize commands with env vars + fill LIBERO results
- Introduce BASE_MODEL / DATASET_ID / REPO_ID / JOB_NAME / OUTPUT_DIR env vars
in the training command and reuse OUTPUT_DIR + BASE_MODEL in the rollout cmd.
- Fill the LIBERO benchmark table with GR00T-LeRobot success rates
(Spatial 94%, Object 98%, Goal 93%, LIBERO 10/Long 90%; avg 93.75%),
drop the OSS column and XX placeholders. LeRobot-focused.
* docs(groot): drop export block, reference env vars directly
Use $DATASET_ID / $BASE_MODEL / $REPO_ID / $OUTPUT_DIR / $JOB_NAME as
bare placeholders in the commands without concrete export assignments.
* docs(groot): keep BASE_MODEL export in training command
* docs(groot): use literal HF repo IDs for dataset/policy repo_id
Public-facing Hub references (--dataset.repo_id, --policy.repo_id) shown as
concrete IDs; local-only values ($OUTPUT_DIR, $JOB_NAME) stay as placeholders.
* docs(groot): add LIBERO training command example
* docs(groot): remove LIBERO checkpoints subdirectory section
* docs(groot): use $BASE_MODEL for base_model_path in LIBERO eval
* docs(groot): drop hf download step from LIBERO eval, fix intro
* docs(groot): restore suite checkpoint download intro sentence
* docs(groot): remove checkpoint download note above LIBERO eval
* docs(groot): update training and rollout commands with new parameters and dependencies
* Add sample so101 training command
* Remove sample so101 training command
* docs(groot): remove optional Flash Attention setup instructions and update base model path for evaluation
* docs(groot): update training command with image transformation parameters
* docs(groot): add note on inference.queue_threshold value for stable inference
* chore(style): pre-commit gr00t
* docs(groot): update
* chore(policies): minor details
* fix(groot): license headers + test guards
* chore(policies): fix tests
* docs(groot): relative actions param doc
* chore(policy): address some of the AI review items
---------
Co-authored-by: Andrew Wrenn <awrenn@nvidia.com>
Co-authored-by: Ryan Halabi <ryhalabi@nvidia.com>
Co-authored-by: nv-sachdevkartik <ksachdev@nvidia.com>
Co-authored-by: groot-validation <groot-validation@localhost>
Co-authored-by: johnnynunez <johnnynuca14@gmail.com>
Co-authored-by: lbenhorin <lbenhorin@nvidia.com>
* feat(policies): add LingBot-VA autoregressive video-action world model
Port the LingBot-VA policy (Wan2.2 dual-stream video+action world model) into
LeRobot, following the EO-1 / VLA-JEPA conventions. Covers inference, checkpoint
conversion, and predicted-video saving (training is deferred to a follow-up PR).
- Vendored Wan transformer/attention/flex/VAE/scheduler modules (key names preserved
for near-identity conversion); torch SDPA default, flashattn/flex lazy-guarded.
- LingBotVAConfig (registered "lingbot_va") + processor with fixed-quantile action
unnormalization; full dual-stream sampling loop with CFG, two flow-matching
schedulers and KV cache, mapped onto select_action with observed-keyframe feedback.
- convert_lingbot_va_checkpoints.py (libero/robotwin variants): bundles the ~5B
transformer, lazy-pulls the frozen VAE+UMT5 from the source repo.
- Predicted-video plumbing in lerobot_eval (predicted_frames_callback; opt-in via
--policy.save_predicted_video) and ConstantWithWarmupSchedulerConfig.
- pyproject: widen diffusers-dep to <0.37, add lingbot_va + imageio-dep extras,
add lingbot_va and (missing) eo1 to `all`.
- Factory + policies/__init__ wiring, docs page + toctree, and tests.
Note: the LIBERO success-rate correctness gate must be validated on a CUDA GPU
with the converted checkpoint.
* feat(lingbot_va): RoboTwin eef-pose eval, single-file model, Hub checkpoints
Make the LingBot-VA port runnable on both LIBERO and RoboTwin and clean up the
package to LeRobot conventions.
- Consolidate all vendored Wan2.2 model code (transformer, attention, VAE helpers,
flow-matching scheduler, grid utils, flex-attention) into a single
modeling_lingbot_va.py; remove the separate wan_*/schedulers modules.
- Move the fixed action (un)normalization quantiles out of the config and into the
post-processor (LIBERO 7-DoF + RoboTwin 16-d eef); remove the conversion script in
favour of ready-to-use LeRobot-format checkpoints on the Hub.
- Fixes found via on-sim validation: undo LIBERO's 180-degree image flip
(image_hflip), encode obs as a multi-frame streaming-VAE clip, reset the streaming
VAE cache between episodes, run the transformer in config.dtype, lazy-load frozen
VAE/UMT5 by subfolder with the text encoder on CPU.
- RoboTwin: add an end-effector-pose action mode to RoboTwinEnv (16-d per-arm
xyz+quat+gripper deltas composed onto the initial eef pose, executed via CuRobo IK)
and the robotwin_tshape latent layout (full-res head + half-res wrists via a second
streaming VAE) with the upstream RoboTwin action quantiles + camera mapping.
- Predicted-video saving works for both benchmarks; docs + tests updated.
* feat(lingbot_va): implement training / fine-tuning (flow-matching loss)
- Implement LingBotVAPolicy.forward(): dual-stream flow-matching training loss
(latent + action, timestep-weighted, action-masked) ported from upstream train.py;
VAE-encodes camera clips, UMT5-encodes the task, noises both streams, runs the
block-causal flex-attention training pass (forward_train).
- training_loss_from_streams() core + _build_training_streams() data prep (action
scatter into the 30-d space, multi-frame VAE encode incl. robotwin_tshape).
- get_optim_params returns only trainable transformer params (LoRA/PEFT friendly);
VAE/UMT5 stay frozen. Training needs attn_mode='flex'.
- Add a tiny-config single-training-step test (forward->loss->backward->AdamW) and a
Training/fine-tuning section in the docs.
* fix(lingbot_va): CI quality gate + fast-test collection
- Add tests/policies/lingbot_va/__init__.py so the test files don't clash by basename
with tests/policies/vla_jepa/* under pytest's default import mode (fast-test collection error).
- Fix vendored typos flagged by the typos hook (pach_scale->patch_scale, total_tolen->
total_token_len, stablized->stabilized) and a mypy union-attr in RoboTwinEnv._read_eef_pose.
- Apply Prettier formatting to docs/source/lingbot_va.mdx.
* docs(lingbot_va): document EEF action-channel schema + camera order
* Update lingbot_va.mdx
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
* Update pyproject.toml
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
* Update pyproject.toml
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
* refactor(lingbot_va): drop hardcoded action quantiles; source from checkpoint
The LIBERO/RoboTwin action (un)normalization quantiles were hardcoded as module
constants in processor_lingbot_va.py. They are already serialized into each
checkpoint's policy_postprocessor.json (via LingBotVAActionUnnormalizeStep.get_config)
and restored on load by PolicyProcessorPipeline.from_pretrained, so the constants are
dead at eval/load time for the released checkpoints (verified: libero_long/robotwin/base
all carry their quantiles on the Hub).
- Remove LIBERO_ACTION_Q01/Q99, ROBOTWIN_ACTION_Q01/Q99 and _default_action_quantiles.
- make_lingbot_va_pre_post_processors now defaults a fresh (unconverted) build to a
neutral [-1, 1] mapping (identity rescale); real per-benchmark stats come from the
saved checkpoint (or postprocessor_overrides), analogous to dataset-stats normalization.
- Update the config doc comment to point at the checkpoint as the source of truth.
- Tests: replace the LIBERO-default assertion with a neutral-default check, and add a
save_pretrained/from_pretrained round-trip guard for the quantile serialization.
* docs(lingbot_va): trim verbose comments
- configuration_lingbot_va.py: condense multi-line field comments to one-liners
(keep the ── section headers).
- processor_lingbot_va.py: shorten the action-quantile explanation block.
- modeling_lingbot_va.py: drop the bare "# ----" separator rules, keeping the
one-line section headers.
No code changes.
* docs(lingbot_va): trim provenance comments; default wan path to base repo
- configuration_lingbot_va.py: drop the "──" decorations and the
"(from transformer/config.json)" note; default wan_pretrained_path to
robbyant/lingbot-va-base (has the frozen vae/text_encoder/tokenizer subfolders).
- modeling_lingbot_va.py: remove the vendored-code banner and the
"(upstream wan_va/...)" section-header provenance/dash decorations; condense the
transformer-dtype comment to one line.
No code changes.
* refactor(lingbot_va): use built-in UnnormalizerProcessorStep for actions
Replace the bespoke LingBotVAActionUnnormalizeStep with the standard
UnnormalizerProcessorStep in QUANTILES mode, which computes the identical
(action + 1) / 2 * (q99 - q01) + q01 mapping. The per-channel q01/q99 are stored
as the step's saved state (a safetensors file) and restored on load; a fresh build
has no action stats so the step is an identity passthrough.
The 3 Hub checkpoints (lerobot/lingbot_va_{libero_long,robotwin,base}) have been
re-uploaded with the new post-processor (policy_postprocessor.json +
*_unnormalizer_processor.safetensors); reloading from the Hub round-trips q01/q99.
- processor_lingbot_va.py: drop the custom step + registry; build the post-processor
with UnnormalizerProcessorStep (explicit ACTION->QUANTILES norm_map so the
preprocessor / training path is unchanged).
- tests: assert the built-in step is used, identity-when-no-stats, correct quantile
unnormalization, and a save_pretrained/from_pretrained stats round-trip.
* docs(lingbot_va): point checkpoint paths at the lerobot org
The LeRobot-format checkpoints moved from pepijn223/* to lerobot/* (libero_long,
robotwin, base). Update the eval/train --policy.path examples accordingly.
* docs(lingbot_va): condense processor normalization comments
* fix(lingbot-va): align RoboTwin evaluation (#3784)
Thank you for the RoboTwin fix, and alignment!
* applying fixes
* updating uv lock and linting
* adjusting test to match expected values
* cleaning up deps
* cleaning up top level imports, styling, and deps guards
* cleanup
* moving wan utils and loading utils to `utils.py`
* removing ftfy by replicating the prompt_clean function without it (we don't expect to have weird chars given in the prompt anyway)
* removing unused function
* guarding for scipy dep, renaming test to avoid collision
* adding back accelerate for peak memory usage optim + justifying robotwin description dep
---------
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: pepijn223 <pepijn223@hf.co>
Co-authored-by: Gangwei XU <gwxu@hust.edu.cn>
Co-authored-by: Maxime Ellerbach <maxime.ellerbach@huggingface.co>
* fix(train): drive Accelerate mixed precision from policy.dtype
`accelerator.autocast()` was always a no-op because `mixed_precision`
was never set, so `--policy.dtype=bfloat16` only cast the model params
(via the policy) while autocast-eligible ops still ran in fp32/tf32.
Map the active policy's `dtype` onto Accelerate's `mixed_precision`
(bfloat16 -> bf16, float16 -> fp16, float32 -> no) so autocast is active
for bf16/fp16 and stays full precision for float32. Policies without a
string `dtype` field fall back to Accelerate's launcher default, so
existing behavior is preserved.
* style(train): condense mixed-precision comment to one line
Drop LOC_SUPPRESS_KINDS. With interactive VQA gone, every runtime text
kind (subtask / memory / interjection) is prose that must never emit
PaliGemma <loc> tokens, so suppress unconditionally. No behavior change:
the only non-suppressed kind (plan) is never generated by the runtime.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make the policy adapter architecturally clean and set up a single general
entry point for any language-conditioned policy.
Adapter architecture (Template Method):
- New lerobot/runtime/adapter.py: BaseLanguageAdapter owns the generic
control loop (throttle → generate → gibberish/empty reject → subtask→memory
cascade → diagnostics) and plan_from_text/handle_interjection. A policy
supplies only select_action + generate_text + build_messages. The
subtask→memory cascade is an overridable hook (_regenerate_context).
- GenerationConfig (typed, constructor-time) replaces config smuggled through
RuntimeState.extra (temperature/top_p/min_new_tokens/chunks_per_regen).
- LanguageDiagnostics (typed, keyed by kind) replaces ~8 loose state.extra
counter keys; the panel reads it via the adapter.
- looks_like_gibberish + split_plan_and_say move to runtime (generic).
Contract:
- LanguageConditionedPolicyAdapter protocol now states the true contract
(select_action, update_language_state, handle_interjection); the runtime
drops both getattr fallbacks.
- PI052PolicyAdapter shrinks to just its primitives (132 → ~half).
General entry point:
- lerobot/runtime/registry.py maps policy type → adapter (lazy import).
- run() resolves the adapter from the registry by policy type and defaults
the panel label to it, so one CLI serves every policy.
- Rename lerobot-pi052-runtime → lerobot-language-runtime (general script);
a new policy just registers its adapter, no new script.
Tests: new tests/runtime/test_adapter.py covers throttle/reject/cascade/
interjection; adapter + runtime + CLI-smoke tests updated for the new shape.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Set up the runtime so a second language-conditioned policy reuses the
CLI/REPL/UI instead of copying pi052's. The tick loop, REPL, panel, and
interactive CLI are now policy-independent in lerobot/runtime/; a policy
plugs in only a LanguageConditionedPolicyAdapter.
- Move repl.py, ui.py, and runtime_cli.py (-> cli.py) from
pi052/inference/ into lerobot/runtime/. Generalize labels/titles
(panel_label param, [runtime] prefixes).
- lerobot.runtime.cli.run(argv, *, adapter_factory, panel_label, prog)
is the shared entry; policy loading already dispatches generically via
the factory on cfg.type.
- lerobot-pi052-runtime is now a thin entry (scripts/lerobot_pi052_runtime.py)
that passes PI052PolicyAdapter into run(). pi052/inference/ keeps only
the adapter.
- Drop PI052Runtime back-compat wrapper (no consumers).
- Drop VQA visualization: delete inference/vqa.py + test_pi052_vqa_loc.py,
remove answer_vqa/VQAResult from the Protocol + adapter, and the
/question command + overlay paths from the CLI/REPL.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(depth unit): storing raw depth units in the dataset metadata for correct depth statistics and depth raw frames handling. The unit is stored as a string ("m","mm") under "depth_unit" at the same level as "is_depth_map". Unit is inferred from the depth frame type.
* feat(raw frame unit): adapting dataset reader so that raw depth frames are scaled according to the requested unit
* feat(stats units): rescaling stats when loading a dataset so that the stats are given in the requested unit
* tests(unit): adapting and extending depth tests to units manipulations
* chore(format): formating code
* feat(warning): adding a warning when depth unit is not specified in the dataset
* chore(infer_depth_unit): moving the depth unit inference utility in a more accessible location
* feat(rerun unit): adding correct depth unit display for rerun (foxglove does not support units yet)
* feat(unit getter): adding a proper output_depth_unit getter to LeRobotDataset for cleaner integration
* fix(streaming dataset): extending support for depth units to streaming datasets
* test(rerun): fixing rerun tests
* Add Foxglove display mode for teleoperate
Add a --display_mode flag (rerun|foxglove) to lerobot-teleoperate. When set
to foxglove, stream observations/actions over a Foxglove WebSocket server:
images as RawImage/CompressedImage, scalars as typed JSON channels with
schemas generated from the feature names (sanitized so paths don't need
quoting). Adds a `foxglove` extra.
* Add Foxglove display mode to lerobot-record
Wire the --display_mode flag (rerun|foxglove) into lerobot-record, matching
lerobot-teleoperate: route init/log through the backend-agnostic dispatchers
and stop the visualization backend on exit.
* update foxglove-sdk to 0.25.1
* Use static lerobot.Scalars schema for Foxglove state topics
Replace the per-topic JSON schema derived from feature names with a single
static lerobot.Scalars schema: a scalars array of {label, value} objects. The
same schema fits any robot regardless of which observation/action features it
reports, and the label field lets Foxglove name each series automatically so
one filtered path plots every feature.
* add foxglove option to dataset viz
* Make Foxglove dataset playback loop the sole frame emitter
Address review: the listener no longer emits frames, it only mutates
playback state and queues a one-shot seek index that the playback loop
services. The loop is now the only caller of emit_frame, so concurrent
random access into the on-disk dataset / video decoder never overlaps.
Also remove the dead server_holder and tighten the _foxglove_safe_name
docstring to state what it does and why.
* Label Foxglove dataset scalars with feature dimension names
Use the dataset's per-dimension feature names (e.g. joint names) as the
Foxglove series labels for /observation/state and /action/state instead
of bare indices. LeRobot stores `names` inconsistently (flat list,
{category: [...]}, or {name: index}), so _feature_dim_names handles each
and falls back to indices on any unknown format or length mismatch.
* Make Foxglove server host bindable and refactor topic/channel handling
Pass display_ip through as the Foxglove WebSocket bind host (127.0.0.1
for local only, 0.0.0.0 for all interfaces) instead of always binding
locally. In lerobot-dataset-viz, fold the separate --port into --web-port
so one flag covers both the Rerun web viewer and the Foxglove server port.
Add a _foxglove_topic() helper and thread a per-topic channel cache
through the log helpers so dataset playback stays self-contained instead
of mutating the module-global cache. Promote SUCCESS to constants.py.
* feat(viz): add support for foxglove in rollout + add to viz tag
* fix(docs): remove misleading installation note
* fix(visualization): no duplicated prefix, consolidated norm + warnings log
* chore(viz): minor improvements
* refactor(viz): split files + autoplay + updated docs + added minimal tests
* fix(viz): right tags + warning
* feat(deprecated ws-port): removing rerun's depreacted ws-port parameter in dataset visualization
* chore(web ports): adding global variables for default foxglove/rerun web ports
* feat(depth): adding depth support to foxglove visualizer. Because of foxglove limitations (min and max values on RawImage cannot be set from the SDK), depth is normalized between [0,1] when a depth range is provided.
* fix(rerun depth range): making rerun depth range computation safe against missing stats
* chore(foxglove depth): make it simple, and make it work.
* fix(scaling): fixing depth frames scaling
---------
Co-authored-by: Roman Shtylman <roman@foxglove.dev>
Co-authored-by: Caroline Pascal <caroline8.pascal@gmail.com>
draccus re-parses each config class's source on every parse() to extract
field help text (~2.5s for TrainPipelineConfig). Memoize it for the test
session; the source is constant within a run.
Fast Tests test time: 664s -> 404s (-39%).
* Add FastWAM policy
* Add FastWAM policy review updates
* big refactor to use models from diffusers and transformers
* changing reproducable results
* preparing for training adding some temporary debug code aswell to visualize model output
* re-parenting of some layers to enable proper zero-3 FSDP
* linting
* small fix for the preprocessor and padded images
* removing some preprocessors
* removing temporary debug code
* cleaning up
* updating uv lock after rebasing
* adding lazy imports
* linting
* fixing stale assertion
* make tokenizer/text-encoder model ids configurable + some nits
* moving and renaming files to have a cleaner file tree
* removed asserts from the model, added guard instead and completely removed useless asserts
* cleaning up imports
* removing is_main_process and custom logging logic
* removing unused / stale attention path, removing some of the stale forwards within wan/models
---------
Co-authored-by: ZibinDong <zibindong@outlook.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Pepijn