mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-29 12:39:41 +00:00
Compare commits
20 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Martino Russi
|
4787cfc7ee | ||
|
Steven Palma
|
a9879e69ed | ||
|
Steven Palma
|
9d82bb9871 | ||
|
Steven Palma
|
c5371d0691 | ||
|
Steven Palma
|
b2c062c0f4 | ||
|
Maxime Ellerbach
|
051b13573e | ||
|
Pepijn
|
7de2e4c1ef | ||
|
Nikodem Bartnik
|
8db50611c2 | ||
|
Maxime Ellerbach
|
92f96f33b3 | ||
|
Steven Palma
|
d4b3ca569c | ||
|
Steven Palma
|
3f2179f3b6 | ||
|
Nikodem Bartnik
|
867b58cfb2 | ||
|
Pepijn
|
279c6c7af3 | ||
|
Lior Ben Horin
|
e40b58a8df | ||
|
Mishig
|
3e538352ca | ||
|
Steven Palma
|
8a74e0ac6d | ||
|
Steven Palma
|
30da8e687a | ||
|
Steven Palma
|
93257e3468 | ||
|
Caroline Pascal
|
b895ed0fe4 | ||
|
Caroline Pascal
|
293a8d9a77 |
@@ -55,7 +55,7 @@ jobs:
|
||||
github.repository == 'huggingface/lerobot'
|
||||
permissions:
|
||||
contents: read
|
||||
uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@2430c1ec91d04667414e2fa31ecfc36c153ea391 # main
|
||||
uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@e60a538eea9817ab312196d0d233604b01697265 # main
|
||||
with:
|
||||
commit_sha: ${{ github.sha }}
|
||||
package: lerobot
|
||||
@@ -78,7 +78,7 @@ jobs:
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@2430c1ec91d04667414e2fa31ecfc36c153ea391 # main
|
||||
uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@e60a538eea9817ab312196d0d233604b01697265 # main
|
||||
with:
|
||||
commit_sha: ${{ github.event.pull_request.head.sha }}
|
||||
pr_number: ${{ github.event.number }}
|
||||
|
||||
@@ -169,6 +169,8 @@
|
||||
- sections:
|
||||
- local: phone_teleop
|
||||
title: Phone
|
||||
- local: isaac_teleop
|
||||
title: Isaac Teleop
|
||||
title: "Teleoperators"
|
||||
- sections:
|
||||
- local: cameras
|
||||
|
||||
@@ -81,6 +81,12 @@ merged. Both prompts also carry a causal **event-boundary** definition (a
|
||||
new event starts when an object becomes held / is released / reaches a new
|
||||
location / a lid changes state / contents move) to sharpen where cuts land.
|
||||
|
||||
Optionally, a third **seeded-relabel** pass (`--plan.subtask_seeded_relabel`)
|
||||
revisits each span with its previous/current/next segment contact sheets and
|
||||
minimally corrects the label, using the first label as a prior — it keeps the
|
||||
boundaries fixed and only sharpens wording, at the cost of one extra call per
|
||||
subtask.
|
||||
|
||||
The resulting spans are then stitched into a gap-free, full-episode
|
||||
cover, so **every frame has exactly one active subtask**. See
|
||||
[`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py)
|
||||
@@ -157,30 +163,33 @@ Every module is on by default and can be toggled independently (set to
|
||||
|
||||
### The VLM (`--vlm.*`)
|
||||
|
||||
| Flag | Default | What it does |
|
||||
| -------------------------- | ------------------ | ----------------------------------------------------------------------------------- |
|
||||
| `--vlm.model_id` | `Qwen/Qwen3.6-27B` | The model to serve and prompt. |
|
||||
| `--vlm.camera_key` | first `images.*` | Which camera every prompt is grounded on. |
|
||||
| `--vlm.serve_command` | auto | The exact `vllm serve …` command (set TP size, GPU memory, `--max-model-len` here). |
|
||||
| `--vlm.parallel_servers` | `1` | Independent servers for round-robin routing (one per GPU). |
|
||||
| `--vlm.num_gpus` | `0` | GPUs per server (`0` = one each). |
|
||||
| `--vlm.client_concurrency` | `16` | In-flight requests across all servers. |
|
||||
| `--vlm.max_new_tokens` | `512` | Generation cap per call. |
|
||||
| `--vlm.temperature` | `0.2` | Sampling temperature. |
|
||||
| Flag | Default | What it does |
|
||||
| -------------------------- | ------------------ | ------------------------------------------------------------------------------------ |
|
||||
| `--vlm.model_id` | `Qwen/Qwen3.6-27B` | The model to serve and prompt. |
|
||||
| `--vlm.camera_key` | first `images.*` | Which camera every prompt is grounded on. |
|
||||
| `--vlm.serve_command` | auto | The exact `vllm serve …` command (set TP size, GPU memory, `--max-model-len` here). |
|
||||
| `--vlm.parallel_servers` | `1` | Independent servers for round-robin routing (one per GPU). |
|
||||
| `--vlm.num_gpus` | `0` | GPUs per server (`0` = one each). |
|
||||
| `--vlm.client_concurrency` | `16` | In-flight requests across all servers. |
|
||||
| `--vlm.max_new_tokens` | `512` | Generation cap per call. |
|
||||
| `--vlm.temperature` | `0.2` | Sampling temperature. |
|
||||
| `--vlm.reasoning_effort` | `null` | Thinking-budget hint (`low`/`medium`/`high`) forwarded to OpenAI-compatible servers. |
|
||||
|
||||
### Subtasks / plan / memory (`--plan.*`)
|
||||
|
||||
| Flag | Default | What it does |
|
||||
| ------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `--plan.frames_per_second` | `2.0` | Frame sampling rate for the contact sheets (`2.0` = one frame every 0.5s). |
|
||||
| `--plan.max_frames_per_prompt` | `60` | Frame budget per VLM call. Episodes whose sampling exceeds this are auto-windowed at the same density, then stitched. |
|
||||
| `--plan.contact_sheet_columns` | `5` | Columns per contact-sheet grid (`contact_sheet_frames_per_sheet` tiles, time row-major). |
|
||||
| `--plan.plan_max_steps` | `8` | Upper bound on subtasks per episode. |
|
||||
| `--plan.subtask_describe_first` | `true` | Run the describe→segment grounding pass (best subtask quality; +1 call/episode). |
|
||||
| `--plan.emit_plan` | `true` | Emit the numbered `plan` rows (`false` = subtasks + memory only). |
|
||||
| `--plan.emit_memory` | `true` | Emit the `memory` rows (`false` = subtasks + plan only); symmetric to `emit_plan`. |
|
||||
| `--plan.n_task_rephrasings` | `10` | How many `task_aug` rephrasings to emit (`0` disables). |
|
||||
| `--plan.derive_task_from_video` | `if_short` | Use the dataset task as-is (`off`), only when it's missing/short (`if_short`), or always re-derive from video (`always`). |
|
||||
| Flag | Default | What it does |
|
||||
| ------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `--plan.frames_per_second` | `2.0` | Frame sampling rate for the contact sheets (`2.0` = one frame every 0.5s). |
|
||||
| `--plan.max_frames_per_prompt` | `60` | Frame budget per VLM call. Episodes whose sampling exceeds this are auto-windowed at the same density, then stitched. |
|
||||
| `--plan.contact_sheet_columns` | `5` | Columns per contact-sheet grid (`contact_sheet_frames_per_sheet` tiles, time row-major). |
|
||||
| `--plan.plan_max_steps` | `8` | Upper bound on subtasks per episode. |
|
||||
| `--plan.subtask_describe_first` | `true` | Run the describe→segment grounding pass (best subtask quality; +1 call/episode). |
|
||||
| `--plan.subtask_seeded_relabel` | `false` | Second pass: re-label each subtask from its prev/current/next contact sheets, seeded with the first label (+1 call/subtask). |
|
||||
| `--plan.subtask_relabel_frames` | `5` | Frames sampled uniformly per segment sheet in the relabel pass (only used when `subtask_seeded_relabel=true`). |
|
||||
| `--plan.emit_plan` | `true` | Emit the numbered `plan` rows (`false` = subtasks + memory only). |
|
||||
| `--plan.emit_memory` | `true` | Emit the `memory` rows (`false` = subtasks + plan only); symmetric to `emit_plan`. |
|
||||
| `--plan.n_task_rephrasings` | `10` | How many `task_aug` rephrasings to emit (`0` disables). |
|
||||
| `--plan.derive_task_from_video` | `if_short` | Use the dataset task as-is (`off`), only when it's missing/short (`if_short`), or always re-derive from video (`always`). |
|
||||
|
||||
### Interjections + VQA
|
||||
|
||||
|
||||
@@ -150,14 +150,14 @@ class MyPolicy(PreTrainedPolicy):
|
||||
|
||||
The methods called by the train/eval loops:
|
||||
|
||||
| Method | Used by | What it does |
|
||||
| ----------------------------------------------------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `reset() -> None` | `lerobot-eval` | Clear per-episode state at the start of each episode. |
|
||||
| `select_action(batch, **kwargs) -> Tensor` | `lerobot-eval` | Return the next action `(B, action_dim)`. Called every step. |
|
||||
| `predict_action_chunk(batch, **kwargs) -> Tensor` | the policy itself | Return an action chunk `(B, chunk_size, action_dim)`. Currently abstract on the base class — raise `NotImplementedError` if your policy doesn't chunk. |
|
||||
| `forward(batch, reduction="mean") -> tuple[Tensor, dict \| None]` | `lerobot-train` | Return `(loss, output_dict)`. Accept `reduction="none"` if you want to support per-sample weighting. |
|
||||
| `get_optim_params() -> dict` | the optimizer | Return `self.parameters()` for simple policies; return a named parameter dict for [multi-optimizer policies](https://github.com/huggingface/lerobot/blob/ecd38c50d7d15b4184cf42649ff1185ee2e11eeb/src/lerobot/policies/sac/modeling_sac.py#L61-L73). |
|
||||
| `update() -> None` _(optional)_ | `lerobot-train` | Called after each optimizer step _if defined_. Use for EMA, target nets, replay buffers (TDMPC uses this). |
|
||||
| Method | Used by | What it does |
|
||||
| ----------------------------------------------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `reset() -> None` | `lerobot-eval` | Clear per-episode state at the start of each episode. |
|
||||
| `select_action(batch, **kwargs) -> Tensor` | `lerobot-eval` | Return the next action `(B, action_dim)`. Called every step. |
|
||||
| `predict_action_chunk(batch, **kwargs) -> Tensor` | the policy itself | Return an action chunk `(B, chunk_size, action_dim)`. Currently abstract on the base class — raise `NotImplementedError` if your policy doesn't chunk. |
|
||||
| `forward(batch, reduction="mean") -> tuple[Tensor, dict \| None]` | `lerobot-train` | Return `(loss, output_dict)`. Accept `reduction="none"` if you want to support per-sample weighting. |
|
||||
| `get_optim_params() -> dict` | the optimizer | Return `self.parameters()` for simple policies; return a named parameter dict for multi-optimizer policies (see `get_optim_params` in [`modeling_act.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/act/modeling_act.py) for a per-group learning-rate example). |
|
||||
| `update() -> None` _(optional)_ | `lerobot-train` | Called after each optimizer step _if defined_. Use for EMA, target nets, replay buffers (TDMPC uses this). |
|
||||
|
||||
Batches are flat dictionaries keyed by the constants in [`lerobot.utils.constants`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/utils/constants.py): `OBS_STATE` (`observation.state.<motor>`), `OBS_IMAGES` (`observation.images.<camera>`), `OBS_LANGUAGE`, `ACTION`, etc. Reuse the constants — don't invent new prefixes.
|
||||
|
||||
@@ -295,12 +295,10 @@ The file names are load-bearing: the factory does lazy imports by name, and the
|
||||
|
||||
### Wiring
|
||||
|
||||
Four places need to know about your policy. All by name.
|
||||
Two places need to know about your policy. All by name.
|
||||
|
||||
1. **`policies/__init__.py`** — re-export `MyPolicyConfig` and add it to `__all__`. **Don't** re-export the modeling class; it loads lazily through the factory (so `import lerobot` stays fast).
|
||||
2. **`factory.py:get_policy_class`** — add a branch returning `MyPolicy` from a lazy import.
|
||||
3. **`factory.py:make_policy_config`** and **`factory.py:make_pre_post_processors`** — same idea, two more branches.
|
||||
4. **`templates/lerobot_modelcard_template.md` and the root `README.md`** — the template is what `push_model_to_hub` renders into the model card of every checkpoint trained with your policy: add a one-line description of your policy in the `model_name` branches, map it in `policy_docs` so cards link to your MDX guide, and optionally add an architecture image to `diagrams`. Then add your policy to the models table in the root `README.md`, under the right category, linking to your doc page.
|
||||
1. **`policies/__init__.py`** — re-export `MyPolicyConfig` and add it to `__all__`. This import is what registers your policy: `@PreTrainedConfig.register_subclass("my_policy")` runs, and from then on the factory resolves everything by convention. **Don't** re-export the modeling class; it loads lazily through the factory (so `import lerobot` stays fast).
|
||||
2. **`templates/lerobot_modelcard_template.md` and the root `README.md`** — the template is what `push_model_to_hub` renders into the model card of every checkpoint trained with your policy: add a one-line description of your policy in the `model_name` branches, map it in `policy_docs` so cards link to your MDX guide, and optionally add an architecture image to `diagrams`. Then add your policy to the models table in the root `README.md`, under the right category, linking to your doc page.
|
||||
|
||||
Mirror an existing policy that's structurally similar to yours; the diff is small.
|
||||
|
||||
@@ -332,6 +330,10 @@ This way:
|
||||
|
||||
Add a matching extra to [`pyproject.toml`](https://github.com/huggingface/lerobot/blob/main/pyproject.toml) `[project.optional-dependencies]` and include it in the `all` extra so `pip install 'lerobot[all]'` keeps installing everything.
|
||||
|
||||
### Avoid copying a modeling file — subclass it
|
||||
|
||||
If your policy needs to modify a backbone that already exists in `transformers` (custom conditioning, extra inputs, a swapped sub-module), **do not vendor a copy of its `modeling_*.py`**. Instead, subclass the smallest upstream unit and override only what changes. [`pi_gemma.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/pi_gemma.py) is the canonical reference: it injects AdaRMS conditioning into PaliGemma/Gemma in ~370 lines by subclassing `GemmaModel`/`PaliGemmaModel` and overriding the decoder-layer forward, instead of forking the ~2,000-line modeling file. Model surgery on a _loaded_ native model is also fine (layer truncation, tokenizer expansion, hidden-state capture — see `evo1/internvl3_embedder.py`, `eo1/modeling_eo1.py`, `groot/groot_n1_7.py` for working examples). Reviewers will ask for this pattern when a PR arrives with a copied modeling file; the only accepted exception is a model that does not exist in `transformers` at all.
|
||||
|
||||
### Benchmarks and a published checkpoint
|
||||
|
||||
A new policy is much easier to review — and far more useful — when it ships with a working checkpoint and at least one number you can reproduce.
|
||||
@@ -367,7 +369,7 @@ If your policy is real-robot-only and no sim benchmark applies, swap the sim eva
|
||||
The general expectations are in [`CONTRIBUTING.md`](https://github.com/huggingface/lerobot/blob/main/CONTRIBUTING.md) and the [PR template](https://github.com/huggingface/lerobot/blob/main/.github/PULL_REQUEST_TEMPLATE.md). On top of those, reviewers will look for:
|
||||
|
||||
- [ ] `MyPolicy` and `MyPolicyConfig` cover the surface above; `__init_subclass__` accepts the class.
|
||||
- [ ] `factory.py` and `policies/__init__.py` are wired (lazy imports for modeling).
|
||||
- [ ] `policies/__init__.py` re-exports the config (this registers the policy; the factory resolves modeling/processor by naming convention).
|
||||
- [ ] `make_my_policy_pre_post_processors` follows the naming convention.
|
||||
- [ ] Optional deps live behind a `[project.optional-dependencies]` extra and the `TYPE_CHECKING + require_package` guard.
|
||||
- [ ] `tests/policies/` updated; backward-compat artifact committed & policy-specific tests.
|
||||
|
||||
@@ -162,11 +162,11 @@ Preliminary LeRobot integration results (GR00T-LeRobot, `eval.n_episodes >= 50`
|
||||
|
||||
| Suite | Success rate | Checkpoint |
|
||||
| ---------------- | -----------: | ------------------------------------------------------------------------------------------------------------- |
|
||||
| LIBERO Spatial | 91% | [nvidia/gr00t17-lerobot-libero_spatial-640](https://huggingface.co/nvidia/gr00t17-lerobot-libero_spatial-640) |
|
||||
| LIBERO Object | 81% | [nvidia/gr00t17-lerobot-libero_object-640](https://huggingface.co/nvidia/gr00t17-lerobot-libero_object-640) |
|
||||
| LIBERO Goal | 97% | [nvidia/gr00t17-lerobot-libero_goal-640](https://huggingface.co/nvidia/gr00t17-lerobot-libero_goal-640) |
|
||||
| LIBERO 10 (Long) | 84% | [nvidia/gr00t17-lerobot-libero_10-640](https://huggingface.co/nvidia/gr00t17-lerobot-libero_10-640) |
|
||||
| **Average** | **88.25%** | |
|
||||
| LIBERO Spatial | 95% | [nvidia/gr00t17-lerobot-libero_spatial-640](https://huggingface.co/nvidia/gr00t17-lerobot-libero_spatial-640) |
|
||||
| LIBERO Object | 100% | [nvidia/gr00t17-lerobot-libero_object-640](https://huggingface.co/nvidia/gr00t17-lerobot-libero_object-640) |
|
||||
| LIBERO Goal | 98% | [nvidia/gr00t17-lerobot-libero_goal-640](https://huggingface.co/nvidia/gr00t17-lerobot-libero_goal-640) |
|
||||
| LIBERO 10 (Long) | 93% | [nvidia/gr00t17-lerobot-libero_10-640](https://huggingface.co/nvidia/gr00t17-lerobot-libero_10-640) |
|
||||
| **Average** | **96.5%** | |
|
||||
|
||||
```bash
|
||||
export MODEL_ID=your_trained_model_on_huggingface
|
||||
|
||||
@@ -117,14 +117,6 @@ middle_dip | 1484 | 1500 | 1547
|
||||
|
||||
Once calibration is complete, the system will save the calibration to `/Users/your_username/.cache/huggingface/lerobot/calibration/teleoperators/homunculus_glove/red.json`
|
||||
|
||||
#### Visualizing Teleoperator Glove
|
||||
|
||||
After calibration, you can visualize the glove movements in real-time. Open the visualizer by navigating to the visualizer directory and opening the HTML file in your browser:
|
||||
|
||||
```bash
|
||||
open examples/hopejr/visualizer/index.html
|
||||
```
|
||||
|
||||
### 1.3 Calibrate Robot Arm
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,397 @@
|
||||
# Isaac Teleop
|
||||
|
||||
Control your robot with NVIDIA [Isaac Teleop](https://github.com/NVIDIA/IsaacTeleop), a
|
||||
multi-modal teleoperation framework. Isaac Teleop drives a single `TeleopSession` from a range
|
||||
of input devices — XR (VR) controllers, hand tracking, full-body tracking, Manus gloves, foot
|
||||
pedals, and more.
|
||||
|
||||
In LeRobot, Isaac Teleop ships as a self-contained example under
|
||||
[`examples/isaac_teleop_to_so101/`](https://github.com/huggingface/lerobot/tree/main/examples/isaac_teleop_to_so101).
|
||||
Each Isaac Teleop input device is its own `Teleoperator` subclass in the example's
|
||||
`isaac_teleop` package, sharing one session lifecycle (see `IsaacTeleopTeleoperator`). The
|
||||
devices available today are the **XR controller** (`XRController`) and a back-drivable
|
||||
**SO-101 leader arm** (`SO101LeaderArm`); Manus gloves and hand/full-body tracking are the
|
||||
natural next devices. This guide focuses on the XR controller; the SO-101 leader is summarized
|
||||
under [Run the example](#step-3-run-the-example).
|
||||
|
||||
**In this guide you'll learn:**
|
||||
|
||||
- How an Isaac Teleop device drives a robot end‑effector (EE) target
|
||||
- How the _clutch_ (squeeze/grip on the XR controller) engages teleoperation without jerking the arm
|
||||
- How to run the SO‑101 teleoperation example and tune motion / gripper / IK
|
||||
|
||||
## Installation
|
||||
|
||||
The example lives in the LeRobot repository (it is not part of the `lerobot` pip package), so
|
||||
clone the repo and install from source. The canonical, always-up-to-date install and usage
|
||||
reference is the example's
|
||||
[`README.md`](https://github.com/huggingface/lerobot/tree/main/examples/isaac_teleop_to_so101/README.md);
|
||||
in short:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/huggingface/lerobot.git
|
||||
cd lerobot
|
||||
uv pip install -e ".[feetech,kinematics,dataset]" "huggingface_hub>=1.5"
|
||||
uv pip install "isaacteleop[cloudxr,retargeters-lite]~=1.3.131" "scipy>=1.14"
|
||||
```
|
||||
|
||||
`isaacteleop` is published on public PyPI (Linux only). The `cloudxr` extra brings the CloudXR
|
||||
runtime bindings; `retargeters-lite` is the scipy-based retargeter path that resolves on both
|
||||
x86_64 and ARM (on aarch64 — e.g. a DGX Spark — the full `retargeters` extra does not resolve
|
||||
because of its `dex-retargeting`/`nlopt` pins, which is why it is not the default here). On
|
||||
x86_64 you can additionally install the full retargeter stack:
|
||||
|
||||
```bash
|
||||
uv pip install "isaacteleop[retargeters]~=1.3.131"
|
||||
```
|
||||
|
||||
### Set up CloudXR and connect a headset
|
||||
|
||||
Isaac Teleop streams the headset to your machine over **NVIDIA CloudXR**, which provides the
|
||||
OpenXR runtime the session connects to. By default LeTeleop **auto-launches the CloudXR runtime
|
||||
for you** when you call `teleop_device.connect()` — you no longer have to run `python -m
|
||||
isaacteleop.cloudxr` and `source cloudxr.env` in a separate shell. All you need is a supported
|
||||
headset connected and the CloudXR firewall ports open. Follow the Isaac Teleop
|
||||
[Quick Start](https://nvidia.github.io/IsaacTeleop/main/getting_started/quick_start.html) for the
|
||||
headset-pairing and firewall details.
|
||||
|
||||
**First run (EULA).** The very first launch must accept the NVIDIA CloudXR EULA. The auto-launch
|
||||
prompts for it **on stdin**, so on a headless machine it will hang waiting for input. Bootstrap
|
||||
the EULA once, interactively, with:
|
||||
|
||||
```bash
|
||||
python -m isaacteleop.cloudxr --accept-eula # one-time: accept the CloudXR EULA
|
||||
```
|
||||
|
||||
After that, `connect()` launches the runtime non-interactively. The launch **blocks for ~30s**
|
||||
while the runtime comes up.
|
||||
|
||||
**Configuration.** Two fields on `IsaacTeleopConfig` (shared by every device) control this:
|
||||
|
||||
- `auto_launch_cloudxr` (default `True`) — whether `connect()` starts the runtime. Set `False`
|
||||
when CloudXR is already running externally.
|
||||
- `cloudxr_env_file` (default `None`) — an optional CloudXR device-profile `.env` selecting the
|
||||
headset transport (e.g. an Apple Vision Pro profile). This is launcher **input**; it is not the
|
||||
`~/.cloudxr/run/cloudxr.env` **output** file the old manual flow told you to `source`. `None`
|
||||
keeps the default auto-WebRTC profile — though the SO-101 example overrides it to the
|
||||
`default.env` shipped next to `teleoperate.py` unless you pass `--teleop.cloudxr_env_file`.
|
||||
|
||||
**Opting out.** To skip the auto-launch (CloudXR already running), either set
|
||||
`auto_launch_cloudxr=False` or export:
|
||||
|
||||
```bash
|
||||
export LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH=1
|
||||
```
|
||||
|
||||
The **env var takes precedence over the config field**: if `LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH=1` is
|
||||
set, the auto-launch is skipped even when `auto_launch_cloudxr=True`. This variable is
|
||||
**independent** of Isaac Lab's `ISAACLAB_CXR_SKIP_AUTOLAUNCH` — setting one does not affect the
|
||||
other.
|
||||
|
||||
**One teleoperator per process.** The CloudXR runtime configures the environment process-wide (a
|
||||
singleton), so run a single Isaac Teleop teleoperator per process.
|
||||
|
||||
**Shutting down.** Always call `teleop_device.disconnect()` on exit — including on Ctrl-C. Wrap
|
||||
your teleoperation loop in `try/finally` and call `disconnect()` in the `finally`. This tears down
|
||||
the OpenXR session **before** the CloudXR runtime, which is the required order; the launcher's
|
||||
`atexit` hook only reaps the runtime and does not run the session's `__exit__`, so without an
|
||||
explicit `disconnect()` an interrupted run shuts down in the wrong order.
|
||||
|
||||
```python
|
||||
teleop_device.connect()
|
||||
try:
|
||||
while True:
|
||||
action = teleop_device.get_action()
|
||||
# ... drive the robot ...
|
||||
finally:
|
||||
teleop_device.disconnect()
|
||||
```
|
||||
|
||||
See [System Requirements](https://nvidia.github.io/IsaacTeleop/main/references/requirements.html)
|
||||
for supported OS / GPU / CloudXR versions and headsets.
|
||||
|
||||
## How it works
|
||||
|
||||
The XR controller is one Isaac Teleop **input** device. `XRController` is a deliberately thin
|
||||
reader: it exposes the **raw** controller grip pose — already statically rebased into the robot
|
||||
base frame — plus the squeeze and trigger analog values. It has **no** retargeters and **no**
|
||||
clutch logic of its own. The clutch (engage latch + delta rebasing onto the EE) and the gripper
|
||||
mapping live downstream in the example loop, which then feeds LeRobot's existing closed‑loop
|
||||
Cartesian IK pipeline — the same one the phone teleoperator uses. The device‑specific pieces are
|
||||
`XRController`, the loop's `Clutch`, and `MapXRControllerActionToRobotAction`; everything downstream
|
||||
(`EEBoundsAndSafety`, `InverseKinematicsEEToJoints`) is shared, and a future device (e.g. Manus
|
||||
gloves) would swap in its own `teleop_<device>.py` + processor while reusing the rest.
|
||||
|
||||
`XRController._build_pipeline` wires Isaac Teleop's `ControllersSource` — statically rebased into
|
||||
the robot base frame by the native `ControllerTransform` (`base_T_anchor`) — and exposes the
|
||||
transformed controller stream verbatim. `get_action()` reads the grip pose, squeeze, and trigger
|
||||
straight off it; the session is always stepped `RUNNING` (there is no clutch retargeter to gate).
|
||||
|
||||
The `Clutch` class (in `examples/isaac_teleop_to_so101/isaac_teleop/clutch.py`, driven by the
|
||||
loop in `common.py`) mirrors Isaac Teleop's `SO101ClutchRetargeter`, but lives in-loop so the
|
||||
device can stay a thin reader:
|
||||
|
||||
- It latches its engage origin on the squeeze **engage edge** (the frame the squeeze first crosses
|
||||
`clutch_threshold`) and rebases both position and orientation around it, so engaging does not
|
||||
teleport the arm. `Clutch.rebase` returns the absolute base-frame target as a `(pos, quat)`
|
||||
pair, which the loop concatenates into the 7D `ee_pose` fed to the processor.
|
||||
- The analog trigger becomes a gripper `closedness` in `[0, 1]` (0 = open, 1 = closed),
|
||||
proportional to the trigger pull, which `MapXRControllerActionToRobotAction` maps to a jaw target.
|
||||
|
||||
See the Isaac Teleop
|
||||
[Retargeting interface](https://nvidia.github.io/IsaacTeleop/main/references/retargeting/index.html)
|
||||
and [architecture overview](https://nvidia.github.io/IsaacTeleop/main/overview/architecture.html)
|
||||
for how source nodes and retargeters compose.
|
||||
|
||||
```text
|
||||
VR controller (OpenXR)
|
||||
│
|
||||
▼
|
||||
XRController.get_action() ── raw base-frame grip_pos / grip_quat + squeeze + trigger
|
||||
│ (TeleopSession always stepped RUNNING; clutch lives downstream)
|
||||
▼
|
||||
Clutch.rebase(grip_pos, grip_quat) ── engage-relative delta applied to the EE home (pos + orient)
|
||||
│ ee_pose (7) / closedness → absolute ee_pose; closedness = trigger
|
||||
▼
|
||||
MapXRControllerActionToRobotAction ── absolute ee.x/y/z; ee.w* = orientation rotvec target;
|
||||
│ ee.x/y/z / ee.w* / ee.gripper_pos ee.gripper_pos = (1 - closedness) * 100
|
||||
▼
|
||||
EEBoundsAndSafety ── workspace clip + per-frame step clamp (clamp+warn)
|
||||
│
|
||||
▼
|
||||
InverseKinematicsEEToJoints ── closed-loop Placo IK; position + soft-orientation
|
||||
│ (orientation_weight=0.01) (passes ee.gripper_pos → gripper.pos)
|
||||
▼
|
||||
SO-101 follower joint targets
|
||||
```
|
||||
|
||||
### The clutch: owned by the example loop
|
||||
|
||||
Unlike the phone pipeline (which splits the clutch across `MapPhoneActionToRobotAction` and
|
||||
`EEReferenceAndDelta`), the XR clutch lives entirely in the example loop's `Clutch` class. It emits
|
||||
an **absolute** EE pose, so there is no `EEReferenceAndDelta` stage and no delta accumulation in the
|
||||
processor — `MapXRControllerActionToRobotAction` is a pure, stateless per‑frame mapping.
|
||||
|
||||
The clutch latches its engage origin on the squeeze **engage edge** (the moment the squeeze crosses
|
||||
`clutch_threshold`) and drives the EE from the motion _relative_ to that origin, so the arm does not
|
||||
teleport on engage. On **every** engage — startup and mid‑task re‑clutch alike — the home
|
||||
_position_ is latched from forward kinematics on the arm's **measured joints**, so the home equals
|
||||
where the arm physically is even if it moved while disengaged, and the engage is jump‑free. The
|
||||
home _orientation_ keeps the last commanded rotation: the 5‑DOF arm tracks orientation only
|
||||
softly, so latching the measured wrist orientation would inject its tracking offset into the
|
||||
command on every re‑clutch.
|
||||
|
||||
## Controls
|
||||
|
||||
- **Squeeze / grip** — the **clutch** (deadman). Hold it past `clutch_threshold` to engage
|
||||
teleoperation; release to pause. Each engage re‑captures the origin, so you can reposition
|
||||
your hand while paused and re‑engage without the arm jumping (index/clutch style).
|
||||
- **Trigger** — the **gripper**, controlled **analog**. The jaw tracks the trigger
|
||||
proportionally — a half‑pressed trigger leaves the jaw half‑closed — via a closedness in
|
||||
`[0, 1]` (0 = open, 1 = closed) that maps to an absolute gripper joint target.
|
||||
- **Controller orientation** — the **wrist**. The clutch rebases the controller orientation
|
||||
(engage‑relative, base‑frame) into a soft IK orientation target the wrist tracks alongside
|
||||
position. On the 5‑DOF SO‑101 the wrist follows the hand only partially by design — see
|
||||
`orientation_weight` below.
|
||||
|
||||
## Get started
|
||||
|
||||
### Step 1: Create the teleoperator
|
||||
|
||||
```python
|
||||
# Run from the repo root so the `examples` package is importable.
|
||||
from examples.isaac_teleop_to_so101.isaac_teleop import XRController, XRControllerConfig
|
||||
|
||||
teleop_config = XRControllerConfig(
|
||||
hand_side="right", # "left" or "right" controller
|
||||
clutch_threshold=0.5, # squeeze value above which the clutch engages
|
||||
)
|
||||
teleop_device = XRController(teleop_config)
|
||||
```
|
||||
|
||||
`XRController.get_action()` returns the **raw** base‑frame controller pose, not a clutch‑rebased
|
||||
target: `grip_pos` (3,) `[x, y, z]` [m] and `grip_quat` (4,) `[qx, qy, qz, qw]` in the robot base
|
||||
frame, plus scalar `squeeze` and `trigger` analog values in `[0, 1]`. The example loop's `Clutch`
|
||||
turns these into the absolute `ee_pose`, and the squeeze is thresholded by the loop against
|
||||
`clutch_threshold` to engage.
|
||||
|
||||
### Step 2: Connect
|
||||
|
||||
Calling `teleop_device.connect()` first auto-launches the CloudXR runtime (unless you opted out —
|
||||
see [Set up CloudXR and connect a headset](#set-up-cloudxr-and-connect-a-headset); this blocks for
|
||||
~30s and on the first run prompts for the EULA on stdin), then starts the Isaac Teleop
|
||||
[`TeleopSession`](https://nvidia.github.io/IsaacTeleop/main/getting_started/teleop_session.html)
|
||||
(opens the OpenXR session and discovers the controllers). XR controllers are self‑calibrating, so
|
||||
there is no manual calibration step — the clutch handles re‑centering each time you engage. Pair
|
||||
`connect()` with a `try/finally` that calls `disconnect()` so the session tears down before the
|
||||
runtime on exit/Ctrl-C.
|
||||
|
||||
### Step 3: Run the example
|
||||
|
||||
The example assumes you configured your robot (SO‑101 follower) and set the correct serial port.
|
||||
|
||||
The **robot URDF and its meshes are fetched automatically** on first run: the XR device downloads
|
||||
the SO-101 URDF from the
|
||||
[`lerobot/robot-urdfs` Hugging Face bucket](https://huggingface.co/buckets/lerobot/robot-urdfs/tree/so101)
|
||||
into the LeRobot cache (`HF_LEROBOT_HOME/robot-urdfs/so101/`) and reuses it after, so there is no
|
||||
separate download step :
|
||||
|
||||
```bash
|
||||
python -m examples.isaac_teleop_to_so101.teleoperate --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
|
||||
--robot.id=so101_follower_arm --teleop.type=xr_controller
|
||||
```
|
||||
|
||||
The CLI is `lerobot-teleoperate`-style (draccus): `--robot.*` configures the SO-101 follower and
|
||||
`--teleop.type` selects the Isaac input device (`xr_controller` | `so101_leader`), with
|
||||
`--teleop.*` its device knobs. `--teleop.type=xr_controller` runs the XR-controller path described
|
||||
above. The startup safety contract: by default it slews all joints to a default reset pose over
|
||||
`--reset_duration` seconds (`--reset_to_origin=false` keeps the arm where it is), then seeds the
|
||||
clutch home from the arm's measured pose so the first engage is jump-free; the follower is
|
||||
commanded only while the clutch is engaged.
|
||||
|
||||
**Customizing the reset pose.** The reset pose ships as a built-in default (a comfortable mid-range
|
||||
pose) and works out of the box — you do **not** need to record anything. To tailor it to your setup,
|
||||
back-drive the arm to the pose you want and run
|
||||
`python -m examples.isaac_teleop_to_so101.override_reset_pose --id <robot.id>`; it writes the
|
||||
current joints to a per-arm file in the LeRobot cache
|
||||
(`HF_LEROBOT_HOME/reset_poses/<robot.name>/<robot.id>.json`, keyed like calibration), which then takes
|
||||
priority over the built-in default on the next run. Because it lives in the user-local cache (not
|
||||
the repo), your override stays on your machine, and both `teleoperate` and `record` honor it
|
||||
when launched with the same `--robot.id`.
|
||||
|
||||
The other device, `--teleop.type=so101_leader`, mirrors the follower 1:1 from a back-drivable
|
||||
SO-101 _leader arm_ whose joints are streamed by Isaac Teleop's native `so101_leader` plugin (no
|
||||
clutch, no IK — the leader and follower share the SO-101 kinematics).
|
||||
|
||||
The `so101_leader_plugin` binary is a C++ plugin that is **not** part of the `isaacteleop` pip
|
||||
package — you build it from the Isaac Teleop source tree. Follow
|
||||
[Build Isaac Teleop from source](https://nvidia.github.io/IsaacTeleop/main/getting_started/build_from_source/index.html)
|
||||
(in short, from your Isaac Teleop checkout: `cmake -B build && cmake --build build --parallel &&
|
||||
cmake --install build`); the build installs the plugins under `<IsaacTeleop>/install/plugins/`, so
|
||||
the binary lands at `install/plugins/so101_leader/so101_leader_plugin` — the `--launch_plugin` path
|
||||
below. See the plugin's own `README.md` (next to the binary) for its serial/calibration details.
|
||||
|
||||
Point `--teleop.port` at the physical leader's serial port and `--launch_plugin` at that plugin
|
||||
binary to have the script spawn it after CloudXR is up:
|
||||
|
||||
```bash
|
||||
python -m examples.isaac_teleop_to_so101.teleoperate --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
|
||||
--robot.id=so101_follower_arm --teleop.type=so101_leader \
|
||||
--teleop.port=/dev/ttyACM1 --teleop.id=so101_leader_arm \
|
||||
--launch_plugin=/code/Teleop/install/plugins/so101_leader/so101_leader_plugin
|
||||
```
|
||||
|
||||
(Note `so101_leader` here is the _Isaac_ leader, resolved against the Isaac Teleop device
|
||||
registry, distinct from `lerobot-teleoperate`'s serial `so101_leader`.) When a `--teleop.port` is
|
||||
set, the plugin's tick→radian calibration is inferred from `--teleop.id` and passed to the plugin
|
||||
as its third positional arg — the LeRobot-format JSON at
|
||||
`HF_LEROBOT_CALIBRATION/teleoperators/so_leader/<id>.json`, the same file the serial SO-101 leader
|
||||
uses (`lerobot-calibrate --teleop.type=so101_leader --teleop.id=<id>`). If it is missing the script
|
||||
warns and the plugin uses built-in defaults. Run `python -m examples.isaac_teleop_to_so101.teleoperate --help` for all flags. Its
|
||||
startup safety contract: by default the follower is
|
||||
slewed to the leader's first reading over `--align_duration` seconds (`--align=false` to skip) so
|
||||
the arm does not snap when the mirror begins, and while the leader stream is stale the follower is
|
||||
held at its measured pose.
|
||||
|
||||
The URDF fetch uses `huggingface_hub` (already a LeRobot dependency) against the public
|
||||
`lerobot/robot-urdfs` bucket, so it needs no login. It is cached under
|
||||
`HF_LEROBOT_HOME/robot-urdfs/so101/`; delete that folder to force a re‑download.
|
||||
|
||||
Then, in your headset: squeeze and hold the grip to engage, move the controller to drive the
|
||||
arm, twist/tilt it to orient the wrist, and press the trigger to close the gripper
|
||||
(proportionally — release to open).
|
||||
|
||||
To record a dataset (not just teleoperate), use `record.py` in the same folder. It dispatches on
|
||||
`--teleop.type` (`xr_controller` | `so101_leader`) exactly like `teleoperate.py`, so either device
|
||||
can drive the follower, and it saves the commanded joints to a LeRobot dataset (`lerobot-record`-style
|
||||
`--dataset.*` flags). See its module docstring for the full CLI and the keyboard recording shortcuts.
|
||||
|
||||
## Important pipeline steps and options
|
||||
|
||||
The clutch already produces an absolute base‑frame pose, so the processor side is a thin
|
||||
**absolute‑pose** path — there is no frame remap, no delta accumulation, and no
|
||||
`EEReferenceAndDelta` stage.
|
||||
|
||||
- `MapXRControllerActionToRobotAction` is a stateless per‑frame mapping from the device output to
|
||||
the IK input contract. It writes the absolute base‑frame position, encodes the absolute
|
||||
orientation as a rotvec target, and inverts the closedness into a motor gripper target:
|
||||
|
||||
```python
|
||||
action["ee.x"], action["ee.y"], action["ee.z"] = ee_pose[:3] # absolute, base frame [m]
|
||||
action["ee.wx"], action["ee.wy"], action["ee.wz"] = orient_rotvec # orientation target (rotvec)
|
||||
action["ee.gripper_pos"] = (1 - closedness) * 100 # motor units; SO-101 calibrates 100 = open
|
||||
```
|
||||
|
||||
The gripper polarity (`100 = open, 0 = closed`) is a hardware‑calibration convention in the source — flip it there if the jaw opens when it should close.
|
||||
|
||||
- `EEBoundsAndSafety` clamps the EE to a workspace and rate‑limits per‑frame jumps. The clutch's
|
||||
no‑teleport keeps frames small, so `max_ee_step_m` mostly catches transient controller tracking
|
||||
glitches. The z floor is `0.0` (the table plane) so a stray target cannot drive the EE below the
|
||||
table; x/y stay at the loose `[-1, 1]` m box. Set `raise_on_jump=False` so an over‑limit frame is
|
||||
**clamped and warned** instead of raising — a crash mid‑loop would leave the arm uncontrolled:
|
||||
|
||||
```python
|
||||
EEBoundsAndSafety(
|
||||
end_effector_bounds={"min": [-1.0, -1.0, 0.0], "max": [1.0, 1.0, 1.0]},
|
||||
max_ee_step_m=0.10,
|
||||
raise_on_jump=False,
|
||||
)
|
||||
```
|
||||
|
||||
- `InverseKinematicsEEToJoints(initial_guess_current_joints=False, orientation_weight=0.01)` solves
|
||||
closed‑loop Placo IK. SO‑101 is a 5‑DOF arm, so the IK is position‑dominant; the small
|
||||
`orientation_weight` lets it softly track the orientation target carried in `ee.w*` so the wrist
|
||||
follows the hand, while the under‑determined roll stays partial by design. There is **no**
|
||||
`GripperVelocityToJoint`: the absolute `ee.gripper_pos` is passed straight to `gripper.pos`.
|
||||
`initial_guess_current_joints=False` warm‑starts each solve from the **previous IK solution**
|
||||
rather than re‑seeding from the measured joints, so the joint trajectory stays continuous
|
||||
frame‑to‑frame. Tune `orientation_weight` on hardware — too high fights position tracking, too
|
||||
low ignores the orientation command.
|
||||
|
||||
The example also gates safety at the loop level: after the startup reset slew (on by default —
|
||||
pass `--reset_to_origin=false` to keep the arm where it is), it commands the robot **only while
|
||||
the clutch is engaged**, and re‑sends the measured joints while disengaged, so releasing the
|
||||
clutch freezes the arm in place.
|
||||
|
||||
See the [Processors for Robots and Teleoperators](./processors_robots_teleop) guide for more on
|
||||
adapting the pipeline to other robots.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`ModuleNotFoundError: isaacteleop`** — the `isaacteleop` package is not installed in the
|
||||
active environment. Re-run the install command at the top of this guide:
|
||||
`uv pip install "isaacteleop[cloudxr,retargeters-lite]~=1.3.131"`.
|
||||
- **No controllers found** — make sure the CloudXR runtime is running, the firewall ports are
|
||||
whitelisted, and the headset is connected (see
|
||||
[Set up CloudXR and connect a headset](#set-up-cloudxr-and-connect-a-headset) and the Isaac
|
||||
Teleop [Quick Start](https://nvidia.github.io/IsaacTeleop/main/getting_started/quick_start.html)).
|
||||
- **CloudXR auto-launch failed** — `connect()` raises a `RuntimeError` if the runtime does not
|
||||
come up within its startup timeout. Check the launcher logs under `~/.cloudxr/logs`. Common
|
||||
causes: the EULA was never accepted (run `python -m isaacteleop.cloudxr --accept-eula` once,
|
||||
interactively — the auto-launch prompts on stdin and hangs headless), or the runtime is already
|
||||
running externally (set `LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH=1` or `auto_launch_cloudxr=False` to
|
||||
skip the auto-launch).
|
||||
- **Arm does not move** — the clutch is a deadman: you must hold the squeeze/grip past
|
||||
`clutch_threshold`. Lower the threshold if your controller's squeeze is reported softly.
|
||||
- **Motion feels misaligned** — confirm the headset/play space orientation. The controller stream
|
||||
is rebased into the robot base frame by the `base_T_anchor` transform on `XRControllerConfig`
|
||||
(default: standard OpenXR → robot axis convention); adjust it if your anchor frame differs.
|
||||
|
||||
## Learn more
|
||||
|
||||
NVIDIA Isaac Teleop documentation ([docs home](https://nvidia.github.io/IsaacTeleop/),
|
||||
[GitHub](https://github.com/NVIDIA/IsaacTeleop)):
|
||||
|
||||
- [Quick Start](https://nvidia.github.io/IsaacTeleop/main/getting_started/quick_start.html) —
|
||||
install, run the CloudXR server, connect a headset, run a teleop example.
|
||||
- [TeleopSession](https://nvidia.github.io/IsaacTeleop/main/getting_started/teleop_session.html) —
|
||||
the session API `XRController` wraps.
|
||||
- [Retargeting interface](https://nvidia.github.io/IsaacTeleop/main/references/retargeting/index.html)
|
||||
and [architecture overview](https://nvidia.github.io/IsaacTeleop/main/overview/architecture.html) —
|
||||
how source nodes and retargeters compose into a pipeline.
|
||||
- [Build from source](https://nvidia.github.io/IsaacTeleop/main/getting_started/build_from_source/index.html) —
|
||||
build `isaacteleop` (and its C++ plugins, including the `so101_leader` plugin used above) from a
|
||||
local checkout.
|
||||
- [System Requirements](https://nvidia.github.io/IsaacTeleop/main/references/requirements.html) and
|
||||
the [CloudXR SDK docs](https://docs.nvidia.com/cloudxr-sdk) — supported platforms, GPUs,
|
||||
CloudXR/OpenXR runtime versions, and headsets.
|
||||
@@ -6,12 +6,11 @@ Encoding frames into an MP4 is a full FFmpeg pipeline: choice of encoder, pixel
|
||||
|
||||
You can set these parameters from the CLI with `--dataset.rgb_encoder.<field>` (e.g. with `lerobot-record` or `lerobot-rollout`). The same block applies to every camera video stream in that run.
|
||||
|
||||
<Tip>
|
||||
Video storage must be on for `rgb_encoder` to have any effect —
|
||||
`use_videos=True` in Python APIs, or `--dataset.video=true` on the CLI (the
|
||||
recording default). With video off, inputs stay as images and `rgb_encoder` is
|
||||
ignored.
|
||||
</Tip>
|
||||
> [!TIP]
|
||||
> Video storage must be on for `rgb_encoder` to have any effect —
|
||||
> `use_videos=True` in Python APIs, or `--dataset.video=true` on the CLI (the
|
||||
> recording default). With video off, inputs stay as images and `rgb_encoder` is
|
||||
> ignored.
|
||||
|
||||
For details on **when** frames are written vs. encoded (streaming vs. post-episode), queues, and other top-level `--dataset.*` switches, see [Streaming Video Encoding](./streaming_video_encoding). For an encoding-parameter comparison and experiments, see the [video-benchmark Space](https://huggingface.co/spaces/lerobot/video-benchmark).
|
||||
|
||||
@@ -43,12 +42,10 @@ lerobot-record \
|
||||
|
||||
## Tuning parameters
|
||||
|
||||
<Tip warning={true}>
|
||||
The defaults are tuned to balance **compression ratio**, **visual quality**, and **decoding/seek speed** for typical robotics datasets. Changing them can affect both recording (CPU load, frame drops) and training (decoding throughput, image quality).
|
||||
|
||||
Only override these parameters if you have a specific reason to, and measure the impact on your pipeline before relying on the new settings.
|
||||
|
||||
</Tip>
|
||||
> [!WARNING]
|
||||
> The defaults are tuned to balance **compression ratio**, **visual quality**, and **decoding/seek speed** for typical robotics datasets. Changing them can affect both recording (CPU load, frame drops) and training (decoding throughput, image quality).
|
||||
>
|
||||
> Only override these parameters if you have a specific reason to, and measure the impact on your pipeline before relying on the new settings.
|
||||
|
||||
All flags below are prefixed with `--dataset.rgb_encoder.` on the CLI.
|
||||
|
||||
@@ -69,25 +66,92 @@ All flags below are prefixed with `--dataset.rgb_encoder.` on the CLI.
|
||||
|
||||
Depth maps (Intel RealSense, Reachy 2) are stored as their **own video streams** alongside the RGB streams. Raw depth (`uint16` millimetres or `float32` metres) can't survive an 8-bit codec, so LeRobot **quantizes** each map to a 12-bit code (`[0, 4095]`) — logarithmically by default, to match the `1/depth` error profile of depth sensors — then packs it into a high-bit-depth pixel format (`gray12le`) and encodes it with a 12-bit codec.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["Raw depth (uint16 mm / float32 m)"] --> B["Clip to depth_min, depth_max"]
|
||||
B --> C["Quantize to 12-bit code 0–4095 (log or linear)"]
|
||||
C --> D["Pack into gray12le"]
|
||||
D --> E["Encode video (hevc Main 12)"]
|
||||
E --> F[("MP4 + metadata: depth_min/max, shift, use_log")]
|
||||
F -. "load time (depth_output_unit)" .-> G["Dequantize to mm or m"]
|
||||
|
||||
classDef input fill:#e3f2fd,stroke:#1565c0,color:#0d47a1;
|
||||
classDef encode fill:#ede7f6,stroke:#5e35b1,color:#311b92;
|
||||
classDef store fill:#fff8e1,stroke:#f9a825,color:#e65100;
|
||||
classDef load fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20;
|
||||
|
||||
class A input;
|
||||
class B,C,D,E encode;
|
||||
class F store;
|
||||
class G load;
|
||||
```
|
||||
<div style="margin:28px 0;padding:14px 0;">
|
||||
<div style="margin:0 auto;display:flex;flex-wrap:wrap;justify-content:center;align-items:stretch;gap:6px;font-family:'Source Sans 3',ui-sans-serif,system-ui,sans-serif;font-size:14px;font-weight:600;color:#1B1B1D;">
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#DBEAFE;color:#1D4ED8;border-radius:9px;padding:8px 12px;">
|
||||
<span>Raw depth</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#3B6FD4;white-space:nowrap;">
|
||||
uint16 mm
|
||||
<br />
|
||||
float32 m
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<div style="border:2px dashed #C4B5FD;border-radius:13px;padding:18px 12px 12px;position:relative;display:flex;align-items:stretch;gap:6px;">
|
||||
<span style="position:absolute;top:-10px;left:12px;background:#fff;padding:0 6px;font-size:11px;font-weight:700;color:#7E22CE;text-transform:uppercase;letter-spacing:0.5px;white-space:nowrap;">
|
||||
Record time
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;">
|
||||
<span>Clip</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">
|
||||
to [depth_min,
|
||||
<br />
|
||||
depth_max]
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;">
|
||||
<span>Quantize</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">
|
||||
12-bit codes 0–4095
|
||||
<br />
|
||||
log (default) or linear
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;">
|
||||
<span>Pack</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">
|
||||
into gray12le
|
||||
<br />
|
||||
plane
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;">
|
||||
<span>Encode</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">
|
||||
HEVC
|
||||
<br />
|
||||
Main 12
|
||||
</span>
|
||||
</span>
|
||||
</div>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#FEF3C7;color:#B45309;border-radius:9px;padding:8px 12px;">
|
||||
<span>MP4</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#C77D18;white-space:nowrap;">
|
||||
stored
|
||||
<br />
|
||||
stream
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#34A06B;">
|
||||
→
|
||||
</span>
|
||||
<div style="border:2px dashed #6EE7B7;border-radius:13px;padding:18px 12px 12px;position:relative;display:flex;align-items:center;gap:6px;">
|
||||
<span style="position:absolute;top:-10px;left:12px;background:#fff;padding:0 6px;font-size:11px;font-weight:700;color:#047857;text-transform:uppercase;letter-spacing:0.5px;white-space:nowrap;">
|
||||
Load time
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#D1FAE5;color:#047857;border-radius:9px;padding:8px 12px;">
|
||||
<span>Dequantize</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#059669;white-space:nowrap;">
|
||||
to mm / m
|
||||
</span>
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Configure the depth pipeline through a parallel **`depth_encoder`** block (`DepthEncoderConfig`). It shares every `RGBEncoderConfig` field (`vcodec`, `pix_fmt`, `crf`, …) and adds four quantizer knobs, set via `--dataset.depth_encoder.<field>`:
|
||||
|
||||
@@ -168,15 +232,16 @@ After the first episode of a video stream is encoded, the encoder configuration
|
||||
|
||||
Two sources contribute to the `info` block:
|
||||
|
||||
- **Stream-derived** (read back from the encoded MP4 with PyAV): `video.height`, `video.width`, `video.codec`, `video.pix_fmt`, `video.fps`, `video.channels`, `is_depth_map`, plus `audio.*` if an audio stream is present.
|
||||
- **Encoder-derived** (taken from `RGBEncoderConfig` or `DepthEncoderConfig`): `video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.video_backend`, `video.extra_options`.
|
||||
| Source | Where it comes from | Fields |
|
||||
| ------------------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Stream-derived** | Read back from the encoded MP4 with PyAV. | `video.height`, `video.width`, `video.codec`, `video.pix_fmt`, `video.fps`, `video.channels`, `is_depth_map`, `audio.*` |
|
||||
| **Encoder-derived** | Taken from `RGBEncoderConfig` / `DepthEncoderConfig`. | `video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.video_backend`, `video.extra_options` |
|
||||
|
||||
<Tip>
|
||||
This block is populated **once**, from the **first** episode. It assumes every
|
||||
episode in the dataset was encoded with the same `rgb_encoder`. Changing
|
||||
encoder settings partway through a recording is not supported — the
|
||||
`info.json` will only reflect the parameters used for the first episode.
|
||||
</Tip>
|
||||
> [!IMPORTANT]
|
||||
> This block is populated **once**, from the **first** episode. It assumes every
|
||||
> episode in the dataset was encoded with the same `rgb_encoder`. Changing
|
||||
> encoder settings partway through a recording is not supported — the
|
||||
> `info.json` will only reflect the parameters used for the first episode.
|
||||
|
||||
---
|
||||
|
||||
@@ -184,5 +249,7 @@ Two sources contribute to the `info` block:
|
||||
|
||||
When aggregating datasets with `merge_datasets`, video files are concatenated as-is (no re-encoding), and encoder fields in `info.json` are merged per-key:
|
||||
|
||||
- **Stream-derived fields must match** across sources: `video.codec`, `video.pix_fmt`, `video.height`, `video.width`, `video.fps`. Otherwise FFmpeg's concat demuxer fails.
|
||||
- **Encoder-tuning fields are merged loosely**: `video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.extra_options`. If every source agrees, the value is kept; if not, it's set to `null` (or `{}` for `video.extra_options`) and a warning is logged.
|
||||
| Merge rule | Fields | Behaviour |
|
||||
| ------------------ | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Must match** | `video.codec`, `video.pix_fmt`, `video.height`, `video.width`, `video.fps` | Stream-derived fields must match across sources, otherwise FFmpeg's concat demuxer fails. |
|
||||
| **Merged loosely** | `video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.extra_options` | Encoder-tuning fields. If every source agrees, the value is kept; if not, it's set to `null` (or `{}` for `video.extra_options`) and a warning is logged. |
|
||||
|
||||
@@ -46,8 +46,11 @@ CMD = (
|
||||
"apt-get update -qq && apt-get install -y -qq git ffmpeg && "
|
||||
"pip install --no-deps "
|
||||
"'lerobot @ git+https://github.com/huggingface/lerobot.git@main' && "
|
||||
# Pins mirror pyproject.toml — unpinned installs pull av 18 / datasets 5 /
|
||||
# draccus 0.11, which break lerobot at import time.
|
||||
"pip install --upgrade-strategy only-if-needed "
|
||||
"datasets pyarrow av jsonlines draccus gymnasium torchcodec mergedeep pyyaml-include toml typing-inspect "
|
||||
"'datasets>=4.7.0,<5.0.0' 'pyarrow>=21.0.0,<30.0.0' 'av>=15.0.0,<16.0.0' 'draccus==0.10.0' "
|
||||
"'pandas>=2.0.0,<3.0.0' jsonlines gymnasium torchcodec mergedeep pyyaml-include toml typing-inspect "
|
||||
"openai && "
|
||||
"export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 && "
|
||||
"export VLLM_VIDEO_BACKEND=pyav && "
|
||||
|
||||
@@ -1,182 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>3D Hand Joint Visualizer</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: Arial, sans-serif;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
overflow: hidden;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
height: 100vh;
|
||||
}
|
||||
|
||||
.controls {
|
||||
padding: 15px;
|
||||
background-color: #f5f5f5;
|
||||
z-index: 100;
|
||||
}
|
||||
|
||||
.status {
|
||||
padding: 10px;
|
||||
border-radius: 5px;
|
||||
margin: 10px 0;
|
||||
}
|
||||
|
||||
.connected {
|
||||
background-color: #d4edda;
|
||||
color: #155724;
|
||||
}
|
||||
|
||||
.disconnected {
|
||||
background-color: #f8d7da;
|
||||
color: #721c24;
|
||||
}
|
||||
|
||||
button {
|
||||
padding: 8px 16px;
|
||||
background-color: #4CAF50;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
cursor: pointer;
|
||||
margin-right: 10px;
|
||||
}
|
||||
|
||||
button:hover {
|
||||
background-color: #45a049;
|
||||
}
|
||||
|
||||
button:disabled {
|
||||
background-color: #cccccc;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.container {
|
||||
display: flex;
|
||||
flex: 1;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
#canvas-container {
|
||||
flex: 3;
|
||||
position: relative;
|
||||
}
|
||||
|
||||
#sidebar {
|
||||
flex: 1;
|
||||
padding: 15px;
|
||||
background-color: #f8f9fa;
|
||||
overflow-y: auto;
|
||||
max-width: 300px;
|
||||
border-left: 1px solid #ddd;
|
||||
}
|
||||
|
||||
.joint-info {
|
||||
margin-bottom: 10px;
|
||||
padding: 8px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.joint-name {
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.joint-value {
|
||||
font-family: monospace;
|
||||
}
|
||||
|
||||
.bar-container {
|
||||
width: 100%;
|
||||
background-color: #e0e0e0;
|
||||
height: 10px;
|
||||
border-radius: 5px;
|
||||
overflow: hidden;
|
||||
margin-top: 5px;
|
||||
}
|
||||
|
||||
.bar {
|
||||
height: 100%;
|
||||
background-color: #4CAF50;
|
||||
width: 0%;
|
||||
transition: width 0.2s ease-in-out;
|
||||
}
|
||||
|
||||
.log-container {
|
||||
margin-top: 20px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 5px;
|
||||
padding: 10px;
|
||||
height: 150px;
|
||||
overflow-y: auto;
|
||||
font-family: monospace;
|
||||
background-color: #f8f9fa;
|
||||
}
|
||||
|
||||
.view-controls {
|
||||
position: absolute;
|
||||
bottom: 10px;
|
||||
left: 10px;
|
||||
z-index: 10;
|
||||
}
|
||||
|
||||
.view-button {
|
||||
background-color: rgba(0, 0, 0, 0.5);
|
||||
color: white;
|
||||
border: none;
|
||||
padding: 5px 10px;
|
||||
margin-right: 5px;
|
||||
border-radius: 3px;
|
||||
cursor: pointer;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="controls">
|
||||
<button id="connectButton">Connect to Device</button>
|
||||
<button id="disconnectButton" disabled>Disconnect</button>
|
||||
<select id="baudRate">
|
||||
<option value="9600">9600</option>
|
||||
<option value="19200">19200</option>
|
||||
<option value="38400">38400</option>
|
||||
<option value="57600">57600</option>
|
||||
<option value="115200" selected>115200</option>
|
||||
</select>
|
||||
<span id="statusIndicator" class="status disconnected">Status: Disconnected</span>
|
||||
</div>
|
||||
|
||||
<div class="container">
|
||||
<div id="canvas-container">
|
||||
<!-- 3D canvas will be inserted here -->
|
||||
<div class="view-controls">
|
||||
<button class="view-button" id="frontView">Front</button>
|
||||
<button class="view-button" id="sideView">Side</button>
|
||||
<button class="view-button" id="topView">Top</button>
|
||||
<button class="view-button" id="resetView">Reset</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="sidebar">
|
||||
<h3>Joint Values</h3>
|
||||
<div id="jointsContainer">
|
||||
<!-- Joint info will be added here -->
|
||||
</div>
|
||||
|
||||
<div class="log-container" id="logContainer">
|
||||
<!-- Log messages will be added here -->
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Import Three.js -->
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.132.2/build/three.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.132.2/examples/js/controls/OrbitControls.js"></script>
|
||||
|
||||
<script src="script.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,669 +0,0 @@
|
||||
// === Hand Visualizer with Pre-Connect Sliders + Per-Joint Angle Limits ===
|
||||
// Assumes your HTML already has elements with the following IDs:
|
||||
// connectButton, disconnectButton, baudRate, statusIndicator, jointsContainer, logContainer,
|
||||
// canvas-container, frontView, sideView, topView, resetView
|
||||
// Requires Three.js + OrbitControls loaded on the page.
|
||||
|
||||
// -------------------- Config --------------------
|
||||
const MAX_JOINTS = 16;
|
||||
const RAW_MIN = 0, RAW_MAX = 4096;
|
||||
const RAW_CENTER = (RAW_MIN + RAW_MAX) / 2;
|
||||
const DEG = Math.PI / 180;
|
||||
const UI_DEG_MIN = -90, UI_DEG_MAX = 90; // UI sliders for angle limits
|
||||
|
||||
// -------------------- State --------------------
|
||||
let port;
|
||||
let reader;
|
||||
let keepReading = false;
|
||||
let isConnected = false;
|
||||
const decoder = new TextDecoder();
|
||||
let inputBuffer = '';
|
||||
|
||||
let jointValues = new Array(MAX_JOINTS).fill(RAW_CENTER);
|
||||
|
||||
// Auto-calibration: track observed min/max per joint
|
||||
let observedMin = new Array(MAX_JOINTS).fill(Infinity);
|
||||
let observedMax = new Array(MAX_JOINTS).fill(-Infinity);
|
||||
let calibrationEnabled = true;
|
||||
|
||||
// Three.js
|
||||
let scene, camera, renderer, controls;
|
||||
let hand = { palm: null, fingers: [] };
|
||||
|
||||
// DOM
|
||||
const connectButton = document.getElementById('connectButton');
|
||||
const disconnectButton = document.getElementById('disconnectButton');
|
||||
const baudRateSelect = document.getElementById('baudRate');
|
||||
const statusIndicator = document.getElementById('statusIndicator');
|
||||
const jointsContainer = document.getElementById('jointsContainer');
|
||||
const logContainer = document.getElementById('logContainer');
|
||||
const canvasContainer = document.getElementById('canvas-container');
|
||||
const frontViewBtn = document.getElementById('frontView');
|
||||
const sideViewBtn = document.getElementById('sideView');
|
||||
const topViewBtn = document.getElementById('topView');
|
||||
const resetViewBtn = document.getElementById('resetView');
|
||||
|
||||
// Helpers
|
||||
const clamp = (x, a, b) => Math.max(a, Math.min(b, x));
|
||||
const invLerp = (a, b, x) => clamp((x - a) / (b - a), 0, 1);
|
||||
|
||||
// -------------------- Joint Map with per-joint angle limits --------------------
|
||||
const fingerJointMap = [
|
||||
// Thumb (4)
|
||||
{ finger:0, joint:0, type:'CMC_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:0, joint:1, type:'CMC_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:0, joint:2, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true }, // +45° only
|
||||
{ finger:0, joint:3, type:'IP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true }, // +45° only
|
||||
|
||||
// Index (3)
|
||||
{ finger:1, joint:0, type:'MCP_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:1, joint:1, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false },
|
||||
{ finger:1, joint:2, type:'PIP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true }, // +45° only
|
||||
|
||||
// Middle (3)
|
||||
{ finger:2, joint:0, type:'MCP_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:2, joint:1, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:2, joint:2, type:'PIP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true }, // +45° only
|
||||
|
||||
// Ring (3)
|
||||
{ finger:3, joint:0, type:'MCP_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:3, joint:1, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false },
|
||||
{ finger:3, joint:2, type:'PIP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false }, // +45° only
|
||||
|
||||
// Pinky (3)
|
||||
{ finger:4, joint:0, type:'MCP_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:false },
|
||||
{ finger:4, joint:1, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false },
|
||||
{ finger:4, joint:2, type:'PIP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false } // +45° only
|
||||
];
|
||||
|
||||
// Assign angle limits (radians) per joint (default ±45°, exceptions: +45° only)
|
||||
for (const j of fingerJointMap) {
|
||||
const isThumb = j.finger === 0;
|
||||
const isPIP = j.type === 'PIP_FLEXION';
|
||||
let minA = -45 * DEG, maxA = +45 * DEG;
|
||||
if ((isThumb && (j.type === 'MCP_FLEXION' || j.type === 'IP_FLEXION')) || (!isThumb && isPIP)) {
|
||||
minA = 0;
|
||||
maxA = +45 * DEG;
|
||||
}
|
||||
j.angleMin = minA;
|
||||
j.angleMax = maxA;
|
||||
}
|
||||
|
||||
// -------------------- UI: Joint Panel --------------------
|
||||
const uiRefs = []; // per joint: { valueLabel, bar, barWrap, slider, invertChk, minDeg, maxDeg }
|
||||
|
||||
function initializeJointElements() {
|
||||
jointsContainer.innerHTML = '';
|
||||
uiRefs.length = 0;
|
||||
|
||||
for (let i = 0; i < MAX_JOINTS; i++) {
|
||||
const wrap = document.createElement('div');
|
||||
wrap.className = 'joint-info';
|
||||
|
||||
const fingerIndex = i < 4 ? 0 : Math.floor((i - 4) / 3) + 1;
|
||||
const jointInfo = fingerJointMap[i];
|
||||
const jointType = jointInfo?.type || 'Unknown';
|
||||
const fingerName = ['Thumb', 'Index', 'Middle', 'Ring', 'Pinky'][fingerIndex];
|
||||
|
||||
// Header
|
||||
const nameEl = document.createElement('div');
|
||||
nameEl.className = 'joint-name';
|
||||
nameEl.textContent = `${fingerName} – ${jointType}`;
|
||||
|
||||
// Value + bar
|
||||
const valueEl = document.createElement('div');
|
||||
valueEl.className = 'joint-value';
|
||||
valueEl.textContent = `Value: ${jointValues[i]}`;
|
||||
|
||||
const barWrap = document.createElement('div');
|
||||
barWrap.className = 'bar-container';
|
||||
const barEl = document.createElement('div');
|
||||
barEl.className = 'bar';
|
||||
barWrap.appendChild(barEl);
|
||||
|
||||
// Slider for pre-connect manual control
|
||||
const slider = document.createElement('input');
|
||||
slider.type = 'range';
|
||||
slider.min = String(RAW_MIN);
|
||||
slider.max = String(RAW_MAX);
|
||||
slider.value = String(jointValues[i]);
|
||||
slider.step = '1';
|
||||
slider.className = 'joint-slider';
|
||||
|
||||
slider.addEventListener('input', () => {
|
||||
if (isConnected) return; // ignore while connected
|
||||
let v = parseInt(slider.value, 10);
|
||||
if (jointInfo?.inverted) v = (jointInfo.min + jointInfo.max) - v;
|
||||
jointValues[i] = clamp(jointInfo ? v : 0, RAW_MIN, RAW_MAX);
|
||||
updateJointDisplay(i, jointValues[i]);
|
||||
updateHandModel();
|
||||
});
|
||||
|
||||
// Invert checkbox
|
||||
const invertLbl = document.createElement('label');
|
||||
invertLbl.className = 'invert-toggle';
|
||||
const invertChk = document.createElement('input');
|
||||
invertChk.type = 'checkbox';
|
||||
invertChk.checked = !!jointInfo?.inverted;
|
||||
invertChk.addEventListener('change', () => {
|
||||
if (jointInfo) jointInfo.inverted = invertChk.checked;
|
||||
addLogMessage(`${fingerName} ${jointType} inversion ${invertChk.checked ? 'enabled' : 'disabled'}`);
|
||||
});
|
||||
invertLbl.appendChild(invertChk);
|
||||
invertLbl.appendChild(document.createTextNode('Invert Values'));
|
||||
|
||||
// Angle limits (deg) controls
|
||||
const limitsRow = document.createElement('div');
|
||||
limitsRow.className = 'limits-row';
|
||||
|
||||
const minDeg = document.createElement('input');
|
||||
minDeg.type = 'number';
|
||||
minDeg.min = String(UI_DEG_MIN);
|
||||
minDeg.max = String(UI_DEG_MAX);
|
||||
minDeg.step = '1';
|
||||
minDeg.value = String(Math.round((jointInfo.angleMin || 0) / DEG));
|
||||
minDeg.className = 'limit-num';
|
||||
|
||||
const maxDeg = document.createElement('input');
|
||||
maxDeg.type = 'number';
|
||||
maxDeg.min = String(UI_DEG_MIN);
|
||||
maxDeg.max = String(UI_DEG_MAX);
|
||||
maxDeg.step = '1';
|
||||
maxDeg.value = String(Math.round((jointInfo.angleMax || 0) / DEG));
|
||||
maxDeg.className = 'limit-num';
|
||||
|
||||
const minLbl = document.createElement('span'); minLbl.textContent = 'min°';
|
||||
const maxLbl = document.createElement('span'); maxLbl.textContent = 'max°';
|
||||
minLbl.className = 'limit-label'; maxLbl.className = 'limit-label';
|
||||
|
||||
function syncLimits() {
|
||||
let mn = parseFloat(minDeg.value);
|
||||
let mx = parseFloat(maxDeg.value);
|
||||
if (isNaN(mn)) mn = -45;
|
||||
if (isNaN(mx)) mx = +45;
|
||||
if (mn > mx) [mn, mx] = [mx, mn];
|
||||
jointInfo.angleMin = clamp(mn, UI_DEG_MIN, UI_DEG_MAX) * DEG;
|
||||
jointInfo.angleMax = clamp(mx, UI_DEG_MIN, UI_DEG_MAX) * DEG;
|
||||
minDeg.value = String(Math.round(jointInfo.angleMin / DEG));
|
||||
maxDeg.value = String(Math.round(jointInfo.angleMax / DEG));
|
||||
updateHandModel();
|
||||
}
|
||||
minDeg.addEventListener('change', syncLimits);
|
||||
maxDeg.addEventListener('change', syncLimits);
|
||||
|
||||
limitsRow.appendChild(minLbl);
|
||||
limitsRow.appendChild(minDeg);
|
||||
limitsRow.appendChild(maxLbl);
|
||||
limitsRow.appendChild(maxDeg);
|
||||
|
||||
// Calibration controls
|
||||
const calibRow = document.createElement('div');
|
||||
calibRow.className = 'calib-row';
|
||||
|
||||
const resetCalibBtn = document.createElement('button');
|
||||
resetCalibBtn.textContent = 'Reset Calib';
|
||||
resetCalibBtn.className = 'calib-btn';
|
||||
resetCalibBtn.addEventListener('click', () => {
|
||||
observedMin[i] = Infinity;
|
||||
observedMax[i] = -Infinity;
|
||||
addLogMessage(`Reset calibration for ${fingerName} ${jointType}`);
|
||||
});
|
||||
|
||||
const calibStatus = document.createElement('span');
|
||||
calibStatus.className = 'calib-status';
|
||||
calibStatus.textContent = `Range: --`;
|
||||
|
||||
calibRow.appendChild(resetCalibBtn);
|
||||
calibRow.appendChild(calibStatus);
|
||||
|
||||
// Compose
|
||||
wrap.appendChild(nameEl);
|
||||
wrap.appendChild(valueEl);
|
||||
wrap.appendChild(barWrap);
|
||||
wrap.appendChild(slider);
|
||||
wrap.appendChild(invertLbl);
|
||||
wrap.appendChild(limitsRow);
|
||||
wrap.appendChild(calibRow);
|
||||
|
||||
jointsContainer.appendChild(wrap);
|
||||
|
||||
uiRefs[i] = { valueLabel: valueEl, bar: barEl, barWrap, slider, invertChk, minDeg, maxDeg, nameEl, calibStatus };
|
||||
}
|
||||
|
||||
setConnectedUI(false); // initial state: sliders active
|
||||
}
|
||||
|
||||
// Toggle UI between pre-connect SLIDERS vs post-connect BARS
|
||||
function setConnectedUI(connected) {
|
||||
isConnected = connected;
|
||||
for (let i = 0; i < uiRefs.length; i++) {
|
||||
const ui = uiRefs[i];
|
||||
if (!ui) continue;
|
||||
// Show bars when connected; sliders disabled/hidden
|
||||
ui.barWrap.style.display = connected ? '' : 'none';
|
||||
ui.slider.disabled = connected;
|
||||
ui.slider.style.display = connected ? 'none' : '';
|
||||
}
|
||||
|
||||
// Reset calibration when connecting
|
||||
if (connected) {
|
||||
observedMin.fill(Infinity);
|
||||
observedMax.fill(-Infinity);
|
||||
addLogMessage('Calibration reset - move joints through full range for best results');
|
||||
}
|
||||
}
|
||||
|
||||
// Update joint display (value text + bar color/width + slider position if needed)
|
||||
function updateJointDisplay(jointIndex, value) {
|
||||
const ui = uiRefs[jointIndex];
|
||||
const info = fingerJointMap[jointIndex];
|
||||
if (!ui || !info) return;
|
||||
|
||||
ui.valueLabel.textContent = `Value: ${value}`;
|
||||
|
||||
// bar
|
||||
const min = info.min, max = info.max;
|
||||
const pct = clamp((value - min) / (max - min), 0, 1) * 100;
|
||||
ui.bar.style.width = `${pct}%`;
|
||||
const hue = Math.floor(pct * 1.2); // 0..120
|
||||
ui.bar.style.backgroundColor = `hsl(${hue}, 80%, 50%)`;
|
||||
|
||||
// slider (only meaningful when not connected; keep in sync anyway)
|
||||
const rawForSlider = info.inverted ? (info.min + info.max) - value : value;
|
||||
if (!isConnected) ui.slider.value = String(clamp(Math.round(rawForSlider), RAW_MIN, RAW_MAX));
|
||||
}
|
||||
|
||||
// -------------------- Serial I/O --------------------
|
||||
async function readSerialData() {
|
||||
while (port?.readable && keepReading) {
|
||||
reader = port.readable.getReader();
|
||||
try {
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value) processData(decoder.decode(value));
|
||||
}
|
||||
} catch (err) {
|
||||
console.error('Error reading:', err);
|
||||
addLogMessage(`Error: ${err.message}`);
|
||||
break;
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function processData(chunk) {
|
||||
inputBuffer += chunk;
|
||||
let idx;
|
||||
while ((idx = inputBuffer.indexOf('\n')) !== -1) {
|
||||
const line = inputBuffer.slice(0, idx).trim();
|
||||
inputBuffer = inputBuffer.slice(idx + 1);
|
||||
|
||||
const vals = line.split(/\s+/).map(v => parseInt(v, 10));
|
||||
if (vals.length === MAX_JOINTS && vals.every(v => Number.isFinite(v))) {
|
||||
for (let i = 0; i < MAX_JOINTS; i++) {
|
||||
const info = fingerJointMap[i];
|
||||
if (!info) continue;
|
||||
|
||||
let rawValue = vals[i];
|
||||
|
||||
// Update calibration tracking
|
||||
if (calibrationEnabled) {
|
||||
observedMin[i] = Math.min(observedMin[i], rawValue);
|
||||
observedMax[i] = Math.max(observedMax[i], rawValue);
|
||||
|
||||
// Update calibration display
|
||||
const ui = uiRefs[i];
|
||||
if (ui && ui.calibStatus) {
|
||||
if (observedMin[i] !== Infinity && observedMax[i] !== -Infinity) {
|
||||
ui.calibStatus.textContent = `Range: ${observedMin[i]}-${observedMax[i]}`;
|
||||
}
|
||||
}
|
||||
|
||||
// Remap observed range to target range
|
||||
if (observedMin[i] !== Infinity && observedMax[i] !== -Infinity && observedMax[i] > observedMin[i]) {
|
||||
const observedRange = observedMax[i] - observedMin[i];
|
||||
const targetRange = info.max - info.min;
|
||||
const normalizedValue = (rawValue - observedMin[i]) / observedRange;
|
||||
rawValue = info.min + (normalizedValue * targetRange);
|
||||
}
|
||||
}
|
||||
|
||||
let v = clamp(rawValue, info.min, info.max);
|
||||
if (info.inverted) v = (info.min + info.max) - v;
|
||||
jointValues[i] = v;
|
||||
updateJointDisplay(i, v);
|
||||
}
|
||||
updateHandModel();
|
||||
} else {
|
||||
addLogMessage(`Received: ${line}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function connectToDevice() {
|
||||
try {
|
||||
port = await navigator.serial.requestPort();
|
||||
const baudRate = parseInt(baudRateSelect.value, 10) || 115200;
|
||||
await port.open({ baudRate });
|
||||
|
||||
keepReading = true;
|
||||
setConnectedUI(true);
|
||||
|
||||
statusIndicator.textContent = 'Status: Connected';
|
||||
statusIndicator.className = 'status connected';
|
||||
connectButton.disabled = true;
|
||||
disconnectButton.disabled = false;
|
||||
baudRateSelect.disabled = true;
|
||||
|
||||
addLogMessage(`Connected at ${baudRate} baud`);
|
||||
readSerialData();
|
||||
} catch (e) {
|
||||
console.error('Connect error:', e);
|
||||
addLogMessage(`Connection error: ${e.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function disconnectFromDevice() {
|
||||
try {
|
||||
keepReading = false;
|
||||
if (reader) {
|
||||
try { reader.cancel(); } catch {}
|
||||
}
|
||||
if (port) {
|
||||
await port.close();
|
||||
port = null;
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('Disconnect error:', e);
|
||||
addLogMessage(`Disconnection error: ${e.message}`);
|
||||
} finally {
|
||||
setConnectedUI(false);
|
||||
statusIndicator.textContent = 'Status: Disconnected';
|
||||
statusIndicator.className = 'status disconnected';
|
||||
connectButton.disabled = false;
|
||||
disconnectButton.disabled = true;
|
||||
baudRateSelect.disabled = false;
|
||||
addLogMessage('Disconnected');
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------- Three.js Scene --------------------
|
||||
function initThreeJS() {
|
||||
scene = new THREE.Scene();
|
||||
scene.background = new THREE.Color(0xf0f0f0);
|
||||
|
||||
camera = new THREE.PerspectiveCamera(
|
||||
75,
|
||||
canvasContainer.clientWidth / canvasContainer.clientHeight,
|
||||
0.1, 1000
|
||||
);
|
||||
camera.position.set(0, 15, 15);
|
||||
camera.lookAt(0, 0, 0);
|
||||
|
||||
renderer = new THREE.WebGLRenderer({ antialias: true });
|
||||
renderer.setSize(canvasContainer.clientWidth, canvasContainer.clientHeight);
|
||||
renderer.setPixelRatio(window.devicePixelRatio);
|
||||
canvasContainer.appendChild(renderer.domElement);
|
||||
|
||||
controls = new THREE.OrbitControls(camera, renderer.domElement);
|
||||
controls.enableDamping = true;
|
||||
controls.dampingFactor = 0.25;
|
||||
|
||||
const ambientLight = new THREE.AmbientLight(0x404040);
|
||||
scene.add(ambientLight);
|
||||
const dir1 = new THREE.DirectionalLight(0xffffff, 0.5);
|
||||
dir1.position.set(1, 1, 1);
|
||||
scene.add(dir1);
|
||||
const dir2 = new THREE.DirectionalLight(0xffffff, 0.3);
|
||||
dir2.position.set(-1, 1, -1);
|
||||
scene.add(dir2);
|
||||
|
||||
const gridHelper = new THREE.GridHelper(20, 20);
|
||||
scene.add(gridHelper);
|
||||
|
||||
createHandModel();
|
||||
window.addEventListener('resize', onWindowResize);
|
||||
animate();
|
||||
}
|
||||
|
||||
function createHandModel() {
|
||||
const palmMaterial = new THREE.MeshPhongMaterial({ color: 0xf5c396 });
|
||||
const fingerMaterial = new THREE.MeshPhongMaterial({ color: 0xf5c396 });
|
||||
const jointMaterial = new THREE.MeshPhongMaterial({ color: 0xe3a977 });
|
||||
|
||||
const palmGeometry = new THREE.BoxGeometry(7, 1, 8);
|
||||
hand.palm = new THREE.Mesh(palmGeometry, palmMaterial);
|
||||
hand.palm.position.set(0, 0, 0);
|
||||
hand.palm.rotation.x = Math.PI / 2; // hand vertical, palm facing forward
|
||||
scene.add(hand.palm);
|
||||
|
||||
const fingerWidth = 1, fingerHeight = 0.8;
|
||||
const fingerSegmentLengths = [3, 2, 1.5];
|
||||
const thumbSegmentLengths = [2, 2, 1.5];
|
||||
|
||||
const fingerBasePositions = [
|
||||
[ 3, 0, -2], // Thumb
|
||||
[ 1.5,-0.5,-4], // Index
|
||||
[ 0, -0.5,-4], // Middle
|
||||
[-1.5,-0.5,-4], // Ring
|
||||
[-3, -0.5,-4], // Pinky
|
||||
];
|
||||
const fingerBaseRot = [
|
||||
{ x:0, y:-Math.PI/3, z: Math.PI/3 }, // Thumb
|
||||
{ x:0, y:-Math.PI/48, z: 0 },
|
||||
{ x:0, y: Math.PI/48, z: 0 },
|
||||
{ x:0, y: Math.PI/32, z: 0 },
|
||||
{ x:0, y: Math.PI/24, z: 0 }
|
||||
];
|
||||
|
||||
for (let fIdx = 0; fIdx < 5; fIdx++) {
|
||||
const finger = { name:['Thumb','Index','Middle','Ring','Pinky'][fIdx], segments:[], joints:[] };
|
||||
const isThumb = fIdx === 0;
|
||||
const segLens = isThumb ? thumbSegmentLengths : fingerSegmentLengths;
|
||||
|
||||
finger.group = new THREE.Group();
|
||||
finger.group.position.set(...fingerBasePositions[fIdx]);
|
||||
finger.group.rotation.x = fingerBaseRot[fIdx].x;
|
||||
finger.group.rotation.y = fingerBaseRot[fIdx].y;
|
||||
finger.group.rotation.z = fingerBaseRot[fIdx].z;
|
||||
finger.group.userData.baseRot = {
|
||||
x:finger.group.rotation.x,
|
||||
y:finger.group.rotation.y,
|
||||
z:finger.group.rotation.z
|
||||
};
|
||||
hand.palm.add(finger.group);
|
||||
|
||||
let parent = finger.group;
|
||||
for (let s = 0; s < segLens.length; s++) {
|
||||
const segGroup = new THREE.Group();
|
||||
|
||||
const jGeom = new THREE.SphereGeometry(fingerWidth * 0.6, 8, 8);
|
||||
const joint = new THREE.Mesh(jGeom, jointMaterial);
|
||||
segGroup.add(joint);
|
||||
|
||||
const segGeom = new THREE.BoxGeometry(fingerWidth, fingerHeight, segLens[s]);
|
||||
const seg = new THREE.Mesh(segGeom, fingerMaterial);
|
||||
seg.position.z = -segLens[s] / 2;
|
||||
segGroup.add(seg);
|
||||
|
||||
parent.add(segGroup);
|
||||
|
||||
finger.segments.push(segGroup);
|
||||
finger.joints.push(joint);
|
||||
|
||||
if (s < segLens.length - 1) {
|
||||
const connector = new THREE.Group();
|
||||
connector.position.z = -segLens[s];
|
||||
segGroup.add(connector);
|
||||
parent = connector;
|
||||
}
|
||||
}
|
||||
|
||||
hand.fingers.push(finger);
|
||||
}
|
||||
|
||||
addFingerLabels();
|
||||
addHandLabel();
|
||||
}
|
||||
|
||||
function addFingerLabels() {
|
||||
const names = ['Thumb','Index','Middle','Ring','Pinky'];
|
||||
for (let i = 0; i < hand.fingers.length; i++) {
|
||||
const finger = hand.fingers[i];
|
||||
const canvas = document.createElement('canvas');
|
||||
const ctx = canvas.getContext('2d');
|
||||
canvas.width = 128; canvas.height = 32;
|
||||
ctx.fillStyle = '#ffffff'; ctx.fillRect(0,0,canvas.width,canvas.height);
|
||||
ctx.font = 'bold 16px Arial';
|
||||
ctx.fillStyle = '#000000';
|
||||
ctx.textAlign = 'center'; ctx.textBaseline = 'middle';
|
||||
ctx.fillText(names[i], canvas.width/2, canvas.height/2);
|
||||
|
||||
const texture = new THREE.CanvasTexture(canvas);
|
||||
const geom = new THREE.PlaneGeometry(2, 0.5);
|
||||
const mat = new THREE.MeshBasicMaterial({ map:texture, transparent:true, side:THREE.DoubleSide });
|
||||
const label = new THREE.Mesh(geom, mat);
|
||||
label.position.set(0, -1.5, -2);
|
||||
label.rotation.x = Math.PI / 2;
|
||||
finger.group.add(label);
|
||||
}
|
||||
}
|
||||
|
||||
function addHandLabel() {
|
||||
const canvas = document.createElement('canvas');
|
||||
const ctx = canvas.getContext('2d');
|
||||
canvas.width = 256; canvas.height = 64;
|
||||
ctx.fillStyle = '#ffffff'; ctx.fillRect(0,0,canvas.width,canvas.height);
|
||||
ctx.font = 'bold 24px Arial';
|
||||
ctx.fillStyle = '#000000';
|
||||
ctx.textAlign = 'center'; ctx.textBaseline = 'middle';
|
||||
ctx.fillText('RIGHT HAND (VERTICAL)', canvas.width/2, canvas.height/2);
|
||||
|
||||
const texture = new THREE.CanvasTexture(canvas);
|
||||
const geom = new THREE.PlaneGeometry(7, 1.75);
|
||||
const mat = new THREE.MeshBasicMaterial({ map:texture, transparent:true, side:THREE.DoubleSide });
|
||||
const label = new THREE.Mesh(geom, mat);
|
||||
label.position.set(0, -2, 0);
|
||||
label.rotation.x = Math.PI / 2;
|
||||
scene.add(label);
|
||||
}
|
||||
|
||||
function updateHandModel() {
|
||||
for (let i = 0; i < MAX_JOINTS; i++) {
|
||||
const info = fingerJointMap[i];
|
||||
if (!info) continue;
|
||||
const { finger, joint, type, min, max, angleMin, angleMax } = info;
|
||||
const raw = jointValues[i];
|
||||
const f = hand.fingers[finger];
|
||||
if (!f) continue;
|
||||
|
||||
const center = (min + max) / 2;
|
||||
let angle = 0;
|
||||
|
||||
if (type.includes('ABDUCTION')) {
|
||||
// symmetric around neutral
|
||||
const k = clamp((raw - center) / ((max - min) / 2), -1, 1);
|
||||
angle = angleMin + (k + 1) * 0.5 * (angleMax - angleMin);
|
||||
|
||||
const base = f.group.userData.baseRot || {x:0,y:0,z:0};
|
||||
if (finger === 0 && joint === 0) {
|
||||
// Thumb: abduction about Z (toward/away from palm)
|
||||
f.group.rotation.z = base.z + angle;
|
||||
} else {
|
||||
// Other fingers: side-to-side about Y
|
||||
f.group.rotation.y = base.y + angle;
|
||||
}
|
||||
} else if (type.includes('FLEXION')) {
|
||||
const isThumb = finger === 0;
|
||||
const isMCP = type === 'MCP_FLEXION';
|
||||
const isPIP = type === 'PIP_FLEXION';
|
||||
const positiveOnly = (isThumb && (type === 'MCP_FLEXION' || type === 'IP_FLEXION')) || (!isThumb && isPIP);
|
||||
|
||||
if (positiveOnly) {
|
||||
const t = raw <= center ? 0 : invLerp(center, max, raw); // 0..1
|
||||
angle = angleMin + t * (angleMax - angleMin); // 0..+limit
|
||||
} else {
|
||||
const k = clamp((raw - center) / ((max - min) / 2), -1, 1);
|
||||
angle = angleMin + (k + 1) * 0.5 * (angleMax - angleMin);
|
||||
}
|
||||
|
||||
if (isMCP) {
|
||||
// MCP flexion applies to the finger base group (same as abduction)
|
||||
const base = f.group.userData.baseRot || {x:0,y:0,z:0};
|
||||
f.group.rotation.x = base.x + angle;
|
||||
} else if (f.segments[joint]) {
|
||||
// PIP/DIP/IP flexion applies to individual segments
|
||||
f.segments[joint].rotation.x = angle;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------- Render Loop --------------------
|
||||
function onWindowResize() {
|
||||
camera.aspect = canvasContainer.clientWidth / canvasContainer.clientHeight;
|
||||
camera.updateProjectionMatrix();
|
||||
renderer.setSize(canvasContainer.clientWidth, canvasContainer.clientHeight);
|
||||
}
|
||||
|
||||
function animate() {
|
||||
requestAnimationFrame(animate);
|
||||
controls.update();
|
||||
renderer.render(scene, camera);
|
||||
}
|
||||
|
||||
// -------------------- Misc UI --------------------
|
||||
function addLogMessage(msg) {
|
||||
const el = document.createElement('div');
|
||||
el.textContent = msg;
|
||||
logContainer.appendChild(el);
|
||||
logContainer.scrollTop = logContainer.scrollHeight;
|
||||
while (logContainer.children.length > 100) {
|
||||
logContainer.removeChild(logContainer.firstChild);
|
||||
}
|
||||
}
|
||||
|
||||
// Camera view controls
|
||||
frontViewBtn?.addEventListener('click', () => { camera.position.set(0, 0, 20); camera.lookAt(0,0,0); controls.update(); });
|
||||
sideViewBtn?.addEventListener('click', () => { camera.position.set(20, 0, 0); camera.lookAt(0,0,0); controls.update(); });
|
||||
topViewBtn?.addEventListener('click', () => { camera.position.set(0, 20, 0); camera.lookAt(0,0,0); controls.update(); });
|
||||
resetViewBtn?.addEventListener('click', () => { camera.position.set(10,10,10); camera.lookAt(0,0,0); controls.update(); });
|
||||
|
||||
// Serial connect buttons
|
||||
connectButton?.addEventListener('click', connectToDevice);
|
||||
disconnectButton?.addEventListener('click', disconnectFromDevice);
|
||||
|
||||
// Web Serial support check
|
||||
if (!navigator.serial) {
|
||||
statusIndicator.textContent = 'Status: Web Serial API not supported in this browser';
|
||||
connectButton.disabled = true;
|
||||
addLogMessage('ERROR: Web Serial API is not supported in this browser. Try Chrome or Edge.');
|
||||
}
|
||||
|
||||
// -------------------- Boot --------------------
|
||||
initThreeJS();
|
||||
initializeJointElements();
|
||||
|
||||
// -------------------- Styles (inline) --------------------
|
||||
const styleElement = document.createElement('style');
|
||||
styleElement.textContent = `
|
||||
.joint-info { border-bottom: 1px solid #eee; padding: 8px 0; }
|
||||
.joint-name { font-weight: 600; margin-bottom: 4px; }
|
||||
.joint-value { font-size: 12px; color: #333; margin-bottom: 4px; }
|
||||
.bar-container { width: 100%; height: 8px; background: #ddd; border-radius: 4px; overflow: hidden; }
|
||||
.bar { height: 100%; width: 0%; background: #4caf50; }
|
||||
.joint-slider { width: 100%; margin: 6px 0; }
|
||||
.invert-toggle { display: inline-flex; align-items: center; gap: 6px; margin-top: 4px; font-size: 12px; color: #555; }
|
||||
.limits-row { display: flex; align-items: center; gap: 6px; margin-top: 6px; flex-wrap: wrap; }
|
||||
.limit-label { font-size: 11px; color: #666; }
|
||||
.limit-num { width: 60px; }
|
||||
.calib-row { display: flex; align-items: center; gap: 8px; margin-top: 4px; }
|
||||
.calib-btn { padding: 2px 6px; font-size: 11px; background: #f44336; color: white; border: none; border-radius: 3px; cursor: pointer; }
|
||||
.calib-btn:hover { background: #d32f2f; }
|
||||
.calib-status { font-size: 11px; color: #666; }
|
||||
.status.connected { color: #0a0; }
|
||||
.status.disconnected { color: #a00; }
|
||||
`;
|
||||
document.head.appendChild(styleElement);
|
||||
@@ -0,0 +1,131 @@
|
||||
# Isaac Teleop → SO-101
|
||||
|
||||
Teleoperate an SO-101/SO-100 follower arm — and record LeRobot datasets — with NVIDIA
|
||||
[Isaac Teleop](https://github.com/NVIDIA/IsaacTeleop). Two input devices ship today:
|
||||
|
||||
- **XR (VR) controller** (`--teleop.type=xr_controller`) — the controller's grip pose drives the
|
||||
end-effector through a squeeze-to-engage clutch and LeRobot's Cartesian IK pipeline; the analog
|
||||
trigger drives the gripper.
|
||||
- **SO-101 leader arm** (`--teleop.type=so101_leader`) — a back-drivable leader arm mirrored 1:1
|
||||
onto the follower via Isaac Teleop's native `so101_leader` plugin (no clutch, no IK).
|
||||
|
||||
The full narrative guide (how the clutch works, CloudXR setup, headset pairing, tuning, and
|
||||
troubleshooting) is in the [LeRobot docs](https://huggingface.co/docs/lerobot/isaac_teleop)
|
||||
(source: `docs/source/isaac_teleop.mdx`). This README is the canonical install and usage
|
||||
reference.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Linux workstation (see NVIDIA's
|
||||
[system requirements](https://nvidia.github.io/IsaacTeleop/main/references/requirements.html)
|
||||
for supported OS/GPU/headset combinations; `isaacteleop` publishes Linux wheels only).
|
||||
- An SO-101 (or SO-100) follower arm, calibrated with `lerobot-calibrate`.
|
||||
- For the XR device: a CloudXR-capable headset (e.g. Quest 3, Pico 4, Apple Vision Pro) on the
|
||||
same network.
|
||||
- For the leader device: a second, back-drivable SO-101 leader arm and the `so101_leader` plugin
|
||||
binary built from the Isaac Teleop source tree (see
|
||||
[Build from source](https://nvidia.github.io/IsaacTeleop/main/getting_started/build_from_source/index.html)).
|
||||
|
||||
## Installation
|
||||
|
||||
This example lives in the LeRobot repository and is not part of the `lerobot` pip package, so
|
||||
work from a source checkout. From the repo root:
|
||||
|
||||
```bash
|
||||
# LeRobot with the extras this example uses:
|
||||
# feetech - SO-101 serial motor bus
|
||||
# kinematics - Placo IK solver (XR controller path)
|
||||
# dataset - dataset recording (record.py)
|
||||
# huggingface_hub >= 1.5 is needed by the automatic URDF fetch (Buckets API).
|
||||
uv pip install -e ".[feetech,kinematics,dataset]" "huggingface_hub>=1.5"
|
||||
|
||||
# Isaac Teleop from public PyPI. `cloudxr` brings the CloudXR runtime bindings;
|
||||
# `retargeters-lite` is the scipy-based retargeter path that resolves on both
|
||||
# x86_64 and ARM (the full `retargeters` extra does not resolve on aarch64).
|
||||
uv pip install "isaacteleop[cloudxr,retargeters-lite]~=1.3.131" "scipy>=1.14"
|
||||
|
||||
# Optional, x86_64 only: the full retargeter stack.
|
||||
uv pip install "isaacteleop[retargeters]~=1.3.131"
|
||||
```
|
||||
|
||||
One-time CloudXR EULA (the auto-launch prompts on stdin and would hang on a headless machine):
|
||||
|
||||
```bash
|
||||
python -m isaacteleop.cloudxr --accept-eula
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
Run everything from the repo root with `python -m` so the `examples` package resolves.
|
||||
|
||||
### Teleoperate — XR controller
|
||||
|
||||
```bash
|
||||
python -m examples.isaac_teleop_to_so101.teleoperate \
|
||||
--robot.type=so101_follower \
|
||||
--robot.port=/dev/ttyACM0 \
|
||||
--robot.id=so101_follower_arm \
|
||||
--teleop.type=xr_controller
|
||||
```
|
||||
|
||||
On startup the script launches the CloudXR runtime (~30 s), prints the workstation IP to enter in
|
||||
the headset's CloudXR web client, waits for the controllers to stream, slews the arm to a reset
|
||||
pose (`--reset_to_origin=false` to skip), and then: **hold the squeeze/grip** to engage, move the
|
||||
controller to drive the arm, pull the trigger to close the gripper. Releasing the squeeze freezes
|
||||
the arm. The SO-101 URDF is fetched automatically from the `lerobot/robot-urdfs` Hugging Face
|
||||
bucket into the LeRobot cache on first run.
|
||||
|
||||
To customize the reset pose: back-drive the arm to the pose you want, then
|
||||
|
||||
```bash
|
||||
python -m examples.isaac_teleop_to_so101.override_reset_pose --port /dev/ttyACM0 --id so101_follower_arm
|
||||
```
|
||||
|
||||
which writes it to `HF_LEROBOT_HOME/reset_poses/<robot.name>/<robot.id>.json`; runs with the same
|
||||
`--robot.id` use it automatically.
|
||||
|
||||
### Teleoperate — SO-101 leader arm
|
||||
|
||||
```bash
|
||||
python -m examples.isaac_teleop_to_so101.teleoperate \
|
||||
--robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm \
|
||||
--teleop.type=so101_leader --teleop.port=/dev/ttyACM1 --teleop.id=so101_leader_arm \
|
||||
--launch_plugin=/path/to/IsaacTeleop/install/plugins/so101_leader/so101_leader_plugin
|
||||
```
|
||||
|
||||
The follower is first slewed to the leader's pose over `--align_duration` seconds
|
||||
(`--align=false` to skip), then mirrors it 1:1. The plugin reuses the serial leader's calibration
|
||||
(`HF_LEROBOT_CALIBRATION/teleoperators/so_leader/<teleop.id>.json`).
|
||||
|
||||
### Record a dataset
|
||||
|
||||
`record.py` takes the same `--robot.*`/`--teleop.*`/loop flags plus `lerobot-record`-style
|
||||
`--dataset.*` flags:
|
||||
|
||||
```bash
|
||||
python -m examples.isaac_teleop_to_so101.record \
|
||||
--robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm \
|
||||
--teleop.type=xr_controller \
|
||||
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
|
||||
--dataset.repo_id=<hf_user>/<dataset_name> \
|
||||
--dataset.single_task="Pick up the cube" \
|
||||
--dataset.num_episodes=3 --dataset.episode_time_s=20 --dataset.reset_time_s=5
|
||||
```
|
||||
|
||||
Keyboard shortcuts (terminal-first, so they work over SSH): **Right/n** end episode early,
|
||||
**Left/r** re-record, **Esc/q** stop after the current episode.
|
||||
|
||||
Run either script with `--help` for all flags.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
isaac_teleop/ device library: session lifecycle (base.py), XRController,
|
||||
SO101LeaderArm, Clutch, configs, and the XR→IK processor step
|
||||
common.py shared loop infra: device bundles, clutch/IK pipeline wiring,
|
||||
reset/align slews, URDF fetch, keyboard listener
|
||||
teleoperate.py teleoperation CLI (device selected via --teleop.type)
|
||||
record.py dataset-recording CLI (same device selection + --dataset.*)
|
||||
override_reset_pose.py save the current joints as the per-arm reset pose
|
||||
default.env CloudXR device-profile overrides passed to the launcher
|
||||
```
|
||||
@@ -0,0 +1,17 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Isaac Teleop -> SO-101 example package."""
|
||||
@@ -0,0 +1,650 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Shared device + control-loop infrastructure for the Isaac Teleop -> SO-101 examples.
|
||||
|
||||
Consumed by ``teleoperate.py`` and ``record.py``, which both build a per-device
|
||||
:class:`Device` bundle and run the same loop: read -> (maybe command) -> hold-when-idle ->
|
||||
sleep. A :class:`Device` bundles three closures: ``compute(obs) -> RobotAction | None``
|
||||
(``None`` = hold at the measured pose while idle), ``startup``, and ``cleanup``. The devices:
|
||||
|
||||
* ``xr_controller`` — a thin :class:`XRController` whose raw grip pose an in-loop
|
||||
:class:`Clutch` turns into an EE target for LeRobot's Cartesian IK pipeline.
|
||||
* ``so101_leader`` — a back-drivable leader arm mirrored 1:1 into the follower.
|
||||
|
||||
Requires the ``isaacteleop`` package and an OpenXR runtime (install instructions in this
|
||||
folder's ``README.md``). User-facing guide: ``docs/source/isaac_teleop.mdx``.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from contextlib import suppress
|
||||
from dataclasses import dataclass
|
||||
from importlib.resources import files
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
|
||||
import numpy as np
|
||||
|
||||
from lerobot.model.kinematics import RobotKinematics
|
||||
from lerobot.processor import (
|
||||
RobotProcessorPipeline,
|
||||
robot_action_observation_to_transition,
|
||||
transition_to_robot_action,
|
||||
)
|
||||
from lerobot.robots import RobotConfig, make_robot_from_config
|
||||
from lerobot.robots.so_follower import SOFollowerConfig # noqa: F401 (registers so101_follower)
|
||||
from lerobot.robots.so_follower.robot_kinematic_processor import (
|
||||
EEBoundsAndSafety,
|
||||
InverseKinematicsEEToJoints,
|
||||
)
|
||||
from lerobot.types import RobotAction, RobotObservation
|
||||
from lerobot.utils.constants import HF_LEROBOT_CALIBRATION, HF_LEROBOT_HOME, TELEOPERATORS
|
||||
from lerobot.utils.robot_utils import precise_sleep
|
||||
|
||||
from .isaac_teleop import (
|
||||
Clutch,
|
||||
IsaacTeleopConfig,
|
||||
MapXRControllerActionToRobotAction,
|
||||
SO101LeaderArm,
|
||||
SO101LeaderArmConfig,
|
||||
XRController,
|
||||
)
|
||||
|
||||
# Fixed rate [Hz] for the teleoperate loop and the pre-loop slews / connect-wait poll sleeps.
|
||||
FPS = 30
|
||||
|
||||
# CloudXR device-profile env file passed to the launcher (see default.env in this package).
|
||||
CLOUDXR_ENV_FILE = str(files(__package__) / "default.env")
|
||||
|
||||
|
||||
class LoopConfig(Protocol):
|
||||
"""Structural type for the loop/launch knobs ``build_device`` and the ``setup_*`` read.
|
||||
|
||||
Both ``TeleoperateConfig`` and ``RecordConfig`` satisfy it, keeping ``common`` decoupled
|
||||
from either entry point's concrete config.
|
||||
"""
|
||||
|
||||
teleop: IsaacTeleopConfig
|
||||
robot: RobotConfig
|
||||
launch_plugin: str | None
|
||||
reset_to_origin: bool
|
||||
reset_duration: float
|
||||
align: bool
|
||||
align_duration: float
|
||||
|
||||
|
||||
# Per-device bundle consumed by the shared loop. ``compute`` returns None to mean
|
||||
# "idle -> hold at the measured pose"; ``startup`` warms up; ``cleanup`` reaps/disconnects.
|
||||
@dataclass(frozen=True)
|
||||
class Device:
|
||||
compute: Callable[[RobotObservation | None], RobotAction | None]
|
||||
startup: Callable[[], None]
|
||||
cleanup: Callable[[], None]
|
||||
|
||||
|
||||
def hold_action(obs: RobotObservation, motor_names: list[str]) -> dict[str, float]:
|
||||
"""Re-send the measured joints — the explicit hold when a device is idle."""
|
||||
return {f"{name}.pos": float(obs[f"{name}.pos"]) for name in motor_names}
|
||||
|
||||
|
||||
class HoldLatch:
|
||||
"""Resolve the per-frame action, holding one LATCHED pose while the device is idle.
|
||||
|
||||
Re-sending the freshly measured joints on every idle frame would ratchet the arm
|
||||
downward: under gravity the P-only servo settles below its goal by a steady-state
|
||||
error, so each re-command of the measurement lowers the goal by that error again.
|
||||
Latching the target once on the active->idle transition holds a fixed pose instead.
|
||||
"""
|
||||
|
||||
def __init__(self, motor_names: list[str]):
|
||||
self._motor_names = motor_names
|
||||
self._held: dict[str, float] | None = None
|
||||
|
||||
def resolve(self, action: RobotAction | None, obs: RobotObservation) -> RobotAction:
|
||||
"""Pass through an active action (clearing the latch); latch + hold when idle."""
|
||||
if action is not None:
|
||||
self._held = None
|
||||
return action
|
||||
if self._held is None:
|
||||
self._held = hold_action(obs, self._motor_names)
|
||||
return self._held
|
||||
|
||||
|
||||
def slew(
|
||||
robot,
|
||||
motor_names: list[str],
|
||||
target_fn: Callable[[], dict[str, float]],
|
||||
duration_s: float,
|
||||
) -> None:
|
||||
"""Linearly slew all joints from their current measured pose toward a target.
|
||||
|
||||
``target_fn`` is called EACH step, so the leader can pass a live re-read (landing on its
|
||||
current pose at ``alpha == 1`` for a continuous handoff) while XR passes a constant.
|
||||
"""
|
||||
obs = robot.get_observation()
|
||||
start = {name: float(obs[f"{name}.pos"]) for name in motor_names}
|
||||
n_steps = max(1, int(duration_s * FPS))
|
||||
for step in range(1, n_steps + 1):
|
||||
alpha = step / n_steps
|
||||
target = target_fn()
|
||||
action = {f"{name}.pos": start[name] + alpha * (target[name] - start[name]) for name in motor_names}
|
||||
robot.send_action(action)
|
||||
precise_sleep(1.0 / FPS)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# XR controller device
|
||||
# ============================================================================
|
||||
|
||||
# Per-frame EE rate limit [m]. With raise_on_jump=False, EEBoundsAndSafety clamps an
|
||||
# over-limit step instead of raising, absorbing a tracking glitch as one slow frame. At
|
||||
# FPS=30, 0.1 m/frame caps EE speed at ~3 m/s. (end_effector_bounds clips the absolute target.)
|
||||
MAX_EE_STEP_M = 0.1
|
||||
|
||||
# Soft-orientation IK weight: small but nonzero so the wrist follows the hand while position
|
||||
# dominates (the 5-DOF SO-101 cannot realize an arbitrary orientation). 0.0 = position-only.
|
||||
IK_ORIENTATION_WEIGHT = 0.01
|
||||
|
||||
|
||||
def _ensure_so101_urdf() -> str:
|
||||
"""Return the cached SO-101 URDF path, fetching the ``so101`` folder (URDF + meshes) from
|
||||
the public ``lerobot/robot-urdfs`` HF bucket into the LeRobot cache on first use."""
|
||||
dest_dir = HF_LEROBOT_HOME / "robot-urdfs" / "so101"
|
||||
urdf_path = dest_dir / "so101_new_calib.urdf"
|
||||
# Completeness marker written only after a FULL sync: the URDF file alone is not a
|
||||
# completeness signal (an interrupted first sync can leave the meshes it references
|
||||
# missing, which the URDF's mere existence would then hide forever). Re-syncing is
|
||||
# idempotent and repairs a partial cache; delete the folder to force a re-download.
|
||||
marker = dest_dir / ".sync_complete"
|
||||
if not marker.exists():
|
||||
from huggingface_hub import sync_bucket
|
||||
|
||||
sync_bucket("hf://buckets/lerobot/robot-urdfs/so101", str(dest_dir), quiet=True)
|
||||
marker.touch()
|
||||
return str(urdf_path)
|
||||
|
||||
|
||||
# Default duration [s] for the startup reset-to-origin slew.
|
||||
RESET_DURATION_S = 5.0
|
||||
|
||||
# Optional cached file written by override_reset_pose.py. When present it takes priority over RESET_ORIGIN_DEG.
|
||||
RESET_POSE_FILE = str(HF_LEROBOT_HOME / "reset_poses" / "{robot_name}" / "{robot_id}.json")
|
||||
|
||||
# Reset target in each motor's native units (arm joints in degrees, gripper RANGE_0_100,
|
||||
# 100 = open). An empirically comfortable pose (elbow/wrist bent) avoiding the singularity of
|
||||
# a fully-extended arm; assumes standard calibration. Override per-arm via override_reset_pose.py.
|
||||
RESET_ORIGIN_DEG: dict[str, float] = {
|
||||
"shoulder_pan": -4.0,
|
||||
"shoulder_lift": -103.0,
|
||||
"elbow_flex": 97.0,
|
||||
"wrist_flex": 78.0,
|
||||
"wrist_roll": -65.0,
|
||||
"gripper": 0.0,
|
||||
}
|
||||
|
||||
|
||||
def _load_reset_target(reset_pose_file: Path, motor_names: list[str]) -> dict[str, float]:
|
||||
"""Return reset targets: the saved reset pose if present, else RESET_ORIGIN_DEG."""
|
||||
if reset_pose_file.exists():
|
||||
saved = json.loads(reset_pose_file.read_text())
|
||||
# Fill any missing motors from the fallback dict.
|
||||
return {name: float(saved.get(name, RESET_ORIGIN_DEG.get(name, 0.0))) for name in motor_names}
|
||||
return {name: RESET_ORIGIN_DEG.get(name, 0.0) for name in motor_names}
|
||||
|
||||
|
||||
# CloudXR web client URL opened in the headset (Isaac Teleop quick start, step 5).
|
||||
_CLOUDXR_WEB_CLIENT_URL = "https://nvidia.github.io/IsaacTeleop/client"
|
||||
# WSS-proxy / self-signed-cert port the operator accepts in-browser before connecting.
|
||||
_CLOUDXR_WSS_PORT = 48322
|
||||
# How often to re-print the connection hint while waiting for the headset [s].
|
||||
_XR_CONNECT_REMINDER_S = 15.0
|
||||
# Virtual / bridge / USB-gadget interfaces a headset can't reach over the network — skip
|
||||
# by name prefix (``docker0``, compose ``br-*``, ``veth*``, libvirt ``virbr*``, and the
|
||||
# Tegra USB device-mode bridge ``l4tbr0``).
|
||||
_SKIP_IFACE_PREFIXES = ("docker", "br-", "veth", "virbr", "l4tbr")
|
||||
|
||||
|
||||
def _primary_ipv4() -> str | None:
|
||||
"""The workstation's primary outbound IPv4, via the UDP-socket trick (``connect()`` on a
|
||||
datagram socket selects the egress interface without sending packets)."""
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
|
||||
try:
|
||||
s.connect(("8.8.8.8", 80))
|
||||
return s.getsockname()[0]
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def _candidate_ipv4s() -> list[tuple[str, str]]:
|
||||
"""Return ``[(interface, ipv4), ...]`` the headset might reach this workstation at.
|
||||
|
||||
Lists each interface's IPv4 via ``psutil`` (dropping loopback, link-local, and the
|
||||
virtual/bridge interfaces in ``_SKIP_IFACE_PREFIXES``), primary outbound first. Falls
|
||||
back to just the primary IP when ``psutil`` is unavailable.
|
||||
"""
|
||||
primary = _primary_ipv4()
|
||||
found: list[tuple[str, str]] = []
|
||||
try:
|
||||
import psutil
|
||||
|
||||
for iface, addrs in psutil.net_if_addrs().items():
|
||||
if iface.startswith(_SKIP_IFACE_PREFIXES):
|
||||
continue
|
||||
for addr in addrs:
|
||||
if addr.family != socket.AF_INET:
|
||||
continue
|
||||
ip = addr.address
|
||||
if ip.startswith("127.") or ip.startswith("169.254."):
|
||||
continue
|
||||
found.append((iface, ip))
|
||||
except Exception:
|
||||
if primary:
|
||||
found.append(("default", primary))
|
||||
found.sort(key=lambda t: t[1] != primary) # primary outbound interface first
|
||||
return found
|
||||
|
||||
|
||||
def _print_xr_connect_help() -> None:
|
||||
"""Print how to connect the headset to this workstation over CloudXR."""
|
||||
ips = _candidate_ipv4s()
|
||||
print("\n" + "=" * 76)
|
||||
print("Connect your XR headset to this workstation over NVIDIA CloudXR:")
|
||||
print(f" 1. In the headset, open the CloudXR web client: {_CLOUDXR_WEB_CLIENT_URL}")
|
||||
print(" 2. Enter this workstation's IP address:")
|
||||
if ips:
|
||||
for iface, ip in ips:
|
||||
print(f" {ip:<15} ({iface})")
|
||||
if len(ips) > 1:
|
||||
print(" (use the address on the same network as your headset)")
|
||||
else:
|
||||
print(" <could not determine — check `hostname -I` / `ip addr`>")
|
||||
print(f" 3. Accept the self-signed cert at https://<that-ip>:{_CLOUDXR_WSS_PORT}/ , then Connect.")
|
||||
print("=" * 76 + "\n")
|
||||
|
||||
|
||||
def _wait_for_xr_controller(teleop_device: XRController) -> None:
|
||||
"""Block until the XR controller is tracked, polling ``get_action()`` and re-printing a
|
||||
reminder every ``_XR_CONNECT_REMINDER_S``. User-paced; ``Ctrl-C`` aborts (no hard timeout).
|
||||
"""
|
||||
_print_xr_connect_help()
|
||||
print("Waiting for the headset controllers to start streaming… (Ctrl-C to abort)")
|
||||
last_reminder = time.time()
|
||||
while True:
|
||||
teleop_device.get_action() # steps the session; updates is_tracking
|
||||
if teleop_device.is_tracking:
|
||||
print("Headset connected — controllers are streaming.")
|
||||
return
|
||||
if time.time() - last_reminder >= _XR_CONNECT_REMINDER_S:
|
||||
print("…still waiting for the headset to connect (Ctrl-C to abort).")
|
||||
last_reminder = time.time()
|
||||
time.sleep(1.0 / FPS)
|
||||
|
||||
|
||||
def setup_xr(cfg: LoopConfig, robot, motor_names: list[str]) -> Device:
|
||||
"""Build the XR controller device bundle (clutch + soft-orientation IK pipeline)."""
|
||||
kinematics_solver = RobotKinematics(
|
||||
urdf_path=_ensure_so101_urdf(),
|
||||
target_frame_name="gripper_frame_link",
|
||||
joint_names=motor_names,
|
||||
)
|
||||
|
||||
teleop_config = cfg.teleop # XRControllerConfig (selected via --teleop.type=xr_controller)
|
||||
teleop_device = XRController(teleop_config)
|
||||
|
||||
# The clutch (below) turns the raw grip pose into an absolute base-frame ee_pose; this
|
||||
# pipeline maps it to joint targets: rename -> bounds/rate-limit -> IK.
|
||||
xr_to_robot_joints_processor = RobotProcessorPipeline[tuple[RobotAction, RobotObservation], RobotAction](
|
||||
steps=[
|
||||
MapXRControllerActionToRobotAction(),
|
||||
# raise_on_jump=False: an over-limit step (e.g. a tracking glitch) is clamped +
|
||||
# warned instead of raised, since a crash mid-loop would leave the arm uncontrolled.
|
||||
# z floor 0.0 keeps a stray target above the table; x/y stay at a loose [-1,1]m box.
|
||||
EEBoundsAndSafety(
|
||||
end_effector_bounds={"min": [-1.0, -1.0, 0.0], "max": [1.0, 1.0, 1.0]},
|
||||
max_ee_step_m=MAX_EE_STEP_M,
|
||||
raise_on_jump=False,
|
||||
),
|
||||
# initial_guess_current_joints=False: warm-start from the previous IK solution so
|
||||
# the joint trajectory stays continuous frame-to-frame.
|
||||
InverseKinematicsEEToJoints(
|
||||
kinematics=kinematics_solver,
|
||||
motor_names=motor_names,
|
||||
initial_guess_current_joints=False,
|
||||
orientation_weight=IK_ORIENTATION_WEIGHT,
|
||||
),
|
||||
],
|
||||
to_transition=robot_action_observation_to_transition,
|
||||
to_output=transition_to_robot_action,
|
||||
)
|
||||
|
||||
# The clutch is built in startup() (after the optional reset slew, seeded from the
|
||||
# post-slew MEASURED pose) and shared with compute() via nonlocal.
|
||||
clutch: Clutch | None = None
|
||||
prev_enabled = False
|
||||
|
||||
def startup() -> None:
|
||||
nonlocal clutch
|
||||
# Connect and wait for the operator to don the headset BEFORE moving the arm, so the
|
||||
# reset slew happens while they are watching in VR.
|
||||
teleop_device.connect()
|
||||
if not teleop_device.is_connected:
|
||||
raise ValueError("Teleop is not connected!")
|
||||
_wait_for_xr_controller(teleop_device)
|
||||
|
||||
if cfg.reset_to_origin:
|
||||
reset_pose_file = Path(RESET_POSE_FILE.format(robot_name=robot.name, robot_id=robot.id))
|
||||
target = _load_reset_target(reset_pose_file, motor_names)
|
||||
source = str(reset_pose_file) if reset_pose_file.exists() else "hardcoded defaults"
|
||||
print(f"Reset target source: {source}")
|
||||
print(f"Resetting to origin over {cfg.reset_duration:.1f} s…")
|
||||
slew(robot, motor_names, lambda: target, cfg.reset_duration)
|
||||
print("Reset complete.")
|
||||
|
||||
# Seed the clutch home from the arm's measured pose (FK of the current joints) so the
|
||||
# first engage is jump-free, whether or not a reset slew ran.
|
||||
obs0 = robot.get_observation()
|
||||
q_measured_deg = np.array([float(obs0[f"{name}.pos"]) for name in motor_names], dtype=float)
|
||||
home_base_T_ee = kinematics_solver.forward_kinematics(q_measured_deg) # noqa: N806
|
||||
clutch = Clutch(home_base_T_ee)
|
||||
|
||||
print("Starting teleop loop. Squeeze and move the controller to teleoperate the robot...")
|
||||
|
||||
def compute(robot_obs: RobotObservation | None) -> RobotAction | None:
|
||||
nonlocal prev_enabled
|
||||
if clutch is None: # set in startup(), which runs before compute()
|
||||
raise RuntimeError("compute() called before startup(); the clutch is not initialized")
|
||||
xr_action = teleop_device.get_action()
|
||||
grip_pos = np.asarray(xr_action["grip_pos"], dtype=float)
|
||||
grip_quat = np.asarray(xr_action["grip_quat"], dtype=float)
|
||||
squeeze = float(xr_action["squeeze"])
|
||||
trigger = float(xr_action["trigger"])
|
||||
enabled = squeeze > teleop_config.clutch_threshold
|
||||
|
||||
# On the engage edge, latch the clutch home at the arm's MEASURED EE pose (FK of
|
||||
# the live joints) and the controller origin so the per-frame delta starts at zero.
|
||||
# Latching the last commanded pose instead would snap the arm back to it at full
|
||||
# servo speed if the arm moved while disengaged (gravity sag, external contact).
|
||||
is_engage_frame = enabled and not prev_enabled
|
||||
if is_engage_frame:
|
||||
q_measured = np.array([float(robot_obs[f"{name}.pos"]) for name in motor_names], dtype=float)
|
||||
measured_base_T_ee = kinematics_solver.forward_kinematics(q_measured) # noqa: N806
|
||||
clutch.engage(grip_pos, grip_quat, measured_base_T_ee=measured_base_T_ee)
|
||||
# Re-anchor the pipeline state at the measured pose as well: EEBoundsAndSafety's
|
||||
# rate limiter and the IK warm start otherwise still reference the stale
|
||||
# pre-disengage command and would fight the fresh home for several frames.
|
||||
xr_to_robot_joints_processor.reset()
|
||||
prev_enabled = enabled
|
||||
|
||||
# SAFETY GATE: command the robot ONLY while the clutch is engaged; otherwise return
|
||||
# None so the loop holds the measured joints (releasing the clutch freezes the arm).
|
||||
if not enabled:
|
||||
return None
|
||||
|
||||
# Rebase the raw grip pose onto the EE, then run the pipeline. closedness = trigger.
|
||||
ee_pos, ee_quat = clutch.rebase(grip_pos, grip_quat)
|
||||
ee_action = {
|
||||
"ee_pose": np.concatenate([ee_pos, ee_quat]).astype(np.float32),
|
||||
"closedness": trigger,
|
||||
}
|
||||
return xr_to_robot_joints_processor((ee_action, robot_obs))
|
||||
|
||||
return Device(compute=compute, startup=startup, cleanup=teleop_device.disconnect)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# SO-101 leader arm device
|
||||
# ============================================================================
|
||||
|
||||
# Default duration [s] for the startup alignment slew (follower current -> leader first pose).
|
||||
ALIGN_DURATION_S = 3.0
|
||||
|
||||
# How long to wait for the leader plugin to start streaming before aligning / looping.
|
||||
LEADER_WARMUP_TIMEOUT_S = 20.0
|
||||
|
||||
# The plugin converts the leader's servo ticks to radians, so it reuses the serial SO-101
|
||||
# leader's calibration, stored by lerobot-calibrate under SO101Leader.name == "so_leader".
|
||||
SO_LEADER_CALIBRATION_NAME = "so_leader"
|
||||
|
||||
|
||||
def _leader_calibration_path(cfg: LoopConfig) -> Path | None:
|
||||
"""Infer the calibration JSON the launched plugin should read, or None.
|
||||
|
||||
Path convention: ``HF_LEROBOT_CALIBRATION / teleoperators / so_leader / {--teleop.id}.json``
|
||||
(or ``--teleop.calibration_dir`` if set). Returns None (plugin falls back to defaults) when
|
||||
it does not exist, warning if an id was given, or when no ``--teleop.id`` is set.
|
||||
"""
|
||||
if not cfg.teleop.id:
|
||||
return None
|
||||
calib_dir = cfg.teleop.calibration_dir or (
|
||||
HF_LEROBOT_CALIBRATION / TELEOPERATORS / SO_LEADER_CALIBRATION_NAME
|
||||
)
|
||||
calib_path = Path(calib_dir) / f"{cfg.teleop.id}.json"
|
||||
if calib_path.is_file():
|
||||
return calib_path
|
||||
print(
|
||||
f"WARNING: no leader calibration at {calib_path}; the plugin will use built-in defaults. "
|
||||
f"Calibrate with the serial leader (`lerobot-calibrate --teleop.type=so101_leader "
|
||||
f"--teleop.id={cfg.teleop.id}`) or the plugin's `calibrate` subcommand."
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _wait_for_leader(teleop: SO101LeaderArm, timeout_s: float) -> dict[str, float]:
|
||||
"""Poll the leader until it streams a live frame; return that frame's ``{joint}.pos``.
|
||||
|
||||
Raises ``SystemExit`` if no live frame arrives within ``timeout_s`` (plugin not pushing,
|
||||
wrong ``--teleop.collection_id``, or CloudXR not up).
|
||||
"""
|
||||
print(f"Waiting up to {timeout_s:.0f}s for the so101_leader plugin to stream…")
|
||||
deadline = time.time() + timeout_s
|
||||
while time.time() < deadline:
|
||||
action = teleop.get_action()
|
||||
if teleop.is_tracking:
|
||||
print("Leader is streaming.")
|
||||
return action
|
||||
time.sleep(1.0 / FPS)
|
||||
raise SystemExit(
|
||||
f"FAILED: leader did not stream within {timeout_s:.0f}s. Is the so101_leader plugin "
|
||||
"running and pushing (check --teleop.collection_id)? Is CloudXR up?"
|
||||
)
|
||||
|
||||
|
||||
def _maybe_launch_plugin(cfg: LoopConfig) -> subprocess.Popen | None:
|
||||
"""Spawn the so101_leader plugin if ``--launch_plugin <path>`` was given (after connect())."""
|
||||
if cfg.launch_plugin is None:
|
||||
return None
|
||||
if not Path(cfg.launch_plugin).exists():
|
||||
raise SystemExit(
|
||||
f"plugin binary not found: {cfg.launch_plugin} (build it in the IsaacTeleop repo first)"
|
||||
)
|
||||
leader_port = cfg.teleop.port # SO101LeaderArmConfig.port, forwarded to the plugin
|
||||
backend = f"leader on {leader_port}" if leader_port else "synthetic trajectory"
|
||||
print(f"launching plugin: {cfg.launch_plugin} ({backend})")
|
||||
# Positional args: [device_path] [collection_id] [calibration_file]. Empty device_path ->
|
||||
# synthetic backend. Calibration (only real hardware needs it) is appended when a port is set.
|
||||
argv = [cfg.launch_plugin, leader_port, cfg.teleop.collection_id]
|
||||
if leader_port:
|
||||
calib_path = _leader_calibration_path(cfg)
|
||||
if calib_path is not None:
|
||||
argv.append(str(calib_path))
|
||||
print(f" leader calibration: {calib_path}")
|
||||
# Spawned after connect() so it inherits the CloudXR runtime env (XR_RUNTIME_JSON, ...).
|
||||
proc = subprocess.Popen(argv)
|
||||
time.sleep(1.5) # let it create its OpenXR session and start pushing
|
||||
return proc
|
||||
|
||||
|
||||
def setup_leader(cfg: LoopConfig, robot, motor_names: list[str]) -> Device:
|
||||
"""Build the SO-101 leader arm device bundle (1:1 joint mirror)."""
|
||||
teleop_config = cfg.teleop # SO101LeaderArmConfig (selected via --teleop.type=so101_leader)
|
||||
teleop = SO101LeaderArm(teleop_config)
|
||||
|
||||
plugin_proc: subprocess.Popen | None = None
|
||||
|
||||
def startup() -> None:
|
||||
nonlocal plugin_proc
|
||||
# connect() auto-launches CloudXR (unless opted out); spawn the plugin AFTER so it
|
||||
# inherits the runtime env. The plugin is reaped in cleanup().
|
||||
teleop.connect()
|
||||
plugin_proc = _maybe_launch_plugin(cfg)
|
||||
|
||||
if not teleop.is_connected:
|
||||
raise ValueError("Teleop is not connected!")
|
||||
|
||||
# Block until the leader streams a live frame (clear error if it never does).
|
||||
_wait_for_leader(teleop, LEADER_WARMUP_TIMEOUT_S)
|
||||
|
||||
if cfg.align:
|
||||
print(f"Aligning follower to leader over {cfg.align_duration:.1f}s…")
|
||||
|
||||
# Re-read the live leader pose once per step so alpha=1 lands on its current pose
|
||||
# from a single coherent frame.
|
||||
def _leader_target() -> dict[str, float]:
|
||||
leader_now = teleop.get_action()
|
||||
return {name: float(leader_now[f"{name}.pos"]) for name in motor_names}
|
||||
|
||||
slew(robot, motor_names, _leader_target, cfg.align_duration)
|
||||
print("Alignment complete.")
|
||||
|
||||
print(
|
||||
"Starting joint-mirror loop. Back-drive the leader to teleoperate the follower… (Ctrl-C to stop)"
|
||||
)
|
||||
|
||||
def compute(robot_obs: RobotObservation | None) -> RobotAction | None:
|
||||
leader_action = teleop.get_action()
|
||||
# Hold the follower at its measured pose when the leader drops out (stale stream)
|
||||
# rather than commanding a possibly-old target.
|
||||
if not teleop.is_tracking:
|
||||
return None
|
||||
return leader_action
|
||||
|
||||
def cleanup() -> None:
|
||||
# A plugin-reaping failure must not skip the session disconnect (and vice versa
|
||||
# the disconnect runs after the plugin stops pushing on it).
|
||||
try:
|
||||
if plugin_proc is not None:
|
||||
plugin_proc.terminate()
|
||||
try:
|
||||
plugin_proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
plugin_proc.kill()
|
||||
finally:
|
||||
teleop.disconnect()
|
||||
|
||||
return Device(compute=compute, startup=startup, cleanup=cleanup)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Shared setup
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def build_device(cfg: LoopConfig) -> tuple:
|
||||
"""Connect the follower, build the selected Isaac device, and run its pre-loop startup.
|
||||
|
||||
Connects the follower FIRST (so the startup slew / clutch-home seed can read live joints),
|
||||
dispatches on ``--teleop.type``, then runs ``device.startup()`` before returning. On any
|
||||
failure after ``connect()`` the follower is disconnected so the connection never leaks.
|
||||
|
||||
Returns ``(robot, device, motor_names)``.
|
||||
"""
|
||||
# Default the CloudXR input profile to this example's default.env unless the user overrode
|
||||
# it via --teleop.cloudxr_env_file.
|
||||
if cfg.teleop.cloudxr_env_file is None:
|
||||
cfg.teleop.cloudxr_env_file = CLOUDXR_ENV_FILE
|
||||
|
||||
# SO-101/SO-100 only (both share the SO-101 URDF), reject other followers.
|
||||
supported_robots = {"so101_follower", "so100_follower"}
|
||||
if cfg.robot.type not in supported_robots:
|
||||
raise ValueError(
|
||||
f"This example only supports SO-101/SO-100 followers ({sorted(supported_robots)}), "
|
||||
f"but got --robot.type={cfg.robot.type}."
|
||||
)
|
||||
|
||||
# The degree-based pipeline relies on --robot.use_degrees (default True).
|
||||
robot = make_robot_from_config(cfg.robot)
|
||||
# Connect FIRST so the startup slew and clutch-home seed can read live joints.
|
||||
robot.connect()
|
||||
# Everything after connect() can fail; this runs outside the callers' try/finally, so
|
||||
# disconnect the follower on any failure to avoid leaking the connection.
|
||||
device: Device | None = None
|
||||
try:
|
||||
# Joint names in action order, read from {name}.pos action features (robot-agnostic).
|
||||
motor_names = [key.removesuffix(".pos") for key in robot.action_features if key.endswith(".pos")]
|
||||
|
||||
if isinstance(cfg.teleop, SO101LeaderArmConfig):
|
||||
device = setup_leader(cfg, robot, motor_names)
|
||||
else:
|
||||
device = setup_xr(cfg, robot, motor_names)
|
||||
|
||||
device.startup()
|
||||
except BaseException:
|
||||
# Reap a partially-started device, then always disconnect the follower.
|
||||
if device is not None:
|
||||
with suppress(Exception):
|
||||
device.cleanup()
|
||||
robot.disconnect()
|
||||
raise
|
||||
|
||||
return robot, device, motor_names
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Keyboard control
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def init_keyboard_listener():
|
||||
"""Recording shortcuts, terminal-first so they work over SSH.
|
||||
|
||||
Whenever stdin is a TTY we use the stdlib :class:`TerminalKeyListener` directly rather
|
||||
than upstream's pynput-first :func:`init_keyboard_listener`, whose global listener would
|
||||
capture the workstation console instead of this (often SSH) terminal. With no TTY we defer
|
||||
to upstream (pynput on a GUI, else headless no-op).
|
||||
"""
|
||||
if not (sys.stdin is not None and sys.stdin.isatty()):
|
||||
from lerobot.utils.keyboard_input import init_keyboard_listener as _upstream
|
||||
|
||||
return _upstream()
|
||||
|
||||
from lerobot.utils.keyboard_input import TerminalKeyListener, apply_recording_control
|
||||
|
||||
events = {"exit_early": False, "rerecord_episode": False, "stop_recording": False}
|
||||
|
||||
# n/r/q are the arrow/Esc equivalents that survive escape-sequence splitting over laggy
|
||||
# SSH/VNC links. Case-insensitive so Shift+letter still works.
|
||||
def on_key(name: str) -> None:
|
||||
key = name.lower()
|
||||
if key in ("right", "n"):
|
||||
apply_recording_control("right", events)
|
||||
elif key in ("left", "r"):
|
||||
apply_recording_control("left", events)
|
||||
elif key in ("esc", "q"):
|
||||
apply_recording_control("esc", events)
|
||||
|
||||
listener = TerminalKeyListener(on_key)
|
||||
listener.start()
|
||||
logging.info(
|
||||
"Keyboard control via terminal — keep this terminal focused: "
|
||||
"Right/n = end episode early, Left/r = re-record, Esc/q = stop."
|
||||
)
|
||||
return listener, events
|
||||
@@ -0,0 +1,21 @@
|
||||
# CloudXR device-profile overrides for the Isaac Teleop XR -> SO-101 example.
|
||||
#
|
||||
# Passed to isaacteleop's CloudXRLauncher as `env_config` (via
|
||||
# XRControllerConfig.cloudxr_env_file). Format: KEY=value, one per line; `#`
|
||||
# comments and blank lines ignored; $VARS / ~ expanded. See
|
||||
# isaacteleop/cloudxr/env_config.py::_load_env_file.
|
||||
#
|
||||
# Runtime-resolved keys (XR_RUNTIME_JSON, XRT_NO_STDIN, NV_CXR_RUNTIME_DIR,
|
||||
# NV_CXR_OUTPUT_DIR) are reserved and ignored if set here.
|
||||
|
||||
# Transport profile the runtime advertises (CloudXR default: auto-webrtc).
|
||||
# "Quest3" also covers the Pico 4. Other values: auto-native, AppleVisionPro.
|
||||
NV_DEVICE_PROFILE=Quest3
|
||||
|
||||
# Input device discovery channels (both default to true; pinned for clarity).
|
||||
NV_CXR_ENABLE_PUSH_DEVICES=true
|
||||
NV_CXR_ENABLE_TENSOR_DATA=true
|
||||
|
||||
# Runtime logs to ~/.cloudxr/logs — helps debug connection issues
|
||||
# (e.g. "Failed to get OpenXR system: -35").
|
||||
NV_CXR_FILE_LOGGING=true
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""NVIDIA Isaac Teleop teleoperators for LeRobot.
|
||||
|
||||
Each input device is an :class:`IsaacTeleopTeleoperator` subclass: :class:`XRController`
|
||||
(XR/VR controller) and :class:`SO101LeaderArm` (back-drivable SO-101 leader arm) ship today.
|
||||
"""
|
||||
|
||||
from .base import IsaacTeleopTeleoperator
|
||||
from .clutch import Clutch
|
||||
from .config_isaac_teleop import IsaacTeleopConfig, SO101LeaderArmConfig, XRControllerConfig
|
||||
from .teleop_so101_leader_arm import SO101LeaderArm, leader_joints_to_robot_action
|
||||
from .teleop_xr_controller import XRController
|
||||
from .xr_controller_processor import MapXRControllerActionToRobotAction
|
||||
|
||||
__all__ = [
|
||||
"Clutch",
|
||||
"IsaacTeleopConfig",
|
||||
"IsaacTeleopTeleoperator",
|
||||
"MapXRControllerActionToRobotAction",
|
||||
"SO101LeaderArm",
|
||||
"SO101LeaderArmConfig",
|
||||
"XRController",
|
||||
"XRControllerConfig",
|
||||
"leader_joints_to_robot_action",
|
||||
]
|
||||
@@ -0,0 +1,282 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Shared base for NVIDIA Isaac Teleop-backed LeRobot teleoperators.
|
||||
|
||||
Isaac Teleop is a multi-modal framework: a single ``TeleopSession`` can be driven by
|
||||
XR controllers, hand tracking, Manus gloves, etc. Each modality is a
|
||||
:class:`Teleoperator` subclass in its own ``teleop_<device>.py``.
|
||||
|
||||
:class:`IsaacTeleopTeleoperator` owns what those devices share — the session
|
||||
lifecycle, the per-step staleness/worker-health guard, and the no-op calibration
|
||||
tracking devices need. A concrete device implements :meth:`_build_pipeline` (its
|
||||
retargeting graph) and :meth:`get_action` (usually via :meth:`_step`).
|
||||
|
||||
``isaacteleop`` is an optional NVIDIA dependency (install instructions in the example's
|
||||
``README.md``); its imports are guarded behind an availability check at module top, so this
|
||||
module imports without it and constructing a device fails fast with install instructions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import abc
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from lerobot.teleoperators.teleoperator import Teleoperator
|
||||
from lerobot.utils.import_utils import is_package_available
|
||||
|
||||
from .config_isaac_teleop import IsaacTeleopConfig
|
||||
|
||||
_isaacteleop_available = is_package_available("isaacteleop")
|
||||
|
||||
if TYPE_CHECKING or _isaacteleop_available:
|
||||
from isaacteleop.cloudxr import CloudXRLauncher
|
||||
from isaacteleop.retargeting_engine.interface import (
|
||||
ExecutionEvents,
|
||||
ExecutionState,
|
||||
GraphExecutable,
|
||||
RetargeterIO,
|
||||
)
|
||||
from isaacteleop.teleop_session_manager import TeleopSession, TeleopSessionConfig
|
||||
else:
|
||||
CloudXRLauncher = None
|
||||
ExecutionEvents = None
|
||||
ExecutionState = None
|
||||
GraphExecutable = None
|
||||
RetargeterIO = None
|
||||
TeleopSession = None
|
||||
TeleopSessionConfig = None
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Gripper closedness [0, 1] -> SO-101 follower motor units [0, 100] (RANGE_0_100, 100 = OPEN).
|
||||
# Shared by the XR processor and leader device, which invert via ``pos = (1 - c) * SCALE``.
|
||||
_GRIPPER_MOTOR_SCALE = 100.0
|
||||
|
||||
|
||||
def _require_isaacteleop() -> None:
|
||||
"""Fail fast with install pointers when the optional ``isaacteleop`` package is missing."""
|
||||
if not _isaacteleop_available:
|
||||
raise ImportError(
|
||||
"The 'isaacteleop' package is required for Isaac Teleop devices but is not "
|
||||
"installed. See examples/isaac_teleop_to_so101/README.md for install instructions."
|
||||
)
|
||||
|
||||
|
||||
class IsaacTeleopTeleoperator(Teleoperator):
|
||||
"""Abstract base for teleoperators backed by an Isaac Teleop ``TeleopSession``.
|
||||
|
||||
Owns the session lifecycle and the per-step health guard; subclasses supply
|
||||
:meth:`_build_pipeline` and :meth:`get_action`.
|
||||
"""
|
||||
|
||||
config_class = IsaacTeleopConfig
|
||||
|
||||
def __init__(self, config: IsaacTeleopConfig):
|
||||
_require_isaacteleop()
|
||||
super().__init__(config)
|
||||
self.config: IsaacTeleopConfig = config
|
||||
self._session: TeleopSession | None = None
|
||||
self._cloudxr_launcher: CloudXRLauncher | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Pipeline construction (device override point)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@abc.abstractmethod
|
||||
def _build_pipeline(self) -> GraphExecutable:
|
||||
"""Build this device's retargeting pipeline (the ``GraphExecutable`` for
|
||||
``TeleopSessionConfig.pipeline``). Called once in :meth:`connect`; its output
|
||||
keys must match what :meth:`get_action` unpacks.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lifecycle (shared)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@property
|
||||
def is_connected(self) -> bool:
|
||||
return self._session is not None
|
||||
|
||||
@property
|
||||
def is_calibrated(self) -> bool:
|
||||
return True # Tracking devices are self-calibrating.
|
||||
|
||||
def calibrate(self) -> None:
|
||||
pass
|
||||
|
||||
def configure(self) -> None:
|
||||
pass
|
||||
|
||||
def connect(self, calibrate: bool = True) -> None:
|
||||
"""Auto-launch the CloudXR runtime (unless opted out) and open the session.
|
||||
|
||||
The CloudXR launch blocks ~30s and, on the first run, prompts on stdin for the
|
||||
EULA (accept once via ``python -m isaacteleop.cloudxr --accept-eula``). Opt out
|
||||
when CloudXR runs externally via ``config.auto_launch_cloudxr=False`` or
|
||||
``LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH=1`` (env var wins).
|
||||
"""
|
||||
if self._session is not None:
|
||||
raise RuntimeError("Already connected. Call disconnect() first.")
|
||||
|
||||
self._ensure_cloudxr_runtime()
|
||||
|
||||
try:
|
||||
pipeline = self._build_pipeline()
|
||||
session_config = TeleopSessionConfig(app_name=self.config.app_name, pipeline=pipeline)
|
||||
self._session = TeleopSession(session_config)
|
||||
self._session.__enter__()
|
||||
except Exception:
|
||||
self._session = None
|
||||
try:
|
||||
self._stop_cloudxr_runtime()
|
||||
except Exception:
|
||||
logger.exception("Failed to stop CloudXR runtime during connect() rollback")
|
||||
raise
|
||||
logger.info("Isaac Teleop session started: %s", self.config.app_name)
|
||||
|
||||
def disconnect(self) -> None:
|
||||
try:
|
||||
if self._session is not None:
|
||||
# Null the handle BEFORE __exit__: even a failed session teardown must not
|
||||
# wedge the device as is_connected (blocking every later connect/disconnect).
|
||||
session = self._session
|
||||
self._session = None
|
||||
session.__exit__(None, None, None)
|
||||
logger.info("Isaac Teleop session ended")
|
||||
finally:
|
||||
# Reap the CloudXR runtime even if session teardown raised, and even if no
|
||||
# session was ever established (e.g. the launcher came up but session creation
|
||||
# failed before this point); a no-op when we never launched CloudXR (opt-out /
|
||||
# externally-owned runtime), so we never stop a runtime we don't own.
|
||||
self._stop_cloudxr_runtime()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# CloudXR runtime (shared)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _ensure_cloudxr_runtime(self) -> None:
|
||||
"""Auto-launch the CloudXR runtime once, unless opted out.
|
||||
|
||||
Idempotent (no-op once the launcher is up). ``LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH``
|
||||
is checked first and wins over ``config.auto_launch_cloudxr``. Constructing
|
||||
:class:`CloudXRLauncher` mutates the process env (``XR_RUNTIME_JSON`` etc.) and
|
||||
blocks until the runtime is ready or raises :class:`RuntimeError`.
|
||||
"""
|
||||
if self._cloudxr_launcher is not None:
|
||||
return
|
||||
|
||||
if os.environ.get("LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH", "").strip() == "1":
|
||||
logger.info(
|
||||
"LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH=1 set; skipping CloudXR auto-launch "
|
||||
"(assuming CloudXR is already running externally)"
|
||||
)
|
||||
return
|
||||
|
||||
if not self.config.auto_launch_cloudxr:
|
||||
logger.info(
|
||||
"config.auto_launch_cloudxr is False; skipping CloudXR auto-launch "
|
||||
"(assuming CloudXR is already running externally)"
|
||||
)
|
||||
return
|
||||
|
||||
logger.info("Launching CloudXR runtime (first run may prompt for EULA and take ~30s)...")
|
||||
|
||||
self._cloudxr_launcher = CloudXRLauncher(
|
||||
install_dir=str(Path.home() / ".cloudxr"),
|
||||
env_config=self.config.cloudxr_env_file,
|
||||
accept_eula=False,
|
||||
)
|
||||
|
||||
def _stop_cloudxr_runtime(self) -> None:
|
||||
"""Stop the auto-launched CloudXR runtime, if any.
|
||||
|
||||
Clean stop nulls the handle. On :class:`RuntimeError` the handle is RETAINED so
|
||||
the launcher's ``atexit`` hook owns the retry — a later :meth:`connect` then
|
||||
treats the retained runtime as still up and will not relaunch.
|
||||
"""
|
||||
if self._cloudxr_launcher is None:
|
||||
return
|
||||
try:
|
||||
self._cloudxr_launcher.stop()
|
||||
except RuntimeError:
|
||||
logger.warning("CloudXR runtime could not be terminated; handle retained for atexit cleanup")
|
||||
else:
|
||||
self._cloudxr_launcher = None
|
||||
logger.info("CloudXR runtime stopped")
|
||||
|
||||
def send_feedback(self, feedback: dict[str, Any]) -> None:
|
||||
pass # Haptic feedback not yet implemented.
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Stepping (shared)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _running_events(self) -> ExecutionEvents:
|
||||
"""Constant ``RUNNING`` ``ExecutionEvents`` for a device with no clutch lifecycle.
|
||||
|
||||
Keeps the stream flowing; ``reset`` stays ``False``. A clutched device that needs
|
||||
a real lifecycle should build its own ``ExecutionEvents`` instead.
|
||||
"""
|
||||
return ExecutionEvents(execution_state=ExecutionState.RUNNING, reset=False)
|
||||
|
||||
def _step(
|
||||
self,
|
||||
*,
|
||||
execution_events: ExecutionEvents | None = None,
|
||||
external_inputs: Mapping[str, Any] | None = None,
|
||||
) -> RetargeterIO:
|
||||
"""Step the session once and return the raw pipeline outputs.
|
||||
|
||||
Applies the shared guard: re-raises a retargeting-worker exception and warns on a
|
||||
stale frame. Subclasses call this from :meth:`get_action`.
|
||||
|
||||
Args:
|
||||
execution_events: The ``ExecutionEvents`` driving the session this frame.
|
||||
Devices with a lifecycle (clutch) MUST pass this every frame — when
|
||||
``None``, ``TeleopSession.step`` auto-fires ``RUNNING`` (the clutch would
|
||||
latch immediately and never stop).
|
||||
external_inputs: Per-step inputs (e.g. a static ``base_T_anchor``) in the
|
||||
``{leaf_node_name: {output_port_name: TensorGroup}}`` shape ``step`` expects.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If not connected, or if the retargeting worker raised.
|
||||
"""
|
||||
if self._session is None:
|
||||
raise RuntimeError("Not connected. Call connect() first.")
|
||||
|
||||
result = self._session.step(
|
||||
execution_events=execution_events,
|
||||
external_inputs=external_inputs,
|
||||
)
|
||||
|
||||
info = self._session.last_step_info
|
||||
if info is not None:
|
||||
if info.worker_exception is not None:
|
||||
raise RuntimeError(
|
||||
"Isaac Teleop retargeting worker raised an exception"
|
||||
) from info.worker_exception
|
||||
if info.frame_deadline_miss:
|
||||
logger.warning(
|
||||
"Isaac Teleop frame deadline miss (returned_age_frames=%s)",
|
||||
info.returned_age_frames,
|
||||
)
|
||||
return result
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Engage-relative clutch for the XR -> SO-101 teleop loop.
|
||||
|
||||
Turns the raw controller grip pose into an absolute base-frame EE target, so the XR
|
||||
device can stay a thin raw-pose reader. Pure numpy + the local ``Rotation`` helper (no
|
||||
``isaacteleop``), so it is unit-testable without the XR runtime.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
|
||||
from lerobot.utils.rotation import Rotation
|
||||
|
||||
|
||||
class Clutch:
|
||||
"""Engage-relative clutch for both position AND orientation.
|
||||
|
||||
Latch an origin on engage, then track the base-frame delta from it, applied
|
||||
independently to position and orientation. State:
|
||||
|
||||
- ``_last_commanded_pos`` / ``_last_commanded_rot``: last commanded EE pose; held
|
||||
while disengaged so the arm freezes where it was left.
|
||||
- ``_home_pos`` / ``_home_rot``: latched on engage — the EE pose the delta applies to.
|
||||
The position comes from the arm's MEASURED pose when the caller provides it (so an
|
||||
arm that moved while disengaged is not snapped back to a stale command); the
|
||||
orientation always comes from the last commanded rotation (see NOTE below).
|
||||
- ``_origin_pos`` / ``_origin_rot``: latched on engage — the controller pose the delta
|
||||
is measured against.
|
||||
|
||||
Each engaged frame :meth:`rebase` returns::
|
||||
|
||||
pos = home_pos + (grip_pos - origin_pos) # 1:1 controller -> EE translation
|
||||
rot = (R_ctrl @ R_origin ^ -1) @ R_home # base-frame delta, left-composed
|
||||
|
||||
On the engage edge the output is exactly the home pose (no teleport). The orientation
|
||||
delta is left-composed (base frame), so hand rotation about base Z maps to EE rotation
|
||||
about base Z. A re-clutch latches a fresh home/origin.
|
||||
|
||||
NOTE: ``_home_rot`` is the last *commanded* orientation even when the measured pose is
|
||||
supplied: the 5-DOF SO-101 tracks orientation only softly, so its measured wrist
|
||||
orientation persistently differs from the command, and latching the measurement would
|
||||
inject that offset into the commanded signal on every re-clutch. Position has no such
|
||||
tracking gap, and there latching the measurement is what prevents the snap-back.
|
||||
"""
|
||||
|
||||
def __init__(self, home_base_T_ee: np.ndarray): # noqa: N803
|
||||
# Seed the held pose from the arm's measured startup EE pose so the first
|
||||
# engage latches home there (no jump on the first squeeze).
|
||||
home = np.asarray(home_base_T_ee, dtype=float)
|
||||
self._last_commanded_pos = home[:3, 3].copy()
|
||||
self._last_commanded_rot = Rotation.from_matrix(home[:3, :3])
|
||||
self._home_pos = self._last_commanded_pos.copy()
|
||||
self._home_rot = self._last_commanded_rot
|
||||
self._origin_pos = np.zeros(3, dtype=float)
|
||||
self._origin_rot = Rotation.from_quat(np.array([0.0, 0.0, 0.0, 1.0]))
|
||||
|
||||
def engage(
|
||||
self,
|
||||
grip_pos: np.ndarray,
|
||||
grip_quat: np.ndarray,
|
||||
measured_base_T_ee: np.ndarray | None = None, # noqa: N803
|
||||
) -> None:
|
||||
"""Latch the engage home (where the arm is now) and controller origin.
|
||||
|
||||
Pass ``measured_base_T_ee`` (FK of the measured joints) so the home POSITION is
|
||||
where the arm physically is — if the arm moved while disengaged (gravity sag,
|
||||
external contact), latching the stale last-commanded position would make the
|
||||
first engaged frame command a full-speed jump back to it. The home ORIENTATION
|
||||
always stays the last commanded one (see the class NOTE).
|
||||
"""
|
||||
if measured_base_T_ee is not None:
|
||||
self._home_pos = np.asarray(measured_base_T_ee, dtype=float)[:3, 3].copy()
|
||||
else:
|
||||
self._home_pos = self._last_commanded_pos.copy()
|
||||
self._home_rot = self._last_commanded_rot
|
||||
self._origin_pos = np.asarray(grip_pos, dtype=float).copy()
|
||||
self._origin_rot = Rotation.from_quat(np.asarray(grip_quat, dtype=float))
|
||||
|
||||
def rebase(self, grip_pos: np.ndarray, grip_quat: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Return the absolute base-frame EE target ``(pos [m], quat [xyzw])`` for this frame."""
|
||||
pos = self._home_pos + (np.asarray(grip_pos, dtype=float) - self._origin_pos)
|
||||
rot_ctrl = Rotation.from_quat(np.asarray(grip_quat, dtype=float))
|
||||
rot = (rot_ctrl * self._origin_rot.inv()) * self._home_rot
|
||||
self._last_commanded_pos = pos.copy()
|
||||
self._last_commanded_rot = rot
|
||||
return pos, rot.as_quat()
|
||||
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Configuration dataclasses for NVIDIA Isaac Teleop-backed teleoperators.
|
||||
|
||||
:class:`IsaacTeleopConfig` holds the shared fields; each device adds its own subclass
|
||||
(e.g. :class:`XRControllerConfig`, :class:`SO101LeaderArmConfig`).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import ClassVar
|
||||
|
||||
from lerobot.teleoperators.config import TeleoperatorConfig
|
||||
|
||||
|
||||
@dataclass(kw_only=True)
|
||||
class IsaacTeleopConfig(TeleoperatorConfig):
|
||||
"""Shared config for all Isaac Teleop-backed teleoperators.
|
||||
|
||||
Uses its own draccus ``_choice_registry`` (decoupled from the global
|
||||
:class:`TeleoperatorConfig` one) so ``--teleop.type`` on a field typed
|
||||
``IsaacTeleopConfig`` resolves against ONLY the Isaac devices — letting them claim
|
||||
short names (``xr_controller``, ``so101_leader``) without colliding with the global
|
||||
registry. These devices are selected by the example scripts, not routed through
|
||||
``make_teleoperator_from_config``.
|
||||
"""
|
||||
|
||||
_choice_registry: ClassVar[dict] = {}
|
||||
|
||||
app_name: str = "LeTeleop"
|
||||
"""Application name for the OpenXR / Isaac Teleop session."""
|
||||
|
||||
auto_launch_cloudxr: bool = True
|
||||
"""Auto-launch the CloudXR runtime on :meth:`connect`. Set ``False`` (or export
|
||||
``LEROBOT_CLOUDXR_SKIP_AUTOLAUNCH=1``, which wins) when CloudXR runs externally.
|
||||
"""
|
||||
|
||||
cloudxr_env_file: str | None = None
|
||||
"""Optional CloudXR device-profile ``.env`` (an INPUT profile selecting the headset
|
||||
transport) passed to ``CloudXRLauncher``. ``None`` keeps the default auto-WebRTC profile.
|
||||
"""
|
||||
|
||||
|
||||
# Static rebase from the OpenXR controller anchor frame (X=Right, Y=Up, Z=Backward) into the
|
||||
# robot base frame (X=Forward, Y=Left, Z=Up). A proper rotation (det=+1): controller motion
|
||||
# forward -> robot +X, right -> robot -Y (i.e. rightward), up -> robot +Z.
|
||||
_DEFAULT_BASE_T_ANCHOR: list[list[float]] = [
|
||||
[0.0, 0.0, -1.0, 0.0],
|
||||
[-1.0, 0.0, 0.0, 0.0],
|
||||
[0.0, 1.0, 0.0, 0.0],
|
||||
[0.0, 0.0, 0.0, 1.0],
|
||||
]
|
||||
|
||||
|
||||
@IsaacTeleopConfig.register_subclass("xr_controller")
|
||||
@dataclass(kw_only=True)
|
||||
class XRControllerConfig(IsaacTeleopConfig):
|
||||
"""Config for Isaac Teleop XR (VR) controller teleoperation.
|
||||
|
||||
Exposes the raw base-frame grip pose, squeeze, and trigger via ``ControllersSource``.
|
||||
No retargeters: the clutch and gripper mapping live in the owning loop.
|
||||
"""
|
||||
|
||||
hand_side: str = "right"
|
||||
"""Which controller hand to use: ``"left"`` or ``"right"``. A plain ``str`` (validated in
|
||||
``__post_init__``) because draccus cannot decode ``Literal``-typed fields from the CLI."""
|
||||
|
||||
clutch_threshold: float = 0.5
|
||||
"""Squeeze value above which the owning loop's clutch engages (held-to-enable). The
|
||||
device reports only the raw squeeze; the threshold is applied by the loop."""
|
||||
|
||||
base_T_anchor: list[list[float]] = field( # noqa: N815 (frameA_T_frameB transform-matrix convention)
|
||||
# Fresh copy per instance: returning the module-level list itself would alias one
|
||||
# mutable matrix across every config.
|
||||
default_factory=lambda: [row.copy() for row in _DEFAULT_BASE_T_ANCHOR]
|
||||
)
|
||||
"""Static 4x4 [row-major] transform rebasing the OpenXR controller anchor frame into
|
||||
the robot base frame. Defaults to OpenXR (X=Right, Y=Up, Z=Backward) -> robot
|
||||
(X=Forward, Y=Left, Z=Up). Plain nested lists so the config stays serializable.
|
||||
"""
|
||||
|
||||
def __post_init__(self):
|
||||
if self.hand_side not in ("left", "right"):
|
||||
raise ValueError(f"hand_side must be 'left' or 'right', got {self.hand_side!r}")
|
||||
|
||||
|
||||
# Provisional gripper open/close endpoints [rad], normalizing the streamed gripper angle
|
||||
# into the follower's RANGE_0_100 jaw target. Derived from the so101_leader plugin README's
|
||||
# example calibration (home_ticks=2048, range 2000..3000; angle = (ticks-home)*2*pi/4096).
|
||||
_DEFAULT_GRIPPER_OPEN_RAD = -0.074
|
||||
_DEFAULT_GRIPPER_CLOSE_RAD = 1.460
|
||||
|
||||
|
||||
@IsaacTeleopConfig.register_subclass("so101_leader")
|
||||
@dataclass(kw_only=True)
|
||||
class SO101LeaderArmConfig(IsaacTeleopConfig):
|
||||
"""Config for an Isaac Teleop SO-101 *leader arm* (generic joint-space device).
|
||||
|
||||
Mirrors the leader's joint angles 1:1 onto a follower SO-101. The leader state is
|
||||
streamed in radians by the native ``so101_leader`` plugin and read via a
|
||||
``JointStateSource``; the device converts arm joints to degrees and the gripper to the
|
||||
follower's RANGE_0_100 jaw target (no IK/clutch/retargeter on the LeRobot side).
|
||||
"""
|
||||
|
||||
port: str = ""
|
||||
"""Serial port of the physical LEADER arm (e.g. ``/dev/ttyACM1``), forwarded to the
|
||||
plugin (which reads the servos) when the example launches it. Empty -> the plugin runs
|
||||
its synthetic trajectory."""
|
||||
|
||||
collection_id: str = "so101_leader"
|
||||
"""Tensor collection id the leader plugin pushes on; must match the running
|
||||
``so101_leader`` plugin (its second positional arg, default ``"so101_leader"``)."""
|
||||
|
||||
gripper_open_rad: float = _DEFAULT_GRIPPER_OPEN_RAD
|
||||
"""Leader gripper angle [rad] at fully OPEN -> follower jaw 100. Provisional default;
|
||||
set from the plugin's ``calibrate`` subcommand. See ``_DEFAULT_GRIPPER_OPEN_RAD``."""
|
||||
|
||||
gripper_close_rad: float = _DEFAULT_GRIPPER_CLOSE_RAD
|
||||
"""Leader gripper angle [rad] at fully CLOSED -> follower jaw 0. Provisional default;
|
||||
set from the plugin's ``calibrate`` subcommand. See ``_DEFAULT_GRIPPER_CLOSE_RAD``."""
|
||||
@@ -0,0 +1,186 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""SO-101 leader-arm device for NVIDIA Isaac Teleop, exposed to LeRobot.
|
||||
|
||||
The leader is a back-drivable SO-101 whose six joint angles are streamed (in radians) by
|
||||
the native ``so101_leader`` plugin; this device reads them via a ``JointStateSource`` and
|
||||
converts them into follower-ready ``{joint}.pos``. Same kinematics as the follower, so it
|
||||
needs no retargeting — a 1:1 joint mirror, direct joint drive.
|
||||
|
||||
Units (converted in the device so the output is always follower-valid):
|
||||
|
||||
* arm joints: ``rad2deg`` — correct only if the leader's calibrated zero and the follower's
|
||||
homing map to the same physical zero (the standard same-hardware assumption).
|
||||
* gripper: normalized from ``[gripper_open_rad, gripper_close_rad]`` to RANGE_0_100.
|
||||
|
||||
``isaacteleop`` imports are guarded behind the availability flag so this module — and the
|
||||
pure :func:`leader_joints_to_robot_action` converter — import without it (construction
|
||||
fails fast via the base class).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import numpy as np
|
||||
|
||||
from lerobot.types import RobotAction
|
||||
|
||||
from .base import _GRIPPER_MOTOR_SCALE, IsaacTeleopTeleoperator, _isaacteleop_available
|
||||
from .config_isaac_teleop import SO101LeaderArmConfig
|
||||
|
||||
if TYPE_CHECKING or _isaacteleop_available:
|
||||
from isaacteleop.retargeting_engine.deviceio_source_nodes import JointStateSource
|
||||
from isaacteleop.retargeting_engine.interface import OutputCombiner
|
||||
else:
|
||||
JointStateSource = None
|
||||
OutputCombiner = None
|
||||
|
||||
# Canonical SO-101 DOF names and order — matches the plugin stream and the follower's motor
|
||||
# order. Passed to the ``JointStateSource`` as its output layout; the source maps by name and
|
||||
# :func:`_joints_group_to_rad` reads back by name, so a layout mismatch can't mislabel a DOF.
|
||||
SO101_LEADER_JOINTS = [
|
||||
"shoulder_pan",
|
||||
"shoulder_lift",
|
||||
"elbow_flex",
|
||||
"wrist_flex",
|
||||
"wrist_roll",
|
||||
"gripper",
|
||||
]
|
||||
|
||||
|
||||
def leader_joints_to_robot_action(
|
||||
joints_rad: dict[str, float],
|
||||
*,
|
||||
gripper_joint: str,
|
||||
gripper_open_rad: float,
|
||||
gripper_close_rad: float,
|
||||
) -> RobotAction:
|
||||
"""Convert streamed leader joint angles [rad] to follower-ready ``{joint}.pos``.
|
||||
|
||||
Pure (no ``isaacteleop``, no I/O). Iteration follows ``joints_rad`` insertion order, so
|
||||
pass it in :data:`SO101_LEADER_JOINTS` order for a stable layout. Arm joints are
|
||||
converted ``rad2deg``; ``gripper_joint`` is normalized from
|
||||
``[gripper_open_rad, gripper_close_rad]`` to RANGE_0_100 (clipped).
|
||||
"""
|
||||
action: RobotAction = {}
|
||||
span = gripper_close_rad - gripper_open_rad
|
||||
for name, rad in joints_rad.items():
|
||||
if name == gripper_joint:
|
||||
# Closedness c=0 at open, c=1 at closed; invert to the follower's 100=open jaw.
|
||||
closedness = 0.0 if span == 0.0 else (rad - gripper_open_rad) / span
|
||||
closedness = min(1.0, max(0.0, closedness))
|
||||
action[f"{name}.pos"] = (1.0 - closedness) * _GRIPPER_MOTOR_SCALE
|
||||
else:
|
||||
action[f"{name}.pos"] = float(np.rad2deg(rad))
|
||||
return action
|
||||
|
||||
|
||||
def _joints_group_to_rad(joints) -> dict[str, float]:
|
||||
"""Read a ``JointStateSource`` output group into ``{joint_name: angle [rad]}``.
|
||||
|
||||
Pure (duck-typed on the group). The group is positional but each slot carries its joint
|
||||
name in ``group.group_type.types``; we key off those names (not a positional index) so a
|
||||
layout mismatch surfaces as a wrong/missing key here rather than a mislabeled DOF.
|
||||
"""
|
||||
names = [t.name for t in joints.group_type.types]
|
||||
return {name: float(joints[i]) for i, name in enumerate(names)}
|
||||
|
||||
|
||||
class SO101LeaderArm(IsaacTeleopTeleoperator):
|
||||
"""SO-101 leader-arm teleoperator (joint-space), direct joint mirror to the follower.
|
||||
|
||||
Reads the six joint angles off a single ``JointStateSource`` each frame; no retargeter,
|
||||
no clutch. When the leader is not streaming, :meth:`get_action` returns the held-last
|
||||
joints and :attr:`is_tracking` is ``False`` so the owning loop can hold the follower.
|
||||
"""
|
||||
|
||||
config_class = SO101LeaderArmConfig
|
||||
name = "isaac_teleop_so101_leader"
|
||||
|
||||
def __init__(self, config: SO101LeaderArmConfig):
|
||||
super().__init__(config)
|
||||
self.config: SO101LeaderArmConfig = config
|
||||
# Held-last joint angles [rad], seeded at zero (URDF/home pose) so the first frames
|
||||
# before the plugin starts pushing read as the home pose, not garbage.
|
||||
self._last_joints_rad: dict[str, float] = dict.fromkeys(SO101_LEADER_JOINTS, 0.0)
|
||||
# Whether the most recent get_action() read live leader data (vs held-last).
|
||||
self._is_tracking = False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Pipeline construction
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _build_pipeline(self) -> OutputCombiner:
|
||||
"""Build the joint-mirror pipeline: a single ``JointStateSource`` leaf that converts
|
||||
the raw stream into a name-keyed joint group. No retargeter (shared kinematics)."""
|
||||
source = JointStateSource(
|
||||
name="so101_leader",
|
||||
collection_id=self.config.collection_id,
|
||||
joint_names=SO101_LEADER_JOINTS,
|
||||
)
|
||||
return OutputCombiner({"joints": source.output(JointStateSource.JOINTS)})
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Action features
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@property
|
||||
def action_features(self) -> dict[str, type]:
|
||||
# Matches the serial SOLeader's action features so this is a drop-in joint-space
|
||||
# leader: one float `{joint}.pos` per DOF, sendable straight to an SO-101 follower.
|
||||
return {f"{name}.pos": float for name in SO101_LEADER_JOINTS}
|
||||
|
||||
@property
|
||||
def feedback_features(self) -> dict[str, type]:
|
||||
return {}
|
||||
|
||||
@property
|
||||
def is_tracking(self) -> bool:
|
||||
"""Whether the last :meth:`get_action` read live leader data (vs held-last)."""
|
||||
return self._is_tracking
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Action extraction
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_action(self) -> RobotAction:
|
||||
"""Step the session and return the leader joints as follower-ready ``{joint}.pos``.
|
||||
|
||||
When the leader is streaming, the live angles are cached and converted; otherwise the
|
||||
held-last angles are reused and :attr:`is_tracking` is set ``False``.
|
||||
"""
|
||||
result = self._step(execution_events=self._running_events())
|
||||
|
||||
joints = result["joints"]
|
||||
# The JointStateSource output is Optional: absent (is_none) when the device is
|
||||
# inactive. Treat that as "not tracking" and reuse the held-last angles.
|
||||
self._is_tracking = not getattr(joints, "is_none", False)
|
||||
if self._is_tracking:
|
||||
try:
|
||||
self._last_joints_rad = _joints_group_to_rad(joints)
|
||||
except (AttributeError, IndexError, KeyError, TypeError, ValueError):
|
||||
# A partially-populated / malformed group on an odd frame: keep held-last, but
|
||||
# report it as not-tracking so the loop holds the follower rather than trusting it.
|
||||
self._is_tracking = False
|
||||
|
||||
return leader_joints_to_robot_action(
|
||||
self._last_joints_rad,
|
||||
gripper_joint="gripper",
|
||||
gripper_open_rad=self.config.gripper_open_rad,
|
||||
gripper_close_rad=self.config.gripper_close_rad,
|
||||
)
|
||||
@@ -0,0 +1,204 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""XR (VR) controller device for NVIDIA Isaac Teleop, exposed to LeRobot.
|
||||
|
||||
A deliberately thin reader: exposes the raw controller grip pose off
|
||||
``ControllersSource`` (statically rebased into the robot base frame by
|
||||
``ControllerTransform``), plus squeeze and trigger. No retargeters and no clutch —
|
||||
the clutch rebasing and gripper mapping live downstream in the owning loop, so this
|
||||
device is stateless across frames.
|
||||
|
||||
``isaacteleop`` imports are guarded behind the availability flag so this module imports
|
||||
without it (construction fails fast via the base class).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
from lerobot.types import RobotAction
|
||||
|
||||
from .base import IsaacTeleopTeleoperator, _isaacteleop_available
|
||||
from .config_isaac_teleop import XRControllerConfig
|
||||
|
||||
if TYPE_CHECKING or _isaacteleop_available:
|
||||
from isaacteleop.retargeting_engine.deviceio_source_nodes import ControllersSource
|
||||
from isaacteleop.retargeting_engine.interface import OutputCombiner, TensorGroup, ValueInput
|
||||
from isaacteleop.retargeting_engine.tensor_types import TransformMatrix
|
||||
from isaacteleop.retargeting_engine.tensor_types.indices import ControllerInputIndex
|
||||
else:
|
||||
ControllersSource = None
|
||||
OutputCombiner = None
|
||||
TensorGroup = None
|
||||
ValueInput = None
|
||||
TransformMatrix = None
|
||||
ControllerInputIndex = None
|
||||
|
||||
# Source-node name for the static base_T_anchor rebase input fed via
|
||||
# ``TeleopSession.step(external_inputs=...)`` each frame.
|
||||
_BASE_T_ANCHOR_INPUT = "base_T_anchor"
|
||||
|
||||
|
||||
class XRController(IsaacTeleopTeleoperator):
|
||||
"""Raw XR controller grip-pose teleoperator (base-frame), no retargeters.
|
||||
|
||||
Reads the raw grip pose + squeeze + trigger off a ``ControllersSource`` rebased into
|
||||
the robot base frame. :meth:`get_action` returns the absolute base-frame grip pose
|
||||
untouched; the owning loop owns the clutch and gripper mapping.
|
||||
"""
|
||||
|
||||
config_class = XRControllerConfig
|
||||
name = "isaac_teleop_controller"
|
||||
|
||||
def __init__(self, config: XRControllerConfig):
|
||||
super().__init__(config)
|
||||
self.config: XRControllerConfig = config
|
||||
|
||||
# Constant base_T_anchor input, built once in connect() (a TensorGroup is heavy and
|
||||
# isaacteleop-backed) and reused every step.
|
||||
self._external_inputs: dict[str, Any] | None = None
|
||||
# Whether the last get_action() read a tracked controller; the owning loop polls this
|
||||
# to wait for the operator to connect before driving the arm.
|
||||
self._is_tracking = False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Pipeline construction
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _build_pipeline(self) -> OutputCombiner:
|
||||
"""Build the raw-grip-pose pipeline: a ``ControllersSource`` rebased into the base
|
||||
frame by ``ControllerTransform``, exposed verbatim as ``"controller"``. No retargeters.
|
||||
"""
|
||||
side = self.config.hand_side
|
||||
controller_key = f"controller_{side}"
|
||||
|
||||
controllers = ControllersSource(name="controllers")
|
||||
# Static base_T_anchor rebase fed via external_inputs each step.
|
||||
xform = ValueInput(_BASE_T_ANCHOR_INPUT, TransformMatrix())
|
||||
transformed = controllers.transformed(xform.output("value"))
|
||||
ctrl = transformed.output(controller_key)
|
||||
|
||||
return OutputCombiner({"controller": ctrl})
|
||||
|
||||
def _build_external_inputs(self) -> dict[str, Any]:
|
||||
"""Materialize the constant ``base_T_anchor`` external input (once, in connect)."""
|
||||
tg = TensorGroup(TransformMatrix())
|
||||
tg[0] = np.asarray(self.config.base_T_anchor, dtype=np.float32)
|
||||
return {_BASE_T_ANCHOR_INPUT: {"value": tg}}
|
||||
|
||||
def connect(self, calibrate: bool = True) -> None:
|
||||
super().connect(calibrate=calibrate)
|
||||
try:
|
||||
self._external_inputs = self._build_external_inputs()
|
||||
except Exception:
|
||||
# Roll the session/runtime back so a failed connect() leaves no half-state
|
||||
# (a live session behind a raised connect would leak the CloudXR runtime).
|
||||
self.disconnect()
|
||||
raise
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Action features
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@property
|
||||
def action_features(self) -> dict:
|
||||
return {
|
||||
"grip_pos": {
|
||||
"dtype": "float32",
|
||||
"shape": (3,),
|
||||
"names": {"x": 0, "y": 1, "z": 2},
|
||||
},
|
||||
"grip_quat": {
|
||||
"dtype": "float32",
|
||||
"shape": (4,),
|
||||
"names": {"qx": 0, "qy": 1, "qz": 2, "qw": 3},
|
||||
},
|
||||
# ``get_action`` returns scalars for these two, so the advertised
|
||||
# shape is () (0-d) to stay consistent with the returned values.
|
||||
"squeeze": {
|
||||
"dtype": "float32",
|
||||
"shape": (),
|
||||
"names": None,
|
||||
},
|
||||
"trigger": {
|
||||
"dtype": "float32",
|
||||
"shape": (),
|
||||
"names": None,
|
||||
},
|
||||
}
|
||||
|
||||
@property
|
||||
def feedback_features(self) -> dict:
|
||||
return {}
|
||||
|
||||
@property
|
||||
def is_tracking(self) -> bool:
|
||||
"""Whether the last :meth:`get_action` read a tracked controller. ``False`` until the
|
||||
headset is connected over CloudXR and its controllers are live; the owning loop polls
|
||||
it to wait for the operator before commanding the arm."""
|
||||
return self._is_tracking
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Action extraction
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_action(self) -> RobotAction:
|
||||
"""Step the session and return the raw base-frame grip pose.
|
||||
|
||||
Reads the grip pose + squeeze + trigger off the transformed controller stream (with
|
||||
the constant ``base_T_anchor`` rebase). When the controller is not tracked, returns
|
||||
identity pose and squeeze/trigger = 0.0 so the owning loop freezes the arm.
|
||||
|
||||
Returns:
|
||||
``{"grip_pos": (3,) [m], "grip_quat": (4,) [qx,qy,qz,qw], "squeeze": float,
|
||||
"trigger": float}`` — pose in the robot base frame; squeeze/trigger in ``[0, 1]``.
|
||||
"""
|
||||
result = self._step(execution_events=self._running_events(), external_inputs=self._external_inputs)
|
||||
|
||||
# Optional controller group is None until the headset is connected and its controllers
|
||||
# are live; expose that as is_tracking so the loop can wait before driving the arm.
|
||||
controller = result["controller"]
|
||||
grip_pos = np.zeros(3, dtype=np.float32)
|
||||
grip_quat = np.array([0.0, 0.0, 0.0, 1.0], dtype=np.float32)
|
||||
squeeze = 0.0
|
||||
trigger = 0.0
|
||||
self._is_tracking = not getattr(controller, "is_none", False)
|
||||
if self._is_tracking:
|
||||
# Read ALL four fields into locals before committing any of them: a failure on a
|
||||
# partially-populated frame must not mix live values with the safe defaults (a
|
||||
# live squeeze paired with a defaulted trigger=0.0 would keep the clutch engaged
|
||||
# while commanding the gripper fully open, dropping whatever is grasped). On
|
||||
# failure the defaults stand untouched and the frame reports not-tracked.
|
||||
try:
|
||||
pos = np.asarray(controller[ControllerInputIndex.GRIP_POSITION], dtype=np.float32)
|
||||
quat = np.asarray(controller[ControllerInputIndex.GRIP_ORIENTATION], dtype=np.float32)
|
||||
squeeze_val = float(controller[ControllerInputIndex.SQUEEZE_VALUE])
|
||||
trigger_val = float(controller[ControllerInputIndex.TRIGGER_VALUE])
|
||||
except (IndexError, KeyError, TypeError, ValueError):
|
||||
self._is_tracking = False
|
||||
else:
|
||||
grip_pos, grip_quat = pos, quat
|
||||
squeeze, trigger = squeeze_val, trigger_val
|
||||
|
||||
return {
|
||||
"grip_pos": grip_pos,
|
||||
"grip_quat": grip_quat,
|
||||
"squeeze": squeeze,
|
||||
"trigger": trigger,
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Processor step that maps XR controller actions to robot EE targets.
|
||||
|
||||
Analogous to ``MapPhoneActionToRobotAction``, this bridges the clutch-rebased EE pose to
|
||||
the IK pipeline's input contract (``EEBoundsAndSafety`` -> ``InverseKinematicsEEToJoints``).
|
||||
Pure (no ``isaacteleop``), so it is unit-testable without the XR runtime.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from lerobot.configs.types import FeatureType, PipelineFeatureType, PolicyFeature
|
||||
from lerobot.processor import ProcessorStepRegistry, RobotActionProcessorStep
|
||||
from lerobot.types import RobotAction
|
||||
from lerobot.utils.rotation import Rotation
|
||||
|
||||
from .base import _GRIPPER_MOTOR_SCALE
|
||||
|
||||
|
||||
@ProcessorStepRegistry.register("map_xr_controller_action_to_robot_action")
|
||||
@dataclass
|
||||
class MapXRControllerActionToRobotAction(RobotActionProcessorStep):
|
||||
"""Maps an absolute base-frame EE pose + gripper closedness to the IK input contract.
|
||||
|
||||
Pure, stateless rename (the owning loop's clutch already produced the absolute base-frame
|
||||
target). Each frame it writes:
|
||||
|
||||
- ``ee.x/y/z`` = ``ee_pose[:3]`` (position [m]);
|
||||
- ``ee.wx/wy/wz`` = rotvec of ``ee_pose[3:7]`` (orientation; the IK tracks it softly at a
|
||||
small ``orientation_weight`` on the 5-DOF SO-101);
|
||||
- ``ee.gripper_pos`` = ``(1 - closedness) * _GRIPPER_MOTOR_SCALE`` (jaw target [0, 100],
|
||||
RANGE_0_100 where 100 = open, so closedness is inverted).
|
||||
|
||||
Input keys: ``ee_pose`` ``(7,)`` ``[x,y,z,qx,qy,qz,qw]``, ``closedness`` float in [0, 1].
|
||||
"""
|
||||
|
||||
def action(self, action: RobotAction) -> RobotAction:
|
||||
ee_pose = action.pop("ee_pose")
|
||||
closedness = float(action.pop("closedness"))
|
||||
|
||||
action["ee.x"] = float(ee_pose[0])
|
||||
action["ee.y"] = float(ee_pose[1])
|
||||
action["ee.z"] = float(ee_pose[2])
|
||||
# Orientation target as a rotvec (quat [qx,qy,qz,qw] -> axis-angle); the IK
|
||||
# consumes ee.w* as a rotvec and tracks it with orientation_weight.
|
||||
rotvec = Rotation.from_quat(ee_pose[3:7]).as_rotvec()
|
||||
action["ee.wx"] = float(rotvec[0])
|
||||
action["ee.wy"] = float(rotvec[1])
|
||||
action["ee.wz"] = float(rotvec[2])
|
||||
# Inverted: closedness c=1 (closed) -> 0, c=0 (open) -> 100 (SO-101 calibration).
|
||||
action["ee.gripper_pos"] = (1.0 - closedness) * _GRIPPER_MOTOR_SCALE
|
||||
return action
|
||||
|
||||
def transform_features(
|
||||
self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
|
||||
) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
|
||||
for feat in ["ee_pose", "closedness"]:
|
||||
features[PipelineFeatureType.ACTION].pop(feat, None)
|
||||
|
||||
for feat in [
|
||||
"ee.x",
|
||||
"ee.y",
|
||||
"ee.z",
|
||||
"ee.wx",
|
||||
"ee.wy",
|
||||
"ee.wz",
|
||||
"ee.gripper_pos",
|
||||
]:
|
||||
features[PipelineFeatureType.ACTION][feat] = PolicyFeature(type=FeatureType.ACTION, shape=(1,))
|
||||
|
||||
return features
|
||||
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Save the current SO-101 joint positions as the reset-origin pose (override).
|
||||
|
||||
Move the arm to the desired reset pose by hand (torque off), then run this script to write
|
||||
those joints to a per-arm file in the LeRobot cache. ``teleoperate.py`` / ``record.py`` load
|
||||
it on startup (matched by ``--robot.id``) as the reset target instead of the defaults.
|
||||
|
||||
Usage::
|
||||
|
||||
# 1. Move arm to desired reset pose by hand
|
||||
python -m examples.isaac_teleop_to_so101.override_reset_pose [--port /dev/ttyACM0] [--id so101_follower_arm]
|
||||
|
||||
# 2. Launch teleop with the SAME --robot.id — it will now reset to this pose on startup
|
||||
python -m examples.isaac_teleop_to_so101.teleoperate --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm --teleop.type=xr_controller
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from lerobot.robots.so_follower import SO100Follower, SO100FollowerConfig
|
||||
|
||||
from .common import RESET_POSE_FILE
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
|
||||
)
|
||||
parser.add_argument("--port", type=str, default="/dev/ttyACM0")
|
||||
parser.add_argument("--id", type=str, default="so101_follower_arm")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
robot = SO100Follower(SO100FollowerConfig(port=args.port, id=args.id, use_degrees=True))
|
||||
robot.connect()
|
||||
# Always disconnect the follower so a failure never leaks the serial connection.
|
||||
try:
|
||||
obs = robot.get_observation()
|
||||
motor_names = list(robot.bus.motors.keys())
|
||||
pose = {name: float(obs[f"{name}.pos"]) for name in motor_names}
|
||||
finally:
|
||||
robot.disconnect()
|
||||
|
||||
print("Current joint positions:")
|
||||
for name, val in pose.items():
|
||||
print(f" {name:20s}: {val:.2f}")
|
||||
|
||||
reset_pose_file = Path(RESET_POSE_FILE.format(robot_name=robot.name, robot_id=robot.id))
|
||||
reset_pose_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
reset_pose_file.write_text(json.dumps(pose, indent=2))
|
||||
print(f"\nSaved to {reset_pose_file}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,321 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Record a LeRobot dataset via NVIDIA Isaac Teleop -> SO-101.
|
||||
|
||||
Runs ``teleoperate.py``'s control loop while also saving each frame to a LeRobot dataset.
|
||||
``--teleop.type`` selects the device (``xr_controller`` | ``so101_leader``) as in
|
||||
``teleoperate.py``.
|
||||
|
||||
Usage::
|
||||
|
||||
# XR (VR) controller: clutch + soft-orientation IK
|
||||
python -m examples.isaac_teleop_to_so101.record \\
|
||||
--robot.type=so101_follower \\
|
||||
--robot.port=/dev/ttyACM0 \\
|
||||
--robot.id=so101_follower_arm \\
|
||||
--teleop.type=xr_controller \\
|
||||
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \\
|
||||
--dataset.repo_id=<hf_user>/<dataset_name> \\
|
||||
--dataset.single_task="Pick up vial from rack on the left side" \\
|
||||
--dataset.num_episodes=3 \\
|
||||
--dataset.episode_time_s=20 \\
|
||||
--dataset.reset_time_s=5
|
||||
|
||||
# SO-101 leader arm: 1:1 joint mirror (real leader on /dev/ttyACM1)
|
||||
python -m examples.isaac_teleop_to_so101.record \\
|
||||
--robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm \\
|
||||
--teleop.type=so101_leader --teleop.port=/dev/ttyACM1 --teleop.id=so101_leader_arm \\
|
||||
--launch_plugin=/path/to/IsaacTeleop/install/plugins/so101_leader/so101_leader_plugin \\
|
||||
--dataset.repo_id=<hf_user>/<dataset_name> --dataset.single_task="Pick up the cube" \\
|
||||
--dataset.num_episodes=3 --dataset.episode_time_s=20 --dataset.reset_time_s=5
|
||||
|
||||
The loop/launch knobs mirror ``teleoperate.py`` (tagged ``[xr]`` / ``[leader]`` below).
|
||||
|
||||
Keyboard shortcuts: Right/n = end episode early and save, Left/r = discard + re-record,
|
||||
Esc/q = stop after the current episode. All frames are recorded (including hold frames).
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import asdict, dataclass
|
||||
from pprint import pformat
|
||||
|
||||
from lerobot.cameras import CameraConfig # noqa: F401
|
||||
from lerobot.cameras.opencv import OpenCVCameraConfig # noqa: F401
|
||||
from lerobot.common.control_utils import sanity_check_dataset_robot_compatibility
|
||||
from lerobot.configs import parser
|
||||
from lerobot.configs.dataset import DatasetRecordConfig
|
||||
from lerobot.datasets import (
|
||||
LeRobotDataset,
|
||||
VideoEncodingManager,
|
||||
aggregate_pipeline_dataset_features,
|
||||
create_initial_features,
|
||||
safe_stop_image_writer,
|
||||
)
|
||||
from lerobot.processor import make_default_processors
|
||||
from lerobot.robots import RobotConfig
|
||||
from lerobot.robots.so_follower import SOFollowerConfig # noqa: F401 (registers so101_follower)
|
||||
from lerobot.utils.constants import ACTION, OBS_STR
|
||||
from lerobot.utils.feature_utils import build_dataset_frame, combine_feature_dicts
|
||||
from lerobot.utils.robot_utils import precise_sleep
|
||||
from lerobot.utils.utils import init_logging
|
||||
|
||||
from .common import (
|
||||
ALIGN_DURATION_S,
|
||||
RESET_DURATION_S,
|
||||
Device,
|
||||
HoldLatch,
|
||||
build_device,
|
||||
init_keyboard_listener,
|
||||
)
|
||||
from .isaac_teleop import IsaacTeleopConfig
|
||||
|
||||
|
||||
@dataclass
|
||||
class RecordConfig:
|
||||
"""CLI config for Isaac Teleop -> SO-101 dataset recording.
|
||||
|
||||
``--robot.*`` / ``--teleop.*`` / ``--dataset.*`` configure the follower, device, and
|
||||
recording; the loop/launch knobs below carry the same ``[xr]`` / ``[leader]`` tags as
|
||||
``teleoperate.py``. Use ``--flag=false`` for booleans (draccus style).
|
||||
"""
|
||||
|
||||
robot: RobotConfig
|
||||
# --teleop.type=xr_controller|so101_leader, resolved against IsaacTeleopConfig's registry.
|
||||
teleop: IsaacTeleopConfig
|
||||
dataset: DatasetRecordConfig
|
||||
|
||||
# [leader] Path to the so101_leader plugin binary to spawn after CloudXR is up (it then
|
||||
# inherits the runtime env). None (default) -> assume the plugin already runs externally.
|
||||
launch_plugin: str | None = None
|
||||
|
||||
# [xr] Slew all joints to the reset pose before the first episode (--reset_to_origin=false to
|
||||
# keep the arm where it is). After the slew the clutch seeds its home from the measured pose.
|
||||
reset_to_origin: bool = True
|
||||
# [xr] Duration [s] of the reset-to-origin slew (passed through to setup_xr).
|
||||
reset_duration: float = RESET_DURATION_S
|
||||
|
||||
# [leader] Slew the follower to the leader's first pose before mirroring (--align=false to
|
||||
# begin the 1:1 mirror immediately; the follower may snap).
|
||||
align: bool = True
|
||||
# [leader] Duration [s] of the startup alignment slew.
|
||||
align_duration: float = ALIGN_DURATION_S
|
||||
|
||||
# Resume recording on an existing (previously interrupted) dataset.
|
||||
resume: bool = False
|
||||
|
||||
|
||||
@safe_stop_image_writer
|
||||
def _record_loop(
|
||||
robot,
|
||||
device: Device,
|
||||
motor_names: list[str],
|
||||
events: dict,
|
||||
fps: int,
|
||||
dataset: LeRobotDataset | None = None,
|
||||
control_time_s: float = 0.0,
|
||||
single_task: str | None = None,
|
||||
) -> None:
|
||||
"""Run one episode (or reset phase) of the control loop.
|
||||
|
||||
When ``dataset`` is None the loop still controls the robot (so the operator
|
||||
can reposition the arm during the reset window) but does not record frames.
|
||||
"""
|
||||
control_interval = 1.0 / fps
|
||||
timestamp = 0.0
|
||||
start_t = time.perf_counter()
|
||||
record_frames = dataset is not None
|
||||
hold = HoldLatch(motor_names)
|
||||
|
||||
while timestamp < control_time_s:
|
||||
loop_start = time.perf_counter()
|
||||
|
||||
if events["exit_early"]:
|
||||
events["exit_early"] = False
|
||||
break
|
||||
|
||||
obs = robot.get_observation()
|
||||
|
||||
if record_frames:
|
||||
observation_frame = build_dataset_frame(dataset.features, obs, prefix=OBS_STR)
|
||||
|
||||
# Device idle (XR clutch disengaged, or leader stream stale) -> hold the pose
|
||||
# latched on the active->idle edge.
|
||||
action = hold.resolve(device.compute(obs), obs)
|
||||
|
||||
robot.send_action(action)
|
||||
|
||||
if record_frames:
|
||||
action_frame = build_dataset_frame(dataset.features, action, prefix=ACTION)
|
||||
dataset.add_frame({**observation_frame, **action_frame, "task": single_task})
|
||||
|
||||
dt_s = time.perf_counter() - loop_start
|
||||
precise_sleep(max(control_interval - dt_s, 0.0))
|
||||
timestamp = time.perf_counter() - start_t
|
||||
|
||||
|
||||
@parser.wrap()
|
||||
def record(cfg: RecordConfig) -> LeRobotDataset:
|
||||
init_logging()
|
||||
logging.info(pformat(asdict(cfg)))
|
||||
|
||||
# Connect the follower, build the selected Isaac device, and run its pre-loop startup
|
||||
# (reset slew / leader align) — shared with teleoperate.py.
|
||||
robot, device, motor_names = build_device(cfg)
|
||||
|
||||
# Build dataset feature spec. The IK pipeline lives inside device.compute(), so the
|
||||
# action features are exactly robot.action_features (joint positions in degrees).
|
||||
teleop_proc, _, obs_proc = make_default_processors()
|
||||
dataset_features = combine_feature_dicts(
|
||||
aggregate_pipeline_dataset_features(
|
||||
pipeline=teleop_proc,
|
||||
initial_features=create_initial_features(action=robot.action_features),
|
||||
use_videos=cfg.dataset.video,
|
||||
),
|
||||
aggregate_pipeline_dataset_features(
|
||||
pipeline=obs_proc,
|
||||
initial_features=create_initial_features(observation=robot.observation_features),
|
||||
use_videos=cfg.dataset.video,
|
||||
),
|
||||
)
|
||||
|
||||
num_cameras = len(robot.cameras) if hasattr(robot, "cameras") else 0
|
||||
image_writer_threads = cfg.dataset.num_image_writer_threads_per_camera * num_cameras
|
||||
|
||||
dataset: LeRobotDataset | None = None
|
||||
listener = None
|
||||
try:
|
||||
if cfg.resume:
|
||||
dataset = LeRobotDataset.resume(
|
||||
cfg.dataset.repo_id,
|
||||
root=cfg.dataset.root,
|
||||
batch_encoding_size=cfg.dataset.video_encoding_batch_size,
|
||||
rgb_encoder=cfg.dataset.rgb_encoder,
|
||||
depth_encoder=cfg.dataset.depth_encoder,
|
||||
encoder_threads=cfg.dataset.encoder_threads,
|
||||
streaming_encoding=cfg.dataset.streaming_encoding,
|
||||
encoder_queue_maxsize=cfg.dataset.encoder_queue_maxsize,
|
||||
image_writer_processes=cfg.dataset.num_image_writer_processes if num_cameras > 0 else 0,
|
||||
image_writer_threads=image_writer_threads if num_cameras > 0 else 0,
|
||||
)
|
||||
sanity_check_dataset_robot_compatibility(dataset, robot, cfg.dataset.fps, dataset_features)
|
||||
else:
|
||||
cfg.dataset.stamp_repo_id()
|
||||
dataset = LeRobotDataset.create(
|
||||
cfg.dataset.repo_id,
|
||||
cfg.dataset.fps,
|
||||
root=cfg.dataset.root,
|
||||
robot_type=robot.name,
|
||||
features=dataset_features,
|
||||
use_videos=cfg.dataset.video,
|
||||
image_writer_processes=cfg.dataset.num_image_writer_processes,
|
||||
image_writer_threads=image_writer_threads,
|
||||
batch_encoding_size=cfg.dataset.video_encoding_batch_size,
|
||||
rgb_encoder=cfg.dataset.rgb_encoder,
|
||||
depth_encoder=cfg.dataset.depth_encoder,
|
||||
encoder_threads=cfg.dataset.encoder_threads,
|
||||
streaming_encoding=cfg.dataset.streaming_encoding,
|
||||
encoder_queue_maxsize=cfg.dataset.encoder_queue_maxsize,
|
||||
)
|
||||
|
||||
listener, events = init_keyboard_listener()
|
||||
|
||||
loop_kwargs = {
|
||||
"robot": robot,
|
||||
"device": device,
|
||||
"motor_names": motor_names,
|
||||
"events": events,
|
||||
"fps": cfg.dataset.fps,
|
||||
"single_task": cfg.dataset.single_task,
|
||||
}
|
||||
|
||||
with VideoEncodingManager(dataset):
|
||||
recorded_episodes = 0
|
||||
while recorded_episodes < cfg.dataset.num_episodes and not events["stop_recording"]:
|
||||
logging.info(f"Recording episode {dataset.num_episodes}")
|
||||
_record_loop(
|
||||
**loop_kwargs,
|
||||
dataset=dataset,
|
||||
control_time_s=cfg.dataset.episode_time_s,
|
||||
)
|
||||
|
||||
# Reset window: give the operator time to reposition the scene.
|
||||
# Skipped for the last episode (or if stop_recording was set).
|
||||
if not events["stop_recording"] and (
|
||||
recorded_episodes < cfg.dataset.num_episodes - 1 or events["rerecord_episode"]
|
||||
):
|
||||
logging.info("Reset the environment")
|
||||
_record_loop(
|
||||
**loop_kwargs,
|
||||
dataset=None,
|
||||
control_time_s=cfg.dataset.reset_time_s,
|
||||
)
|
||||
|
||||
if events["rerecord_episode"]:
|
||||
logging.info("Re-record episode")
|
||||
events["rerecord_episode"] = False
|
||||
events["exit_early"] = False
|
||||
dataset.clear_episode_buffer()
|
||||
continue
|
||||
|
||||
dataset.save_episode()
|
||||
recorded_episodes += 1
|
||||
|
||||
finally:
|
||||
logging.info("Stop recording")
|
||||
|
||||
# Hardware teardown FIRST, each step guarded: the arm must be freed promptly (not
|
||||
# after a potentially long finalize/encode), a cleanup failure must not skip the
|
||||
# follower disconnect (which is what disables torque), and neither must prevent
|
||||
# the dataset from being finalized below.
|
||||
try:
|
||||
device.cleanup()
|
||||
except Exception:
|
||||
logging.exception("Device cleanup failed")
|
||||
try:
|
||||
if robot.is_connected:
|
||||
robot.disconnect()
|
||||
except Exception:
|
||||
logging.exception("Robot disconnect failed")
|
||||
|
||||
# Restore the terminal before the (potentially long) finalize/encode.
|
||||
if listener is not None:
|
||||
try:
|
||||
listener.stop()
|
||||
except Exception:
|
||||
logging.exception("Keyboard listener stop failed")
|
||||
|
||||
if dataset is not None:
|
||||
dataset.finalize()
|
||||
|
||||
if cfg.dataset.push_to_hub:
|
||||
if dataset is not None and dataset.num_episodes > 0:
|
||||
dataset.push_to_hub(tags=cfg.dataset.tags, private=cfg.dataset.private)
|
||||
else:
|
||||
logging.warning("No episodes saved — skipping push to hub")
|
||||
|
||||
logging.info("Exiting")
|
||||
|
||||
return dataset
|
||||
|
||||
|
||||
def main():
|
||||
record()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 NVIDIA Corporation and The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Teleoperate an SO-101 follower arm via NVIDIA Isaac Teleop.
|
||||
|
||||
``lerobot-teleoperate``-style CLI (draccus): ``--teleop.type`` selects the Isaac device
|
||||
(``xr_controller`` | ``so101_leader``), ``--robot.*`` the follower::
|
||||
|
||||
# XR (VR) controller: clutch + soft-orientation IK
|
||||
python -m examples.isaac_teleop_to_so101.teleoperate --robot.type=so101_follower \
|
||||
--robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm --teleop.type=xr_controller
|
||||
|
||||
# SO-101 leader arm: 1:1 joint mirror (real leader on /dev/ttyACM1)
|
||||
python -m examples.isaac_teleop_to_so101.teleoperate --robot.type=so101_follower \
|
||||
--robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm --teleop.type=so101_leader \
|
||||
--teleop.port=/dev/ttyACM1 --teleop.id=so101_leader_arm \
|
||||
--launch_plugin=/code/Teleop/install/plugins/so101_leader/so101_leader_plugin
|
||||
|
||||
``--teleop.type`` resolves against the Isaac device registry (see :class:`IsaacTeleopConfig`),
|
||||
distinct from the serial ``so101_leader``. The pipelines, clutch/IK/align internals, and
|
||||
reset-pose behavior live in ``common.py``. Requires the ``isaacteleop`` package and an OpenXR
|
||||
runtime (install instructions in this folder's ``README.md``).
|
||||
"""
|
||||
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
from lerobot.configs import parser
|
||||
from lerobot.robots import RobotConfig
|
||||
from lerobot.robots.so_follower import SOFollowerConfig # noqa: F401 (registers so101_follower)
|
||||
from lerobot.utils.robot_utils import precise_sleep
|
||||
|
||||
from .common import (
|
||||
ALIGN_DURATION_S,
|
||||
FPS,
|
||||
RESET_DURATION_S,
|
||||
HoldLatch,
|
||||
build_device,
|
||||
)
|
||||
from .isaac_teleop import IsaacTeleopConfig
|
||||
|
||||
|
||||
@dataclass
|
||||
class TeleoperateConfig:
|
||||
"""``lerobot-teleoperate``-style CLI for the Isaac Teleop -> SO-101 example.
|
||||
|
||||
The fields below are the loop/launch knobs (not part of either device's config); the
|
||||
``[xr]`` / ``[leader]`` tags mark which device a knob applies to. Use ``--flag=false``
|
||||
for booleans (draccus style).
|
||||
"""
|
||||
|
||||
# Isaac Teleop input device + its knobs (--teleop.type=xr_controller|so101_leader,
|
||||
# then --teleop.<field>=...). Resolved against IsaacTeleopConfig's own choice registry.
|
||||
teleop: IsaacTeleopConfig
|
||||
# SO-101 FOLLOWER arm (--robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=...).
|
||||
robot: RobotConfig
|
||||
|
||||
# [leader] Path to the so101_leader plugin binary to spawn AFTER CloudXR is up (it then
|
||||
# inherits the runtime env). None (default) -> assume the plugin already runs externally.
|
||||
# The leader's serial port is --teleop.port (forwarded to the plugin; empty -> synthetic).
|
||||
launch_plugin: str | None = None
|
||||
|
||||
# [xr] Slew all joints to a default reset pose before the loop (--reset_to_origin=false to
|
||||
# keep the arm where it is). After the slew the clutch seeds its home from the measured pose.
|
||||
reset_to_origin: bool = True
|
||||
# [xr] Duration [s] of the reset-to-origin slew.
|
||||
reset_duration: float = RESET_DURATION_S
|
||||
|
||||
# [leader] Slew the follower to the leader's first pose before mirroring (--align=false to
|
||||
# begin the 1:1 mirror immediately; the follower may snap).
|
||||
align: bool = True
|
||||
# [leader] Duration [s] of the startup alignment slew.
|
||||
align_duration: float = ALIGN_DURATION_S
|
||||
|
||||
|
||||
@parser.wrap()
|
||||
def teleoperate(cfg: TeleoperateConfig):
|
||||
robot, device, motor_names = build_device(cfg)
|
||||
hold = HoldLatch(motor_names)
|
||||
try:
|
||||
while True:
|
||||
t0 = time.perf_counter()
|
||||
obs = robot.get_observation()
|
||||
# Idle (compute() -> None) holds the pose latched on the active->idle edge.
|
||||
action = hold.resolve(device.compute(obs), obs)
|
||||
robot.send_action(action)
|
||||
precise_sleep(max(1.0 / FPS - (time.perf_counter() - t0), 0.0))
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
# A failing device cleanup must not skip the follower disconnect (which is what
|
||||
# disables torque on the arm).
|
||||
try:
|
||||
device.cleanup()
|
||||
finally:
|
||||
robot.disconnect()
|
||||
|
||||
|
||||
def main():
|
||||
teleoperate()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+1
-3
@@ -25,7 +25,7 @@ discord = "https://discord.gg/s3KuuzsPFb"
|
||||
|
||||
[project]
|
||||
name = "lerobot"
|
||||
version = "0.5.2"
|
||||
version = "0.6.1"
|
||||
description = "🤗 LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch"
|
||||
dynamic = ["readme"]
|
||||
license = { text = "Apache-2.0" }
|
||||
@@ -413,8 +413,6 @@ ignore = [
|
||||
"__init__.py" = ["F401", "F403", "E402"]
|
||||
# E402: conditional-import guards (TYPE_CHECKING / is_package_available) must precede the imports they protect
|
||||
"src/lerobot/scripts/convert_dataset_v21_to_v30.py" = ["E402"]
|
||||
"src/lerobot/policies/wall_x/**" = ["N801", "N812", "SIM102", "SIM108", "SIM210", "SIM211", "B006", "B007", "SIM118"] # Supprese these as they are coming from original Qwen2_5_vl code TODO(pepijn): refactor original
|
||||
|
||||
[tool.ruff.lint.isort]
|
||||
combine-as-imports = true
|
||||
known-first-party = ["lerobot"]
|
||||
|
||||
@@ -65,6 +65,14 @@ class PlanConfig:
|
||||
# invented from the task text (+1 VLM call/episode).
|
||||
subtask_describe_first: bool = True
|
||||
|
||||
# Seeded relabeling: after segmentation, re-label each span with a focused
|
||||
# pass that sees the previous / current / next segment contact sheets and
|
||||
# minimally corrects the seed label (macrodata's best end-to-end labeling
|
||||
# step). Costs +1 VLM call per subtask; off by default.
|
||||
subtask_seeded_relabel: bool = False
|
||||
# Frames sampled uniformly per segment sheet in the relabel pass.
|
||||
subtask_relabel_frames: int = 5
|
||||
|
||||
# Emit ``style="plan"`` rows at each boundary; False = subtasks + memory only.
|
||||
emit_plan: bool = True
|
||||
|
||||
@@ -160,6 +168,11 @@ class VlmConfig:
|
||||
# Forwarded as extra_body.chat_template_kwargs (e.g. {"enable_thinking": false}).
|
||||
chat_template_kwargs: dict[str, Any] | None = None
|
||||
|
||||
# OpenAI-style thinking budget hint ("low"/"medium"/"high"); forwarded to
|
||||
# the server when set. Used to cap a thinking model's reasoning so it
|
||||
# leaves tokens for the actual JSON answer on OpenAI-compatible endpoints.
|
||||
reasoning_effort: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ExecutorConfig:
|
||||
|
||||
@@ -413,7 +413,16 @@ def _draw_timestamp_badge(image: PIL.Image.Image, timestamp: float) -> PIL.Image
|
||||
|
||||
result = image.copy()
|
||||
draw = ImageDraw.Draw(result)
|
||||
font = ImageFont.load_default()
|
||||
# Scale the timestamp to the tile so it stays legible after the model
|
||||
# downsamples the full sheet into 768px tiles — a tiny bitmap font blurs
|
||||
# at contact-sheet resolution and the VLM can no longer read the exact
|
||||
# source time, which is what the boundary score depends on. ``size=`` is
|
||||
# supported by Pillow's bitmap default since 10.1; fall back otherwise.
|
||||
badge_px = max(14, round(image.height * 0.12))
|
||||
try:
|
||||
font = ImageFont.load_default(size=badge_px)
|
||||
except TypeError:
|
||||
font = ImageFont.load_default()
|
||||
label = f"{timestamp:06.2f}s"
|
||||
left, top, right, bottom = draw.textbbox((0, 0), label, font=font)
|
||||
text_w, text_h = right - left, bottom - top
|
||||
|
||||
@@ -116,6 +116,8 @@ class PlanSubtasksMemoryModule:
|
||||
rows.extend(self._task_aug_rows([effective_task, *variants], t0))
|
||||
|
||||
subtask_spans = self._generate_subtasks(record, task=effective_task)
|
||||
if self.config.subtask_seeded_relabel and subtask_spans:
|
||||
subtask_spans = self._seeded_relabel(record, subtask_spans, effective_task)
|
||||
|
||||
# subtask rows
|
||||
for span in subtask_spans:
|
||||
@@ -509,6 +511,51 @@ class PlanSubtasksMemoryModule:
|
||||
|
||||
return cleaned
|
||||
|
||||
def _seeded_relabel(
|
||||
self, record: EpisodeRecord, spans: list[dict[str, Any]], task: str
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Re-label each span using prev/current/next segment contact sheets.
|
||||
|
||||
Boundaries are kept fixed; only ``text`` is refined. The original
|
||||
("seed") label is passed as a strong prior so the model verifies and
|
||||
minimally corrects it rather than re-describing from scratch — the
|
||||
macrodata seeded-relabeling step. One VLM call per span.
|
||||
"""
|
||||
n = len(spans)
|
||||
out: list[dict[str, Any]] = []
|
||||
for i, span in enumerate(spans):
|
||||
content: list[dict[str, Any]] = []
|
||||
if i > 0:
|
||||
content += self._segment_sheet(record, spans[i - 1])
|
||||
content += self._segment_sheet(record, span)
|
||||
if i < n - 1:
|
||||
content += self._segment_sheet(record, spans[i + 1])
|
||||
prompt = load_prompt("plan_subtask_relabel").format(
|
||||
episode_task=task,
|
||||
seed_label=span["text"],
|
||||
segment_index=i + 1,
|
||||
segment_count=n,
|
||||
start=float(span["start"]),
|
||||
end=float(span["end"]),
|
||||
)
|
||||
content.append({"type": "text", "text": prompt})
|
||||
label = self._vlm_field([{"role": "user", "content": content}], "label")
|
||||
text = label.strip() if isinstance(label, str) and label.strip() else span["text"]
|
||||
out.append({**span, "text": text})
|
||||
return out
|
||||
|
||||
def _segment_sheet(self, record: EpisodeRecord, span: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""Contact-sheet block(s) for one span: up to N frames sampled uniformly."""
|
||||
s, e = float(span["start"]), float(span["end"])
|
||||
n = max(1, int(self.config.subtask_relabel_frames))
|
||||
if e <= s or n == 1:
|
||||
timestamps = [s]
|
||||
else:
|
||||
step = (e - s) / (n - 1)
|
||||
timestamps = [s + i * step for i in range(n)]
|
||||
frames = self.frame_provider.frames_at(record, timestamps)
|
||||
return self._contact_sheet_blocks(frames, timestamps[: len(frames)])
|
||||
|
||||
def _generate_subtasks_windowed(
|
||||
self, record: EpisodeRecord, task: str, window_s: float
|
||||
) -> list[dict[str, Any]]:
|
||||
|
||||
@@ -22,12 +22,23 @@ plain editors and roundtrip cleanly through ``ruff format``.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
_DIR = Path(__file__).parent
|
||||
|
||||
|
||||
def load(name: str) -> str:
|
||||
"""Read prompt template ``name.txt`` from the ``prompts/`` directory."""
|
||||
"""Read prompt template ``name.txt`` from the ``prompts/`` directory.
|
||||
|
||||
A ``LEROBOT_PROMPT_OVERRIDE_<name>`` environment variable, when set to a
|
||||
non-empty value, takes precedence over the packaged file. This lets prompt
|
||||
search (e.g. GEPA) inject candidate templates into a remote job without
|
||||
rebuilding the package; the override must keep the same ``{placeholder}``
|
||||
fields the call site formats in.
|
||||
"""
|
||||
override = os.environ.get(f"LEROBOT_PROMPT_OVERRIDE_{name}")
|
||||
if override and override.strip():
|
||||
return override
|
||||
path = _DIR / f"{name}.txt"
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
Annotate one fixed segment from a longer robot demonstration.
|
||||
|
||||
Return only JSON:
|
||||
{{"label": "<short descriptive subtask label>"}}
|
||||
|
||||
You are shown up to three timestamped contact sheets, in order:
|
||||
- The FIRST sheet is the PREVIOUS segment (context only); it may be absent.
|
||||
- The SECOND sheet is the CURRENT target segment.
|
||||
- The THIRD sheet is the NEXT segment (context only); it may be absent.
|
||||
Each tile has its timestamp (seconds, absolute video time) burned into its
|
||||
top-left corner.
|
||||
|
||||
Episode instruction: "{episode_task}"
|
||||
Target segment: {segment_index} of {segment_count}
|
||||
Target time: {start:.2f}s to {end:.2f}s
|
||||
Original predicted label for this exact segment: "{seed_label}"
|
||||
|
||||
Rules:
|
||||
- Label ONLY the current target segment (the second sheet). Use the
|
||||
previous/next sheets only to disambiguate what changed.
|
||||
- Treat the original predicted label as a STRONG PRIOR, not ground truth:
|
||||
verify it against the current segment and correct it minimally.
|
||||
- If it already names the right action and main object, keep it; only fix
|
||||
grammar or add a clearly visible essential detail.
|
||||
- If it is vague but directionally correct, make it more specific.
|
||||
- If it describes the previous/next segment, the wrong action, wrong
|
||||
object, wrong destination, or a wrong state change, replace it.
|
||||
- Do not describe the previous or next segment, and do not split, merge,
|
||||
or move the fixed segment.
|
||||
- Do not introduce an action that is not clearly visible in the current
|
||||
target segment.
|
||||
- Use one concise imperative phrase. Name the manipulated object and the
|
||||
action / state change. Include source, destination, side, direction,
|
||||
final placement, or opened/closed state when visible and central.
|
||||
- Do not mention timestamps, frame numbers, uncertainty, or intent.
|
||||
@@ -1,112 +1,68 @@
|
||||
You are labeling a teleoperated robot demonstration.
|
||||
You are annotating a teleoperated robot demonstration shown as
|
||||
timestamped contact sheets (each tile has its time in seconds burned
|
||||
into the top-left corner). The operator's goal was: "{episode_task}"
|
||||
|
||||
The user originally asked: "{episode_task}"
|
||||
{observation_block}Reconstruct the sequence of COMPLETED manipulation events the robot
|
||||
performs, in chronological order. Output one segment per event with a
|
||||
[start, end] time in seconds and a short action label.
|
||||
|
||||
You are shown the entire demonstration as a single video. Watch the
|
||||
whole clip, then segment it into a list of consecutive atomic subtasks
|
||||
the robot performs.
|
||||
GROUNDING — read first, it overrides everything below:
|
||||
- Label ONLY events you can SEE in the frames. The instruction is the
|
||||
goal; the VIDEO is the ground truth for what actually happened.
|
||||
- Do NOT invent, anticipate, or pad steps that are not shown.
|
||||
|
||||
{observation_block}GROUNDING — read this first, it overrides everything below:
|
||||
- Label ONLY what the robot actually does in the video. Every subtask
|
||||
you emit must correspond to motion you can SEE in specific frames.
|
||||
- Do NOT invent, anticipate, or pad. If the robot only does one thing
|
||||
(e.g. it just navigates to a location and the clip ends), emit
|
||||
EXACTLY ONE subtask. Many demonstrations are a single atomic skill.
|
||||
- ``max_steps`` below is a hard CEILING, not a target. Emitting fewer
|
||||
subtasks than the ceiling is not just allowed, it is expected for
|
||||
short / atomic demonstrations. One correct subtask is far better
|
||||
than several invented ones.
|
||||
- If the video does not clearly show the action implied by the task,
|
||||
describe what you actually see — do NOT fabricate the task's steps
|
||||
from the instruction text. The instruction tells you the goal; the
|
||||
VIDEO is the ground truth for what happened.
|
||||
Granularity — segment by completed events, not by motion:
|
||||
- Start a NEW segment whenever the world state changes: an object is
|
||||
grasped, lifted, transported, placed, or released; a held object
|
||||
changes; a drawer/door/lid/container opens or closes; contents move
|
||||
between containers (poured); a tool starts or stops acting on a
|
||||
surface. Watch the gripper open/close transitions — they usually mark
|
||||
boundaries.
|
||||
- Do NOT split approach, reach, grasp adjustment, small repositioning,
|
||||
hesitation, or retreat into their own segments. Fold each into the
|
||||
event it belongs to (the approach is part of the pick; the retreat is
|
||||
part of the place).
|
||||
- Do NOT merge separate completed events. Each distinct pick, place,
|
||||
open, close, pour, push, wipe, or insert is its own segment, even when
|
||||
they repeat on different objects or locations.
|
||||
- Most segments last 2-10 seconds. Shorter segments are okay ONLY for
|
||||
fast pick / place / open / close / release events. Never emit a
|
||||
segment shorter than {min_subtask_seconds} seconds; merge a too-short
|
||||
candidate into its neighbour instead.
|
||||
- Skip idle time, pure camera motion, and tiny hand jitter.
|
||||
|
||||
Authoring rules — Hi Robot atom granularity, pi0.7-style short prompts:
|
||||
Labels — short imperative phrases:
|
||||
- One concise command naming the action and the manipulated object, e.g.
|
||||
"pick up the red cup", "put the cup on the shelf", "open the top
|
||||
drawer", "pour water into the glass", "insert the plug into the
|
||||
socket".
|
||||
- Include source, destination, side, direction, or the final
|
||||
open/closed state when it is visible and central to the event.
|
||||
- Prefer these verbs (extend only when none fits): pick up, put, place,
|
||||
push, pull, turn, press, open, close, pour, insert, wipe, stack.
|
||||
Disambiguate by what you SEE:
|
||||
* STACK vs PUT: object placed ON TOP OF another object -> "stack".
|
||||
* INSERT vs PUT: object pushed INTO a fitted slot/hole/socket -> "insert".
|
||||
* PICK UP vs PUT (direction): gripper CLOSES and object moves WITH
|
||||
the hand -> "pick up"; gripper OPENS and object stays -> "put".
|
||||
* POUR vs PUT: source is tilted and contents flow -> "pour".
|
||||
- Use the exact object nouns implied by the task; stay consistent across
|
||||
the episode (don't switch "cube" to "block").
|
||||
- Write imperative commands, never third person ("the robot ..."), and
|
||||
drop articles/adverbs.
|
||||
|
||||
- Each subtask = one COMPOSITE atomic skill the low-level policy can
|
||||
execute end-to-end. A "skill" bundles its own approach motion with
|
||||
its terminal action — do NOT split the approach off as its own
|
||||
subtask. The whole-arm policy already learns to reach as part of
|
||||
every manipulation primitive.
|
||||
- Write each subtask as an IMPERATIVE COMMAND, starting with one of
|
||||
these verbs (extend only when none fits):
|
||||
pick up <obj> — approach + grasp + lift in one subtask
|
||||
put <obj> on/in <loc> — transport + release in one subtask
|
||||
place <obj> on/in <loc> — synonym of "put"; pick one and stay consistent
|
||||
push <obj> — contact + linear shove
|
||||
pull <obj> — contact + linear retract
|
||||
turn <knob/dial/handle> — rotary actuation
|
||||
press <button> — single-press contact
|
||||
open <drawer/door/lid> — full open motion
|
||||
close <drawer/door/lid> — full close motion
|
||||
pour <src> into <dst> — tilt + flow
|
||||
insert <obj> into <slot>— alignment + push-fit
|
||||
go to <loc> — ONLY when no grasp / actuation follows
|
||||
(e.g. a pure relocation between phases).
|
||||
If the next subtask grasps something at
|
||||
that location, drop "go to ..." and just
|
||||
write "pick up ..." instead.
|
||||
- Forbidden ultra-fine splits — the VLM is NOT allowed to emit these
|
||||
as standalone subtasks; fold them into the parent composite:
|
||||
"move to X" → fold into "pick up X" (or whatever follows)
|
||||
"reach for X" → fold into "pick up X"
|
||||
"grasp X" → fold into "pick up X"
|
||||
"lift X" → fold into "pick up X" (or "put X on Y" if it's
|
||||
the transport phase of a place)
|
||||
"release X" → fold into "put X on Y" (or "place X in Y")
|
||||
- Keep it SHORT — a verb phrase, not a sentence. Drop articles
|
||||
("the", "a") and adverbs ("carefully", "slowly"). Add a "how"
|
||||
detail (which hand, which grasp point) ONLY when it is needed to
|
||||
disambiguate. Every subtask must begin with one of the verbs
|
||||
above (no leading nouns, no "then", no "first").
|
||||
- NEVER use third person. Never write "the robot", "the arm", "the
|
||||
gripper moves", "it picks up" — the robot is implied. Command it,
|
||||
do not describe it.
|
||||
- Use the exact object nouns from the task above. If the task says
|
||||
"cube", every subtask says "cube" — never switch to "block". If it
|
||||
says "box", never switch to "bin"/"container". Keep vocabulary
|
||||
consistent across the whole episode.
|
||||
- Good: "pick up blue cube", "put blue cube in box", "open drawer",
|
||||
"turn red knob", "press start button", "go to sink".
|
||||
- Bad: "move to blue cube" (approach as its own subtask — forbidden,
|
||||
must be folded into "pick up blue cube"); "the robot arm moves
|
||||
towards the blue cube" (third person, too long); "carefully pick
|
||||
up the cube" (adverb, article); "release the yellow block"
|
||||
("block" when the task said "cube", and "release" must be folded
|
||||
into a "put"/"place" subtask).
|
||||
- Subtasks are non-overlapping and cover the full episode in order.
|
||||
Choose the cut points yourself based on what you see in the video
|
||||
(gripper open/close events, contact, regrasps, transitions).
|
||||
- Each subtask spans at least {min_subtask_seconds} seconds. If a
|
||||
candidate span would be shorter, merge it into its neighbour
|
||||
rather than emitting it.
|
||||
- Do not exceed {max_steps} subtasks total. Fewer, larger composites
|
||||
are preferred over many micro-steps.
|
||||
- Every subtask's [start_time, end_time] must lie within
|
||||
[0.0, {episode_duration}] seconds.
|
||||
|
||||
SPECIAL CASES — verb disambiguation (each rule is narrowly visual and
|
||||
fires ONLY on the spatial situation it names; it must not change how you
|
||||
label any other situation):
|
||||
- STACK vs PUT: if an object is placed ON TOP OF another specific object
|
||||
(not on a flat table / shelf / counter), use "stack ... on ...", not
|
||||
"put". "stack blue book on green book", NOT "put blue book on table".
|
||||
- INSERT vs PUT: if an object goes INTO a fitted slot / hole / socket /
|
||||
receptacle (push-fit), use "insert ... into ...", not "put".
|
||||
- RETRIEVE/PICK-UP vs PUT (direction): watch the gripper. If it CLOSES
|
||||
on the object and the object moves WITH the hand, it is "pick up" /
|
||||
"retrieve" (object leaves its location). If the gripper OPENS and the
|
||||
object stays where the hand left it, it is "put" / "place" (object
|
||||
arrives at a location). Decide by which way the object moves, not by
|
||||
where the hand ends up.
|
||||
- POUR vs PUT: only use "pour" when the source is tilted and contents
|
||||
flow out; moving a full container without tilting is "put"/"place".
|
||||
Timing:
|
||||
- Use the burned-in timestamps to set start and end. Boundaries should
|
||||
land on or near a printed time, and every [start, end] must lie within
|
||||
[0.0, {episode_duration}] seconds, be non-overlapping, and cover the
|
||||
episode in order.
|
||||
- Emit at most {max_steps} segments.
|
||||
|
||||
Output strictly valid JSON of shape:
|
||||
|
||||
{{
|
||||
"subtasks": [
|
||||
{{"text": "<short imperative verb phrase>", "start": <float>, "end": <float>}},
|
||||
{{"text": "<short imperative action label>", "start": <float>, "end": <float>}},
|
||||
...
|
||||
]
|
||||
}}
|
||||
|
||||
@@ -285,6 +285,8 @@ def _make_openai_client(config: VlmConfig) -> VlmClient:
|
||||
"max_tokens": max_tok,
|
||||
"temperature": temp,
|
||||
}
|
||||
if config.reasoning_effort:
|
||||
kwargs["reasoning_effort"] = config.reasoning_effort
|
||||
extra_body: dict[str, Any] = {}
|
||||
if send_mm_kwargs and mm_kwargs:
|
||||
extra_body["mm_processor_kwargs"] = {**mm_kwargs, "do_sample_frames": True}
|
||||
@@ -296,7 +298,13 @@ def _make_openai_client(config: VlmConfig) -> VlmClient:
|
||||
chosen = clients[rr_counter["i"] % len(clients)]
|
||||
rr_counter["i"] += 1
|
||||
response = chosen.chat.completions.create(**kwargs)
|
||||
return response.choices[0].message.content or ""
|
||||
# Some OpenAI-compatible servers can return a choice with no message
|
||||
# (safety filter, or a "thinking" model that spends the whole budget
|
||||
# before emitting content). Treat that as an empty reply so the
|
||||
# JSON-retry path handles it instead of crashing the run.
|
||||
choice = response.choices[0] if response.choices else None
|
||||
message = choice.message if choice is not None else None
|
||||
return (message.content if message is not None else None) or ""
|
||||
|
||||
def _gen(batch: Sequence[Sequence[dict[str, Any]]], max_tok: int, temp: float) -> list[str]:
|
||||
if len(batch) <= 1 or config.client_concurrency <= 1:
|
||||
|
||||
@@ -205,24 +205,30 @@ class PreTrainedConfig(draccus.ChoiceRegistry, HubMixin, abc.ABC): # type: igno
|
||||
f"{CONFIG_NAME} not found on the HuggingFace Hub in {model_id}"
|
||||
) from e
|
||||
|
||||
# HACK: Parse the original config to get the config subclass, so that we can
|
||||
# apply cli overrides.
|
||||
# This is very ugly, ideally we'd like to be able to do that natively with draccus
|
||||
# something like --policy.path (in addition to --policy.type)
|
||||
with draccus.config_type("json"):
|
||||
orig_config = draccus.parse(cls, config_file, args=[])
|
||||
|
||||
if config_file is None:
|
||||
raise FileNotFoundError(f"{CONFIG_NAME} not found in {model_id}")
|
||||
|
||||
with open(config_file) as f:
|
||||
config = json.load(f)
|
||||
|
||||
config.pop("type")
|
||||
# Resolve the concrete config subclass from the serialized "type" tag, then parse
|
||||
# the config (with CLI overrides) directly for that class. The "type" key is
|
||||
# stripped because draccus only consumes it when parsing the registry base class.
|
||||
policy_type = config.pop("type", None)
|
||||
if policy_type is None:
|
||||
raise ValueError(f"Missing 'type' field in {CONFIG_NAME} of {model_id}")
|
||||
try:
|
||||
config_cls = cls.get_choice_class(policy_type)
|
||||
except Exception as e:
|
||||
raise ValueError(
|
||||
f"Policy type '{policy_type}' (from {CONFIG_NAME} of {model_id}) is not registered. "
|
||||
f"Available policy types: {cls.get_known_choices()}"
|
||||
) from e
|
||||
|
||||
with tempfile.NamedTemporaryFile("w+", delete=False, suffix=".json") as f:
|
||||
json.dump(config, f)
|
||||
config_file = f.name
|
||||
|
||||
cli_overrides = policy_kwargs.pop("cli_overrides", [])
|
||||
with draccus.config_type("json"):
|
||||
return draccus.parse(orig_config.__class__, config_file, args=cli_overrides)
|
||||
return draccus.parse(config_cls, config_file, args=cli_overrides)
|
||||
|
||||
@@ -32,6 +32,7 @@ from .pretrained import PreTrainedPolicy as PreTrainedPolicy
|
||||
from .smolvla.configuration_smolvla import SmolVLAConfig as SmolVLAConfig
|
||||
from .tdmpc.configuration_tdmpc import TDMPCConfig as TDMPCConfig
|
||||
from .utils import make_robot_action, prepare_observation_for_inference
|
||||
from .vla_jepa.configuration_vla_jepa import VLAJEPAConfig as VLAJEPAConfig
|
||||
from .vqbet.configuration_vqbet import VQBeTConfig as VQBeTConfig
|
||||
from .wall_x.configuration_wall_x import WallXConfig as WallXConfig
|
||||
from .xvla.configuration_xvla import XVLAConfig as XVLAConfig
|
||||
@@ -57,6 +58,7 @@ __all__ = [
|
||||
"PI05Config",
|
||||
"SmolVLAConfig",
|
||||
"TDMPCConfig",
|
||||
"VLAJEPAConfig",
|
||||
"VQBeTConfig",
|
||||
"WallXConfig",
|
||||
"XVLAConfig",
|
||||
|
||||
@@ -18,17 +18,10 @@ from typing import Any
|
||||
import torch
|
||||
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
RenameObservationsProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_pre_post_processors,
|
||||
)
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .configuration_act import ACTConfig
|
||||
|
||||
@@ -54,34 +47,4 @@ def make_act_pre_post_processors(
|
||||
tuple[PolicyProcessorPipeline[dict[str, Any], dict[str, Any]], PolicyProcessorPipeline[PolicyAction, PolicyAction]]: A tuple containing the
|
||||
pre-processor pipeline and the post-processor pipeline.
|
||||
"""
|
||||
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
device=config.device,
|
||||
),
|
||||
]
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
]
|
||||
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_default_pre_post_processors(config, dataset_stats, normalizer_device=config.device)
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Flow-matching sampling primitives shared across policies.
|
||||
|
||||
Canonical versions of the beta-distributed timestep sampler and the forward-Euler
|
||||
denoising loop (with its real-time-chunking hook) that the openpi-derived policies
|
||||
(pi0, pi05, smolvla, eo1) historically each carried a copy of. All functions are
|
||||
stateless; adopting them does not affect checkpoints.
|
||||
"""
|
||||
|
||||
from collections.abc import Callable
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
from torch import Tensor
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from lerobot.policies.rtc.modeling_rtc import RTCProcessor
|
||||
|
||||
|
||||
def sample_beta(alpha: float, beta: float, bsize: int, device) -> Tensor: # see openpi (exact copy)
|
||||
# Beta sampling uses _sample_dirichlet which isn't implemented for MPS, so sample on CPU
|
||||
alpha_t = torch.tensor(alpha, dtype=torch.float32)
|
||||
beta_t = torch.tensor(beta, dtype=torch.float32)
|
||||
dist = torch.distributions.Beta(alpha_t, beta_t)
|
||||
return dist.sample((bsize,)).to(device)
|
||||
|
||||
|
||||
def sample_noise(shape, device) -> Tensor:
|
||||
"""Standard-normal float32 noise, the flow-matching x_1 sample."""
|
||||
return torch.normal(
|
||||
mean=0.0,
|
||||
std=1.0,
|
||||
size=shape,
|
||||
dtype=torch.float32,
|
||||
device=device,
|
||||
)
|
||||
|
||||
|
||||
def sample_time_beta(bsize: int, device, *, alpha: float, beta: float, scale: float, offset: float) -> Tensor:
|
||||
"""Beta-distributed flow-matching timesteps: ``Beta(alpha, beta) * scale + offset`` (openpi convention)."""
|
||||
time_beta = sample_beta(alpha, beta, bsize, device)
|
||||
time = time_beta * scale + offset
|
||||
return time.to(dtype=torch.float32, device=device)
|
||||
|
||||
|
||||
def euler_integrate(
|
||||
denoise_fn: Callable[[Tensor, Tensor], Tensor],
|
||||
noise: Tensor,
|
||||
num_steps: int,
|
||||
*,
|
||||
rtc_processor: "RTCProcessor | None" = None,
|
||||
rtc_enabled: bool = False,
|
||||
inference_delay: int | None = None,
|
||||
prev_chunk_left_over: Tensor | None = None,
|
||||
execution_horizon: int | None = None,
|
||||
) -> Tensor:
|
||||
"""Forward-Euler integration of a velocity field from t=1 (noise) to t=0 (actions).
|
||||
|
||||
This is the openpi sampling loop: ``dt = -1/num_steps``, ``time = 1.0 + step*dt``,
|
||||
``x_t <- x_t + dt * v_t``, with the optional real-time-chunking (RTC) guidance hook
|
||||
wrapping the velocity computation and debug tracking after each step.
|
||||
|
||||
Args:
|
||||
denoise_fn: Computes the velocity ``v_t`` from ``(x_t, time_tensor)`` where
|
||||
``time_tensor`` is a float32 tensor of shape ``(batch_size,)``. The returned
|
||||
velocity must have the same shape and dtype as ``x_t``.
|
||||
noise: Initial sample ``x_1`` of shape ``(batch_size, ...)``.
|
||||
num_steps: Number of Euler steps.
|
||||
rtc_processor: Optional RTC processor. Debug tracking fires whenever it is set and
|
||||
has debugging enabled, even if RTC guidance itself is disabled (this mirrors
|
||||
the historical per-policy loops).
|
||||
rtc_enabled: Whether to route the velocity computation through
|
||||
``rtc_processor.denoise_step`` (requires ``rtc_processor``).
|
||||
inference_delay: RTC guidance parameter, forwarded verbatim.
|
||||
prev_chunk_left_over: RTC guidance parameter, forwarded verbatim.
|
||||
execution_horizon: RTC guidance parameter, forwarded verbatim.
|
||||
"""
|
||||
bsize = noise.shape[0]
|
||||
device = noise.device
|
||||
|
||||
dt = -1.0 / num_steps
|
||||
x_t = noise
|
||||
for step in range(num_steps):
|
||||
time = 1.0 + step * dt
|
||||
time_tensor = torch.tensor(time, dtype=torch.float32, device=device).expand(bsize)
|
||||
|
||||
def denoise_step_partial_call(input_x_t, current_timestep=time_tensor):
|
||||
return denoise_fn(input_x_t, current_timestep)
|
||||
|
||||
if rtc_enabled:
|
||||
v_t = rtc_processor.denoise_step(
|
||||
x_t=x_t,
|
||||
prev_chunk_left_over=prev_chunk_left_over,
|
||||
inference_delay=inference_delay,
|
||||
time=time,
|
||||
original_denoise_step_partial=denoise_step_partial_call,
|
||||
execution_horizon=execution_horizon,
|
||||
)
|
||||
else:
|
||||
v_t = denoise_step_partial_call(x_t)
|
||||
|
||||
x_t = x_t + dt * v_t
|
||||
|
||||
if rtc_processor is not None and rtc_processor.is_debug_enabled():
|
||||
rtc_processor.track(time=time, x_t=x_t, v_t=v_t)
|
||||
|
||||
return x_t
|
||||
@@ -0,0 +1,302 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Helpers shared by the openpi-derived VLA policies (pi0, pi05, pi0_fast, smolvla, eo1, xvla).
|
||||
|
||||
These are the canonical versions of functions that historically were copy-pasted per
|
||||
policy. They are pure (no parameters, no module state), so importing them from here
|
||||
instead of a policy-local copy has no effect on checkpoints.
|
||||
"""
|
||||
|
||||
import math
|
||||
from collections.abc import Callable
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F # noqa: N812
|
||||
from torch import Tensor
|
||||
|
||||
from lerobot.utils.constants import OPENPI_ATTENTION_MASK_VALUE
|
||||
from lerobot.utils.device_utils import get_safe_dtype
|
||||
from lerobot.utils.import_utils import _transformers_available, require_package
|
||||
|
||||
if TYPE_CHECKING or _transformers_available:
|
||||
from transformers import DynamicCache
|
||||
else:
|
||||
DynamicCache = None
|
||||
|
||||
|
||||
def create_sinusoidal_pos_embedding( # see openpi `create_sinusoidal_pos_embedding` (exact copy)
|
||||
time: torch.Tensor, dimension: int, min_period: float, max_period: float, device="cpu"
|
||||
) -> Tensor:
|
||||
"""Computes sine-cosine positional embedding vectors for scalar positions."""
|
||||
if dimension % 2 != 0:
|
||||
raise ValueError(f"dimension ({dimension}) must be divisible by 2")
|
||||
|
||||
if time.ndim != 1:
|
||||
raise ValueError("The time tensor is expected to be of shape `(batch_size, )`.")
|
||||
|
||||
dtype = get_safe_dtype(torch.float64, device.type)
|
||||
fraction = torch.linspace(0.0, 1.0, dimension // 2, dtype=dtype, device=device)
|
||||
period = min_period * (max_period / min_period) ** fraction
|
||||
|
||||
# Compute the outer product
|
||||
scaling_factor = 1.0 / period * 2 * math.pi
|
||||
sin_input = scaling_factor[None, :] * time[:, None]
|
||||
return torch.cat([torch.sin(sin_input), torch.cos(sin_input)], dim=1)
|
||||
|
||||
|
||||
def fuse_action_time_embedding(
|
||||
noisy_actions: Tensor,
|
||||
timestep: Tensor,
|
||||
*,
|
||||
action_in_proj: nn.Module,
|
||||
action_time_mlp_in: nn.Module,
|
||||
action_time_mlp_out: nn.Module,
|
||||
min_period: float,
|
||||
max_period: float,
|
||||
time_emb_dtype: torch.dtype | None = None,
|
||||
apply: Callable[[Callable, Tensor], Tensor] | None = None,
|
||||
) -> Tensor:
|
||||
"""Fuse noisy actions and a diffusion timestep into the action-expert input embedding.
|
||||
|
||||
This is the block copy-pasted across the openpi action-expert policies (pi0, pi05,
|
||||
eo1, smolvla): project actions, add a sine-cosine timestep embedding, concatenate, and
|
||||
run ``mlp_in -> SiLU -> mlp_out``. The ``nn.Linear`` layers are passed in (not owned by
|
||||
this helper) so adopting it does not rename any checkpoint keys.
|
||||
|
||||
Args:
|
||||
noisy_actions: ``(batch, horizon, action_dim)`` noised action chunk.
|
||||
timestep: ``(batch,)`` diffusion timestep.
|
||||
action_in_proj: ``Linear(action_dim, width)``.
|
||||
action_time_mlp_in: ``Linear(2*width, width)``.
|
||||
action_time_mlp_out: ``Linear(width, width)``.
|
||||
min_period / max_period: sine-cosine embedding periods.
|
||||
time_emb_dtype: dtype to cast the time embedding to. ``None`` (default, the
|
||||
smolvla/eo1 convention) uses the projected action dtype; pass ``timestep.dtype``
|
||||
for the pi0/pi05 convention.
|
||||
apply: optional wrapper ``apply(fn, arg) -> fn(arg)`` used to route the two
|
||||
sub-computations through gradient checkpointing (pi0/eo1). Defaults to a direct
|
||||
call (smolvla).
|
||||
"""
|
||||
if apply is None:
|
||||
|
||||
def apply(fn, arg):
|
||||
return fn(arg)
|
||||
|
||||
action_emb = apply(action_in_proj, noisy_actions)
|
||||
|
||||
time_emb = create_sinusoidal_pos_embedding(
|
||||
timestep,
|
||||
action_in_proj.out_features,
|
||||
min_period=min_period,
|
||||
max_period=max_period,
|
||||
device=timestep.device,
|
||||
)
|
||||
time_emb = time_emb.type(dtype=time_emb_dtype if time_emb_dtype is not None else action_emb.dtype)
|
||||
time_emb = time_emb[:, None, :].expand_as(action_emb)
|
||||
action_time_emb = torch.cat([action_emb, time_emb], dim=2)
|
||||
|
||||
def _mlp(x):
|
||||
return action_time_mlp_out(F.silu(action_time_mlp_in(x)))
|
||||
|
||||
return apply(_mlp, action_time_emb)
|
||||
|
||||
|
||||
def make_att_2d_masks(pad_masks: Tensor, att_masks: Tensor) -> Tensor: # see openpi (exact copy)
|
||||
"""Copied from big_vision.
|
||||
|
||||
Tokens can attend to valid inputs tokens which have a cumulative mask_ar
|
||||
smaller or equal to theirs. This way `mask_ar` int[B, N] can be used to
|
||||
setup several types of attention, for example:
|
||||
|
||||
[[1 1 1 1 1 1]]: pure causal attention.
|
||||
|
||||
[[0 0 0 1 1 1]]: prefix-lm attention. The first 3 tokens can attend between
|
||||
themselves and the last 3 tokens have a causal attention. The first
|
||||
entry could also be a 1 without changing behaviour.
|
||||
|
||||
[[1 0 1 0 1 0 0 1 0 0]]: causal attention between 4 blocks. Tokens of a
|
||||
block can attend all previous blocks and all tokens on the same block.
|
||||
|
||||
Args:
|
||||
input_mask: bool[B, N] true if its part of the input, false if padding.
|
||||
mask_ar: int32[B, N] mask that's 1 where previous tokens cannot depend on
|
||||
it and 0 where it shares the same attention mask as the previous token.
|
||||
"""
|
||||
if att_masks.ndim != 2:
|
||||
raise ValueError(att_masks.ndim)
|
||||
if pad_masks.ndim != 2:
|
||||
raise ValueError(pad_masks.ndim)
|
||||
|
||||
cumsum = torch.cumsum(att_masks, dim=1)
|
||||
att_2d_masks = cumsum[:, None, :] <= cumsum[:, :, None]
|
||||
pad_2d_masks = pad_masks[:, None, :] * pad_masks[:, :, None]
|
||||
return att_2d_masks & pad_2d_masks
|
||||
|
||||
|
||||
def prepare_attention_masks_4d(att_2d_masks: Tensor, dtype: torch.dtype | None = None) -> Tensor:
|
||||
"""Expand boolean 2D attention masks to the additive 4D layout expected by transformers.
|
||||
|
||||
Valid positions become 0.0 and masked positions the large negative openpi constant.
|
||||
"""
|
||||
att_2d_masks_4d = att_2d_masks[:, None, :, :]
|
||||
result = torch.where(att_2d_masks_4d, 0.0, OPENPI_ATTENTION_MASK_VALUE)
|
||||
if dtype is not None:
|
||||
result = result.to(dtype=dtype)
|
||||
return result
|
||||
|
||||
|
||||
def clone_past_key_values(past_key_values):
|
||||
"""Clone the DynamicCache returned by prefix prefill for compiled denoising."""
|
||||
if DynamicCache is None:
|
||||
require_package("transformers", extra="transformers-dep")
|
||||
|
||||
return DynamicCache(
|
||||
tuple(
|
||||
(keys.clone(), values.clone(), sliding_window) for keys, values, sliding_window in past_key_values
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def pad_vector(vector: Tensor, new_dim: int, *, truncate: bool = False) -> Tensor:
|
||||
"""Pad the last dimension of a vector to new_dim with zeros.
|
||||
|
||||
Can be (batch_size x sequence_length x features_dimension)
|
||||
or (batch_size x features_dimension)
|
||||
|
||||
With ``truncate=False`` (openpi behavior), vectors whose last dimension is already
|
||||
>= new_dim are returned unchanged. With ``truncate=True`` (xVLA behavior), the last
|
||||
dimension is truncated to exactly ``new_dim`` (which may be 0).
|
||||
"""
|
||||
if vector.shape[-1] == new_dim:
|
||||
return vector
|
||||
if not truncate:
|
||||
if vector.shape[-1] >= new_dim:
|
||||
return vector
|
||||
return F.pad(vector, (0, new_dim - vector.shape[-1]))
|
||||
shape = list(vector.shape)
|
||||
current_dim = shape[-1]
|
||||
shape[-1] = new_dim
|
||||
new_vector = vector.new_zeros(*shape)
|
||||
length = min(current_dim, new_dim)
|
||||
new_vector[..., :length] = vector[..., :length]
|
||||
return new_vector
|
||||
|
||||
|
||||
def resize_with_pad_torch( # see openpi `resize_with_pad_torch` (exact copy)
|
||||
images: torch.Tensor,
|
||||
height: int,
|
||||
width: int,
|
||||
mode: str = "bilinear",
|
||||
) -> torch.Tensor:
|
||||
"""PyTorch version of resize_with_pad. Resizes an image to a target height and width without distortion
|
||||
by padding with black. If the image is float32, it must be in the range [-1, 1].
|
||||
|
||||
Padding is centered (openpi convention). For the top-left-padding variant used by
|
||||
smolvla/xvla, see :func:`resize_with_pad`.
|
||||
|
||||
Args:
|
||||
images: Tensor of shape [*b, h, w, c] or [*b, c, h, w]
|
||||
height: Target height
|
||||
width: Target width
|
||||
mode: Interpolation mode ('bilinear', 'nearest', etc.)
|
||||
|
||||
Returns:
|
||||
Resized and padded tensor with same shape format as input
|
||||
"""
|
||||
# Check if input is in channels-last format [*b, h, w, c] or channels-first [*b, c, h, w]
|
||||
if images.shape[-1] <= 4: # Assume channels-last format
|
||||
channels_last = True
|
||||
if images.dim() == 3:
|
||||
images = images.unsqueeze(0) # Add batch dimension
|
||||
images = images.permute(0, 3, 1, 2) # [b, h, w, c] -> [b, c, h, w]
|
||||
else:
|
||||
channels_last = False
|
||||
if images.dim() == 3:
|
||||
images = images.unsqueeze(0) # Add batch dimension
|
||||
|
||||
batch_size, channels, cur_height, cur_width = images.shape
|
||||
|
||||
# Calculate resize ratio
|
||||
ratio = max(cur_width / width, cur_height / height)
|
||||
resized_height = int(cur_height / ratio)
|
||||
resized_width = int(cur_width / ratio)
|
||||
|
||||
# Resize
|
||||
resized_images = F.interpolate(
|
||||
images,
|
||||
size=(resized_height, resized_width),
|
||||
mode=mode,
|
||||
align_corners=False if mode == "bilinear" else None,
|
||||
)
|
||||
|
||||
# Handle dtype-specific clipping
|
||||
if images.dtype == torch.uint8:
|
||||
resized_images = torch.round(resized_images).clamp(0, 255).to(torch.uint8)
|
||||
elif images.dtype == torch.float32:
|
||||
resized_images = resized_images.clamp(0.0, 1.0)
|
||||
else:
|
||||
raise ValueError(f"Unsupported image dtype: {images.dtype}")
|
||||
|
||||
# Calculate padding
|
||||
pad_h0, remainder_h = divmod(height - resized_height, 2)
|
||||
pad_h1 = pad_h0 + remainder_h
|
||||
pad_w0, remainder_w = divmod(width - resized_width, 2)
|
||||
pad_w1 = pad_w0 + remainder_w
|
||||
|
||||
# Pad
|
||||
constant_value = 0 if images.dtype == torch.uint8 else 0.0
|
||||
padded_images = F.pad(
|
||||
resized_images,
|
||||
(pad_w0, pad_w1, pad_h0, pad_h1), # left, right, top, bottom
|
||||
mode="constant",
|
||||
value=constant_value,
|
||||
)
|
||||
|
||||
# Convert back to original format if needed
|
||||
if channels_last:
|
||||
padded_images = padded_images.permute(0, 2, 3, 1) # [b, c, h, w] -> [b, h, w, c]
|
||||
|
||||
return padded_images
|
||||
|
||||
|
||||
def resize_with_pad(img: torch.Tensor, height: int, width: int, *, pad_value: float) -> torch.Tensor:
|
||||
"""Resize a (b, c, h, w) image without distortion, padding on the LEFT and TOP.
|
||||
|
||||
This is the smolvla/xvla convention. For the centered-padding openpi variant, see
|
||||
:func:`resize_with_pad_torch`. ``pad_value`` is keyword-only on purpose: callers
|
||||
historically used different values (0, -1) and must state their choice explicitly.
|
||||
"""
|
||||
if img.ndim != 4:
|
||||
raise ValueError(f"(b,c,h,w) expected, but got {img.shape}")
|
||||
|
||||
current_height, current_width = img.shape[2:]
|
||||
if current_height == height and current_width == width:
|
||||
return img
|
||||
|
||||
ratio = max(current_width / width, current_height / height)
|
||||
resized_height = int(current_height / ratio)
|
||||
resized_width = int(current_width / ratio)
|
||||
resized_img = F.interpolate(
|
||||
img, size=(resized_height, resized_width), mode="bilinear", align_corners=False
|
||||
)
|
||||
|
||||
pad_height = max(0, height - resized_height)
|
||||
pad_width = max(0, width - resized_width)
|
||||
padded_img = F.pad(resized_img, (pad_width, 0, pad_height, 0), value=pad_value)
|
||||
return padded_img
|
||||
@@ -19,17 +19,10 @@ from typing import Any
|
||||
import torch
|
||||
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
RenameObservationsProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_pre_post_processors,
|
||||
)
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .configuration_diffusion import DiffusionConfig
|
||||
|
||||
@@ -63,32 +56,4 @@ def make_diffusion_pre_post_processors(
|
||||
Returns:
|
||||
A tuple containing the configured pre-processor and post-processor pipelines.
|
||||
"""
|
||||
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
]
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
]
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_default_pre_post_processors(config, dataset_stats)
|
||||
|
||||
@@ -23,24 +23,16 @@ import torch
|
||||
|
||||
from lerobot.configs.types import FeatureType, PipelineFeatureType, PolicyFeature
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
ComplementaryDataProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStep,
|
||||
ProcessorStepRegistry,
|
||||
RenameObservationsProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from lerobot.processor.converters import policy_action_to_transition, transition_to_policy_action
|
||||
from lerobot.types import TransitionKey
|
||||
from lerobot.utils.constants import (
|
||||
OBS_STATE,
|
||||
POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
)
|
||||
from lerobot.utils.constants import OBS_STATE
|
||||
from lerobot.utils.import_utils import _transformers_available, require_package
|
||||
|
||||
from .configuration_eo1 import EO1Config
|
||||
@@ -242,14 +234,12 @@ def make_eo1_pre_post_processors(
|
||||
]:
|
||||
"""Build pre/post processor pipelines for EO1."""
|
||||
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats)
|
||||
|
||||
input_steps: list[ProcessorStep] = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
steps.rename_observations,
|
||||
steps.add_batch_dim,
|
||||
steps.normalize,
|
||||
EO1ConversationTemplateStep(input_features=config.input_features, chunk_size=config.chunk_size),
|
||||
EO1QwenProcessorStep(
|
||||
processor_name=config.vlm_base,
|
||||
@@ -257,27 +247,12 @@ def make_eo1_pre_post_processors(
|
||||
image_max_pixels=config.image_max_pixels,
|
||||
use_fast_processor=config.use_fast_processor,
|
||||
),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
steps.to_device,
|
||||
]
|
||||
|
||||
output_steps: list[ProcessorStep] = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features,
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
steps.unnormalize,
|
||||
steps.to_cpu,
|
||||
]
|
||||
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
@@ -27,9 +27,11 @@ from lerobot.utils.import_utils import _transformers_available, require_package
|
||||
|
||||
if TYPE_CHECKING or _transformers_available:
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
from transformers.utils import is_flash_attn_2_available
|
||||
else:
|
||||
AutoModel = None
|
||||
AutoTokenizer = None
|
||||
is_flash_attn_2_available = None
|
||||
|
||||
IMAGENET_MEAN = (0.485, 0.456, 0.406)
|
||||
IMAGENET_STD = (0.229, 0.224, 0.225)
|
||||
@@ -135,9 +137,13 @@ class InternVL3Embedder(nn.Module):
|
||||
raise ValueError(f"Unsupported EVO1 vlm_dtype '{model_dtype}'") from exc
|
||||
self.model_dtype = model_dtype
|
||||
|
||||
attn_implementation = "flash_attention_2" if (use_flash_attn and _flash_attn_available()) else "eager"
|
||||
attn_implementation = (
|
||||
"flash_attention_2" if (use_flash_attn and is_flash_attn_2_available()) else "eager"
|
||||
)
|
||||
if use_flash_attn and attn_implementation == "eager":
|
||||
logger.warning("flash_attn is not installed. Falling back to eager attention.")
|
||||
logger.warning(
|
||||
"Flash Attention 2 is unavailable on this runtime. Falling back to eager attention."
|
||||
)
|
||||
|
||||
self.model = AutoModel.from_pretrained(
|
||||
model_name,
|
||||
@@ -359,11 +365,3 @@ class InternVL3Embedder(nn.Module):
|
||||
@property
|
||||
def device(self) -> torch.device:
|
||||
return next(self.model.parameters()).device
|
||||
|
||||
|
||||
def _flash_attn_available() -> bool:
|
||||
try:
|
||||
import flash_attn # noqa: F401
|
||||
except ModuleNotFoundError:
|
||||
return False
|
||||
return True
|
||||
|
||||
+66
-318
@@ -17,6 +17,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import inspect
|
||||
import logging
|
||||
from typing import TYPE_CHECKING, Any, TypedDict, Unpack
|
||||
|
||||
@@ -44,26 +45,10 @@ from lerobot.utils.constants import (
|
||||
)
|
||||
from lerobot.utils.feature_utils import dataset_to_policy_features
|
||||
|
||||
from .act.configuration_act import ACTConfig
|
||||
from .diffusion.configuration_diffusion import DiffusionConfig
|
||||
from .eo1.configuration_eo1 import EO1Config
|
||||
from .evo1.configuration_evo1 import Evo1Config
|
||||
from .fastwam.configuration_fastwam import FastWAMConfig
|
||||
from .gaussian_actor.configuration_gaussian_actor import GaussianActorConfig
|
||||
from .groot.configuration_groot import GrootConfig
|
||||
from .lingbot_va.configuration_lingbot_va import LingBotVAConfig
|
||||
from .molmoact2.configuration_molmoact2 import MolmoAct2Config
|
||||
from .multi_task_dit.configuration_multi_task_dit import MultiTaskDiTConfig
|
||||
from .pi0.configuration_pi0 import PI0Config
|
||||
from .pi05.configuration_pi05 import PI05Config
|
||||
from .pretrained import PreTrainedPolicy
|
||||
from .smolvla.configuration_smolvla import SmolVLAConfig
|
||||
from .tdmpc.configuration_tdmpc import TDMPCConfig
|
||||
from .utils import validate_visual_features_consistency
|
||||
from .vla_jepa.configuration_vla_jepa import VLAJEPAConfig
|
||||
from .vqbet.configuration_vqbet import VQBeTConfig
|
||||
from .wall_x.configuration_wall_x import WallXConfig
|
||||
from .xvla.configuration_xvla import XVLAConfig
|
||||
|
||||
|
||||
def _reconnect_relative_absolute_steps(
|
||||
@@ -88,100 +73,23 @@ def get_policy_class(name: str) -> type[PreTrainedPolicy]:
|
||||
"""
|
||||
Retrieves a policy class by its registered name.
|
||||
|
||||
This function uses dynamic imports to avoid loading all policy classes into memory
|
||||
at once, improving startup time and reducing dependencies.
|
||||
Resolution is convention-based: the draccus-registered config class of ``name`` is
|
||||
looked up, its ``configuration_*`` module path is rewritten to ``modeling_*``, and
|
||||
the ``<X>Policy`` class is imported from there. The modeling module is only imported
|
||||
at call time, keeping heavy optional dependencies lazy. This works for both built-in
|
||||
policies and third-party lerobot plugins (anything registered via
|
||||
``@PreTrainedConfig.register_subclass``).
|
||||
|
||||
Args:
|
||||
name: The name of the policy. Supported names are "tdmpc", "diffusion", "act",
|
||||
"multi_task_dit", "vqbet", "pi0", "pi05", "gaussian_actor", "smolvla", "wall_x",
|
||||
"molmoact2", "eo1", "evo1".
|
||||
name: The registered name of the policy (e.g. "act", "diffusion", "pi0").
|
||||
Returns:
|
||||
The policy class corresponding to the given name.
|
||||
|
||||
Raises:
|
||||
NotImplementedError: If the policy name is not recognized.
|
||||
ValueError: If the policy name is not registered.
|
||||
ImportError: If the policy's optional dependencies are not installed.
|
||||
"""
|
||||
if name == "tdmpc":
|
||||
from .tdmpc.modeling_tdmpc import TDMPCPolicy
|
||||
|
||||
return TDMPCPolicy
|
||||
elif name == "diffusion":
|
||||
from .diffusion.modeling_diffusion import DiffusionPolicy
|
||||
|
||||
return DiffusionPolicy
|
||||
elif name == "act":
|
||||
from .act.modeling_act import ACTPolicy
|
||||
|
||||
return ACTPolicy
|
||||
elif name == "multi_task_dit":
|
||||
from .multi_task_dit.modeling_multi_task_dit import MultiTaskDiTPolicy
|
||||
|
||||
return MultiTaskDiTPolicy
|
||||
elif name == "vqbet":
|
||||
from .vqbet.modeling_vqbet import VQBeTPolicy
|
||||
|
||||
return VQBeTPolicy
|
||||
elif name == "pi0":
|
||||
from .pi0.modeling_pi0 import PI0Policy
|
||||
|
||||
return PI0Policy
|
||||
elif name == "pi0_fast":
|
||||
from .pi0_fast.modeling_pi0_fast import PI0FastPolicy
|
||||
|
||||
return PI0FastPolicy
|
||||
elif name == "pi05":
|
||||
from .pi05.modeling_pi05 import PI05Policy
|
||||
|
||||
return PI05Policy
|
||||
elif name == "gaussian_actor":
|
||||
from .gaussian_actor.modeling_gaussian_actor import GaussianActorPolicy
|
||||
|
||||
return GaussianActorPolicy
|
||||
elif name == "smolvla":
|
||||
from .smolvla.modeling_smolvla import SmolVLAPolicy
|
||||
|
||||
return SmolVLAPolicy
|
||||
elif name == "groot":
|
||||
from .groot.modeling_groot import GrootPolicy
|
||||
|
||||
return GrootPolicy
|
||||
elif name == "xvla":
|
||||
from .xvla.modeling_xvla import XVLAPolicy
|
||||
|
||||
return XVLAPolicy
|
||||
elif name == "wall_x":
|
||||
from .wall_x.modeling_wall_x import WallXPolicy
|
||||
|
||||
return WallXPolicy
|
||||
elif name == "eo1":
|
||||
from .eo1.modeling_eo1 import EO1Policy
|
||||
|
||||
return EO1Policy
|
||||
elif name == "molmoact2":
|
||||
from .molmoact2.modeling_molmoact2 import MolmoAct2Policy
|
||||
|
||||
return MolmoAct2Policy
|
||||
elif name == "vla_jepa":
|
||||
from .vla_jepa.modeling_vla_jepa import VLAJEPAPolicy
|
||||
|
||||
return VLAJEPAPolicy
|
||||
elif name == "lingbot_va":
|
||||
from .lingbot_va.modeling_lingbot_va import LingBotVAPolicy
|
||||
|
||||
return LingBotVAPolicy
|
||||
elif name == "fastwam":
|
||||
from .fastwam.modeling_fastwam import FastWAMPolicy
|
||||
|
||||
return FastWAMPolicy
|
||||
elif name == "evo1":
|
||||
from .evo1.modeling_evo1 import Evo1Policy
|
||||
|
||||
return Evo1Policy
|
||||
else:
|
||||
try:
|
||||
return _get_policy_cls_from_policy_name(name=name)
|
||||
except Exception as e:
|
||||
raise ValueError(f"Policy type '{name}' is not available.") from e
|
||||
return _get_policy_cls_from_policy_name(name=name)
|
||||
|
||||
|
||||
def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
|
||||
@@ -192,9 +100,8 @@ def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
|
||||
mapping a string identifier to the corresponding config class.
|
||||
|
||||
Args:
|
||||
policy_type: The type of the policy. Supported types include "tdmpc",
|
||||
"multi_task_dit", "diffusion", "act", "vqbet", "pi0", "pi05", "gaussian_actor",
|
||||
"smolvla", "wall_x", "molmoact2", "eo1", "evo1".
|
||||
policy_type: The registered type of the policy (any name registered via
|
||||
``@PreTrainedConfig.register_subclass``, e.g. "act", "diffusion", "pi0").
|
||||
**kwargs: Keyword arguments to be passed to the configuration class constructor.
|
||||
|
||||
Returns:
|
||||
@@ -203,48 +110,11 @@ def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
|
||||
Raises:
|
||||
ValueError: If the `policy_type` is not recognized.
|
||||
"""
|
||||
if policy_type == "tdmpc":
|
||||
return TDMPCConfig(**kwargs)
|
||||
elif policy_type == "diffusion":
|
||||
return DiffusionConfig(**kwargs)
|
||||
elif policy_type == "act":
|
||||
return ACTConfig(**kwargs)
|
||||
elif policy_type == "multi_task_dit":
|
||||
return MultiTaskDiTConfig(**kwargs)
|
||||
elif policy_type == "vqbet":
|
||||
return VQBeTConfig(**kwargs)
|
||||
elif policy_type == "pi0":
|
||||
return PI0Config(**kwargs)
|
||||
elif policy_type == "pi05":
|
||||
return PI05Config(**kwargs)
|
||||
elif policy_type == "gaussian_actor":
|
||||
return GaussianActorConfig(**kwargs)
|
||||
elif policy_type == "smolvla":
|
||||
return SmolVLAConfig(**kwargs)
|
||||
elif policy_type == "groot":
|
||||
return GrootConfig(**kwargs)
|
||||
elif policy_type == "xvla":
|
||||
return XVLAConfig(**kwargs)
|
||||
elif policy_type == "wall_x":
|
||||
return WallXConfig(**kwargs)
|
||||
elif policy_type == "eo1":
|
||||
return EO1Config(**kwargs)
|
||||
elif policy_type == "molmoact2":
|
||||
return MolmoAct2Config(**kwargs)
|
||||
elif policy_type == "vla_jepa":
|
||||
return VLAJEPAConfig(**kwargs)
|
||||
elif policy_type == "lingbot_va":
|
||||
return LingBotVAConfig(**kwargs)
|
||||
elif policy_type == "fastwam":
|
||||
return FastWAMConfig(**kwargs)
|
||||
elif policy_type == "evo1":
|
||||
return Evo1Config(**kwargs)
|
||||
else:
|
||||
try:
|
||||
config_cls = PreTrainedConfig.get_choice_class(policy_type)
|
||||
return config_cls(**kwargs)
|
||||
except Exception as e:
|
||||
raise ValueError(f"Policy type '{policy_type}' is not available.") from e
|
||||
try:
|
||||
config_cls = PreTrainedConfig.get_choice_class(policy_type)
|
||||
except Exception as e:
|
||||
raise ValueError(f"Policy type '{policy_type}' is not available.") from e
|
||||
return config_cls(**kwargs)
|
||||
|
||||
|
||||
class ProcessorConfigKwargs(TypedDict, total=False):
|
||||
@@ -298,8 +168,7 @@ def make_pre_post_processors(
|
||||
A tuple containing the input (pre-processor) and output (post-processor) pipelines.
|
||||
|
||||
Raises:
|
||||
NotImplementedError: If a processor factory is not implemented for the given
|
||||
policy configuration type.
|
||||
ValueError: If no processor factory exists for the given policy configuration type.
|
||||
"""
|
||||
if pretrained_path:
|
||||
if isinstance(policy_cfg, GrootConfig):
|
||||
@@ -351,166 +220,13 @@ def make_pre_post_processors(
|
||||
)
|
||||
return preprocessor, postprocessor
|
||||
|
||||
# Create a new processor based on policy type
|
||||
if isinstance(policy_cfg, TDMPCConfig):
|
||||
from .tdmpc.processor_tdmpc import make_tdmpc_pre_post_processors
|
||||
|
||||
processors = make_tdmpc_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, DiffusionConfig):
|
||||
from .diffusion.processor_diffusion import make_diffusion_pre_post_processors
|
||||
|
||||
processors = make_diffusion_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, ACTConfig):
|
||||
from .act.processor_act import make_act_pre_post_processors
|
||||
|
||||
processors = make_act_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, MultiTaskDiTConfig):
|
||||
from .multi_task_dit.processor_multi_task_dit import (
|
||||
make_multi_task_dit_pre_post_processors,
|
||||
)
|
||||
|
||||
processors = make_multi_task_dit_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, VQBeTConfig):
|
||||
from .vqbet.processor_vqbet import make_vqbet_pre_post_processors
|
||||
|
||||
processors = make_vqbet_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, PI0Config):
|
||||
from .pi0.processor_pi0 import make_pi0_pre_post_processors
|
||||
|
||||
processors = make_pi0_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, PI05Config):
|
||||
from .pi05.processor_pi05 import make_pi05_pre_post_processors
|
||||
|
||||
processors = make_pi05_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, GaussianActorConfig):
|
||||
from .gaussian_actor.processor_gaussian_actor import make_gaussian_actor_pre_post_processors
|
||||
|
||||
processors = make_gaussian_actor_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, SmolVLAConfig):
|
||||
from .smolvla.processor_smolvla import make_smolvla_pre_post_processors
|
||||
|
||||
processors = make_smolvla_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, GrootConfig):
|
||||
from .groot.processor_groot import make_groot_pre_post_processors
|
||||
|
||||
processors = make_groot_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
dataset_meta=kwargs.get("dataset_meta"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, XVLAConfig):
|
||||
from .xvla.processor_xvla import (
|
||||
make_xvla_pre_post_processors,
|
||||
)
|
||||
|
||||
processors = make_xvla_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, WallXConfig):
|
||||
from .wall_x.processor_wall_x import make_wall_x_pre_post_processors
|
||||
|
||||
processors = make_wall_x_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, EO1Config):
|
||||
from .eo1.processor_eo1 import make_eo1_pre_post_processors
|
||||
|
||||
processors = make_eo1_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
elif isinstance(policy_cfg, Evo1Config):
|
||||
from .evo1.processor_evo1 import make_evo1_pre_post_processors
|
||||
|
||||
processors = make_evo1_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, MolmoAct2Config):
|
||||
from .molmoact2.processor_molmoact2 import make_molmoact2_pre_post_processors
|
||||
|
||||
processors = make_molmoact2_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
dataset_meta=kwargs.get("dataset_meta"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, VLAJEPAConfig):
|
||||
from .vla_jepa.processor_vla_jepa import make_vla_jepa_pre_post_processors
|
||||
|
||||
processors = make_vla_jepa_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, LingBotVAConfig):
|
||||
from .lingbot_va.processor_lingbot_va import make_lingbot_va_pre_post_processors
|
||||
|
||||
processors = make_lingbot_va_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
elif isinstance(policy_cfg, FastWAMConfig):
|
||||
from .fastwam.processor_fastwam import make_fastwam_pre_post_processors
|
||||
|
||||
processors = make_fastwam_pre_post_processors(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
|
||||
else:
|
||||
try:
|
||||
processors = _make_processors_from_policy_config(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
)
|
||||
except Exception as e:
|
||||
raise ValueError(f"Processor for policy type '{policy_cfg.type}' is not implemented.") from e
|
||||
|
||||
return processors
|
||||
# Create new processors from the policy config, resolving the per-policy factory
|
||||
# function by naming convention (lazy import keeps optional dependencies optional).
|
||||
return _make_processors_from_policy_config(
|
||||
config=policy_cfg,
|
||||
dataset_stats=kwargs.get("dataset_stats"),
|
||||
dataset_meta=kwargs.get("dataset_meta"),
|
||||
)
|
||||
|
||||
|
||||
def make_policy(
|
||||
@@ -654,10 +370,12 @@ def make_policy(
|
||||
return policy
|
||||
|
||||
|
||||
def _get_policy_cls_from_policy_name(name: str) -> type[PreTrainedConfig]:
|
||||
def _get_policy_cls_from_policy_name(name: str) -> type[PreTrainedPolicy]:
|
||||
"""Get policy class from its registered name using dynamic imports.
|
||||
|
||||
This is used as a helper function to import policies from 3rd party lerobot plugins.
|
||||
Works for built-in policies and 3rd party lerobot plugins alike: the config class
|
||||
registered under ``name`` is resolved via the draccus ChoiceRegistry, and the policy
|
||||
class is imported from the sibling ``modeling_*`` module by naming convention.
|
||||
|
||||
Args:
|
||||
name: The name of the policy.
|
||||
@@ -683,22 +401,39 @@ def _get_policy_cls_from_policy_name(name: str) -> type[PreTrainedConfig]:
|
||||
"configuration_", "modeling_"
|
||||
) # e.g., configuration_diffusion -> modeling_diffusion
|
||||
|
||||
module = importlib.import_module(module_path)
|
||||
policy_cls = getattr(module, cls_name)
|
||||
try:
|
||||
module = importlib.import_module(module_path)
|
||||
except ModuleNotFoundError as e:
|
||||
if e.name == module_path:
|
||||
# The modeling_* module itself does not exist for this policy type. A missing
|
||||
# optional dependency inside an existing module propagates unchanged instead,
|
||||
# so its actionable install hint stays visible.
|
||||
raise ValueError(f"Policy class for '{name}' is not implemented.") from e
|
||||
raise
|
||||
policy_cls = getattr(module, cls_name, None)
|
||||
if policy_cls is None:
|
||||
raise ValueError(
|
||||
f"Policy class '{cls_name}' not found in '{module_path}'. "
|
||||
f"Policies must expose '<Name>Policy' in the sibling 'modeling_*' module by naming convention."
|
||||
)
|
||||
return policy_cls
|
||||
|
||||
|
||||
def _make_processors_from_policy_config(
|
||||
config: PreTrainedConfig,
|
||||
dataset_stats: dict[str, dict[str, torch.Tensor]] | None = None,
|
||||
dataset_meta: Any | None = None,
|
||||
) -> tuple[Any, Any]:
|
||||
"""Create pre- and post-processors from a policy configuration using dynamic imports.
|
||||
|
||||
This is used as a helper function to import processor factories from 3rd party lerobot plugins.
|
||||
Resolves ``make_{type}_pre_post_processors`` from the policy's ``processor_*`` module
|
||||
by naming convention. Works for built-in policies and 3rd party lerobot plugins.
|
||||
|
||||
Args:
|
||||
config: The policy configuration object.
|
||||
dataset_stats: Dataset statistics for normalization.
|
||||
dataset_meta: Dataset metadata, forwarded only to factories that declare a
|
||||
``dataset_meta`` parameter (e.g. groot, molmoact2).
|
||||
Returns:
|
||||
A tuple containing the input (pre-processor) and output (post-processor) pipelines.
|
||||
"""
|
||||
@@ -711,6 +446,19 @@ def _make_processors_from_policy_config(
|
||||
logging.debug(
|
||||
f"Instantiating pre/post processors using function '{function_name}' from module '{module_path}'"
|
||||
)
|
||||
module = importlib.import_module(module_path)
|
||||
function = getattr(module, function_name)
|
||||
return function(config, dataset_stats=dataset_stats)
|
||||
try:
|
||||
module = importlib.import_module(module_path)
|
||||
except ModuleNotFoundError as e:
|
||||
if e.name == module_path:
|
||||
# The processor_* module itself does not exist for this policy type. A missing
|
||||
# optional dependency inside an existing module propagates unchanged instead,
|
||||
# so its actionable install hint stays visible.
|
||||
raise ValueError(f"Processor for policy type '{policy_type}' is not implemented.") from e
|
||||
raise
|
||||
function = getattr(module, function_name, None)
|
||||
if function is None:
|
||||
raise ValueError(f"Processor for policy type '{policy_type}' is not implemented.")
|
||||
call_kwargs: dict[str, Any] = {"dataset_stats": dataset_stats}
|
||||
if "dataset_meta" in inspect.signature(function).parameters:
|
||||
call_kwargs["dataset_meta"] = dataset_meta
|
||||
return function(config, **call_kwargs)
|
||||
|
||||
@@ -22,20 +22,11 @@ import torch
|
||||
from lerobot.configs import PipelineFeatureType, PolicyFeature
|
||||
from lerobot.processor import (
|
||||
ActionProcessorStep,
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStepRegistry,
|
||||
RenameObservationsProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
)
|
||||
from lerobot.utils.constants import (
|
||||
POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
|
||||
from .configuration_fastwam import FastWAMConfig
|
||||
@@ -105,38 +96,20 @@ def make_fastwam_pre_post_processors(
|
||||
# anyway) and unsafe across fine-tuning: its `resize_size` would be inherited from the base
|
||||
# checkpoint's camera geometry, not this dataset's, making the concatenation N_cameras x too wide.
|
||||
|
||||
steps = make_default_policy_processor_steps(config, normalization_stats, normalizer_device=config.device)
|
||||
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=normalization_stats,
|
||||
device=config.device,
|
||||
),
|
||||
steps.rename_observations,
|
||||
steps.add_batch_dim,
|
||||
steps.to_device,
|
||||
steps.normalize,
|
||||
]
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features,
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=normalization_stats,
|
||||
),
|
||||
steps.unnormalize,
|
||||
]
|
||||
if config.toggle_action_dimensions:
|
||||
output_steps.append(
|
||||
FastWAMActionToggleProcessorStep(toggle_dimensions=config.toggle_action_dimensions)
|
||||
)
|
||||
output_steps.append(DeviceProcessorStep(device="cpu"))
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
output_steps.append(steps.to_cpu)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
@@ -20,17 +20,10 @@ from typing import Any
|
||||
import torch
|
||||
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
RenameObservationsProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_pre_post_processors,
|
||||
)
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .configuration_gaussian_actor import GaussianActorConfig
|
||||
|
||||
@@ -62,33 +55,4 @@ def make_gaussian_actor_pre_post_processors(
|
||||
Returns:
|
||||
A tuple containing the configured pre-processor and post-processor pipelines.
|
||||
"""
|
||||
|
||||
# Add remaining processors
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
]
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
]
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_default_pre_post_processors(config, dataset_stats)
|
||||
|
||||
@@ -25,19 +25,12 @@ import torch
|
||||
|
||||
from lerobot.configs.types import FeatureType, NormalizationMode
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStep,
|
||||
RenameObservationsProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
)
|
||||
from lerobot.processor.converters import policy_action_to_transition, transition_to_policy_action
|
||||
from lerobot.utils.constants import (
|
||||
POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
|
||||
from .configuration_lingbot_va import LingBotVAConfig
|
||||
@@ -52,15 +45,13 @@ def make_lingbot_va_pre_post_processors(
|
||||
]:
|
||||
"""Build the pre/post processor pipelines for LingBot-VA."""
|
||||
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats)
|
||||
|
||||
input_steps: list[ProcessorStep] = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
steps.rename_observations,
|
||||
steps.add_batch_dim,
|
||||
steps.normalize,
|
||||
steps.to_device,
|
||||
]
|
||||
|
||||
# Unnormalize actions from [-1, 1] to physical units (QUANTILES) using q01/q99 restored from the checkpoint.
|
||||
@@ -70,18 +61,7 @@ def make_lingbot_va_pre_post_processors(
|
||||
norm_map={FeatureType.ACTION: NormalizationMode.QUANTILES},
|
||||
stats=dataset_stats,
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
steps.to_cpu,
|
||||
]
|
||||
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
@@ -19,18 +19,12 @@ from typing import Any
|
||||
import torch
|
||||
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
RenameObservationsProcessorStep,
|
||||
TokenizerProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .configuration_multi_task_dit import MultiTaskDiTConfig
|
||||
|
||||
@@ -66,9 +60,11 @@ def make_multi_task_dit_pre_post_processors(
|
||||
A tuple containing the configured pre-processor and post-processor pipelines.
|
||||
"""
|
||||
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats, normalizer_device=config.device)
|
||||
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
steps.rename_observations,
|
||||
steps.add_batch_dim,
|
||||
TokenizerProcessorStep(
|
||||
tokenizer_name=config.text_encoder_name,
|
||||
padding=config.tokenizer_padding,
|
||||
@@ -76,32 +72,12 @@ def make_multi_task_dit_pre_post_processors(
|
||||
max_length=config.tokenizer_max_length,
|
||||
truncation=config.tokenizer_truncation,
|
||||
),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
device=config.device,
|
||||
),
|
||||
steps.to_device,
|
||||
steps.normalize,
|
||||
]
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features,
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
steps.unnormalize,
|
||||
steps.to_cpu,
|
||||
]
|
||||
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
@@ -21,22 +21,16 @@ import torch
|
||||
from lerobot.configs import PipelineFeatureType, PolicyFeature
|
||||
from lerobot.processor import (
|
||||
AbsoluteActionsProcessorStep,
|
||||
AddBatchDimensionProcessorStep,
|
||||
ComplementaryDataProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStep,
|
||||
ProcessorStepRegistry,
|
||||
RelativeActionsProcessorStep,
|
||||
RenameObservationsProcessorStep,
|
||||
TokenizerProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .configuration_pi0 import PI0Config
|
||||
|
||||
@@ -136,10 +130,12 @@ def make_pi0_pre_post_processors(
|
||||
action_names=getattr(config, "action_feature_names", None),
|
||||
)
|
||||
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats)
|
||||
|
||||
# OpenPI order: raw → relative → normalize → model → unnormalize → absolute
|
||||
input_steps: list[ProcessorStep] = [
|
||||
RenameObservationsProcessorStep(rename_map={}), # To mimic the same processor as pretrained one
|
||||
AddBatchDimensionProcessorStep(),
|
||||
steps.rename_observations, # To mimic the same processor as pretrained one
|
||||
steps.add_batch_dim,
|
||||
Pi0NewLineProcessor(), # Add newlines before tokenization for PaliGemma
|
||||
TokenizerProcessorStep(
|
||||
tokenizer_name="google/paligemma-3b-pt-224",
|
||||
@@ -147,32 +143,15 @@ def make_pi0_pre_post_processors(
|
||||
padding_side="right",
|
||||
padding="max_length",
|
||||
),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
steps.to_device,
|
||||
relative_step,
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
steps.normalize,
|
||||
]
|
||||
|
||||
output_steps: list[ProcessorStep] = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
steps.unnormalize,
|
||||
AbsoluteActionsProcessorStep(enabled=config.use_relative_actions, relative_step=relative_step),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
steps.to_cpu,
|
||||
]
|
||||
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
@@ -24,26 +24,17 @@ import torch
|
||||
from lerobot.configs import PipelineFeatureType, PolicyFeature
|
||||
from lerobot.processor import (
|
||||
AbsoluteActionsProcessorStep,
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStep,
|
||||
ProcessorStepRegistry,
|
||||
RelativeActionsProcessorStep,
|
||||
RenameObservationsProcessorStep,
|
||||
TokenizerProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from lerobot.types import EnvTransition, TransitionKey
|
||||
from lerobot.utils.constants import (
|
||||
OBS_STATE,
|
||||
POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
)
|
||||
from lerobot.utils.constants import OBS_STATE
|
||||
|
||||
from .configuration_pi05 import PI05Config
|
||||
|
||||
@@ -135,18 +126,16 @@ def make_pi05_pre_post_processors(
|
||||
action_names=getattr(config, "action_feature_names", None),
|
||||
)
|
||||
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats)
|
||||
|
||||
# OpenPI order: raw → relative → normalize → model → unnormalize → absolute
|
||||
input_steps: list[ProcessorStep] = [
|
||||
RenameObservationsProcessorStep(rename_map={}), # To mimic the same processor as pretrained one
|
||||
AddBatchDimensionProcessorStep(),
|
||||
steps.rename_observations, # To mimic the same processor as pretrained one
|
||||
steps.add_batch_dim,
|
||||
relative_step,
|
||||
# NOTE: NormalizerProcessorStep MUST come before Pi05PrepareStateTokenizerProcessorStep
|
||||
# because the tokenizer step expects normalized state in [-1, 1] range for discretization
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
steps.normalize,
|
||||
Pi05PrepareStateTokenizerProcessorStep(max_state_dim=config.max_state_dim),
|
||||
TokenizerProcessorStep(
|
||||
tokenizer_name="google/paligemma-3b-pt-224",
|
||||
@@ -154,26 +143,13 @@ def make_pi05_pre_post_processors(
|
||||
padding_side="right",
|
||||
padding="max_length",
|
||||
),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
steps.to_device,
|
||||
]
|
||||
|
||||
output_steps: list[ProcessorStep] = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
steps.unnormalize,
|
||||
AbsoluteActionsProcessorStep(enabled=config.use_relative_actions, relative_step=relative_step),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
steps.to_cpu,
|
||||
]
|
||||
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
@@ -25,26 +25,17 @@ from lerobot.configs import PipelineFeatureType, PolicyFeature
|
||||
from lerobot.processor import (
|
||||
AbsoluteActionsProcessorStep,
|
||||
ActionTokenizerProcessorStep,
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStep,
|
||||
ProcessorStepRegistry,
|
||||
RelativeActionsProcessorStep,
|
||||
RenameObservationsProcessorStep,
|
||||
TokenizerProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from lerobot.types import EnvTransition, TransitionKey
|
||||
from lerobot.utils.constants import (
|
||||
OBS_STATE,
|
||||
POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
)
|
||||
from lerobot.utils.constants import OBS_STATE
|
||||
|
||||
from .configuration_pi0_fast import PI0FastConfig
|
||||
|
||||
@@ -135,6 +126,8 @@ def make_pi0_fast_pre_post_processors(
|
||||
action_names=getattr(config, "action_feature_names", None),
|
||||
)
|
||||
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats)
|
||||
|
||||
# Pi0Fast order: relative → normalize → tokenize → model → unnormalize → absolute
|
||||
# This matches pi0/pi0.5: RelativeActionsProcessorStep runs first on raw absolute actions,
|
||||
# caching the raw state. NormalizerProcessorStep then normalizes the raw relative actions,
|
||||
@@ -144,14 +137,10 @@ def make_pi0_fast_pre_post_processors(
|
||||
# before Pi0FastPrepareStateAndLanguageTokenizerProcessorStep, so the state tokenizer
|
||||
# continues to receive normalized state in [-1, 1] as expected.
|
||||
input_steps: list[ProcessorStep] = [
|
||||
RenameObservationsProcessorStep(rename_map={}), # To mimic the same processor as pretrained one
|
||||
AddBatchDimensionProcessorStep(),
|
||||
steps.rename_observations, # To mimic the same processor as pretrained one
|
||||
steps.add_batch_dim,
|
||||
relative_step,
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
steps.normalize,
|
||||
Pi0FastPrepareStateAndLanguageTokenizerProcessorStep(max_state_dim=config.max_state_dim),
|
||||
TokenizerProcessorStep(
|
||||
tokenizer_name=config.text_tokenizer_name,
|
||||
@@ -165,26 +154,13 @@ def make_pi0_fast_pre_post_processors(
|
||||
fast_skip_tokens=config.fast_skip_tokens,
|
||||
paligemma_tokenizer_name=config.text_tokenizer_name,
|
||||
),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
steps.to_device,
|
||||
]
|
||||
|
||||
output_steps: list[ProcessorStep] = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
steps.unnormalize,
|
||||
AbsoluteActionsProcessorStep(enabled=config.use_relative_actions, relative_step=relative_step),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
steps.to_cpu,
|
||||
]
|
||||
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
@@ -23,8 +23,6 @@ from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
from typing import TYPE_CHECKING, TypedDict, TypeVar, Unpack
|
||||
|
||||
import packaging
|
||||
import safetensors
|
||||
from huggingface_hub import HfApi, ModelCard, ModelCardData, hf_hub_download, save_torch_state_dict
|
||||
from huggingface_hub.constants import SAFETENSORS_SINGLE_FILE
|
||||
from huggingface_hub.errors import HfHubHTTPError
|
||||
@@ -34,6 +32,7 @@ from torch import Tensor, nn
|
||||
from lerobot.__version__ import __version__
|
||||
from lerobot.configs import PreTrainedConfig
|
||||
from lerobot.configs.train import TrainPipelineConfig
|
||||
from lerobot.utils.device_utils import resolve_safetensors_device
|
||||
from lerobot.utils.hub import HubMixin
|
||||
|
||||
from .utils import log_model_loading_keys
|
||||
@@ -221,26 +220,10 @@ class PreTrainedPolicy(nn.Module, HubMixin, abc.ABC):
|
||||
|
||||
@classmethod
|
||||
def _load_as_safetensor(cls, model: T, model_file: str, map_location: str, strict: bool) -> T:
|
||||
# Create base kwargs
|
||||
kwargs = {"strict": strict}
|
||||
|
||||
# Add device parameter for newer versions that support it
|
||||
if packaging.version.parse(safetensors.__version__) >= packaging.version.parse("0.4.3"):
|
||||
kwargs["device"] = map_location
|
||||
|
||||
# Load the model with appropriate kwargs
|
||||
missing_keys, unexpected_keys = load_model_as_safetensor(model, model_file, **kwargs)
|
||||
missing_keys, unexpected_keys = load_model_as_safetensor(
|
||||
model, model_file, strict=strict, device=resolve_safetensors_device(map_location)
|
||||
)
|
||||
log_model_loading_keys(missing_keys, unexpected_keys)
|
||||
|
||||
# For older versions, manually move to device if needed
|
||||
if "device" not in kwargs and map_location != "cpu":
|
||||
logging.warning(
|
||||
"Loading model weights on other devices than 'cpu' is not supported natively in your version of safetensors."
|
||||
" This means that the model is loaded on 'cpu' first and then copied to the device."
|
||||
" This leads to a slower loading time."
|
||||
" Please update safetensors to version 0.4.3 or above for improved performance."
|
||||
)
|
||||
model.to(map_location)
|
||||
return model
|
||||
|
||||
@abc.abstractmethod
|
||||
|
||||
@@ -19,19 +19,13 @@ from typing import Any
|
||||
import torch
|
||||
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NewLineTaskProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
RenameObservationsProcessorStep,
|
||||
TokenizerProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .configuration_smolvla import SmolVLAConfig
|
||||
|
||||
@@ -66,9 +60,11 @@ def make_smolvla_pre_post_processors(
|
||||
A tuple containing the configured pre-processor and post-processor pipelines.
|
||||
"""
|
||||
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats)
|
||||
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}), # To mimic the same processor as pretrained one
|
||||
AddBatchDimensionProcessorStep(),
|
||||
steps.rename_observations, # To mimic the same processor as pretrained one
|
||||
steps.add_batch_dim,
|
||||
NewLineTaskProcessorStep(),
|
||||
TokenizerProcessorStep(
|
||||
tokenizer_name=config.vlm_model_name,
|
||||
@@ -76,28 +72,11 @@ def make_smolvla_pre_post_processors(
|
||||
padding_side="right",
|
||||
max_length=config.tokenizer_max_length,
|
||||
),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
steps.to_device,
|
||||
steps.normalize,
|
||||
]
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
steps.unnormalize,
|
||||
steps.to_cpu,
|
||||
]
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
@@ -19,17 +19,10 @@ from typing import Any
|
||||
import torch
|
||||
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
RenameObservationsProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_pre_post_processors,
|
||||
)
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .configuration_tdmpc import TDMPCConfig
|
||||
|
||||
@@ -61,32 +54,4 @@ def make_tdmpc_pre_post_processors(
|
||||
Returns:
|
||||
A tuple containing the configured pre-processor and post-processor pipelines.
|
||||
"""
|
||||
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
]
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
]
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_default_pre_post_processors(config, dataset_stats)
|
||||
|
||||
@@ -20,20 +20,16 @@ import torch
|
||||
|
||||
from lerobot.policies.vla_jepa.configuration_vla_jepa import VLAJEPAConfig
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
EnvTransition,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStep,
|
||||
ProcessorStepRegistry,
|
||||
RenameObservationsProcessorStep,
|
||||
TransitionKey,
|
||||
UnnormalizerProcessorStep,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from lerobot.processor.converters import policy_action_to_transition, transition_to_policy_action
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
|
||||
@ProcessorStepRegistry.register(name="vla_jepa_clip_actions")
|
||||
@@ -112,15 +108,12 @@ def make_vla_jepa_pre_post_processors(
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction],
|
||||
]:
|
||||
features = {**config.input_features, **config.output_features}
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats)
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features=features,
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
steps.rename_observations,
|
||||
steps.add_batch_dim,
|
||||
steps.to_device,
|
||||
steps.normalize,
|
||||
]
|
||||
output_steps: list[ProcessorStep] = []
|
||||
if config.clip_normalized_actions:
|
||||
@@ -129,6 +122,8 @@ def make_vla_jepa_pre_post_processors(
|
||||
output_steps.append(
|
||||
PreSnapGripperProcessorStep(gripper_dim=config.gripper_dim, threshold=config.gripper_threshold)
|
||||
)
|
||||
# NOTE: unlike the default policy unnormalizer (output features only), VLA-JEPA
|
||||
# unnormalizes over BOTH input and output features.
|
||||
output_steps.append(
|
||||
UnnormalizerProcessorStep(
|
||||
features=features,
|
||||
@@ -140,16 +135,5 @@ def make_vla_jepa_pre_post_processors(
|
||||
output_steps.append(
|
||||
BinarizeGripperProcessorStep(gripper_dim=config.gripper_dim, threshold=config.gripper_threshold)
|
||||
)
|
||||
output_steps.append(DeviceProcessorStep(device="cpu"))
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
output_steps.append(steps.to_cpu)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
@@ -20,17 +20,10 @@ from typing import Any
|
||||
import torch
|
||||
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
RenameObservationsProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_pre_post_processors,
|
||||
)
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .configuration_vqbet import VQBeTConfig
|
||||
|
||||
@@ -62,32 +55,4 @@ def make_vqbet_pre_post_processors(
|
||||
Returns:
|
||||
A tuple containing the configured pre-processor and post-processor pipelines.
|
||||
"""
|
||||
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}), # Let the possibility to the user to rename the keys
|
||||
AddBatchDimensionProcessorStep(),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
]
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
]
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_default_pre_post_processors(config, dataset_stats)
|
||||
|
||||
@@ -58,10 +58,14 @@ class WallXConfig(PreTrainedConfig):
|
||||
# Action prediction mode: "diffusion" or "fast"
|
||||
prediction_mode: str = "diffusion"
|
||||
|
||||
# Attention Implementation, options: "eager", "flash_attention_2", "sdpa"
|
||||
# NOTE: flash-attn==2.7.4.post1 is required for flash_attention_2 implementation
|
||||
# Wall-X's bidirectional action-token islands currently require eager attention.
|
||||
attn_implementation: str = "eager"
|
||||
|
||||
# Vision attention is independent from the text action-token mask. ``auto`` uses
|
||||
# PyTorch's packed variable-length attention when the runtime supports it and
|
||||
# otherwise falls back to the native per-chunk SDPA implementation.
|
||||
vision_attn_implementation: str = "auto"
|
||||
|
||||
# ==================== Optimizer Presets ====================
|
||||
optimizer_lr: float = 2e-5
|
||||
optimizer_betas: tuple[float, float] = (0.9, 0.95)
|
||||
@@ -86,6 +90,18 @@ class WallXConfig(PreTrainedConfig):
|
||||
if self.prediction_mode not in ["diffusion", "fast"]:
|
||||
raise ValueError(f"prediction_mode must be 'diffusion' or 'fast', got {self.prediction_mode}")
|
||||
|
||||
if self.attn_implementation != "eager":
|
||||
raise ValueError(
|
||||
"Wall-X currently supports only attn_implementation='eager' because its "
|
||||
"bidirectional action-token islands require an explicit attention mask."
|
||||
)
|
||||
|
||||
if self.vision_attn_implementation not in {"auto", "sdpa", "varlen"}:
|
||||
raise ValueError(
|
||||
"vision_attn_implementation must be one of 'auto', 'sdpa', or 'varlen', got "
|
||||
f"{self.vision_attn_implementation!r}"
|
||||
)
|
||||
|
||||
# Assign use_fast_tokenizer based on prediction_mode
|
||||
if self.prediction_mode == "fast":
|
||||
self.use_fast_tokenizer = True
|
||||
|
||||
@@ -43,11 +43,14 @@ from typing import TYPE_CHECKING, Any
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from PIL import Image
|
||||
import torch.nn.functional as functional
|
||||
from safetensors import SafetensorError
|
||||
from safetensors.torch import load_file
|
||||
from torch import Tensor
|
||||
from torch.distributions import Beta
|
||||
from torch.nn import CrossEntropyLoss
|
||||
from torchvision.transforms import InterpolationMode
|
||||
from torchvision.transforms.v2 import functional as tv_functional
|
||||
|
||||
from lerobot.utils.constants import ACTION, OBS_STATE
|
||||
from lerobot.utils.import_utils import (
|
||||
@@ -74,17 +77,17 @@ if TYPE_CHECKING or _wallx_deps_available:
|
||||
from qwen_vl_utils.vision_process import smart_resize
|
||||
from torchdiffeq import odeint
|
||||
from transformers import AutoProcessor, BatchFeature
|
||||
from transformers.cache_utils import StaticCache
|
||||
from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import (
|
||||
Qwen2_5_VisionTransformerPretrainedModel,
|
||||
Qwen2_5_VLForConditionalGeneration,
|
||||
)
|
||||
from transformers.utils import is_torchdynamo_compiling
|
||||
from transformers.utils import cached_file, is_torchdynamo_compiling
|
||||
|
||||
from .qwen_model.configuration_qwen2_5_vl import Qwen2_5_VLConfig
|
||||
from .qwen_model.qwen2_5_vl_moe import (
|
||||
Qwen2_5_VisionTransformerPretrainedModel,
|
||||
from .qwen_model import (
|
||||
Qwen2_5_VLACausalLMOutputWithPast,
|
||||
Qwen2_5_VLConfig,
|
||||
Qwen2_5_VLMoEModel,
|
||||
configure_wall_x_vision_attention,
|
||||
)
|
||||
else:
|
||||
LoraConfig = None
|
||||
@@ -93,13 +96,14 @@ else:
|
||||
odeint = None
|
||||
AutoProcessor = None
|
||||
BatchFeature = None
|
||||
StaticCache = None
|
||||
Qwen2_5_VLForConditionalGeneration = None
|
||||
cached_file = None
|
||||
is_torchdynamo_compiling = None
|
||||
Qwen2_5_VLConfig = None
|
||||
Qwen2_5_VisionTransformerPretrainedModel = None
|
||||
Qwen2_5_VLACausalLMOutputWithPast = None
|
||||
Qwen2_5_VLMoEModel = None
|
||||
configure_wall_x_vision_attention = None
|
||||
|
||||
from .utils import (
|
||||
get_wallx_normal_text,
|
||||
@@ -111,6 +115,75 @@ from .utils import (
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _wall_x_resize_dimensions(height: int, width: int) -> tuple[int, int, int, int]:
|
||||
"""Return the intermediate and final Wall-X resize dimensions as ``(H, W, H, W)``."""
|
||||
if RESOLUTION == -1:
|
||||
intermediate_height, intermediate_width = height, width
|
||||
elif width > height:
|
||||
intermediate_width = RESOLUTION
|
||||
intermediate_height = int(RESOLUTION * height / width)
|
||||
else:
|
||||
intermediate_height = RESOLUTION
|
||||
intermediate_width = int(RESOLUTION * width / height)
|
||||
|
||||
resized_height, resized_width = smart_resize(
|
||||
intermediate_height,
|
||||
intermediate_width,
|
||||
factor=IMAGE_FACTOR,
|
||||
min_pixels=MIN_PIXELS,
|
||||
max_pixels=MAX_PIXELS,
|
||||
)
|
||||
return intermediate_height, intermediate_width, resized_height, resized_width
|
||||
|
||||
|
||||
def _resize_wall_x_image_batch(images: Tensor) -> tuple[Tensor, tuple[int, int, int, int]]:
|
||||
"""Quantize and resize a BCHW camera batch without leaving its current device."""
|
||||
if images.ndim != 4:
|
||||
raise ValueError(f"Wall-X images must be BCHW tensors, got shape {tuple(images.shape)}")
|
||||
|
||||
original_height, original_width = images.shape[-2:]
|
||||
intermediate_height, intermediate_width, resized_height, resized_width = _wall_x_resize_dimensions(
|
||||
original_height, original_width
|
||||
)
|
||||
|
||||
if images.is_floating_point():
|
||||
# Match the previous PIL path, which quantized via `(image * 255).to(torch.uint8)`.
|
||||
images = (images * 255).to(torch.uint8)
|
||||
elif images.dtype != torch.uint8:
|
||||
raise TypeError(f"Wall-X images must be floating point or uint8, got {images.dtype}")
|
||||
|
||||
if images.shape[-2:] != (intermediate_height, intermediate_width):
|
||||
images = tv_functional.resize(
|
||||
images,
|
||||
[intermediate_height, intermediate_width],
|
||||
interpolation=InterpolationMode.BICUBIC,
|
||||
antialias=True,
|
||||
)
|
||||
if images.shape[-2:] != (resized_height, resized_width):
|
||||
images = tv_functional.resize(
|
||||
images,
|
||||
[resized_height, resized_width],
|
||||
interpolation=InterpolationMode.BICUBIC,
|
||||
antialias=True,
|
||||
)
|
||||
|
||||
return images, (original_height, original_width, resized_height, resized_width)
|
||||
|
||||
|
||||
def _prepare_wall_x_image_inputs(
|
||||
batch: dict[str, Any], img_keys: list[str]
|
||||
) -> tuple[list[list[Tensor]], dict[str, tuple[int, int, int, int]]]:
|
||||
"""Resize each camera as a batch, then restore sample-major/camera-minor ordering."""
|
||||
resized_by_key: dict[str, Tensor] = {}
|
||||
dimensions_by_key: dict[str, tuple[int, int, int, int]] = {}
|
||||
for key in img_keys:
|
||||
resized_by_key[key], dimensions_by_key[key] = _resize_wall_x_image_batch(batch[key])
|
||||
|
||||
batch_size = batch[img_keys[0]].shape[0]
|
||||
image_inputs = [[resized_by_key[key][i] for key in img_keys] for i in range(batch_size)]
|
||||
return image_inputs, dimensions_by_key
|
||||
|
||||
|
||||
class SinusoidalPosEmb(nn.Module):
|
||||
"""Sinusoidal positional embedding for diffusion timesteps."""
|
||||
|
||||
@@ -246,7 +319,7 @@ class ActionHead(nn.Module):
|
||||
flow = flow.to(torch.float32)
|
||||
|
||||
action_pred = self.action_proj_back(action_hidden_states)
|
||||
loss = F.mse_loss(action_pred, flow, reduction="none")
|
||||
loss = functional.mse_loss(action_pred, flow, reduction="none")
|
||||
|
||||
if dof_mask is not None:
|
||||
dof_mask = dof_mask.reshape(-1, dof_mask.shape[-1]).to(torch.float32)
|
||||
@@ -254,7 +327,7 @@ class ActionHead(nn.Module):
|
||||
|
||||
return loss
|
||||
|
||||
def proprioception_proj(self, proprioception, dof_mask=None, use_history=False):
|
||||
def proprioception_proj(self, proprioception, dof_mask=None):
|
||||
"""Project proprioceptive data to hidden space."""
|
||||
# Ensure proper device and dtype alignment
|
||||
proprioception = proprioception.to(device=self.propri_proj.weight.device).to(
|
||||
@@ -264,10 +337,7 @@ class ActionHead(nn.Module):
|
||||
if dof_mask is not None:
|
||||
# Concatenate proprioception with DOF mask
|
||||
# TODO: Use variable-based dimension checking for better flexibility
|
||||
if use_history:
|
||||
proprioception = torch.cat([proprioception, dof_mask], dim=-1)
|
||||
else:
|
||||
proprioception = torch.cat([proprioception, dof_mask], dim=-1)
|
||||
proprioception = torch.cat([proprioception, dof_mask], dim=-1)
|
||||
|
||||
proprioception = proprioception.to(device=self.propri_proj.weight.device).to(
|
||||
dtype=self.propri_proj.weight.dtype
|
||||
@@ -281,7 +351,7 @@ class ActionHead(nn.Module):
|
||||
_Qwen2_5_VLForAction_Base = Qwen2_5_VLForConditionalGeneration if _wallx_deps_available else nn.Module
|
||||
|
||||
|
||||
class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base): # noqa: N801
|
||||
"""
|
||||
Qwen2.5 Vision-Language Mixture of Experts model for action processing.
|
||||
|
||||
@@ -305,6 +375,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
config=None,
|
||||
action_tokenizer_path=None,
|
||||
attn_implementation: str = "eager",
|
||||
vision_attn_implementation: str = "auto",
|
||||
cache_dir: str | PathLike | None = None,
|
||||
force_download: bool = False,
|
||||
local_files_only: bool = False,
|
||||
@@ -321,11 +392,14 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
config_path (str, optional): Configuration file path, if None will look for qwen25_config.json in pretrained_model_path
|
||||
action_tokenizer_path (str, optional): Action tokenizer path, if None will load from default config
|
||||
attn_implementation (str, optional): Attention implementation, if None will load from default config
|
||||
vision_attn_implementation (str, optional): Vision attention backend. ``auto`` uses packed
|
||||
variable-length attention when supported and otherwise falls back to SDPA.
|
||||
**kwargs: Additional arguments
|
||||
|
||||
Returns:
|
||||
Qwen2_5_VLMoEForAction: Loaded model instance
|
||||
"""
|
||||
Qwen2_5_VLMoEModel._require_eager_attention(attn_implementation)
|
||||
if config is None:
|
||||
config = cls.config_class.from_pretrained(
|
||||
pretrained_name_or_path,
|
||||
@@ -339,7 +413,15 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
)
|
||||
if attn_implementation is not None:
|
||||
config._attn_implementation = attn_implementation
|
||||
processor = AutoProcessor.from_pretrained(pretrained_name_or_path, use_fast=True)
|
||||
processor = AutoProcessor.from_pretrained(
|
||||
pretrained_name_or_path,
|
||||
cache_dir=cache_dir,
|
||||
force_download=force_download,
|
||||
local_files_only=local_files_only,
|
||||
token=token,
|
||||
revision=revision,
|
||||
use_fast=True,
|
||||
)
|
||||
if action_tokenizer_path is not None:
|
||||
action_tokenizer = AutoProcessor.from_pretrained(action_tokenizer_path, trust_remote_code=True)
|
||||
processor.action_processor = action_tokenizer
|
||||
@@ -351,41 +433,41 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
config.text_config.pad_token_id = processor.tokenizer.pad_token_id
|
||||
|
||||
# Initialize model with configuration and processor
|
||||
model = cls(config, processor=processor, action_tokenizer=action_tokenizer, **kwargs)
|
||||
model = cls(
|
||||
config,
|
||||
processor=processor,
|
||||
action_tokenizer=action_tokenizer,
|
||||
vision_attn_implementation=vision_attn_implementation,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
# Resize token embeddings to match processor tokenizer vocabulary size
|
||||
model.resize_token_embeddings(len(processor.tokenizer))
|
||||
|
||||
# Try to load the model.safetensors file
|
||||
print(f"Loading model from: {pretrained_name_or_path}")
|
||||
logger.info("Loading Wall-X model from %s", pretrained_name_or_path)
|
||||
try:
|
||||
from transformers.utils import cached_file
|
||||
|
||||
# Try safetensors first
|
||||
resolved_file = cached_file(
|
||||
pretrained_name_or_path,
|
||||
"model.safetensors",
|
||||
cache_dir=kwargs.get("cache_dir"),
|
||||
force_download=kwargs.get("force_download", False),
|
||||
cache_dir=cache_dir,
|
||||
force_download=force_download,
|
||||
resume_download=kwargs.get("resume_download"),
|
||||
proxies=kwargs.get("proxies"),
|
||||
token=kwargs.get("token"),
|
||||
revision=kwargs.get("revision"),
|
||||
local_files_only=kwargs.get("local_files_only", False),
|
||||
token=token,
|
||||
revision=revision,
|
||||
local_files_only=local_files_only,
|
||||
)
|
||||
from safetensors.torch import load_file
|
||||
|
||||
sd = load_file(resolved_file)
|
||||
print("✓ Loaded state dict from model.safetensors")
|
||||
except Exception as e:
|
||||
print(f"Could not load state dict from remote files: {e}")
|
||||
print("Returning model without loading pretrained weights")
|
||||
return model
|
||||
except (OSError, SafetensorError) as error:
|
||||
raise OSError(
|
||||
f"Failed to load pretrained Wall-X weights from {pretrained_name_or_path!r}"
|
||||
) from error
|
||||
logger.info("Loaded Wall-X state dict from model.safetensors")
|
||||
|
||||
state_dict = {}
|
||||
# filter normalizer statistic params
|
||||
del_keys = []
|
||||
for key in sd.keys():
|
||||
for key in sd:
|
||||
if "action_preprocessor.normalizer" in key:
|
||||
del_keys.append(key)
|
||||
for key in del_keys:
|
||||
@@ -404,6 +486,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
action_tokenizer=None,
|
||||
action_mapper=None,
|
||||
flow_loss_weight=1.0,
|
||||
vision_attn_implementation: str = "auto",
|
||||
):
|
||||
"""
|
||||
Initialize the Qwen2.5 VLMoE model for action processing.
|
||||
@@ -416,10 +499,16 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
action_mapper: Action mapping utility
|
||||
flow_loss_weight (float): Weight for flow loss computation
|
||||
"""
|
||||
Qwen2_5_VLMoEModel._require_eager_attention(config._attn_implementation)
|
||||
config._attn_implementation = "eager"
|
||||
# Text needs eager attention for action-token islands. Vision has no such
|
||||
# constraint, so keep its portable native fallback on SDPA.
|
||||
config.vision_config._attn_implementation = "sdpa"
|
||||
super().__init__(config)
|
||||
|
||||
# Initialize vision transformer and language model components
|
||||
self.visual = Qwen2_5_VisionTransformerPretrainedModel._from_config(config.vision_config)
|
||||
configure_wall_x_vision_attention(self.visual, vision_attn_implementation)
|
||||
self.model = Qwen2_5_VLMoEModel(config)
|
||||
self.vocab_size = config.vocab_size
|
||||
self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
|
||||
@@ -457,7 +546,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
|
||||
params_to_keep_float32 = []
|
||||
|
||||
for name, param in self.named_parameters():
|
||||
for name, _param in self.named_parameters():
|
||||
if "input_layernorm" in name or "post_attention_layernorm" in name or "model.norm" in name:
|
||||
params_to_keep_float32.append(name)
|
||||
if "action_preprocessor" in name:
|
||||
@@ -491,7 +580,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
"action_token_id": action_token_id,
|
||||
}
|
||||
|
||||
def add_lora(self, r=8, lora_alpha=32, target_modules=["q_proj", "v_proj"], lora_dropout=0.1):
|
||||
def add_lora(self, r=8, lora_alpha=32, target_modules=None, lora_dropout=0.1):
|
||||
"""
|
||||
Add LoRA (Low-Rank Adaptation) adapters to the model.
|
||||
|
||||
@@ -501,6 +590,9 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
target_modules (list): List of module names to apply LoRA to
|
||||
lora_dropout (float): Dropout probability for LoRA layers
|
||||
"""
|
||||
if target_modules is None:
|
||||
target_modules = ["q_proj", "v_proj"]
|
||||
|
||||
config = LoraConfig(
|
||||
r=r,
|
||||
lora_alpha=lora_alpha,
|
||||
@@ -795,6 +887,9 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
)
|
||||
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
|
||||
|
||||
if rope_deltas is not None:
|
||||
self.rope_deltas = rope_deltas
|
||||
|
||||
# Calculate RoPE position IDs if not provided
|
||||
# Note: Cannot calculate rope deltas with 4D attention mask. TODO: Fix this limitation
|
||||
if position_ids is None and (attention_mask is None or attention_mask.ndim == 2):
|
||||
@@ -833,7 +928,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
# Process image embeddings
|
||||
if pixel_values is not None:
|
||||
pixel_values = pixel_values.type(self.visual.dtype)
|
||||
image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw)
|
||||
image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw).pooler_output
|
||||
mask = input_ids == self.config.image_token_id
|
||||
mask_unsqueezed = mask.unsqueeze(-1)
|
||||
mask_expanded = mask_unsqueezed.expand_as(inputs_embeds)
|
||||
@@ -845,7 +940,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
# Process video embeddings
|
||||
if pixel_values_videos is not None:
|
||||
pixel_values_videos = pixel_values_videos.type(self.visual.dtype)
|
||||
video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw)
|
||||
video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw).pooler_output
|
||||
n_video_tokens = (input_ids == self.config.video_token_id).sum().item()
|
||||
n_video_features = video_embeds.shape[0]
|
||||
|
||||
@@ -869,7 +964,6 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
proprioception = self.action_preprocessor.proprioception_proj(
|
||||
proprioception,
|
||||
agent_pos_mask,
|
||||
use_history=proprioception.shape[1] > 1,
|
||||
)
|
||||
mask = input_ids == self.action_token_id_set["propri_token_id"]
|
||||
mask_unsqueezed = mask.unsqueeze(-1)
|
||||
@@ -919,6 +1013,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
output_attentions=output_attentions,
|
||||
output_hidden_states=output_hidden_states,
|
||||
return_dict=return_dict,
|
||||
cache_position=cache_position,
|
||||
)
|
||||
|
||||
hidden_states = outputs[0]
|
||||
@@ -1107,7 +1202,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
# Process image embeddings
|
||||
if pixel_values is not None:
|
||||
pixel_values = pixel_values.type(self.visual.dtype)
|
||||
image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw)
|
||||
image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw).pooler_output
|
||||
n_image_tokens = (input_ids == self.config.image_token_id).sum().item()
|
||||
n_image_features = image_embeds.shape[0]
|
||||
|
||||
@@ -1128,7 +1223,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
# Process video embeddings
|
||||
if pixel_values_videos is not None:
|
||||
pixel_values_videos = pixel_values_videos.type(self.visual.dtype)
|
||||
video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw)
|
||||
video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw).pooler_output
|
||||
n_video_tokens = (input_ids == self.config.video_token_id).sum().item()
|
||||
n_video_features = video_embeds.shape[0]
|
||||
|
||||
@@ -1153,7 +1248,6 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
proprio_embed = self.action_preprocessor.proprioception_proj(
|
||||
proprioception,
|
||||
agent_pos_mask,
|
||||
use_history=proprioception.shape[1] > 1,
|
||||
)
|
||||
proprioception_mask = input_ids == self.action_token_id_set["propri_token_id"]
|
||||
proprio_embed = proprio_embed.to(torch.bfloat16)
|
||||
@@ -1202,25 +1296,37 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
|
||||
# Split input sequence for text and fast modes (not needed for diffusion)
|
||||
if predict_mode == "text" or predict_mode == "fast":
|
||||
# Look for generation prompt tokens: <|im_start|>assistant
|
||||
generation_prompt = "<|im_start|>assistant\n"
|
||||
generation_prompt_ids = torch.tensor(
|
||||
[151644, 77091], device=input_ids.device, dtype=input_ids.dtype
|
||||
)
|
||||
matches = (input_ids[0, :-1] == generation_prompt_ids[0]) & (
|
||||
input_ids[0, 1:] == generation_prompt_ids[1]
|
||||
self.processor.tokenizer.encode(generation_prompt, add_special_tokens=False),
|
||||
device=input_ids.device,
|
||||
dtype=input_ids.dtype,
|
||||
)
|
||||
prompt_length = generation_prompt_ids.numel()
|
||||
if prompt_length == 0:
|
||||
raise ValueError(f"Tokenizer produced no tokens for generation prompt {generation_prompt!r}")
|
||||
if input_ids.shape[1] < prompt_length:
|
||||
matches = torch.empty(0, device=input_ids.device, dtype=torch.bool)
|
||||
else:
|
||||
matches = (
|
||||
input_ids[0]
|
||||
.unfold(dimension=0, size=prompt_length, step=1)
|
||||
.eq(generation_prompt_ids)
|
||||
.all(dim=-1)
|
||||
)
|
||||
|
||||
if matches.any():
|
||||
split_pos = torch.nonzero(matches, as_tuple=True)[0][0].item()
|
||||
prompt_end = split_pos + prompt_length
|
||||
# Extract ground truth output tokens (including newline)
|
||||
gt_output_ids = input_ids[:, split_pos + 3 :]
|
||||
gt_output_ids = input_ids[:, prompt_end:]
|
||||
# Remove output part from input, keeping prompt
|
||||
input_ids = input_ids[:, : split_pos + 3]
|
||||
inputs_embeds = inputs_embeds[:, : split_pos + 3, :]
|
||||
input_ids = input_ids[:, :prompt_end]
|
||||
inputs_embeds = inputs_embeds[:, :prompt_end, :]
|
||||
if attention_mask is not None:
|
||||
attention_mask = attention_mask[:, : split_pos + 3]
|
||||
attention_mask = attention_mask[:, :prompt_end]
|
||||
if labels is not None:
|
||||
labels = labels[:, split_pos + 3 :]
|
||||
labels = labels[:, prompt_end:]
|
||||
else:
|
||||
raise ValueError(
|
||||
"input_ids does not contain the generation prompt tokens <|im_start|>assistant"
|
||||
@@ -1255,7 +1361,7 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
use_cache=True,
|
||||
pad_token_id=self.processor.tokenizer.pad_token_id,
|
||||
temperature=(1.0 if not re_generate else 0.7), # Higher temperature for regeneration
|
||||
do_sample=(False if not re_generate else True), # Enable sampling for regeneration
|
||||
do_sample=re_generate, # Enable sampling for regeneration
|
||||
)
|
||||
|
||||
# Decode generated and ground truth text
|
||||
@@ -1524,27 +1630,6 @@ class Qwen2_5_VLMoEForAction(_Qwen2_5_VLForAction_Base):
|
||||
else:
|
||||
model_inputs = {"input_ids": input_ids, "inputs_embeds": None}
|
||||
|
||||
# Prepare 4D causal attention mask for static cache
|
||||
if isinstance(past_key_values, StaticCache) and attention_mask.ndim == 2:
|
||||
if model_inputs["inputs_embeds"] is not None:
|
||||
batch_size, sequence_length, _ = inputs_embeds.shape
|
||||
device = inputs_embeds.device
|
||||
else:
|
||||
batch_size, sequence_length = input_ids.shape
|
||||
device = input_ids.device
|
||||
|
||||
attention_mask = self.model._prepare_4d_causal_attention_mask_with_cache_position(
|
||||
attention_mask,
|
||||
sequence_length=sequence_length,
|
||||
target_length=past_key_values.get_max_cache_shape(),
|
||||
dtype=self.lm_head.weight.dtype,
|
||||
device=device,
|
||||
cache_position=cache_position,
|
||||
batch_size=batch_size,
|
||||
config=self.config,
|
||||
past_key_values=past_key_values,
|
||||
)
|
||||
|
||||
# Assemble all model inputs for generation
|
||||
model_inputs.update(
|
||||
{
|
||||
@@ -1749,6 +1834,7 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
pretrained_name_or_path=config.pretrained_name_or_path,
|
||||
action_tokenizer_path=config.action_tokenizer_path,
|
||||
attn_implementation=config.attn_implementation,
|
||||
vision_attn_implementation=config.vision_attn_implementation,
|
||||
)
|
||||
self.model.to(config.device)
|
||||
self.model.to_bfloat16_for_selected_params()
|
||||
@@ -1768,6 +1854,8 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
def preprocess_inputs(
|
||||
self,
|
||||
batch: dict[str, Any],
|
||||
*,
|
||||
compute_position_ids: bool = False,
|
||||
) -> BatchFeature:
|
||||
"""
|
||||
Convert a batch of LeRobot dataset items to Wall-X model input format.
|
||||
@@ -1789,50 +1877,21 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
# Get batch size from state tensor
|
||||
batch_size = batch[OBS_STATE].shape[0]
|
||||
|
||||
# ==================== PROCESS ALL SAMPLES ====================
|
||||
all_image_inputs = []
|
||||
all_texts = []
|
||||
|
||||
# Find image keys in batch
|
||||
img_keys = [key for key in self.config.image_features if key in batch]
|
||||
if not img_keys:
|
||||
raise ValueError("Wall-X requires at least one image feature in each batch")
|
||||
|
||||
# Resize one camera batch at a time on the tensors' current device. Reassembling
|
||||
# sample-major keeps image_grid_thw aligned with each sample's image placeholders.
|
||||
all_image_inputs, dimensions_by_key = _prepare_wall_x_image_inputs(batch, img_keys)
|
||||
all_texts = []
|
||||
|
||||
# Preserve the existing grounding behavior for multi-camera inputs: the old camera
|
||||
# loop left these values set to the final configured camera's dimensions.
|
||||
orig_height, orig_width, resized_height, resized_width = dimensions_by_key[img_keys[-1]]
|
||||
|
||||
for i in range(batch_size):
|
||||
# Vision preprocessing per sample
|
||||
processed_frames = []
|
||||
orig_height, orig_width = None, None
|
||||
resized_height, resized_width = None, None
|
||||
|
||||
for key in img_keys:
|
||||
current_obs = batch[key][i].clone() # (C, H, W)
|
||||
if current_obs.dim() == 3:
|
||||
current_obs = current_obs.permute(1, 2, 0) # (H, W, C)
|
||||
|
||||
img_pil = Image.fromarray((current_obs * 255).to(torch.uint8).cpu().numpy())
|
||||
orig_width, orig_height = img_pil.size
|
||||
|
||||
target_size = RESOLUTION
|
||||
if target_size != -1:
|
||||
if orig_width > orig_height:
|
||||
new_width = target_size
|
||||
new_height = int(target_size * orig_height / orig_width)
|
||||
else:
|
||||
new_height = target_size
|
||||
new_width = int(target_size * orig_width / orig_height)
|
||||
img_pil = img_pil.resize((new_width, new_height))
|
||||
|
||||
current_width, current_height = img_pil.size
|
||||
resized_height, resized_width = smart_resize(
|
||||
current_height,
|
||||
current_width,
|
||||
factor=IMAGE_FACTOR,
|
||||
min_pixels=MIN_PIXELS,
|
||||
max_pixels=MAX_PIXELS,
|
||||
)
|
||||
resized_img = img_pil.resize((resized_width, resized_height))
|
||||
processed_frames.append(resized_img)
|
||||
|
||||
all_image_inputs.append(processed_frames)
|
||||
|
||||
# Text preprocessing
|
||||
task_text = batch["task"][i] if isinstance(batch["task"], list) else batch["task"]
|
||||
instruction_info = {"instruction": task_text}
|
||||
@@ -1859,8 +1918,8 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
agent_pos_mask = (~torch.isnan(agent_pos)).float()
|
||||
agent_pos = agent_pos.nan_to_num(nan=0.0)
|
||||
|
||||
if agent_pos.shape[-1] != 20:
|
||||
pad_size = 20 - agent_pos.shape[-1]
|
||||
if agent_pos.shape[-1] < self.config.max_state_dim:
|
||||
pad_size = self.config.max_state_dim - agent_pos.shape[-1]
|
||||
agent_pos = torch.cat(
|
||||
[
|
||||
agent_pos,
|
||||
@@ -1880,6 +1939,10 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
],
|
||||
dim=-1,
|
||||
)
|
||||
elif agent_pos.shape[-1] > self.config.max_state_dim:
|
||||
raise ValueError(
|
||||
f"State dimension {agent_pos.shape[-1]} exceeds max_state_dim {self.config.max_state_dim}"
|
||||
)
|
||||
|
||||
# ==================== PROCESS ACTIONS ====================
|
||||
action = batch.get(ACTION) # (batch_size, chunk_size, action_dim)
|
||||
@@ -1889,8 +1952,8 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
dof_mask = (~torch.isnan(action)).float()
|
||||
action = action.nan_to_num(nan=0.0)
|
||||
|
||||
if action.shape[-1] != 20:
|
||||
pad_size = 20 - action.shape[-1]
|
||||
if action.shape[-1] < self.config.max_action_dim:
|
||||
pad_size = self.config.max_action_dim - action.shape[-1]
|
||||
action = torch.cat(
|
||||
[action, torch.zeros(action.shape[0], action.shape[1], pad_size, device=action.device)],
|
||||
dim=-1,
|
||||
@@ -1902,6 +1965,10 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
],
|
||||
dim=-1,
|
||||
)
|
||||
elif action.shape[-1] > self.config.max_action_dim:
|
||||
raise ValueError(
|
||||
f"Action dimension {action.shape[-1]} exceeds max_action_dim {self.config.max_action_dim}"
|
||||
)
|
||||
else:
|
||||
action_dim = self.config.output_features[ACTION].shape[0]
|
||||
dof_mask = torch.cat(
|
||||
@@ -1910,7 +1977,10 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
batch_size, self.config.chunk_size, action_dim, device=batch[OBS_STATE].device
|
||||
),
|
||||
torch.zeros(
|
||||
batch_size, self.config.chunk_size, 20 - action_dim, device=batch[OBS_STATE].device
|
||||
batch_size,
|
||||
self.config.chunk_size,
|
||||
self.config.max_action_dim - action_dim,
|
||||
device=batch[OBS_STATE].device,
|
||||
),
|
||||
],
|
||||
dim=-1,
|
||||
@@ -1930,12 +2000,26 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
text=all_texts,
|
||||
images=all_image_inputs,
|
||||
videos=None,
|
||||
device=batch[OBS_STATE].device,
|
||||
padding=True,
|
||||
truncation=True,
|
||||
return_tensors="pt",
|
||||
max_length=TOKENIZER_MAX_LENGTH,
|
||||
)
|
||||
|
||||
if compute_position_ids:
|
||||
# Qwen's RoPE indexing uses Python list/scalar conversions. Run it while the
|
||||
# tokenizer and grid metadata are still on CPU, then move the compact result.
|
||||
position_ids, rope_deltas = self.model.get_rope_index(
|
||||
inputs.input_ids,
|
||||
inputs.get("image_grid_thw"),
|
||||
inputs.get("video_grid_thw"),
|
||||
inputs.get("second_per_grid_ts"),
|
||||
inputs.attention_mask,
|
||||
)
|
||||
inputs["position_ids"] = position_ids
|
||||
inputs["rope_deltas"] = rope_deltas
|
||||
|
||||
# ==================== ADDITIONAL INPUTS ====================
|
||||
action_token_id = self.model.processor.tokenizer.convert_tokens_to_ids("<|action|>")
|
||||
moe_token_types = inputs.input_ids == action_token_id
|
||||
@@ -1952,7 +2036,7 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
)
|
||||
|
||||
# Move all tensors to the correct device
|
||||
device = self.config.device
|
||||
device = batch[OBS_STATE].device
|
||||
for key, value in inputs.items():
|
||||
if isinstance(value, torch.Tensor):
|
||||
inputs[key] = value.to(device)
|
||||
@@ -1972,9 +2056,7 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
Returns:
|
||||
tuple: (loss, loss_dict)
|
||||
"""
|
||||
batch = self.preprocess_inputs(
|
||||
batch,
|
||||
)
|
||||
batch = self.preprocess_inputs(batch, compute_position_ids=True)
|
||||
|
||||
# Call the underlying model's forward with mode="train"
|
||||
outputs = self.model(**batch, mode="train")
|
||||
@@ -1982,19 +2064,19 @@ class WallXPolicy(PreTrainedPolicy):
|
||||
# Extract losses from output
|
||||
loss = outputs.loss
|
||||
loss_dict = {
|
||||
"loss": loss.item() if loss is not None else 0.0,
|
||||
"loss": loss.detach() if loss is not None else 0.0,
|
||||
}
|
||||
|
||||
if outputs.flow_loss is not None:
|
||||
loss_dict["flow_loss"] = outputs.flow_loss.item()
|
||||
loss_dict["flow_loss"] = outputs.flow_loss.detach()
|
||||
if outputs.cross_entropy_loss is not None:
|
||||
loss_dict["cross_entropy_loss"] = outputs.cross_entropy_loss.item()
|
||||
loss_dict["cross_entropy_loss"] = outputs.cross_entropy_loss.detach()
|
||||
|
||||
# Add channel losses if available
|
||||
if outputs.channel_loss_dict is not None:
|
||||
for key, value in outputs.channel_loss_dict.items():
|
||||
if isinstance(value, torch.Tensor):
|
||||
loss_dict[f"channel_{key}"] = value.item()
|
||||
loss_dict[f"channel_{key}"] = value.detach()
|
||||
|
||||
return loss, loss_dict
|
||||
|
||||
|
||||
@@ -20,19 +20,13 @@ import torch
|
||||
|
||||
from lerobot.configs import PipelineFeatureType, PolicyFeature
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
ComplementaryDataProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStepRegistry,
|
||||
RenameObservationsProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .configuration_wall_x import WallXConfig
|
||||
|
||||
@@ -65,37 +59,22 @@ def make_wall_x_pre_post_processors(
|
||||
A tuple containing the configured pre-processor and post-processor pipelines
|
||||
"""
|
||||
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats)
|
||||
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
steps.rename_observations,
|
||||
steps.add_batch_dim,
|
||||
WallXTaskProcessor(), # Process task description
|
||||
NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
steps.normalize,
|
||||
steps.to_device,
|
||||
]
|
||||
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
steps.unnormalize,
|
||||
steps.to_cpu,
|
||||
]
|
||||
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
|
||||
@ProcessorStepRegistry.register(name="wall_x_task_processor")
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2025 HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from .configuration_qwen2_5_vl import (
|
||||
Qwen2_5_VLConfig,
|
||||
Qwen2_5_VLTextConfig,
|
||||
Qwen2_5_VLVisionConfig,
|
||||
)
|
||||
from .qwen2_5_vl_moe import (
|
||||
BlockSparseMLP,
|
||||
Qwen2_5_VLACausalLMOutputWithPast,
|
||||
Qwen2_5_VLDecoderLayer_with_MoE,
|
||||
Qwen2_5_VLMoEModel,
|
||||
SparseMoeBlock,
|
||||
)
|
||||
from .vision_attention import (
|
||||
WallXVisionAttention,
|
||||
configure_wall_x_vision_attention,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"BlockSparseMLP",
|
||||
"Qwen2_5_VLACausalLMOutputWithPast",
|
||||
"Qwen2_5_VLConfig",
|
||||
"Qwen2_5_VLDecoderLayer_with_MoE",
|
||||
"Qwen2_5_VLMoEModel",
|
||||
"Qwen2_5_VLTextConfig",
|
||||
"Qwen2_5_VLVisionConfig",
|
||||
"SparseMoeBlock",
|
||||
"WallXVisionAttention",
|
||||
"configure_wall_x_vision_attention",
|
||||
]
|
||||
@@ -1,250 +1,114 @@
|
||||
from transformers.configuration_utils import PretrainedConfig
|
||||
from transformers.modeling_rope_utils import rope_config_validation
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2025 HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Wall-X configuration extensions for the native Transformers Qwen2.5-VL config."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from huggingface_hub.dataclasses import strict
|
||||
|
||||
from lerobot.utils.import_utils import _transformers_available
|
||||
|
||||
if TYPE_CHECKING or _transformers_available:
|
||||
from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import (
|
||||
Qwen2_5_VLConfig as TransformersQwen2_5_VLConfig,
|
||||
Qwen2_5_VLTextConfig as TransformersQwen2_5_VLTextConfig,
|
||||
Qwen2_5_VLVisionConfig,
|
||||
)
|
||||
else:
|
||||
|
||||
@dataclass
|
||||
class _TransformersConfigFallback:
|
||||
"""Import-safe stand-in used only when Transformers is unavailable."""
|
||||
|
||||
TransformersQwen2_5_VLConfig = _TransformersConfigFallback
|
||||
TransformersQwen2_5_VLTextConfig = _TransformersConfigFallback
|
||||
Qwen2_5_VLVisionConfig = None
|
||||
|
||||
# Wall-X checkpoints pre0.6.0 use the legacy, flat Qwen2.5-VL config layout. The native
|
||||
# ``Qwen2_5_VLConfig`` accepts that layout and moves text-model fields into its
|
||||
# ``text_config`` sub-config, so only the Wall-X-specific MoE fields need to be
|
||||
# declared here.
|
||||
_LEGACY_TEXT_ATTRIBUTES = {
|
||||
"attention_dropout",
|
||||
"attention_moe",
|
||||
"dim_inputs",
|
||||
"dof_config",
|
||||
"experts",
|
||||
"hidden_act",
|
||||
"hidden_size",
|
||||
"initializer_range",
|
||||
"intermediate_size",
|
||||
"layer_types",
|
||||
"max_position_embeddings",
|
||||
"max_window_layers",
|
||||
"mlp_moe",
|
||||
"noise_scheduler",
|
||||
"num_attention_heads",
|
||||
"num_experts",
|
||||
"num_hidden_layers",
|
||||
"num_key_value_heads",
|
||||
"pad_token_id",
|
||||
"rms_norm_eps",
|
||||
"sliding_window",
|
||||
"use_cache",
|
||||
"use_sliding_window",
|
||||
"vocab_size",
|
||||
}
|
||||
|
||||
|
||||
class Qwen2_5_VLVisionConfig(PretrainedConfig):
|
||||
model_type = "qwen2_5_vl"
|
||||
base_config_key = "vision_config"
|
||||
@strict
|
||||
class Qwen2_5_VLTextConfig(TransformersQwen2_5_VLTextConfig): # noqa: N801
|
||||
"""Native Qwen2.5-VL text config plus Wall-X's hard-routed MoE settings."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
depth=32,
|
||||
hidden_size=3584,
|
||||
hidden_act="silu",
|
||||
intermediate_size=3420,
|
||||
num_heads=16,
|
||||
in_channels=3,
|
||||
patch_size=14,
|
||||
spatial_merge_size=2,
|
||||
temporal_patch_size=2,
|
||||
tokens_per_second=4,
|
||||
window_size=112,
|
||||
out_hidden_size=3584,
|
||||
fullatt_block_indexes=[7, 15, 23, 31],
|
||||
initializer_range=0.02,
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(**kwargs)
|
||||
num_experts: int = 4
|
||||
experts: list[dict] | None = None
|
||||
dof_config: dict | None = None
|
||||
noise_scheduler: dict | None = None
|
||||
dim_inputs: tuple[int, ...] | list[int] = (1536, 1536)
|
||||
attention_moe: bool = False
|
||||
mlp_moe: bool = False
|
||||
|
||||
self.depth = depth
|
||||
self.hidden_size = hidden_size
|
||||
self.hidden_act = hidden_act
|
||||
self.intermediate_size = intermediate_size
|
||||
self.num_heads = num_heads
|
||||
self.in_channels = in_channels
|
||||
self.patch_size = patch_size
|
||||
self.spatial_merge_size = spatial_merge_size
|
||||
self.temporal_patch_size = temporal_patch_size
|
||||
self.tokens_per_second = tokens_per_second
|
||||
self.window_size = window_size
|
||||
self.fullatt_block_indexes = fullatt_block_indexes
|
||||
self.out_hidden_size = out_hidden_size
|
||||
self.initializer_range = initializer_range
|
||||
def __post_init__(self, **kwargs):
|
||||
self.dim_inputs = tuple(self.dim_inputs)
|
||||
super().__post_init__(**kwargs)
|
||||
|
||||
|
||||
class Qwen2_5_VLConfig(PretrainedConfig):
|
||||
r"""
|
||||
This is the configuration class to store the configuration of a [`Qwen2_5_VLModel`]. It is used to instantiate a
|
||||
Qwen2-VL model according to the specified arguments, defining the model architecture. Instantiating a configuration
|
||||
with the defaults will yield a similar configuration to that of
|
||||
Qwen2-VL-7B-Instruct [Qwen/Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct).
|
||||
@strict
|
||||
class Qwen2_5_VLConfig(TransformersQwen2_5_VLConfig): # noqa: N801
|
||||
"""Native composite Qwen2.5-VL config with a Wall-X text sub-config.
|
||||
|
||||
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
|
||||
documentation from [`PretrainedConfig`] for more information.
|
||||
The native composite loader supports both current nested configs and the
|
||||
flat layout used by existing ``wall-oss-flow`` checkpoints.
|
||||
"""
|
||||
|
||||
|
||||
Args:
|
||||
vocab_size (`int`, *optional*, defaults to 152064):
|
||||
Vocabulary size of the Qwen2_5_VL model. Defines the number of different tokens that can be represented by the
|
||||
`inputs_ids` passed when calling [`Qwen2_5_VLModel`]
|
||||
hidden_size (`int`, *optional*, defaults to 8192):
|
||||
Dimension of the hidden representations.
|
||||
intermediate_size (`int`, *optional*, defaults to 29568):
|
||||
Dimension of the MLP representations.
|
||||
num_hidden_layers (`int`, *optional*, defaults to 80):
|
||||
Number of hidden layers in the Transformer encoder.
|
||||
num_attention_heads (`int`, *optional*, defaults to 64):
|
||||
Number of attention heads for each attention layer in the Transformer encoder.
|
||||
num_key_value_heads (`int`, *optional*, defaults to 8):
|
||||
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
|
||||
`num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
|
||||
`num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
|
||||
converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
|
||||
by meanpooling all the original heads within that group. For more details checkout [this
|
||||
paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `32`.
|
||||
hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
|
||||
The non-linear activation function (function or string) in the decoder.
|
||||
max_position_embeddings (`int`, *optional*, defaults to 32768):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
initializer_range (`float`, *optional*, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
rms_norm_eps (`float`, *optional*, defaults to 1e-05):
|
||||
The epsilon used by the rms normalization layers.
|
||||
use_cache (`bool`, *optional*, defaults to `True`):
|
||||
Whether or not the model should return the last key/values attentions (not used by all models). Only
|
||||
relevant if `config.is_decoder=True`.
|
||||
tie_word_embeddings (`bool`, *optional*, defaults to `False`):
|
||||
Whether the model's input and output word embeddings should be tied.
|
||||
rope_theta (`float`, *optional*, defaults to 1000000.0):
|
||||
The base period of the RoPE embeddings.
|
||||
use_sliding_window (`bool`, *optional*, defaults to `False`):
|
||||
Whether to use sliding window attention.
|
||||
sliding_window (`int`, *optional*, defaults to 4096):
|
||||
Sliding window attention (SWA) window size. If not specified, will default to `4096`.
|
||||
max_window_layers (`int`, *optional*, defaults to 80):
|
||||
The number of layers that use SWA (Sliding Window Attention). The bottom layers use SWA while the top use full attention.
|
||||
attention_dropout (`float`, *optional*, defaults to 0.0):
|
||||
The dropout ratio for the attention probabilities.
|
||||
vision_config (`Dict`, *optional*):
|
||||
The config for the visual encoder initialization.
|
||||
rope_scaling (`Dict`, *optional*):
|
||||
Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
|
||||
and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
|
||||
accordingly.
|
||||
Expected contents:
|
||||
`rope_type` (`str`):
|
||||
The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
|
||||
'llama3'], with 'default' being the original RoPE implementation.
|
||||
`factor` (`float`, *optional*):
|
||||
Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
|
||||
most scaling types, a `factor` of x will enable the model to handle sequences of length x *
|
||||
original maximum pre-trained length.
|
||||
`original_max_position_embeddings` (`int`, *optional*):
|
||||
Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during
|
||||
pretraining.
|
||||
`attention_factor` (`float`, *optional*):
|
||||
Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
|
||||
computation. If unspecified, it defaults to value recommended by the implementation, using the
|
||||
`factor` field to infer the suggested value.
|
||||
`beta_fast` (`float`, *optional*):
|
||||
Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
|
||||
ramp function. If unspecified, it defaults to 32.
|
||||
`beta_slow` (`float`, *optional*):
|
||||
Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
|
||||
ramp function. If unspecified, it defaults to 1.
|
||||
`short_factor` (`List[float]`, *optional*):
|
||||
Only used with 'longrope'. The scaling factor to be applied to short contexts (<
|
||||
`original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
|
||||
size divided by the number of attention heads divided by 2
|
||||
`long_factor` (`List[float]`, *optional*):
|
||||
Only used with 'longrope'. The scaling factor to be applied to long contexts (<
|
||||
`original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
|
||||
size divided by the number of attention heads divided by 2
|
||||
`low_freq_factor` (`float`, *optional*):
|
||||
Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
|
||||
`high_freq_factor` (`float`, *optional*):
|
||||
Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
|
||||
|
||||
```python
|
||||
>>> from transformers import Qwen2_5_VLForConditionalGeneration, Qwen2_5_VLConfig
|
||||
|
||||
>>> # Initializing a Qwen2_5_VL style configuration
|
||||
>>> configuration = Qwen2_5_VLConfig()
|
||||
|
||||
>>> # Initializing a model from the Qwen2-VL-7B style configuration
|
||||
>>> model = Qwen2_5_VLForConditionalGeneration(configuration)
|
||||
|
||||
>>> # Accessing the model configuration
|
||||
>>> configuration = model.config
|
||||
```"""
|
||||
|
||||
model_type = "qwen2_5_vl"
|
||||
sub_configs = {"vision_config": Qwen2_5_VLVisionConfig}
|
||||
keys_to_ignore_at_inference = ["past_key_values"]
|
||||
# Default tensor parallel plan for base model `Qwen2_5_VL`
|
||||
base_model_tp_plan = {
|
||||
"layers.*.self_attn.q_proj": "colwise",
|
||||
"layers.*.self_attn.k_proj": "colwise",
|
||||
"layers.*.self_attn.v_proj": "colwise",
|
||||
"layers.*.self_attn.o_proj": "rowwise",
|
||||
"layers.*.mlp.gate_proj": "colwise",
|
||||
"layers.*.mlp.up_proj": "colwise",
|
||||
"layers.*.mlp.down_proj": "rowwise",
|
||||
}
|
||||
base_model_pp_plan = {
|
||||
"embed_tokens": (["input_ids"], ["inputs_embeds"]),
|
||||
"layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
|
||||
"norm": (["hidden_states"], ["hidden_states"]),
|
||||
sub_configs = {
|
||||
"vision_config": Qwen2_5_VLVisionConfig,
|
||||
"text_config": Qwen2_5_VLTextConfig,
|
||||
}
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_size=152064,
|
||||
hidden_size=8192,
|
||||
intermediate_size=29568,
|
||||
num_hidden_layers=80,
|
||||
num_attention_heads=64,
|
||||
num_key_value_heads=8,
|
||||
hidden_act="silu",
|
||||
max_position_embeddings=32768,
|
||||
initializer_range=0.02,
|
||||
rms_norm_eps=1e-05,
|
||||
use_cache=True,
|
||||
tie_word_embeddings=False,
|
||||
rope_theta=1000000.0,
|
||||
use_sliding_window=False,
|
||||
sliding_window=4096,
|
||||
max_window_layers=80,
|
||||
attention_dropout=0.0,
|
||||
vision_config=None,
|
||||
rope_scaling=None,
|
||||
num_experts=4,
|
||||
experts=None,
|
||||
dof_config=None,
|
||||
noise_scheduler=None,
|
||||
dim_inputs=(1536, 1536),
|
||||
attention_moe=False,
|
||||
mlp_moe=False,
|
||||
**kwargs,
|
||||
):
|
||||
if isinstance(vision_config, dict):
|
||||
self.vision_config = self.sub_configs["vision_config"](**vision_config)
|
||||
elif vision_config is None:
|
||||
self.vision_config = self.sub_configs["vision_config"]()
|
||||
def __getattr__(self, name):
|
||||
"""Keep legacy direct access to fields now owned by ``text_config``.
|
||||
|
||||
self.vocab_size = vocab_size
|
||||
self.max_position_embeddings = max_position_embeddings
|
||||
self.hidden_size = hidden_size
|
||||
self.intermediate_size = intermediate_size
|
||||
self.num_hidden_layers = num_hidden_layers
|
||||
self.num_attention_heads = num_attention_heads
|
||||
self.use_sliding_window = use_sliding_window
|
||||
self.sliding_window = sliding_window
|
||||
self.max_window_layers = max_window_layers
|
||||
self.layer_types = ["dense"] * num_hidden_layers
|
||||
|
||||
# for backward compatibility
|
||||
if num_key_value_heads is None:
|
||||
num_key_value_heads = num_attention_heads
|
||||
|
||||
self.num_key_value_heads = num_key_value_heads
|
||||
self.hidden_act = hidden_act
|
||||
self.initializer_range = initializer_range
|
||||
self.rms_norm_eps = rms_norm_eps
|
||||
self.use_cache = use_cache
|
||||
self.rope_theta = rope_theta
|
||||
self.attention_dropout = attention_dropout
|
||||
self.rope_scaling = rope_scaling
|
||||
|
||||
self.num_experts = num_experts
|
||||
self.experts = experts
|
||||
self.dof_config = dof_config
|
||||
self.noise_scheduler = noise_scheduler
|
||||
self.dim_inputs = tuple(dim_inputs)
|
||||
self.attention_moe = attention_moe
|
||||
self.mlp_moe = mlp_moe
|
||||
|
||||
if self.rope_scaling is not None and "type" in self.rope_scaling:
|
||||
if self.rope_scaling["type"] == "mrope":
|
||||
self.rope_scaling["type"] = "default"
|
||||
self.rope_scaling["rope_type"] = self.rope_scaling["type"]
|
||||
rope_config_validation(self, ignore_keys={"mrope_section"})
|
||||
|
||||
super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)
|
||||
|
||||
@property
|
||||
def text_config(self):
|
||||
return self
|
||||
|
||||
|
||||
__all__ = ["Qwen2_5_VLConfig"]
|
||||
Wall-X historically used a flat config and accesses fields such as
|
||||
``hidden_size`` and ``num_experts`` directly. Forwarding unknown
|
||||
attributes preserves that API without duplicating the native config.
|
||||
"""
|
||||
text_config = self.__dict__.get("text_config")
|
||||
if name in _LEGACY_TEXT_ATTRIBUTES and text_config is not None and hasattr(text_config, name):
|
||||
return getattr(text_config, name)
|
||||
raise AttributeError(f"{type(self).__name__!s} has no attribute {name!r}")
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,208 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2025 HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Wall-X vision attention backends.
|
||||
|
||||
Qwen2.5-VL's native non-Flash vision path splits a packed image sequence into
|
||||
Python-level chunks before calling attention. Wall-X batches many camera frames,
|
||||
so that path launches thousands of tiny attention operations per training step.
|
||||
This module keeps the native SDPA path as a portable fallback and adds a packed
|
||||
``torch.nn.attention.varlen`` path that consumes Qwen's existing ``cu_seqlens``
|
||||
metadata directly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
import logging
|
||||
from functools import lru_cache
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
from lerobot.utils.import_utils import _transformers_available
|
||||
|
||||
if TYPE_CHECKING or _transformers_available:
|
||||
from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import (
|
||||
Qwen2_5_VLVisionAttention,
|
||||
apply_rotary_pos_emb_vision,
|
||||
)
|
||||
else:
|
||||
Qwen2_5_VLVisionAttention = nn.Module
|
||||
apply_rotary_pos_emb_vision = None
|
||||
|
||||
try:
|
||||
from torch.nn.attention.varlen import varlen_attn as _varlen_attn
|
||||
except ImportError: # torch<2.10
|
||||
_varlen_attn = None
|
||||
|
||||
_VARLEN_USES_WINDOW_SIZE = (
|
||||
_varlen_attn is not None and "window_size" in inspect.signature(_varlen_attn).parameters
|
||||
)
|
||||
|
||||
|
||||
VisionAttentionBackend = Literal["auto", "sdpa", "varlen"]
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@lru_cache
|
||||
def _log_resolved_backend(requested: str, resolved: str) -> None:
|
||||
logger.info("Wall-X vision attention backend: %s (requested: %s)", resolved, requested)
|
||||
|
||||
|
||||
def _varlen_unavailable_reason(
|
||||
hidden_states: torch.Tensor,
|
||||
position_embeddings: tuple[torch.Tensor, torch.Tensor] | None,
|
||||
) -> str | None:
|
||||
if _varlen_attn is None:
|
||||
return "torch.nn.attention.varlen is unavailable (PyTorch 2.10 or newer is required)"
|
||||
if position_embeddings is None:
|
||||
return "precomputed vision position embeddings were not provided"
|
||||
if hidden_states.device.type != "cuda" or torch.version.cuda is None:
|
||||
return "packed varlen attention requires an NVIDIA CUDA device"
|
||||
if hidden_states.dtype not in {torch.float16, torch.bfloat16}:
|
||||
return f"packed varlen attention requires float16 or bfloat16 inputs, got {hidden_states.dtype}"
|
||||
major, _minor = torch.cuda.get_device_capability(hidden_states.device)
|
||||
if major < 8:
|
||||
return "packed varlen attention requires an NVIDIA Ampere GPU or newer"
|
||||
return None
|
||||
|
||||
|
||||
def _supports_varlen_attention(
|
||||
hidden_states: torch.Tensor,
|
||||
position_embeddings: tuple[torch.Tensor, torch.Tensor] | None,
|
||||
) -> bool:
|
||||
return _varlen_unavailable_reason(hidden_states, position_embeddings) is None
|
||||
|
||||
|
||||
class WallXVisionAttention(Qwen2_5_VLVisionAttention):
|
||||
"""Qwen2.5-VL vision attention with packed varlen and native SDPA fallback."""
|
||||
|
||||
def __init__(self, config, backend: VisionAttentionBackend):
|
||||
super().__init__(config)
|
||||
self.wallx_backend = backend
|
||||
self._resolved_backend_key = None
|
||||
self._resolved_backend = None
|
||||
|
||||
def _resolve_backend(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
position_embeddings: tuple[torch.Tensor, torch.Tensor] | None,
|
||||
) -> str:
|
||||
key = (
|
||||
hidden_states.device.type,
|
||||
hidden_states.device.index,
|
||||
hidden_states.dtype,
|
||||
position_embeddings is not None,
|
||||
)
|
||||
if self._resolved_backend_key == key:
|
||||
return self._resolved_backend
|
||||
|
||||
use_varlen = self.wallx_backend != "sdpa" and _supports_varlen_attention(
|
||||
hidden_states, position_embeddings
|
||||
)
|
||||
if self.wallx_backend == "varlen" and not use_varlen:
|
||||
reason = _varlen_unavailable_reason(hidden_states, position_embeddings)
|
||||
raise RuntimeError(f"Wall-X vision_attn_implementation='varlen' cannot be used: {reason}")
|
||||
|
||||
resolved_backend = "varlen" if use_varlen else "sdpa"
|
||||
self._resolved_backend_key = key
|
||||
self._resolved_backend = resolved_backend
|
||||
_log_resolved_backend(self.wallx_backend, resolved_backend)
|
||||
return resolved_backend
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
cu_seqlens: torch.Tensor,
|
||||
rotary_pos_emb: torch.Tensor | None = None,
|
||||
position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
|
||||
**kwargs,
|
||||
) -> torch.Tensor:
|
||||
del rotary_pos_emb
|
||||
|
||||
if self._resolve_backend(hidden_states, position_embeddings) == "sdpa":
|
||||
return super().forward(
|
||||
hidden_states=hidden_states,
|
||||
cu_seqlens=cu_seqlens,
|
||||
position_embeddings=position_embeddings,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
seq_length = hidden_states.shape[0]
|
||||
query_states, key_states, value_states = (
|
||||
self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
|
||||
)
|
||||
|
||||
cos, sin = position_embeddings
|
||||
query_states, key_states = apply_rotary_pos_emb_vision(
|
||||
query_states,
|
||||
key_states,
|
||||
cos,
|
||||
sin,
|
||||
)
|
||||
|
||||
if cu_seqlens.dtype != torch.int32:
|
||||
cu_seqlens = cu_seqlens.to(dtype=torch.int32)
|
||||
max_seqlen = int((cu_seqlens[1:] - cu_seqlens[:-1]).max().item())
|
||||
varlen_kwargs = {"scale": self.scaling}
|
||||
if _VARLEN_USES_WINDOW_SIZE:
|
||||
varlen_kwargs["window_size"] = (-1, -1)
|
||||
else: # Stable PyTorch 2.10 API; pre-release variants used window_size.
|
||||
varlen_kwargs["is_causal"] = False
|
||||
attn_output = _varlen_attn(
|
||||
query_states,
|
||||
key_states,
|
||||
value_states,
|
||||
cu_seqlens,
|
||||
cu_seqlens,
|
||||
max_seqlen,
|
||||
max_seqlen,
|
||||
**varlen_kwargs,
|
||||
)
|
||||
attn_output = attn_output.reshape(seq_length, -1).contiguous()
|
||||
return self.proj(attn_output)
|
||||
|
||||
|
||||
def configure_wall_x_vision_attention(
|
||||
vision_model: nn.Module,
|
||||
backend: VisionAttentionBackend,
|
||||
) -> None:
|
||||
"""Install Wall-X's scoped packed attention without changing checkpoint keys."""
|
||||
if backend == "sdpa":
|
||||
_log_resolved_backend(backend, "sdpa")
|
||||
return
|
||||
if backend == "varlen" and _varlen_attn is None:
|
||||
raise RuntimeError(
|
||||
"Wall-X vision_attn_implementation='varlen' requires torch.nn.attention.varlen "
|
||||
"from PyTorch 2.10 or newer"
|
||||
)
|
||||
if backend == "auto" and _varlen_attn is None:
|
||||
_log_resolved_backend(backend, "sdpa")
|
||||
return
|
||||
|
||||
for block in vision_model.blocks:
|
||||
previous_attention = block.attn
|
||||
replacement = WallXVisionAttention(previous_attention.config, backend=backend)
|
||||
replacement.to(
|
||||
device=previous_attention.qkv.weight.device,
|
||||
dtype=previous_attention.qkv.weight.dtype,
|
||||
)
|
||||
replacement.load_state_dict(previous_attention.state_dict(), strict=True)
|
||||
replacement.train(previous_attention.training)
|
||||
block.attn = replacement
|
||||
@@ -116,6 +116,7 @@ def preprocesser_call(
|
||||
images: list | Any | None = None,
|
||||
text: str | list[str] | None = None,
|
||||
videos: list | Any | None = None,
|
||||
device: torch.device | str | None = None,
|
||||
padding: bool | str = False,
|
||||
truncation: bool | None = None,
|
||||
max_length: int | None = None,
|
||||
@@ -134,6 +135,7 @@ def preprocesser_call(
|
||||
images: Input images (PIL, numpy arrays, or torch tensors)
|
||||
text: Text or list of texts to tokenize
|
||||
videos: Input videos (numpy arrays or torch tensors)
|
||||
device: Device on which image/video preprocessing should run
|
||||
padding: Whether to pad sequences to same length
|
||||
truncation: Whether to truncate sequences longer than max_length
|
||||
max_length: Maximum length for truncation/padding
|
||||
@@ -151,7 +153,11 @@ def preprocesser_call(
|
||||
"""
|
||||
# Process image inputs
|
||||
if images is not None and len(images) > 0:
|
||||
image_inputs = processor.image_processor(images=images, return_tensors=return_tensors)
|
||||
image_inputs = processor.image_processor(
|
||||
images=images,
|
||||
return_tensors=return_tensors,
|
||||
device=device,
|
||||
)
|
||||
image_grid_thw = image_inputs["image_grid_thw"]
|
||||
else:
|
||||
image_inputs = {}
|
||||
@@ -159,7 +165,11 @@ def preprocesser_call(
|
||||
|
||||
# Process video inputs
|
||||
if videos is not None:
|
||||
videos_inputs = processor.image_processor(videos=videos, return_tensors=return_tensors)
|
||||
videos_inputs = processor.image_processor(
|
||||
videos=videos,
|
||||
return_tensors=return_tensors,
|
||||
device=device,
|
||||
)
|
||||
video_grid_thw = videos_inputs["video_grid_thw"]
|
||||
else:
|
||||
videos_inputs = {}
|
||||
@@ -413,10 +423,7 @@ def get_task_instruction(
|
||||
}
|
||||
)
|
||||
|
||||
if priority_order is not None:
|
||||
priority_order = OrderedDict(priority_order)
|
||||
else:
|
||||
priority_order = default_priority_order
|
||||
priority_order = OrderedDict(priority_order) if priority_order is not None else default_priority_order
|
||||
|
||||
got_instruction = False
|
||||
task_instruction = ""
|
||||
@@ -424,9 +431,8 @@ def get_task_instruction(
|
||||
# Sample instruction components based on priority probabilities
|
||||
for key, prob in priority_order.items():
|
||||
if key in frame_instruction_info and frame_instruction_info[key] != "":
|
||||
if got_instruction:
|
||||
if random.random() >= prob:
|
||||
continue
|
||||
if got_instruction and random.random() >= prob:
|
||||
continue
|
||||
|
||||
task_instruction += f"\n{frame_instruction_info[key]}"
|
||||
got_instruction = True
|
||||
@@ -538,10 +544,7 @@ def img_key_mapping(img_keys: list[str]) -> list[str]:
|
||||
if key in CAMERA_NAME_MAPPING:
|
||||
key = CAMERA_NAME_MAPPING[key]
|
||||
else:
|
||||
if "view" in key:
|
||||
key = key.replace("_", " ")
|
||||
else:
|
||||
key = key + " view"
|
||||
key = key.replace("_", " ") if "view" in key else key + " view"
|
||||
processed_img_keys.append(key)
|
||||
return processed_img_keys
|
||||
|
||||
|
||||
@@ -22,19 +22,14 @@ import torch
|
||||
|
||||
from lerobot.configs import PipelineFeatureType, PolicyFeature
|
||||
from lerobot.processor import (
|
||||
AddBatchDimensionProcessorStep,
|
||||
DeviceProcessorStep,
|
||||
NormalizerProcessorStep,
|
||||
ObservationProcessorStep,
|
||||
PolicyAction,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStep,
|
||||
ProcessorStepRegistry,
|
||||
RenameObservationsProcessorStep,
|
||||
TokenizerProcessorStep,
|
||||
UnnormalizerProcessorStep,
|
||||
policy_action_to_transition,
|
||||
transition_to_policy_action,
|
||||
make_default_policy_processor_steps,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from lerobot.types import EnvTransition, TransitionKey
|
||||
from lerobot.utils.constants import (
|
||||
@@ -42,8 +37,6 @@ from lerobot.utils.constants import (
|
||||
OBS_IMAGES,
|
||||
OBS_PREFIX,
|
||||
OBS_STATE,
|
||||
POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
)
|
||||
|
||||
from .configuration_xvla import XVLAConfig
|
||||
@@ -61,10 +54,11 @@ def make_xvla_pre_post_processors(
|
||||
Build the LeRobot processor pipelines for XVLA.
|
||||
"""
|
||||
|
||||
features = {**config.input_features, **config.output_features}
|
||||
steps = make_default_policy_processor_steps(config, dataset_stats)
|
||||
|
||||
input_steps = [
|
||||
RenameObservationsProcessorStep(rename_map={}),
|
||||
AddBatchDimensionProcessorStep(),
|
||||
steps.rename_observations,
|
||||
steps.add_batch_dim,
|
||||
TokenizerProcessorStep(
|
||||
tokenizer_name=config.tokenizer_name,
|
||||
max_length=config.tokenizer_max_length,
|
||||
@@ -74,32 +68,15 @@ def make_xvla_pre_post_processors(
|
||||
XVLAImageToFloatProcessorStep(),
|
||||
XVLAImageNetNormalizeProcessorStep(),
|
||||
XVLAAddDomainIdProcessorStep(),
|
||||
DeviceProcessorStep(device=config.device),
|
||||
NormalizerProcessorStep(
|
||||
features=features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
steps.to_device,
|
||||
steps.normalize,
|
||||
]
|
||||
output_steps = [
|
||||
UnnormalizerProcessorStep(
|
||||
features=config.output_features,
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
),
|
||||
DeviceProcessorStep(device="cpu"),
|
||||
steps.unnormalize,
|
||||
steps.to_cpu,
|
||||
]
|
||||
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
|
||||
|
||||
|
||||
# Custom XVLA processor steps
|
||||
|
||||
@@ -42,10 +42,14 @@ from .delta_action_processor import MapDeltaActionToRobotActionStep, MapTensorTo
|
||||
from .device_processor import DeviceProcessorStep
|
||||
from .env_processor import IsaaclabArenaProcessorStep, LiberoProcessorStep
|
||||
from .factory import (
|
||||
DefaultPolicyProcessorSteps,
|
||||
make_default_policy_processor_steps,
|
||||
make_default_pre_post_processors,
|
||||
make_default_processors,
|
||||
make_default_robot_action_processor,
|
||||
make_default_robot_observation_processor,
|
||||
make_default_teleop_action_processor,
|
||||
make_policy_processor_pipelines,
|
||||
)
|
||||
from .gym_action_processor import (
|
||||
Numpy2TorchActionProcessorStep,
|
||||
@@ -129,10 +133,14 @@ __all__ = [
|
||||
"ImageCropResizeProcessorStep",
|
||||
"InfoProcessorStep",
|
||||
"InterventionActionProcessorStep",
|
||||
"DefaultPolicyProcessorSteps",
|
||||
"make_default_policy_processor_steps",
|
||||
"make_default_pre_post_processors",
|
||||
"make_default_processors",
|
||||
"make_default_teleop_action_processor",
|
||||
"make_default_robot_action_processor",
|
||||
"make_default_robot_observation_processor",
|
||||
"make_policy_processor_pipelines",
|
||||
"AbsoluteActionsProcessorStep",
|
||||
"RelativeActionsProcessorStep",
|
||||
"MapDeltaActionToRobotActionStep",
|
||||
|
||||
@@ -14,15 +14,33 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from lerobot.types import RobotAction, RobotObservation
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
from lerobot.configs.policies import PreTrainedConfig
|
||||
from lerobot.types import PolicyAction, RobotAction, RobotObservation
|
||||
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
|
||||
|
||||
from .batch_processor import AddBatchDimensionProcessorStep
|
||||
from .converters import (
|
||||
observation_to_transition,
|
||||
policy_action_to_transition,
|
||||
robot_action_observation_to_transition,
|
||||
transition_to_observation,
|
||||
transition_to_policy_action,
|
||||
transition_to_robot_action,
|
||||
)
|
||||
from .pipeline import IdentityProcessorStep, RobotProcessorPipeline
|
||||
from .device_processor import DeviceProcessorStep
|
||||
from .normalize_processor import NormalizerProcessorStep, UnnormalizerProcessorStep
|
||||
from .pipeline import (
|
||||
IdentityProcessorStep,
|
||||
PolicyProcessorPipeline,
|
||||
ProcessorStep,
|
||||
RobotProcessorPipeline,
|
||||
)
|
||||
from .rename_processor import RenameObservationsProcessorStep
|
||||
|
||||
|
||||
def make_default_teleop_action_processor() -> RobotProcessorPipeline[
|
||||
@@ -61,3 +79,97 @@ def make_default_processors():
|
||||
robot_action_processor = make_default_robot_action_processor()
|
||||
robot_observation_processor = make_default_robot_observation_processor()
|
||||
return (teleop_action_processor, robot_action_processor, robot_observation_processor)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DefaultPolicyProcessorSteps:
|
||||
"""The canonical processor steps shared by most policies' pre/post pipelines.
|
||||
|
||||
Policies compose these in their own order (step ORDER is a Hub-serialized contract
|
||||
and intentionally stays explicit per policy) and interleave their custom steps.
|
||||
"""
|
||||
|
||||
rename_observations: RenameObservationsProcessorStep
|
||||
add_batch_dim: AddBatchDimensionProcessorStep
|
||||
to_device: DeviceProcessorStep
|
||||
normalize: NormalizerProcessorStep
|
||||
unnormalize: UnnormalizerProcessorStep
|
||||
to_cpu: DeviceProcessorStep
|
||||
|
||||
|
||||
def make_default_policy_processor_steps(
|
||||
config: PreTrainedConfig,
|
||||
dataset_stats: dict[str, dict[str, torch.Tensor]] | None = None,
|
||||
*,
|
||||
normalizer_device: torch.device | str | None = None,
|
||||
) -> DefaultPolicyProcessorSteps:
|
||||
"""Construct the canonical policy processor steps from a policy config.
|
||||
|
||||
Args:
|
||||
config: A `PreTrainedConfig` providing `device`, `input_features`,
|
||||
`output_features` and `normalization_mapping`.
|
||||
dataset_stats: Dataset statistics used for (un)normalization.
|
||||
normalizer_device: Device passed to `NormalizerProcessorStep` (some policies pin
|
||||
their normalization stats to the policy device; most leave it unset).
|
||||
"""
|
||||
return DefaultPolicyProcessorSteps(
|
||||
rename_observations=RenameObservationsProcessorStep(rename_map={}),
|
||||
add_batch_dim=AddBatchDimensionProcessorStep(),
|
||||
to_device=DeviceProcessorStep(device=config.device),
|
||||
normalize=NormalizerProcessorStep(
|
||||
features={**config.input_features, **config.output_features},
|
||||
norm_map=config.normalization_mapping,
|
||||
stats=dataset_stats,
|
||||
device=normalizer_device,
|
||||
),
|
||||
unnormalize=UnnormalizerProcessorStep(
|
||||
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
|
||||
),
|
||||
to_cpu=DeviceProcessorStep(device="cpu"),
|
||||
)
|
||||
|
||||
|
||||
def make_policy_processor_pipelines(
|
||||
input_steps: list[ProcessorStep],
|
||||
output_steps: list[ProcessorStep],
|
||||
) -> tuple[
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction],
|
||||
]:
|
||||
"""Wrap pre/post step lists into the canonical policy pipeline pair.
|
||||
|
||||
Uses the standard pipeline names (which determine the serialized JSON filenames on
|
||||
the Hub) and the standard policy-action converters on the postprocessor.
|
||||
"""
|
||||
return (
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
|
||||
steps=input_steps,
|
||||
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
|
||||
),
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction](
|
||||
steps=output_steps,
|
||||
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
|
||||
to_transition=policy_action_to_transition,
|
||||
to_output=transition_to_policy_action,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def make_default_pre_post_processors(
|
||||
config: PreTrainedConfig,
|
||||
dataset_stats: dict[str, dict[str, torch.Tensor]] | None = None,
|
||||
*,
|
||||
normalizer_device: torch.device | str | None = None,
|
||||
) -> tuple[
|
||||
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
|
||||
PolicyProcessorPipeline[PolicyAction, PolicyAction],
|
||||
]:
|
||||
"""The pure-scaffold policy pipeline pair: Rename -> Batch -> Device -> Normalize,
|
||||
and Unnormalize -> Device(cpu). Policies with custom steps or a different step order
|
||||
compose `make_default_policy_processor_steps` themselves instead.
|
||||
"""
|
||||
s = make_default_policy_processor_steps(config, dataset_stats, normalizer_device=normalizer_device)
|
||||
return make_policy_processor_pipelines(
|
||||
input_steps=[s.rename_observations, s.add_batch_dim, s.to_device, s.normalize],
|
||||
output_steps=[s.unnormalize, s.to_cpu],
|
||||
)
|
||||
|
||||
@@ -21,8 +21,6 @@ from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
from typing import TYPE_CHECKING, Any, TypeVar
|
||||
|
||||
import packaging
|
||||
import safetensors
|
||||
from huggingface_hub import HfApi, ModelCard, ModelCardData, hf_hub_download
|
||||
from huggingface_hub.constants import SAFETENSORS_SINGLE_FILE
|
||||
from huggingface_hub.errors import HfHubHTTPError
|
||||
@@ -30,6 +28,7 @@ from safetensors.torch import load_model as load_model_as_safetensor, save_model
|
||||
from torch import Tensor, nn
|
||||
|
||||
from lerobot.configs.rewards import RewardModelConfig
|
||||
from lerobot.utils.device_utils import resolve_safetensors_device
|
||||
from lerobot.utils.hub import HubMixin
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -129,29 +128,13 @@ class PreTrainedRewardModel(nn.Module, HubMixin, abc.ABC):
|
||||
|
||||
@classmethod
|
||||
def _load_as_safetensor(cls, model: T, model_file: str, map_location: str, strict: bool) -> T:
|
||||
# Create base kwargs
|
||||
kwargs = {"strict": strict}
|
||||
|
||||
# Add device parameter for newer versions that support it
|
||||
if packaging.version.parse(safetensors.__version__) >= packaging.version.parse("0.4.3"):
|
||||
kwargs["device"] = map_location
|
||||
|
||||
# Load the model with appropriate kwargs
|
||||
missing_keys, unexpected_keys = load_model_as_safetensor(model, model_file, **kwargs)
|
||||
missing_keys, unexpected_keys = load_model_as_safetensor(
|
||||
model, model_file, strict=strict, device=resolve_safetensors_device(map_location)
|
||||
)
|
||||
if missing_keys:
|
||||
logging.warning(f"Missing key(s) when loading model: {missing_keys}")
|
||||
if unexpected_keys:
|
||||
logging.warning(f"Unexpected key(s) when loading model: {unexpected_keys}")
|
||||
|
||||
# For older versions, manually move to device if needed
|
||||
if "device" not in kwargs and map_location != "cpu":
|
||||
logging.warning(
|
||||
"Loading model weights on other devices than 'cpu' is not supported natively in your version of safetensors."
|
||||
" This means that the model is loaded on 'cpu' first and then copied to the device."
|
||||
" This leads to a slower loading time."
|
||||
" Please update safetensors to version 0.4.3 or above for improved performance."
|
||||
)
|
||||
model.to(map_location)
|
||||
return model
|
||||
|
||||
def get_optim_params(self):
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
@@ -33,6 +34,8 @@ from lerobot.processor import (
|
||||
)
|
||||
from lerobot.utils.rotation import Rotation
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@ProcessorStepRegistry.register("ee_reference_and_delta")
|
||||
@dataclass
|
||||
@@ -194,11 +197,17 @@ class EEBoundsAndSafety(RobotActionProcessorStep):
|
||||
Attributes:
|
||||
end_effector_bounds: A dictionary with "min" and "max" keys for position clipping.
|
||||
max_ee_step_m: The maximum allowed change in position (in meters) between steps.
|
||||
raise_on_jump: When ``True`` (default) an over-limit per-frame step raises
|
||||
``ValueError`` (aborting the control loop). When ``False`` the step is
|
||||
rate-limited to ``max_ee_step_m`` and a warning is logged instead — the
|
||||
safer choice for live teleoperation, where a transient tracking glitch
|
||||
should not crash the loop and leave the robot uncontrolled.
|
||||
_last_pos: Internal state storing the last commanded position.
|
||||
"""
|
||||
|
||||
end_effector_bounds: dict
|
||||
max_ee_step_m: float = 0.05
|
||||
raise_on_jump: bool = True
|
||||
_last_pos: np.ndarray | None = field(default=None, init=False, repr=False)
|
||||
|
||||
def action(self, action: RobotAction) -> RobotAction:
|
||||
@@ -226,8 +235,19 @@ class EEBoundsAndSafety(RobotActionProcessorStep):
|
||||
dpos = pos - self._last_pos
|
||||
n = float(np.linalg.norm(dpos))
|
||||
if n > self.max_ee_step_m and n > 0:
|
||||
# Clamp the step to the per-frame limit (rate-limit). The clamped
|
||||
# value is computed either way; raise_on_jump only decides whether
|
||||
# an over-limit step aborts the loop or is rate-limited + warned.
|
||||
pos = self._last_pos + dpos * (self.max_ee_step_m / n)
|
||||
raise ValueError(f"EE jump {n:.3f}m > {self.max_ee_step_m}m")
|
||||
if self.raise_on_jump:
|
||||
raise ValueError(f"EE jump {n:.3f}m > {self.max_ee_step_m}m")
|
||||
logger.warning(
|
||||
"EE jump %.3fm > %.3fm; rate-limited to the per-frame step "
|
||||
"(likely a transient tracking glitch; if it recurs every frame "
|
||||
"the commanded target is systematically out of workspace).",
|
||||
n,
|
||||
self.max_ee_step_m,
|
||||
)
|
||||
|
||||
self._last_pos = pos
|
||||
|
||||
@@ -264,12 +284,18 @@ class InverseKinematicsEEToJoints(RobotActionProcessorStep):
|
||||
q_curr: Internal state storing the last joint positions, used as an initial guess for the IK solver.
|
||||
initial_guess_current_joints: If True, use the robot's current joint state as the IK guess.
|
||||
If False, use the solution from the previous step.
|
||||
orientation_weight: Weight for the orientation constraint passed to
|
||||
``RobotKinematics.inverse_kinematics``. Defaults to ``0.01`` (matching the solver
|
||||
default, so existing callers are unchanged). Set to ``0.0`` for position-only IK on
|
||||
under-actuated arms; a small nonzero weight gives soft-orientation IK on the 5-DOF
|
||||
SO-101, where the wrist tracks orientation only partially (position dominates).
|
||||
"""
|
||||
|
||||
kinematics: RobotKinematics
|
||||
motor_names: list[str]
|
||||
q_curr: np.ndarray | None = field(default=None, init=False, repr=False)
|
||||
initial_guess_current_joints: bool = True
|
||||
orientation_weight: float = 0.01
|
||||
|
||||
def action(self, action: RobotAction) -> RobotAction:
|
||||
x = action.pop("ee.x")
|
||||
@@ -308,7 +334,9 @@ class InverseKinematicsEEToJoints(RobotActionProcessorStep):
|
||||
t_des[:3, 3] = [x, y, z]
|
||||
|
||||
# Compute inverse kinematics
|
||||
q_target = self.kinematics.inverse_kinematics(self.q_curr, t_des)
|
||||
q_target = self.kinematics.inverse_kinematics(
|
||||
self.q_curr, t_des, orientation_weight=self.orientation_weight
|
||||
)
|
||||
self.q_curr = q_target
|
||||
|
||||
# TODO: This is sentitive to order of motor_names = q_target mapping
|
||||
|
||||
@@ -28,7 +28,12 @@ For distributed runs, see ``examples/annotations/run_hf_job.py``.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from contextlib import suppress
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from huggingface_hub import HfApi, snapshot_download
|
||||
from huggingface_hub.errors import RevisionNotFoundError
|
||||
|
||||
from lerobot.annotations.steerable_pipeline.config import AnnotationPipelineConfig
|
||||
from lerobot.annotations.steerable_pipeline.executor import Executor
|
||||
@@ -42,6 +47,12 @@ from lerobot.annotations.steerable_pipeline.validator import StagingValidator
|
||||
from lerobot.annotations.steerable_pipeline.vlm_client import make_vlm_client
|
||||
from lerobot.annotations.steerable_pipeline.writer import LanguageColumnsWriter
|
||||
from lerobot.configs import parser
|
||||
from lerobot.utils.import_utils import _datasets_available, require_package
|
||||
|
||||
if TYPE_CHECKING or _datasets_available:
|
||||
from lerobot.datasets.dataset_metadata import CODEBASE_VERSION
|
||||
from lerobot.datasets.io_utils import load_info
|
||||
from lerobot.datasets.utils import create_lerobot_dataset_card
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -50,8 +61,6 @@ def _resolve_root(cfg: AnnotationPipelineConfig) -> Path:
|
||||
if cfg.root is not None:
|
||||
return Path(cfg.root)
|
||||
if cfg.repo_id is not None:
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
return Path(snapshot_download(repo_id=cfg.repo_id, repo_type="dataset"))
|
||||
raise ValueError("Either --root or --repo_id must be provided.")
|
||||
|
||||
@@ -125,7 +134,7 @@ def _push_to_hub(root: Path, cfg: AnnotationPipelineConfig) -> None:
|
||||
|
||||
Pushes to ``cfg.new_repo_id`` when set, otherwise back to ``cfg.repo_id``.
|
||||
"""
|
||||
from huggingface_hub import HfApi # noqa: PLC0415
|
||||
require_package("datasets", "dataset")
|
||||
|
||||
repo_id = cfg.new_repo_id or cfg.repo_id
|
||||
commit_message = cfg.push_commit_message or "Add steerable annotations (lerobot-annotate)"
|
||||
@@ -143,33 +152,26 @@ def _push_to_hub(root: Path, cfg: AnnotationPipelineConfig) -> None:
|
||||
repo_id=repo_id,
|
||||
repo_type="dataset",
|
||||
commit_message=commit_message,
|
||||
ignore_patterns=[".annotate_staging/**", "**/.DS_Store"],
|
||||
# README.md is excluded because when pushing to ``new_repo_id`` the
|
||||
# source card's links (e.g. the visualize badge) would keep pointing
|
||||
# at the source dataset; a fresh card is generated below instead.
|
||||
ignore_patterns=[".annotate_staging/**", "**/.DS_Store", "README.md"],
|
||||
)
|
||||
print(f"[lerobot-annotate] uploaded to https://huggingface.co/datasets/{repo_id}", flush=True)
|
||||
|
||||
dataset_info = load_info(root)
|
||||
card = create_lerobot_dataset_card(dataset_info=dataset_info, license="apache-2.0", repo_id=repo_id)
|
||||
card.push_to_hub(repo_id=repo_id, repo_type="dataset")
|
||||
|
||||
# Tag the upload with the codebase version. ``LeRobotDatasetMetadata``
|
||||
# resolves the dataset revision via ``get_safe_version`` which scans
|
||||
# for tags like ``v3.0``; without a tag it raises
|
||||
# ``RevisionNotFoundError``. Read the version straight from the
|
||||
# dataset's own ``meta/info.json`` so we tag whatever the writer
|
||||
# actually wrote (no accidental drift if the codebase floor moves).
|
||||
from lerobot.datasets.dataset_metadata import CODEBASE_VERSION # noqa: PLC0415
|
||||
|
||||
info_path = root / "meta" / "info.json"
|
||||
version_tag = CODEBASE_VERSION
|
||||
if info_path.exists():
|
||||
try:
|
||||
from lerobot.utils.io_utils import load_json # noqa: PLC0415
|
||||
|
||||
info = load_json(info_path)
|
||||
ds_version = info.get("codebase_version")
|
||||
if isinstance(ds_version, str) and ds_version.startswith("v"):
|
||||
version_tag = ds_version
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(
|
||||
f"[lerobot-annotate] could not read codebase_version from info.json ({exc}); falling back to {version_tag}",
|
||||
flush=True,
|
||||
)
|
||||
version_tag = (
|
||||
dataset_info.codebase_version if dataset_info.codebase_version.startswith("v") else CODEBASE_VERSION
|
||||
)
|
||||
revision = getattr(commit_info, "oid", None)
|
||||
tag_kwargs = {
|
||||
"repo_id": repo_id,
|
||||
@@ -180,10 +182,6 @@ def _push_to_hub(root: Path, cfg: AnnotationPipelineConfig) -> None:
|
||||
tag_kwargs["revision"] = revision
|
||||
|
||||
try:
|
||||
from contextlib import suppress # noqa: PLC0415
|
||||
|
||||
from huggingface_hub.errors import RevisionNotFoundError # noqa: PLC0415
|
||||
|
||||
with suppress(RevisionNotFoundError):
|
||||
api.delete_tag(repo_id, tag=version_tag, repo_type="dataset")
|
||||
api.create_tag(**tag_kwargs)
|
||||
|
||||
@@ -171,6 +171,9 @@ def update_policy(
|
||||
train_metrics.update_s = time.perf_counter() - start_time
|
||||
if torch.cuda.is_available():
|
||||
train_metrics.gpu_mem_gb = torch.cuda.max_memory_allocated() / (1024**3)
|
||||
# Aggregate the policy's scalar outputs for logging and rank-reduction across the log window.
|
||||
if output_dict:
|
||||
train_metrics.update_metrics(output_dict)
|
||||
return train_metrics, output_dict
|
||||
|
||||
|
||||
@@ -572,7 +575,7 @@ def train(cfg: TrainPipelineConfig, accelerator: "Accelerator | None" = None):
|
||||
batch = preprocessor(batch)
|
||||
train_tracker.dataloading_s = time.perf_counter() - start_time
|
||||
|
||||
train_tracker, output_dict = update_policy(
|
||||
train_tracker, _ = update_policy(
|
||||
train_tracker,
|
||||
policy,
|
||||
batch,
|
||||
@@ -605,9 +608,10 @@ def train(cfg: TrainPipelineConfig, accelerator: "Accelerator | None" = None):
|
||||
train_tracker.samples_per_s = effective_batch_size / step_time
|
||||
logging.info(train_tracker)
|
||||
if wandb_logger:
|
||||
# Policy sub-losses (latent_loss, action_loss, ...) are aggregated into the
|
||||
# tracker by update_policy, so to_dict() already carries their windowed,
|
||||
# rank-reduced averages — no per-step output_dict passthrough needed.
|
||||
wandb_log_dict = train_tracker.to_dict()
|
||||
if output_dict:
|
||||
wandb_log_dict.update(output_dict)
|
||||
# Log sample weighting statistics if enabled
|
||||
if sample_weighter is not None:
|
||||
weighter_stats = sample_weighter.get_stats()
|
||||
|
||||
@@ -59,6 +59,20 @@ def get_safe_torch_device(try_device: str, log: bool = False) -> torch.device:
|
||||
return device
|
||||
|
||||
|
||||
def resolve_safetensors_device(map_location: str | torch.device) -> str:
|
||||
"""Resolve a device string for a safetensors load, working around a device-mapping quirk.
|
||||
|
||||
safetensors' load maps the bare string "cuda" to cuda:0 regardless of the current device
|
||||
(unlike torch's .to("cuda"), which honors torch.cuda.current_device()). Under multi-GPU
|
||||
accelerate/FSDP every rank would then load its weights onto GPU 0, OOMing it before sharding.
|
||||
Resolve "cuda" to the concrete current-device index so each rank loads onto its own GPU.
|
||||
"""
|
||||
map_location = str(map_location)
|
||||
if map_location == "cuda" and torch.cuda.is_available():
|
||||
return f"cuda:{torch.cuda.current_device()}"
|
||||
return map_location
|
||||
|
||||
|
||||
def get_safe_dtype(dtype: torch.dtype, device: str | torch.device):
|
||||
"""
|
||||
mps is currently not compatible with float64
|
||||
|
||||
@@ -104,6 +104,7 @@ class MetricsTracker:
|
||||
"episodes",
|
||||
"epochs",
|
||||
"accelerator",
|
||||
"_caller_metrics",
|
||||
]
|
||||
|
||||
def __init__(
|
||||
@@ -129,6 +130,9 @@ class MetricsTracker:
|
||||
self.episodes = self.samples / self._avg_samples_per_ep
|
||||
self.epochs = self.samples / self._num_frames
|
||||
self.accelerator = accelerator
|
||||
# Meter names the caller registered up front. update_metrics() leaves these untouched, so a
|
||||
# policy that echoes e.g. "loss" in its output dict can't clobber the aggregated meter.
|
||||
self._caller_metrics: set[str] = set(self.metrics)
|
||||
|
||||
def __getattr__(self, name: str) -> int | dict[str, AverageMeter] | AverageMeter | Any:
|
||||
if name in self.__dict__:
|
||||
@@ -156,6 +160,21 @@ class MetricsTracker:
|
||||
self.episodes = self.samples / self._avg_samples_per_ep
|
||||
self.epochs = self.samples / self._num_frames
|
||||
|
||||
def update_metrics(self, values: dict[str, Any]) -> None:
|
||||
"""Accumulate a dict of scalar metrics, auto-registering a meter for each new key.
|
||||
|
||||
Non-numeric values and bools are ignored.
|
||||
Caller-registered metrics (those passed to the constructor) are never overridden.
|
||||
"""
|
||||
for name, value in values.items():
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||
continue
|
||||
if name in self._caller_metrics:
|
||||
continue
|
||||
if name not in self.metrics:
|
||||
self.metrics[name] = AverageMeter(name, ":.3f", reduction="mean")
|
||||
self.metrics[name].update(float(value))
|
||||
|
||||
def reduce_across_ranks(self) -> None:
|
||||
"""
|
||||
Synchronises the running averages of every metric whose ``reduction`` is not ``"none"``
|
||||
|
||||
@@ -85,7 +85,7 @@ def _spy_responder(captured: list[list[dict[str, Any]]], reply: Any):
|
||||
def test_module1_plan_memory_subtask_smoke(fixture_dataset_root: Path, tmp_path: Path) -> None:
|
||||
vlm = make_canned_responder(
|
||||
{
|
||||
"atomic subtasks": {
|
||||
"COMPLETED manipulation events": {
|
||||
"subtasks": [
|
||||
{"text": "grasp the handle of the sponge", "start": 0.0, "end": 0.4},
|
||||
{"text": "wipe the counter from left to right", "start": 0.4, "end": 0.8},
|
||||
@@ -126,7 +126,7 @@ def test_module1_emit_memory_false_skips_memory_keeps_subtasks_and_plan(
|
||||
leaving subtask + plan generation intact — symmetric to ``emit_plan``."""
|
||||
vlm = make_canned_responder(
|
||||
{
|
||||
"atomic subtasks": {
|
||||
"COMPLETED manipulation events": {
|
||||
"subtasks": [
|
||||
{"text": "grasp the handle of the sponge", "start": 0.0, "end": 0.4},
|
||||
{"text": "wipe the counter from left to right", "start": 0.4, "end": 0.8},
|
||||
@@ -318,7 +318,7 @@ def test_module1_attaches_contact_sheets_to_subtask_prompt(
|
||||
return block.get("text", "")
|
||||
return ""
|
||||
|
||||
subtask_calls = [m for m in captured if "atomic subtasks" in _prompt_text(m)]
|
||||
subtask_calls = [m for m in captured if "COMPLETED manipulation events" in _prompt_text(m)]
|
||||
assert len(subtask_calls) == 1, "expected exactly one subtask-prompt VLM call"
|
||||
content = subtask_calls[0][0]["content"]
|
||||
video_blocks = [b for b in content if isinstance(b, dict) and b.get("type") == "video"]
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Behavior-pinning tests for the shared flow-matching sampling primitives.
|
||||
|
||||
``euler_integrate`` is compared against a verbatim copy of the historical pi0/pi05/
|
||||
smolvla sampling loop (including its RTC hook semantics): any divergence from that
|
||||
reference is a behavior change for released checkpoints.
|
||||
"""
|
||||
|
||||
import torch
|
||||
|
||||
from lerobot.policies.common.flow_matching import (
|
||||
euler_integrate,
|
||||
sample_beta,
|
||||
sample_noise,
|
||||
sample_time_beta,
|
||||
)
|
||||
|
||||
|
||||
def test_sample_beta_range_dtype_and_reproducibility():
|
||||
torch.manual_seed(0)
|
||||
s1 = sample_beta(1.5, 1.0, 4096, "cpu")
|
||||
torch.manual_seed(0)
|
||||
s2 = sample_beta(1.5, 1.0, 4096, "cpu")
|
||||
assert torch.equal(s1, s2)
|
||||
assert s1.shape == (4096,) and s1.dtype == torch.float32
|
||||
assert s1.min() >= 0.0 and s1.max() <= 1.0
|
||||
# Beta(1.5, 1.0) mean is 1.5/2.5 = 0.6.
|
||||
assert abs(s1.mean().item() - 0.6) < 0.02
|
||||
|
||||
|
||||
def test_sample_time_beta_openpi_convention():
|
||||
torch.manual_seed(1)
|
||||
time = sample_time_beta(4096, "cpu", alpha=1.5, beta=1.0, scale=0.999, offset=0.001)
|
||||
assert time.dtype == torch.float32
|
||||
assert time.min() >= 0.001 and time.max() <= 1.0
|
||||
# Exact composition: Beta sample * scale + offset, same RNG stream.
|
||||
torch.manual_seed(1)
|
||||
expected = sample_beta(1.5, 1.0, 4096, "cpu") * 0.999 + 0.001
|
||||
torch.testing.assert_close(time, expected, rtol=0, atol=0)
|
||||
|
||||
|
||||
def test_sample_noise_seeded():
|
||||
torch.manual_seed(2)
|
||||
n1 = sample_noise((2, 8, 4), "cpu")
|
||||
torch.manual_seed(2)
|
||||
n2 = sample_noise((2, 8, 4), "cpu")
|
||||
assert torch.equal(n1, n2)
|
||||
assert n1.dtype == torch.float32 and n1.shape == (2, 8, 4)
|
||||
|
||||
|
||||
def test_euler_integrate_constant_velocity_is_exact():
|
||||
# With v_t == c constant, x_0 = x_1 + sum(dt * c) = x_1 - c exactly (num_steps * dt = -1).
|
||||
noise = torch.randn(3, 5, 2)
|
||||
c = torch.randn(3, 5, 2)
|
||||
out = euler_integrate(lambda x_t, time: c, noise, num_steps=10)
|
||||
torch.testing.assert_close(out, noise - c, rtol=0, atol=1e-6)
|
||||
|
||||
|
||||
def _reference_pi0_loop(denoise_fn, noise, num_steps, rtc_enabled, rtc_processor, kw):
|
||||
"""Verbatim structure of the historical pi0/pi05/smolvla sample_actions loop."""
|
||||
bsize = noise.shape[0]
|
||||
device = noise.device
|
||||
dt = -1.0 / num_steps
|
||||
x_t = noise
|
||||
for step in range(num_steps):
|
||||
time = 1.0 + step * dt
|
||||
time_tensor = torch.tensor(time, dtype=torch.float32, device=device).expand(bsize)
|
||||
|
||||
def denoise_step_partial_call(input_x_t, current_timestep=time_tensor):
|
||||
return denoise_fn(input_x_t, current_timestep)
|
||||
|
||||
if rtc_enabled:
|
||||
v_t = rtc_processor.denoise_step(
|
||||
x_t=x_t,
|
||||
prev_chunk_left_over=kw.get("prev_chunk_left_over"),
|
||||
inference_delay=kw.get("inference_delay"),
|
||||
time=time,
|
||||
original_denoise_step_partial=denoise_step_partial_call,
|
||||
execution_horizon=kw.get("execution_horizon"),
|
||||
)
|
||||
else:
|
||||
v_t = denoise_step_partial_call(x_t)
|
||||
x_t = x_t + dt * v_t
|
||||
if rtc_processor is not None and rtc_processor.is_debug_enabled():
|
||||
rtc_processor.track(time=time, x_t=x_t, v_t=v_t)
|
||||
return x_t
|
||||
|
||||
|
||||
class _StubRTCProcessor:
|
||||
def __init__(self, debug_enabled: bool):
|
||||
self._debug = debug_enabled
|
||||
self.tracked = []
|
||||
self.guidance_calls = []
|
||||
|
||||
def is_debug_enabled(self):
|
||||
return self._debug
|
||||
|
||||
def denoise_step(
|
||||
self,
|
||||
x_t,
|
||||
prev_chunk_left_over,
|
||||
inference_delay,
|
||||
time,
|
||||
original_denoise_step_partial,
|
||||
execution_horizon,
|
||||
):
|
||||
self.guidance_calls.append(
|
||||
{
|
||||
"time": time,
|
||||
"inference_delay": inference_delay,
|
||||
"execution_horizon": execution_horizon,
|
||||
"x_t": x_t.clone(),
|
||||
}
|
||||
)
|
||||
return original_denoise_step_partial(x_t) * 0.5
|
||||
|
||||
def track(self, time, x_t, v_t):
|
||||
self.tracked.append({"time": time, "x_t": x_t.clone(), "v_t": v_t.clone()})
|
||||
|
||||
|
||||
def _make_denoise_fn():
|
||||
weight = torch.randn(4, 4) * 0.1
|
||||
|
||||
def denoise_fn(x_t, time_tensor):
|
||||
return x_t @ weight + time_tensor[:, None, None]
|
||||
|
||||
return denoise_fn
|
||||
|
||||
|
||||
def test_euler_integrate_matches_historical_loop():
|
||||
torch.manual_seed(3)
|
||||
denoise_fn = _make_denoise_fn()
|
||||
noise = torch.randn(2, 6, 4)
|
||||
ref = _reference_pi0_loop(denoise_fn, noise, 10, rtc_enabled=False, rtc_processor=None, kw={})
|
||||
out = euler_integrate(denoise_fn, noise, 10)
|
||||
assert torch.equal(out, ref)
|
||||
|
||||
|
||||
def test_euler_integrate_rtc_guidance_and_kwarg_forwarding():
|
||||
torch.manual_seed(4)
|
||||
denoise_fn = _make_denoise_fn()
|
||||
noise = torch.randn(2, 6, 4)
|
||||
leftover = torch.randn(2, 6, 4)
|
||||
kw = {"inference_delay": 3, "prev_chunk_left_over": leftover, "execution_horizon": 25}
|
||||
|
||||
ref_proc, new_proc = _StubRTCProcessor(False), _StubRTCProcessor(False)
|
||||
ref = _reference_pi0_loop(denoise_fn, noise, 6, rtc_enabled=True, rtc_processor=ref_proc, kw=kw)
|
||||
out = euler_integrate(
|
||||
denoise_fn,
|
||||
noise,
|
||||
6,
|
||||
rtc_processor=new_proc,
|
||||
rtc_enabled=True,
|
||||
inference_delay=3,
|
||||
prev_chunk_left_over=leftover,
|
||||
execution_horizon=25,
|
||||
)
|
||||
assert torch.equal(out, ref)
|
||||
assert len(new_proc.guidance_calls) == 6
|
||||
for ref_call, new_call in zip(ref_proc.guidance_calls, new_proc.guidance_calls, strict=True):
|
||||
assert ref_call["time"] == new_call["time"]
|
||||
assert new_call["inference_delay"] == 3 and new_call["execution_horizon"] == 25
|
||||
# Guidance sees the PRE-update x_t.
|
||||
assert torch.equal(ref_call["x_t"], new_call["x_t"])
|
||||
|
||||
|
||||
def test_euler_integrate_debug_tracking_fires_even_when_rtc_disabled():
|
||||
# Historical behavior: track() fires whenever the processor exists and has debugging
|
||||
# enabled, independent of whether RTC guidance is active.
|
||||
torch.manual_seed(5)
|
||||
denoise_fn = _make_denoise_fn()
|
||||
noise = torch.randn(2, 6, 4)
|
||||
proc = _StubRTCProcessor(True)
|
||||
out = euler_integrate(denoise_fn, noise, 4, rtc_processor=proc, rtc_enabled=False)
|
||||
assert len(proc.guidance_calls) == 0
|
||||
assert len(proc.tracked) == 4
|
||||
# track() receives the POST-update x_t; the last one is the returned sample.
|
||||
assert torch.equal(proc.tracked[-1]["x_t"], out)
|
||||
@@ -0,0 +1,289 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Behavior-pinning tests for the shared VLA helpers.
|
||||
|
||||
These helpers are the canonical versions of functions that used to be copy-pasted across
|
||||
the openpi-derived policies (pi0, pi05, pi0_fast, smolvla, eo1, xvla). The expected
|
||||
values below encode the historical per-policy behavior exactly; a failure here means a
|
||||
behavior change that would silently affect released checkpoints.
|
||||
"""
|
||||
|
||||
import math
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
import torch.nn.functional as F # noqa: N812
|
||||
|
||||
from lerobot.policies.common.vla_utils import (
|
||||
create_sinusoidal_pos_embedding,
|
||||
fuse_action_time_embedding,
|
||||
make_att_2d_masks,
|
||||
pad_vector,
|
||||
prepare_attention_masks_4d,
|
||||
resize_with_pad,
|
||||
resize_with_pad_torch,
|
||||
)
|
||||
from lerobot.utils.constants import OPENPI_ATTENTION_MASK_VALUE
|
||||
|
||||
|
||||
def test_create_sinusoidal_pos_embedding_matches_openpi_formula():
|
||||
time = torch.tensor([0.0, 0.25, 1.0])
|
||||
dim, min_period, max_period = 8, 4e-3, 4.0
|
||||
emb = create_sinusoidal_pos_embedding(time, dim, min_period, max_period, device=torch.device("cpu"))
|
||||
|
||||
assert emb.shape == (3, dim)
|
||||
# Independent recomputation of the openpi formula in float64.
|
||||
fraction = torch.linspace(0.0, 1.0, dim // 2, dtype=torch.float64)
|
||||
period = min_period * (max_period / min_period) ** fraction
|
||||
scaling = 1.0 / period * 2 * math.pi
|
||||
sin_input = scaling[None, :] * time.to(torch.float64)[:, None]
|
||||
expected = torch.cat([torch.sin(sin_input), torch.cos(sin_input)], dim=1)
|
||||
torch.testing.assert_close(emb, expected, rtol=1e-9, atol=1e-9)
|
||||
|
||||
|
||||
def test_create_sinusoidal_pos_embedding_validation():
|
||||
with pytest.raises(ValueError, match="divisible by 2"):
|
||||
create_sinusoidal_pos_embedding(torch.zeros(2), 7, 4e-3, 4.0, device=torch.device("cpu"))
|
||||
with pytest.raises(ValueError, match="batch_size"):
|
||||
create_sinusoidal_pos_embedding(torch.zeros(2, 2), 8, 4e-3, 4.0, device=torch.device("cpu"))
|
||||
|
||||
|
||||
def test_make_att_2d_masks_docstring_cases():
|
||||
# Pure causal attention: [[1 1 1]]
|
||||
pad = torch.ones(1, 3, dtype=torch.bool)
|
||||
att = torch.tensor([[1, 1, 1]], dtype=torch.int32)
|
||||
expected = torch.tensor([[[1, 0, 0], [1, 1, 0], [1, 1, 1]]], dtype=torch.bool)
|
||||
assert torch.equal(make_att_2d_masks(pad, att), expected)
|
||||
|
||||
# Prefix-LM: [[0 0 1 1]] -> first two tokens attend bidirectionally, rest causal.
|
||||
att = torch.tensor([[0, 0, 1, 1]], dtype=torch.int32)
|
||||
pad = torch.ones(1, 4, dtype=torch.bool)
|
||||
expected = torch.tensor([[[1, 1, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]]], dtype=torch.bool)
|
||||
assert torch.equal(make_att_2d_masks(pad, att), expected)
|
||||
|
||||
# Padding removes rows and columns.
|
||||
pad = torch.tensor([[True, True, False]])
|
||||
att = torch.tensor([[0, 1, 1]], dtype=torch.int32)
|
||||
out = make_att_2d_masks(pad, att)
|
||||
assert not out[0, :, 2].any() and not out[0, 2, :].any()
|
||||
|
||||
|
||||
def test_make_att_2d_masks_validation():
|
||||
with pytest.raises(ValueError):
|
||||
make_att_2d_masks(torch.ones(3, dtype=torch.bool), torch.ones(1, 3, dtype=torch.int32))
|
||||
with pytest.raises(ValueError):
|
||||
make_att_2d_masks(torch.ones(1, 3, dtype=torch.bool), torch.ones(3, dtype=torch.int32))
|
||||
|
||||
|
||||
def test_prepare_attention_masks_4d():
|
||||
masks = torch.tensor([[[True, False], [False, True]]])
|
||||
out = prepare_attention_masks_4d(masks)
|
||||
assert out.shape == (1, 1, 2, 2)
|
||||
expected = torch.tensor([[[[0.0, OPENPI_ATTENTION_MASK_VALUE], [OPENPI_ATTENTION_MASK_VALUE, 0.0]]]])
|
||||
assert torch.equal(out, expected)
|
||||
|
||||
out_bf16 = prepare_attention_masks_4d(masks, dtype=torch.bfloat16)
|
||||
assert out_bf16.dtype == torch.bfloat16
|
||||
assert torch.equal(out_bf16, expected.to(torch.bfloat16))
|
||||
|
||||
|
||||
def test_pad_vector_openpi_semantics():
|
||||
v = torch.arange(6.0).reshape(2, 3)
|
||||
padded = pad_vector(v, 5)
|
||||
assert padded.shape == (2, 5)
|
||||
assert torch.equal(padded[:, :3], v) and not padded[:, 3:].any()
|
||||
# Already large enough (>=): returned unchanged, same object.
|
||||
assert pad_vector(v, 3) is v
|
||||
assert pad_vector(v, 2) is v
|
||||
# 3D input.
|
||||
v3 = torch.ones(2, 4, 3)
|
||||
assert pad_vector(v3, 7).shape == (2, 4, 7)
|
||||
|
||||
|
||||
def test_pad_vector_truncate_semantics():
|
||||
v = torch.arange(6.0).reshape(2, 3)
|
||||
out = pad_vector(v, 2, truncate=True)
|
||||
assert out.shape == (2, 2) and torch.equal(out, v[:, :2])
|
||||
out = pad_vector(v, 5, truncate=True)
|
||||
assert out.shape == (2, 5) and torch.equal(out[:, :3], v) and not out[:, 3:].any()
|
||||
assert pad_vector(v, 0, truncate=True).shape == (2, 0)
|
||||
assert pad_vector(v, 3, truncate=True) is v
|
||||
|
||||
|
||||
@pytest.mark.parametrize("channels_last", [True, False])
|
||||
def test_resize_with_pad_torch_centered(channels_last):
|
||||
img = torch.rand(2, 3, 30, 60) if not channels_last else torch.rand(2, 30, 60, 3)
|
||||
out = resize_with_pad_torch(img, 64, 64)
|
||||
if channels_last:
|
||||
assert out.shape == (2, 64, 64, 3)
|
||||
# Aspect ratio preserved: 30x60 -> 32x64, padded 16 top and 16 bottom (centered).
|
||||
assert not out[:, :16].any() and not out[:, -16:].any()
|
||||
assert out[:, 16:48].abs().sum() > 0
|
||||
else:
|
||||
assert out.shape == (2, 3, 64, 64)
|
||||
assert not out[:, :, :16].any() and not out[:, :, -16:].any()
|
||||
|
||||
|
||||
def test_resize_with_pad_torch_uint8_roundtrip():
|
||||
img = (torch.rand(1, 3, 20, 20) * 255).to(torch.uint8)
|
||||
out = resize_with_pad_torch(img, 40, 40)
|
||||
assert out.dtype == torch.uint8 and out.shape == (1, 3, 40, 40)
|
||||
with pytest.raises(ValueError, match="Unsupported image dtype"):
|
||||
resize_with_pad_torch(torch.rand(1, 3, 8, 8, dtype=torch.float64), 16, 16)
|
||||
|
||||
|
||||
def test_resize_with_pad_top_left():
|
||||
img = torch.rand(2, 3, 30, 60)
|
||||
out = resize_with_pad(img, 64, 64, pad_value=-1.0)
|
||||
assert out.shape == (2, 3, 64, 64)
|
||||
# 30x60 -> 32x64; this variant pads on the TOP only (32 rows of pad_value).
|
||||
assert torch.equal(out[:, :, :32], torch.full((2, 3, 32, 64), -1.0))
|
||||
assert out[:, :, 32:].min() >= 0
|
||||
# No-op fast path returns the same object.
|
||||
assert resize_with_pad(img, 30, 60, pad_value=0.0) is img
|
||||
with pytest.raises(ValueError, match="expected"):
|
||||
resize_with_pad(torch.rand(3, 8, 8), 16, 16, pad_value=0.0)
|
||||
|
||||
|
||||
def test_clone_past_key_values():
|
||||
pytest.importorskip("transformers")
|
||||
from transformers import DynamicCache
|
||||
|
||||
from lerobot.policies.common.vla_utils import clone_past_key_values
|
||||
|
||||
cache = DynamicCache()
|
||||
keys, values = torch.rand(1, 2, 4, 8), torch.rand(1, 2, 4, 8)
|
||||
cache.update(keys, values, 0)
|
||||
cloned = clone_past_key_values(cache)
|
||||
(ck, cv, _), (ok, ov, _) = next(iter(cloned)), next(iter(cache))
|
||||
assert torch.equal(ck, ok) and torch.equal(cv, ov)
|
||||
# Deep copy: mutating the clone must not touch the original.
|
||||
ck.zero_()
|
||||
assert not torch.equal(ck, ok)
|
||||
|
||||
|
||||
def test_clone_past_key_values_is_fullgraph_compilable():
|
||||
pytest.importorskip("transformers")
|
||||
from transformers import DynamicCache
|
||||
|
||||
from lerobot.policies.common.vla_utils import clone_past_key_values
|
||||
|
||||
cache = DynamicCache()
|
||||
keys, values = torch.rand(1, 2, 4, 8), torch.rand(1, 2, 4, 8)
|
||||
cache.update(keys, values, 0)
|
||||
|
||||
compiled_clone = torch.compile(clone_past_key_values, backend="eager", fullgraph=True)
|
||||
cloned = compiled_clone(cache)
|
||||
|
||||
(cloned_keys, cloned_values, _), (original_keys, original_values, _) = (
|
||||
next(iter(cloned)),
|
||||
next(iter(cache)),
|
||||
)
|
||||
assert torch.equal(cloned_keys, original_keys)
|
||||
assert torch.equal(cloned_values, original_values)
|
||||
|
||||
|
||||
def _make_action_time_layers(action_dim=7, width=16, seed=0):
|
||||
torch.manual_seed(seed)
|
||||
return (
|
||||
torch.nn.Linear(action_dim, width),
|
||||
torch.nn.Linear(2 * width, width),
|
||||
torch.nn.Linear(width, width),
|
||||
)
|
||||
|
||||
|
||||
def test_fuse_action_time_embedding_matches_pi0_inline_loop():
|
||||
# Verbatim pi0 block: time_emb cast to timestep.dtype, no gradient checkpointing.
|
||||
in_proj, mlp_in, mlp_out = _make_action_time_layers()
|
||||
min_period, max_period = 4e-3, 4.0
|
||||
noisy_actions = torch.randn(3, 5, 7)
|
||||
timestep = torch.rand(3)
|
||||
|
||||
time_emb = create_sinusoidal_pos_embedding(
|
||||
timestep, in_proj.out_features, min_period=min_period, max_period=max_period, device=timestep.device
|
||||
)
|
||||
time_emb = time_emb.type(dtype=timestep.dtype)
|
||||
action_emb = in_proj(noisy_actions)
|
||||
time_emb = time_emb[:, None, :].expand_as(action_emb)
|
||||
action_time_emb = torch.cat([action_emb, time_emb], dim=2)
|
||||
ref = mlp_out(F.silu(mlp_in(action_time_emb)))
|
||||
|
||||
out = fuse_action_time_embedding(
|
||||
noisy_actions,
|
||||
timestep,
|
||||
action_in_proj=in_proj,
|
||||
action_time_mlp_in=mlp_in,
|
||||
action_time_mlp_out=mlp_out,
|
||||
min_period=min_period,
|
||||
max_period=max_period,
|
||||
time_emb_dtype=timestep.dtype,
|
||||
)
|
||||
assert torch.equal(out, ref)
|
||||
|
||||
|
||||
def test_fuse_action_time_embedding_matches_smolvla_inline_loop():
|
||||
# Verbatim smolvla block: time_emb cast to action_emb.dtype (default), no checkpointing.
|
||||
in_proj, mlp_in, mlp_out = _make_action_time_layers(seed=1)
|
||||
min_period, max_period = 4e-3, 4.0
|
||||
noisy_actions = torch.randn(2, 4, 7)
|
||||
timestep = torch.rand(2)
|
||||
|
||||
action_emb = in_proj(noisy_actions)
|
||||
dtype = action_emb.dtype
|
||||
time_emb = create_sinusoidal_pos_embedding(
|
||||
timestep, in_proj.out_features, min_period, max_period, device=action_emb.device
|
||||
)
|
||||
time_emb = time_emb.type(dtype=dtype)
|
||||
time_emb = time_emb[:, None, :].expand_as(action_emb)
|
||||
action_time_emb = torch.cat([action_emb, time_emb], dim=2)
|
||||
action_time_emb = mlp_in(action_time_emb)
|
||||
action_time_emb = F.silu(action_time_emb)
|
||||
ref = mlp_out(action_time_emb)
|
||||
|
||||
out = fuse_action_time_embedding(
|
||||
noisy_actions,
|
||||
timestep,
|
||||
action_in_proj=in_proj,
|
||||
action_time_mlp_in=mlp_in,
|
||||
action_time_mlp_out=mlp_out,
|
||||
min_period=min_period,
|
||||
max_period=max_period,
|
||||
)
|
||||
assert torch.equal(out, ref)
|
||||
|
||||
|
||||
def test_fuse_action_time_embedding_apply_hook_is_invoked():
|
||||
# The apply hook (gradient-checkpoint wrapper in pi0/eo1) must wrap both sub-steps.
|
||||
in_proj, mlp_in, mlp_out = _make_action_time_layers(seed=2)
|
||||
calls = []
|
||||
|
||||
def apply(fn, arg):
|
||||
calls.append(fn)
|
||||
return fn(arg)
|
||||
|
||||
out = fuse_action_time_embedding(
|
||||
torch.randn(2, 4, 7),
|
||||
torch.rand(2),
|
||||
action_in_proj=in_proj,
|
||||
action_time_mlp_in=mlp_in,
|
||||
action_time_mlp_out=mlp_out,
|
||||
min_period=4e-3,
|
||||
max_period=4.0,
|
||||
apply=apply,
|
||||
)
|
||||
assert len(calls) == 2
|
||||
assert out.shape == (2, 4, 16)
|
||||
@@ -25,13 +25,57 @@ pytest.importorskip("transformers")
|
||||
pytest.importorskip("torchdiffeq")
|
||||
|
||||
from lerobot.policies.factory import make_policy_config # noqa: E402
|
||||
from lerobot.policies.wall_x import WallXConfig # noqa: E402
|
||||
from lerobot.policies.wall_x import (
|
||||
WallXConfig, # noqa: E402
|
||||
)
|
||||
from lerobot.policies.wall_x.modeling_wall_x import WallXPolicy # noqa: E402
|
||||
from lerobot.policies.wall_x.processor_wall_x import make_wall_x_pre_post_processors # noqa: E402
|
||||
from lerobot.policies.wall_x.qwen_model import Qwen2_5_VLMoEModel, Qwen2_5_VLTextConfig # noqa: E402
|
||||
from lerobot.utils.random_utils import set_seed # noqa: E402
|
||||
from tests.utils import require_cuda, require_hf_token # noqa: E402
|
||||
|
||||
|
||||
def test_moe_model_captures_requested_hidden_states_and_attentions():
|
||||
hidden_size = 16
|
||||
expert_config = {
|
||||
"hidden_size": hidden_size,
|
||||
"intermediate_size": 32,
|
||||
"hidden_act": "silu",
|
||||
}
|
||||
config = Qwen2_5_VLTextConfig(
|
||||
vocab_size=32,
|
||||
hidden_size=hidden_size,
|
||||
intermediate_size=32,
|
||||
num_hidden_layers=2,
|
||||
num_attention_heads=4,
|
||||
num_key_value_heads=4,
|
||||
max_position_embeddings=32,
|
||||
layer_types=["full_attention", "full_attention"],
|
||||
rope_parameters={
|
||||
"rope_type": "default",
|
||||
"rope_theta": 1_000_000.0,
|
||||
"mrope_section": [1, 1, 0],
|
||||
},
|
||||
num_experts=2,
|
||||
experts=[expert_config, expert_config],
|
||||
dim_inputs=(hidden_size, hidden_size),
|
||||
mlp_moe=True,
|
||||
)
|
||||
config._attn_implementation = "eager"
|
||||
model = Qwen2_5_VLMoEModel(config)
|
||||
input_ids = torch.tensor([[1, 2, 3]])
|
||||
|
||||
output = model(
|
||||
input_ids=input_ids,
|
||||
moe_token_types=torch.zeros_like(input_ids),
|
||||
output_hidden_states=True,
|
||||
output_attentions=True,
|
||||
)
|
||||
|
||||
assert len(output.hidden_states) == config.num_hidden_layers + 1
|
||||
assert len(output.attentions) == config.num_hidden_layers
|
||||
|
||||
|
||||
@require_cuda
|
||||
@require_hf_token
|
||||
def test_policy_instantiation():
|
||||
|
||||
@@ -18,6 +18,8 @@ import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
from huggingface_hub.errors import RevisionNotFoundError
|
||||
|
||||
# ``lerobot.scripts.lerobot_annotate`` (and the ``_push_to_hub`` path it
|
||||
# exercises) imports ``lerobot.datasets``, which only ships under the
|
||||
@@ -26,11 +28,13 @@ pytest.importorskip("datasets", reason="datasets is required (install lerobot[da
|
||||
|
||||
|
||||
def test_push_to_hub_tags_uploaded_dataset_revision(tmp_path, monkeypatch):
|
||||
from lerobot.scripts.lerobot_annotate import _push_to_hub
|
||||
from lerobot.scripts import lerobot_annotate
|
||||
|
||||
root = tmp_path / "dataset"
|
||||
(root / "meta").mkdir(parents=True)
|
||||
(root / "meta" / "info.json").write_text(json.dumps({"codebase_version": "v3.0"}))
|
||||
(root / "meta" / "info.json").write_text(
|
||||
json.dumps({"codebase_version": "v3.0", "fps": 30, "features": {}})
|
||||
)
|
||||
|
||||
calls = {}
|
||||
|
||||
@@ -43,9 +47,6 @@ def test_push_to_hub_tags_uploaded_dataset_revision(tmp_path, monkeypatch):
|
||||
return SimpleNamespace(oid="abc123")
|
||||
|
||||
def delete_tag(self, repo_id, **kwargs):
|
||||
import requests
|
||||
from huggingface_hub.errors import RevisionNotFoundError
|
||||
|
||||
calls["delete_tag"] = {"repo_id": repo_id, **kwargs}
|
||||
# Simulate the common case: no stale tag to delete.
|
||||
raise RevisionNotFoundError("no such tag", response=requests.Response())
|
||||
@@ -53,7 +54,12 @@ def test_push_to_hub_tags_uploaded_dataset_revision(tmp_path, monkeypatch):
|
||||
def create_tag(self, **kwargs):
|
||||
calls["create_tag"] = kwargs
|
||||
|
||||
monkeypatch.setattr("huggingface_hub.HfApi", FakeHfApi)
|
||||
monkeypatch.setattr(lerobot_annotate, "HfApi", FakeHfApi)
|
||||
|
||||
def fake_card_push(self, **kwargs):
|
||||
calls["card_push"] = {"content": str(self), **kwargs}
|
||||
|
||||
monkeypatch.setattr("huggingface_hub.DatasetCard.push_to_hub", fake_card_push)
|
||||
|
||||
cfg = SimpleNamespace(
|
||||
repo_id="source/dataset",
|
||||
@@ -62,7 +68,7 @@ def test_push_to_hub_tags_uploaded_dataset_revision(tmp_path, monkeypatch):
|
||||
push_commit_message=None,
|
||||
)
|
||||
|
||||
_push_to_hub(root, cfg)
|
||||
lerobot_annotate._push_to_hub(root, cfg)
|
||||
|
||||
assert calls["create_repo"] == {
|
||||
"repo_id": "annotated/dataset",
|
||||
@@ -71,6 +77,13 @@ def test_push_to_hub_tags_uploaded_dataset_revision(tmp_path, monkeypatch):
|
||||
"exist_ok": True,
|
||||
}
|
||||
assert calls["upload_folder"]["repo_id"] == "annotated/dataset"
|
||||
# The source README must not be copied over: its links (e.g. the
|
||||
# visualize badge) point at the source dataset. A card regenerated for
|
||||
# the target repo is pushed instead.
|
||||
assert "README.md" in calls["upload_folder"]["ignore_patterns"]
|
||||
assert calls["card_push"]["repo_id"] == "annotated/dataset"
|
||||
assert "visualize_dataset?path=annotated/dataset" in calls["card_push"]["content"]
|
||||
assert "source/dataset" not in calls["card_push"]["content"]
|
||||
# A stale tag (e.g. from a previous annotation run) is deleted first so
|
||||
# the new tag always points at the upload we just made.
|
||||
assert calls["delete_tag"] == {
|
||||
|
||||
@@ -233,3 +233,37 @@ def test_metrics_tracker_reduce_across_ranks_invokes_reduce():
|
||||
# accumulate against the cluster view rather than the stale per-rank sum.
|
||||
meter = tracker.update_s
|
||||
assert meter.sum / meter.count == pytest.approx(meter.avg)
|
||||
|
||||
|
||||
def test_metrics_tracker_update_metrics_registers_and_averages():
|
||||
tracker = MetricsTracker(batch_size=32, num_frames=1000, num_episodes=50, metrics={})
|
||||
tracker.update_metrics({"latent_loss": 0.2, "action_loss": 0.4})
|
||||
tracker.update_metrics({"latent_loss": 0.4, "action_loss": 0.6})
|
||||
|
||||
# New keys are auto-registered as mean-reduced meters and averaged over the window.
|
||||
assert tracker.metrics["latent_loss"].reduction == "mean"
|
||||
assert tracker.metrics["latent_loss"].avg == pytest.approx(0.3)
|
||||
assert tracker.metrics["action_loss"].avg == pytest.approx(0.5)
|
||||
assert tracker.to_dict()["latent_loss"] == pytest.approx(0.3)
|
||||
|
||||
|
||||
def test_metrics_tracker_update_metrics_skips_non_numeric():
|
||||
tracker = MetricsTracker(batch_size=32, num_frames=1000, num_episodes=50, metrics={})
|
||||
tracker.update_metrics({"loss": 0.5, "head_mode": "sparse", "enabled": True})
|
||||
|
||||
# strings and bools ignored
|
||||
assert "loss" in tracker.metrics
|
||||
assert "head_mode" not in tracker.metrics
|
||||
assert "enabled" not in tracker.metrics
|
||||
|
||||
|
||||
def test_metrics_tracker_update_metrics_does_not_override_caller_meter():
|
||||
# A policy that echoes "loss" in its output dict must not overwrite the caller-owned,
|
||||
# already-aggregated loss meter.
|
||||
metrics = {"loss": AverageMeter("loss", ":.3f", reduction="mean")}
|
||||
tracker = MetricsTracker(batch_size=32, num_frames=1000, num_episodes=50, metrics=metrics)
|
||||
tracker.loss = 1.0 # caller-set optimized loss
|
||||
tracker.update_metrics({"loss": 99.0, "latent_loss": 0.2})
|
||||
|
||||
assert tracker.metrics["loss"].avg == pytest.approx(1.0) # snapshot ignored
|
||||
assert tracker.metrics["latent_loss"].avg == pytest.approx(0.2)
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
version = 1
|
||||
revision = 2
|
||||
revision = 3
|
||||
requires-python = ">=3.12"
|
||||
resolution-markers = [
|
||||
"(python_full_version >= '3.15' and platform_machine == 'AMD64' and sys_platform == 'linux') or (python_full_version >= '3.15' and platform_machine == 'x86_64' and sys_platform == 'linux')",
|
||||
@@ -2127,16 +2127,16 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "hydra-core"
|
||||
version = "1.3.3"
|
||||
version = "1.3.4"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "antlr4-python3-runtime", marker = "sys_platform == 'linux'" },
|
||||
{ name = "omegaconf", marker = "sys_platform == 'linux'" },
|
||||
{ name = "packaging", marker = "sys_platform == 'linux'" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/0d/0b/7c0d941311aadc6479ec01767edba9c8a07db1452685de3567ed3058d0c9/hydra_core-1.3.3.tar.gz", hash = "sha256:b7477ee21f08b62f71bf0126d44695c048dc7e9c0cc79e2d593b707cb1e44048", size = 3262532, upload-time = "2026-06-11T05:54:26.835Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/10/dd/220f0e91743136725352497e98540772a01fc7c3ab96ff16c3c74424e984/hydra_core-1.3.4.tar.gz", hash = "sha256:ad0f7b05a0242255a8984d5a4ed2f6847f7b783ed727368a2c0155ec52d6c34c", size = 3263348, upload-time = "2026-07-04T16:25:38.891Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e5/57/4e39f85347f77144d2ad12e87d5df8fb8f17023f9bd9e8c6e903a128382c/hydra_core-1.3.3-py3-none-any.whl", hash = "sha256:cf349fc393f486f250e5825592c3d0a50c0af3effd726cf8dd5b637a7cb464e3", size = 154706, upload-time = "2026-06-11T05:54:24.917Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ed/cd/a568610bafe991fdd3f628fb606316b3b2be52ded019284e895d9beb3a1e/hydra_core-1.3.4-py3-none-any.whl", hash = "sha256:e58683692904a09f1fdfffa1a9b86bfd94e215b59f1ee17e7cd7d92738090d33", size = 155478, upload-time = "2026-07-04T16:25:37.291Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -2823,7 +2823,7 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "lerobot"
|
||||
version = "0.5.2"
|
||||
version = "0.6.1"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "cmake" },
|
||||
@@ -6934,15 +6934,15 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "uvicorn"
|
||||
version = "0.49.0"
|
||||
version = "0.50.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "click" },
|
||||
{ name = "h11" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c4/1f/fa18009dea8469069cca78a4e877a008ab78f08b064bfc9ab891579077ff/uvicorn-0.49.0.tar.gz", hash = "sha256:ebf4271aa580d9de97f93192d4595176df6e91f9aae919ca73e4fc07df1e66a3", size = 91284, upload-time = "2026-06-03T22:01:30.448Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/2e/41/06cce5dbb9f77591512957710ac709e60b12e6216a2f2d0d607fd49706e8/uvicorn-0.50.0.tar.gz", hash = "sha256:0c92e1bc2259cb7faa4fcef774a5966588f2e88542744550b66799fba10b76f1", size = 93257, upload-time = "2026-07-04T05:03:26.33Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/88/fa/e1388bbcf24ef3274f45c0c1c7b501fd14971037c1b6ee23610553307497/uvicorn-0.49.0-py3-none-any.whl", hash = "sha256:ba3d14c3ee7e41c6c654c46c9eb489d33213cdd30aa1696eab1374337c13f68f", size = 71376, upload-time = "2026-06-03T22:01:29.037Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a0/3a/eb70620ca2bf8213603d5c731460687c49fee38b0072f0b4a637781f0a53/uvicorn-0.50.0-py3-none-any.whl", hash = "sha256:05f0eb19edf38208f79f43df8a63081b48df31b0cd1e5997be957a4dc97d1b19", size = 72716, upload-time = "2026-07-04T05:03:24.848Z" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
|
||||
@@ -1,182 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>3D Hand Joint Visualizer</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: Arial, sans-serif;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
overflow: hidden;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
height: 100vh;
|
||||
}
|
||||
|
||||
.controls {
|
||||
padding: 15px;
|
||||
background-color: #f5f5f5;
|
||||
z-index: 100;
|
||||
}
|
||||
|
||||
.status {
|
||||
padding: 10px;
|
||||
border-radius: 5px;
|
||||
margin: 10px 0;
|
||||
}
|
||||
|
||||
.connected {
|
||||
background-color: #d4edda;
|
||||
color: #155724;
|
||||
}
|
||||
|
||||
.disconnected {
|
||||
background-color: #f8d7da;
|
||||
color: #721c24;
|
||||
}
|
||||
|
||||
button {
|
||||
padding: 8px 16px;
|
||||
background-color: #4CAF50;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
cursor: pointer;
|
||||
margin-right: 10px;
|
||||
}
|
||||
|
||||
button:hover {
|
||||
background-color: #45a049;
|
||||
}
|
||||
|
||||
button:disabled {
|
||||
background-color: #cccccc;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.container {
|
||||
display: flex;
|
||||
flex: 1;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
#canvas-container {
|
||||
flex: 3;
|
||||
position: relative;
|
||||
}
|
||||
|
||||
#sidebar {
|
||||
flex: 1;
|
||||
padding: 15px;
|
||||
background-color: #f8f9fa;
|
||||
overflow-y: auto;
|
||||
max-width: 300px;
|
||||
border-left: 1px solid #ddd;
|
||||
}
|
||||
|
||||
.joint-info {
|
||||
margin-bottom: 10px;
|
||||
padding: 8px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.joint-name {
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.joint-value {
|
||||
font-family: monospace;
|
||||
}
|
||||
|
||||
.bar-container {
|
||||
width: 100%;
|
||||
background-color: #e0e0e0;
|
||||
height: 10px;
|
||||
border-radius: 5px;
|
||||
overflow: hidden;
|
||||
margin-top: 5px;
|
||||
}
|
||||
|
||||
.bar {
|
||||
height: 100%;
|
||||
background-color: #4CAF50;
|
||||
width: 0%;
|
||||
transition: width 0.2s ease-in-out;
|
||||
}
|
||||
|
||||
.log-container {
|
||||
margin-top: 20px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 5px;
|
||||
padding: 10px;
|
||||
height: 150px;
|
||||
overflow-y: auto;
|
||||
font-family: monospace;
|
||||
background-color: #f8f9fa;
|
||||
}
|
||||
|
||||
.view-controls {
|
||||
position: absolute;
|
||||
bottom: 10px;
|
||||
left: 10px;
|
||||
z-index: 10;
|
||||
}
|
||||
|
||||
.view-button {
|
||||
background-color: rgba(0, 0, 0, 0.5);
|
||||
color: white;
|
||||
border: none;
|
||||
padding: 5px 10px;
|
||||
margin-right: 5px;
|
||||
border-radius: 3px;
|
||||
cursor: pointer;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="controls">
|
||||
<button id="connectButton">Connect to Device</button>
|
||||
<button id="disconnectButton" disabled>Disconnect</button>
|
||||
<select id="baudRate">
|
||||
<option value="9600">9600</option>
|
||||
<option value="19200">19200</option>
|
||||
<option value="38400">38400</option>
|
||||
<option value="57600">57600</option>
|
||||
<option value="115200" selected>115200</option>
|
||||
</select>
|
||||
<span id="statusIndicator" class="status disconnected">Status: Disconnected</span>
|
||||
</div>
|
||||
|
||||
<div class="container">
|
||||
<div id="canvas-container">
|
||||
<!-- 3D canvas will be inserted here -->
|
||||
<div class="view-controls">
|
||||
<button class="view-button" id="frontView">Front</button>
|
||||
<button class="view-button" id="sideView">Side</button>
|
||||
<button class="view-button" id="topView">Top</button>
|
||||
<button class="view-button" id="resetView">Reset</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="sidebar">
|
||||
<h3>Joint Values</h3>
|
||||
<div id="jointsContainer">
|
||||
<!-- Joint info will be added here -->
|
||||
</div>
|
||||
|
||||
<div class="log-container" id="logContainer">
|
||||
<!-- Log messages will be added here -->
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Import Three.js -->
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.132.2/build/three.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.132.2/examples/js/controls/OrbitControls.js"></script>
|
||||
|
||||
<script src="script.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,669 +0,0 @@
|
||||
// === Hand Visualizer with Pre-Connect Sliders + Per-Joint Angle Limits ===
|
||||
// Assumes your HTML already has elements with the following IDs:
|
||||
// connectButton, disconnectButton, baudRate, statusIndicator, jointsContainer, logContainer,
|
||||
// canvas-container, frontView, sideView, topView, resetView
|
||||
// Requires Three.js + OrbitControls loaded on the page.
|
||||
|
||||
// -------------------- Config --------------------
|
||||
const MAX_JOINTS = 16;
|
||||
const RAW_MIN = 0, RAW_MAX = 4096;
|
||||
const RAW_CENTER = (RAW_MIN + RAW_MAX) / 2;
|
||||
const DEG = Math.PI / 180;
|
||||
const UI_DEG_MIN = -90, UI_DEG_MAX = 90; // UI sliders for angle limits
|
||||
|
||||
// -------------------- State --------------------
|
||||
let port;
|
||||
let reader;
|
||||
let keepReading = false;
|
||||
let isConnected = false;
|
||||
const decoder = new TextDecoder();
|
||||
let inputBuffer = '';
|
||||
|
||||
let jointValues = new Array(MAX_JOINTS).fill(RAW_CENTER);
|
||||
|
||||
// Auto-calibration: track observed min/max per joint
|
||||
let observedMin = new Array(MAX_JOINTS).fill(Infinity);
|
||||
let observedMax = new Array(MAX_JOINTS).fill(-Infinity);
|
||||
let calibrationEnabled = true;
|
||||
|
||||
// Three.js
|
||||
let scene, camera, renderer, controls;
|
||||
let hand = { palm: null, fingers: [] };
|
||||
|
||||
// DOM
|
||||
const connectButton = document.getElementById('connectButton');
|
||||
const disconnectButton = document.getElementById('disconnectButton');
|
||||
const baudRateSelect = document.getElementById('baudRate');
|
||||
const statusIndicator = document.getElementById('statusIndicator');
|
||||
const jointsContainer = document.getElementById('jointsContainer');
|
||||
const logContainer = document.getElementById('logContainer');
|
||||
const canvasContainer = document.getElementById('canvas-container');
|
||||
const frontViewBtn = document.getElementById('frontView');
|
||||
const sideViewBtn = document.getElementById('sideView');
|
||||
const topViewBtn = document.getElementById('topView');
|
||||
const resetViewBtn = document.getElementById('resetView');
|
||||
|
||||
// Helpers
|
||||
const clamp = (x, a, b) => Math.max(a, Math.min(b, x));
|
||||
const invLerp = (a, b, x) => clamp((x - a) / (b - a), 0, 1);
|
||||
|
||||
// -------------------- Joint Map with per-joint angle limits --------------------
|
||||
const fingerJointMap = [
|
||||
// Thumb (4)
|
||||
{ finger:0, joint:0, type:'CMC_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:0, joint:1, type:'CMC_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:0, joint:2, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true }, // +45° only
|
||||
{ finger:0, joint:3, type:'IP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true }, // +45° only
|
||||
|
||||
// Index (3)
|
||||
{ finger:1, joint:0, type:'MCP_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:1, joint:1, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false },
|
||||
{ finger:1, joint:2, type:'PIP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true }, // +45° only
|
||||
|
||||
// Middle (3)
|
||||
{ finger:2, joint:0, type:'MCP_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:2, joint:1, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:2, joint:2, type:'PIP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:true }, // +45° only
|
||||
|
||||
// Ring (3)
|
||||
{ finger:3, joint:0, type:'MCP_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:true },
|
||||
{ finger:3, joint:1, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false },
|
||||
{ finger:3, joint:2, type:'PIP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false }, // +45° only
|
||||
|
||||
// Pinky (3)
|
||||
{ finger:4, joint:0, type:'MCP_ABDUCTION', min:RAW_MIN, max:RAW_MAX, inverted:false },
|
||||
{ finger:4, joint:1, type:'MCP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false },
|
||||
{ finger:4, joint:2, type:'PIP_FLEXION', min:RAW_MIN, max:RAW_MAX, inverted:false } // +45° only
|
||||
];
|
||||
|
||||
// Assign angle limits (radians) per joint (default ±45°, exceptions: +45° only)
|
||||
for (const j of fingerJointMap) {
|
||||
const isThumb = j.finger === 0;
|
||||
const isPIP = j.type === 'PIP_FLEXION';
|
||||
let minA = -45 * DEG, maxA = +45 * DEG;
|
||||
if ((isThumb && (j.type === 'MCP_FLEXION' || j.type === 'IP_FLEXION')) || (!isThumb && isPIP)) {
|
||||
minA = 0;
|
||||
maxA = +45 * DEG;
|
||||
}
|
||||
j.angleMin = minA;
|
||||
j.angleMax = maxA;
|
||||
}
|
||||
|
||||
// -------------------- UI: Joint Panel --------------------
|
||||
const uiRefs = []; // per joint: { valueLabel, bar, barWrap, slider, invertChk, minDeg, maxDeg }
|
||||
|
||||
function initializeJointElements() {
|
||||
jointsContainer.innerHTML = '';
|
||||
uiRefs.length = 0;
|
||||
|
||||
for (let i = 0; i < MAX_JOINTS; i++) {
|
||||
const wrap = document.createElement('div');
|
||||
wrap.className = 'joint-info';
|
||||
|
||||
const fingerIndex = i < 4 ? 0 : Math.floor((i - 4) / 3) + 1;
|
||||
const jointInfo = fingerJointMap[i];
|
||||
const jointType = jointInfo?.type || 'Unknown';
|
||||
const fingerName = ['Thumb', 'Index', 'Middle', 'Ring', 'Pinky'][fingerIndex];
|
||||
|
||||
// Header
|
||||
const nameEl = document.createElement('div');
|
||||
nameEl.className = 'joint-name';
|
||||
nameEl.textContent = `${fingerName} – ${jointType}`;
|
||||
|
||||
// Value + bar
|
||||
const valueEl = document.createElement('div');
|
||||
valueEl.className = 'joint-value';
|
||||
valueEl.textContent = `Value: ${jointValues[i]}`;
|
||||
|
||||
const barWrap = document.createElement('div');
|
||||
barWrap.className = 'bar-container';
|
||||
const barEl = document.createElement('div');
|
||||
barEl.className = 'bar';
|
||||
barWrap.appendChild(barEl);
|
||||
|
||||
// Slider for pre-connect manual control
|
||||
const slider = document.createElement('input');
|
||||
slider.type = 'range';
|
||||
slider.min = String(RAW_MIN);
|
||||
slider.max = String(RAW_MAX);
|
||||
slider.value = String(jointValues[i]);
|
||||
slider.step = '1';
|
||||
slider.className = 'joint-slider';
|
||||
|
||||
slider.addEventListener('input', () => {
|
||||
if (isConnected) return; // ignore while connected
|
||||
let v = parseInt(slider.value, 10);
|
||||
if (jointInfo?.inverted) v = (jointInfo.min + jointInfo.max) - v;
|
||||
jointValues[i] = clamp(jointInfo ? v : 0, RAW_MIN, RAW_MAX);
|
||||
updateJointDisplay(i, jointValues[i]);
|
||||
updateHandModel();
|
||||
});
|
||||
|
||||
// Invert checkbox
|
||||
const invertLbl = document.createElement('label');
|
||||
invertLbl.className = 'invert-toggle';
|
||||
const invertChk = document.createElement('input');
|
||||
invertChk.type = 'checkbox';
|
||||
invertChk.checked = !!jointInfo?.inverted;
|
||||
invertChk.addEventListener('change', () => {
|
||||
if (jointInfo) jointInfo.inverted = invertChk.checked;
|
||||
addLogMessage(`${fingerName} ${jointType} inversion ${invertChk.checked ? 'enabled' : 'disabled'}`);
|
||||
});
|
||||
invertLbl.appendChild(invertChk);
|
||||
invertLbl.appendChild(document.createTextNode('Invert Values'));
|
||||
|
||||
// Angle limits (deg) controls
|
||||
const limitsRow = document.createElement('div');
|
||||
limitsRow.className = 'limits-row';
|
||||
|
||||
const minDeg = document.createElement('input');
|
||||
minDeg.type = 'number';
|
||||
minDeg.min = String(UI_DEG_MIN);
|
||||
minDeg.max = String(UI_DEG_MAX);
|
||||
minDeg.step = '1';
|
||||
minDeg.value = String(Math.round((jointInfo.angleMin || 0) / DEG));
|
||||
minDeg.className = 'limit-num';
|
||||
|
||||
const maxDeg = document.createElement('input');
|
||||
maxDeg.type = 'number';
|
||||
maxDeg.min = String(UI_DEG_MIN);
|
||||
maxDeg.max = String(UI_DEG_MAX);
|
||||
maxDeg.step = '1';
|
||||
maxDeg.value = String(Math.round((jointInfo.angleMax || 0) / DEG));
|
||||
maxDeg.className = 'limit-num';
|
||||
|
||||
const minLbl = document.createElement('span'); minLbl.textContent = 'min°';
|
||||
const maxLbl = document.createElement('span'); maxLbl.textContent = 'max°';
|
||||
minLbl.className = 'limit-label'; maxLbl.className = 'limit-label';
|
||||
|
||||
function syncLimits() {
|
||||
let mn = parseFloat(minDeg.value);
|
||||
let mx = parseFloat(maxDeg.value);
|
||||
if (isNaN(mn)) mn = -45;
|
||||
if (isNaN(mx)) mx = +45;
|
||||
if (mn > mx) [mn, mx] = [mx, mn];
|
||||
jointInfo.angleMin = clamp(mn, UI_DEG_MIN, UI_DEG_MAX) * DEG;
|
||||
jointInfo.angleMax = clamp(mx, UI_DEG_MIN, UI_DEG_MAX) * DEG;
|
||||
minDeg.value = String(Math.round(jointInfo.angleMin / DEG));
|
||||
maxDeg.value = String(Math.round(jointInfo.angleMax / DEG));
|
||||
updateHandModel();
|
||||
}
|
||||
minDeg.addEventListener('change', syncLimits);
|
||||
maxDeg.addEventListener('change', syncLimits);
|
||||
|
||||
limitsRow.appendChild(minLbl);
|
||||
limitsRow.appendChild(minDeg);
|
||||
limitsRow.appendChild(maxLbl);
|
||||
limitsRow.appendChild(maxDeg);
|
||||
|
||||
// Calibration controls
|
||||
const calibRow = document.createElement('div');
|
||||
calibRow.className = 'calib-row';
|
||||
|
||||
const resetCalibBtn = document.createElement('button');
|
||||
resetCalibBtn.textContent = 'Reset Calib';
|
||||
resetCalibBtn.className = 'calib-btn';
|
||||
resetCalibBtn.addEventListener('click', () => {
|
||||
observedMin[i] = Infinity;
|
||||
observedMax[i] = -Infinity;
|
||||
addLogMessage(`Reset calibration for ${fingerName} ${jointType}`);
|
||||
});
|
||||
|
||||
const calibStatus = document.createElement('span');
|
||||
calibStatus.className = 'calib-status';
|
||||
calibStatus.textContent = `Range: --`;
|
||||
|
||||
calibRow.appendChild(resetCalibBtn);
|
||||
calibRow.appendChild(calibStatus);
|
||||
|
||||
// Compose
|
||||
wrap.appendChild(nameEl);
|
||||
wrap.appendChild(valueEl);
|
||||
wrap.appendChild(barWrap);
|
||||
wrap.appendChild(slider);
|
||||
wrap.appendChild(invertLbl);
|
||||
wrap.appendChild(limitsRow);
|
||||
wrap.appendChild(calibRow);
|
||||
|
||||
jointsContainer.appendChild(wrap);
|
||||
|
||||
uiRefs[i] = { valueLabel: valueEl, bar: barEl, barWrap, slider, invertChk, minDeg, maxDeg, nameEl, calibStatus };
|
||||
}
|
||||
|
||||
setConnectedUI(false); // initial state: sliders active
|
||||
}
|
||||
|
||||
// Toggle UI between pre-connect SLIDERS vs post-connect BARS
|
||||
function setConnectedUI(connected) {
|
||||
isConnected = connected;
|
||||
for (let i = 0; i < uiRefs.length; i++) {
|
||||
const ui = uiRefs[i];
|
||||
if (!ui) continue;
|
||||
// Show bars when connected; sliders disabled/hidden
|
||||
ui.barWrap.style.display = connected ? '' : 'none';
|
||||
ui.slider.disabled = connected;
|
||||
ui.slider.style.display = connected ? 'none' : '';
|
||||
}
|
||||
|
||||
// Reset calibration when connecting
|
||||
if (connected) {
|
||||
observedMin.fill(Infinity);
|
||||
observedMax.fill(-Infinity);
|
||||
addLogMessage('Calibration reset - move joints through full range for best results');
|
||||
}
|
||||
}
|
||||
|
||||
// Update joint display (value text + bar color/width + slider position if needed)
|
||||
function updateJointDisplay(jointIndex, value) {
|
||||
const ui = uiRefs[jointIndex];
|
||||
const info = fingerJointMap[jointIndex];
|
||||
if (!ui || !info) return;
|
||||
|
||||
ui.valueLabel.textContent = `Value: ${value}`;
|
||||
|
||||
// bar
|
||||
const min = info.min, max = info.max;
|
||||
const pct = clamp((value - min) / (max - min), 0, 1) * 100;
|
||||
ui.bar.style.width = `${pct}%`;
|
||||
const hue = Math.floor(pct * 1.2); // 0..120
|
||||
ui.bar.style.backgroundColor = `hsl(${hue}, 80%, 50%)`;
|
||||
|
||||
// slider (only meaningful when not connected; keep in sync anyway)
|
||||
const rawForSlider = info.inverted ? (info.min + info.max) - value : value;
|
||||
if (!isConnected) ui.slider.value = String(clamp(Math.round(rawForSlider), RAW_MIN, RAW_MAX));
|
||||
}
|
||||
|
||||
// -------------------- Serial I/O --------------------
|
||||
async function readSerialData() {
|
||||
while (port?.readable && keepReading) {
|
||||
reader = port.readable.getReader();
|
||||
try {
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (value) processData(decoder.decode(value));
|
||||
}
|
||||
} catch (err) {
|
||||
console.error('Error reading:', err);
|
||||
addLogMessage(`Error: ${err.message}`);
|
||||
break;
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function processData(chunk) {
|
||||
inputBuffer += chunk;
|
||||
let idx;
|
||||
while ((idx = inputBuffer.indexOf('\n')) !== -1) {
|
||||
const line = inputBuffer.slice(0, idx).trim();
|
||||
inputBuffer = inputBuffer.slice(idx + 1);
|
||||
|
||||
const vals = line.split(/\s+/).map(v => parseInt(v, 10));
|
||||
if (vals.length === MAX_JOINTS && vals.every(v => Number.isFinite(v))) {
|
||||
for (let i = 0; i < MAX_JOINTS; i++) {
|
||||
const info = fingerJointMap[i];
|
||||
if (!info) continue;
|
||||
|
||||
let rawValue = vals[i];
|
||||
|
||||
// Update calibration tracking
|
||||
if (calibrationEnabled) {
|
||||
observedMin[i] = Math.min(observedMin[i], rawValue);
|
||||
observedMax[i] = Math.max(observedMax[i], rawValue);
|
||||
|
||||
// Update calibration display
|
||||
const ui = uiRefs[i];
|
||||
if (ui && ui.calibStatus) {
|
||||
if (observedMin[i] !== Infinity && observedMax[i] !== -Infinity) {
|
||||
ui.calibStatus.textContent = `Range: ${observedMin[i]}-${observedMax[i]}`;
|
||||
}
|
||||
}
|
||||
|
||||
// Remap observed range to target range
|
||||
if (observedMin[i] !== Infinity && observedMax[i] !== -Infinity && observedMax[i] > observedMin[i]) {
|
||||
const observedRange = observedMax[i] - observedMin[i];
|
||||
const targetRange = info.max - info.min;
|
||||
const normalizedValue = (rawValue - observedMin[i]) / observedRange;
|
||||
rawValue = info.min + (normalizedValue * targetRange);
|
||||
}
|
||||
}
|
||||
|
||||
let v = clamp(rawValue, info.min, info.max);
|
||||
if (info.inverted) v = (info.min + info.max) - v;
|
||||
jointValues[i] = v;
|
||||
updateJointDisplay(i, v);
|
||||
}
|
||||
updateHandModel();
|
||||
} else {
|
||||
addLogMessage(`Received: ${line}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function connectToDevice() {
|
||||
try {
|
||||
port = await navigator.serial.requestPort();
|
||||
const baudRate = parseInt(baudRateSelect.value, 10) || 115200;
|
||||
await port.open({ baudRate });
|
||||
|
||||
keepReading = true;
|
||||
setConnectedUI(true);
|
||||
|
||||
statusIndicator.textContent = 'Status: Connected';
|
||||
statusIndicator.className = 'status connected';
|
||||
connectButton.disabled = true;
|
||||
disconnectButton.disabled = false;
|
||||
baudRateSelect.disabled = true;
|
||||
|
||||
addLogMessage(`Connected at ${baudRate} baud`);
|
||||
readSerialData();
|
||||
} catch (e) {
|
||||
console.error('Connect error:', e);
|
||||
addLogMessage(`Connection error: ${e.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function disconnectFromDevice() {
|
||||
try {
|
||||
keepReading = false;
|
||||
if (reader) {
|
||||
try { reader.cancel(); } catch {}
|
||||
}
|
||||
if (port) {
|
||||
await port.close();
|
||||
port = null;
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('Disconnect error:', e);
|
||||
addLogMessage(`Disconnection error: ${e.message}`);
|
||||
} finally {
|
||||
setConnectedUI(false);
|
||||
statusIndicator.textContent = 'Status: Disconnected';
|
||||
statusIndicator.className = 'status disconnected';
|
||||
connectButton.disabled = false;
|
||||
disconnectButton.disabled = true;
|
||||
baudRateSelect.disabled = false;
|
||||
addLogMessage('Disconnected');
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------- Three.js Scene --------------------
|
||||
function initThreeJS() {
|
||||
scene = new THREE.Scene();
|
||||
scene.background = new THREE.Color(0xf0f0f0);
|
||||
|
||||
camera = new THREE.PerspectiveCamera(
|
||||
75,
|
||||
canvasContainer.clientWidth / canvasContainer.clientHeight,
|
||||
0.1, 1000
|
||||
);
|
||||
camera.position.set(0, 15, 15);
|
||||
camera.lookAt(0, 0, 0);
|
||||
|
||||
renderer = new THREE.WebGLRenderer({ antialias: true });
|
||||
renderer.setSize(canvasContainer.clientWidth, canvasContainer.clientHeight);
|
||||
renderer.setPixelRatio(window.devicePixelRatio);
|
||||
canvasContainer.appendChild(renderer.domElement);
|
||||
|
||||
controls = new THREE.OrbitControls(camera, renderer.domElement);
|
||||
controls.enableDamping = true;
|
||||
controls.dampingFactor = 0.25;
|
||||
|
||||
const ambientLight = new THREE.AmbientLight(0x404040);
|
||||
scene.add(ambientLight);
|
||||
const dir1 = new THREE.DirectionalLight(0xffffff, 0.5);
|
||||
dir1.position.set(1, 1, 1);
|
||||
scene.add(dir1);
|
||||
const dir2 = new THREE.DirectionalLight(0xffffff, 0.3);
|
||||
dir2.position.set(-1, 1, -1);
|
||||
scene.add(dir2);
|
||||
|
||||
const gridHelper = new THREE.GridHelper(20, 20);
|
||||
scene.add(gridHelper);
|
||||
|
||||
createHandModel();
|
||||
window.addEventListener('resize', onWindowResize);
|
||||
animate();
|
||||
}
|
||||
|
||||
function createHandModel() {
|
||||
const palmMaterial = new THREE.MeshPhongMaterial({ color: 0xf5c396 });
|
||||
const fingerMaterial = new THREE.MeshPhongMaterial({ color: 0xf5c396 });
|
||||
const jointMaterial = new THREE.MeshPhongMaterial({ color: 0xe3a977 });
|
||||
|
||||
const palmGeometry = new THREE.BoxGeometry(7, 1, 8);
|
||||
hand.palm = new THREE.Mesh(palmGeometry, palmMaterial);
|
||||
hand.palm.position.set(0, 0, 0);
|
||||
hand.palm.rotation.x = Math.PI / 2; // hand vertical, palm facing forward
|
||||
scene.add(hand.palm);
|
||||
|
||||
const fingerWidth = 1, fingerHeight = 0.8;
|
||||
const fingerSegmentLengths = [3, 2, 1.5];
|
||||
const thumbSegmentLengths = [2, 2, 1.5];
|
||||
|
||||
const fingerBasePositions = [
|
||||
[ 3, 0, -2], // Thumb
|
||||
[ 1.5,-0.5,-4], // Index
|
||||
[ 0, -0.5,-4], // Middle
|
||||
[-1.5,-0.5,-4], // Ring
|
||||
[-3, -0.5,-4], // Pinky
|
||||
];
|
||||
const fingerBaseRot = [
|
||||
{ x:0, y:-Math.PI/3, z: Math.PI/3 }, // Thumb
|
||||
{ x:0, y:-Math.PI/48, z: 0 },
|
||||
{ x:0, y: Math.PI/48, z: 0 },
|
||||
{ x:0, y: Math.PI/32, z: 0 },
|
||||
{ x:0, y: Math.PI/24, z: 0 }
|
||||
];
|
||||
|
||||
for (let fIdx = 0; fIdx < 5; fIdx++) {
|
||||
const finger = { name:['Thumb','Index','Middle','Ring','Pinky'][fIdx], segments:[], joints:[] };
|
||||
const isThumb = fIdx === 0;
|
||||
const segLens = isThumb ? thumbSegmentLengths : fingerSegmentLengths;
|
||||
|
||||
finger.group = new THREE.Group();
|
||||
finger.group.position.set(...fingerBasePositions[fIdx]);
|
||||
finger.group.rotation.x = fingerBaseRot[fIdx].x;
|
||||
finger.group.rotation.y = fingerBaseRot[fIdx].y;
|
||||
finger.group.rotation.z = fingerBaseRot[fIdx].z;
|
||||
finger.group.userData.baseRot = {
|
||||
x:finger.group.rotation.x,
|
||||
y:finger.group.rotation.y,
|
||||
z:finger.group.rotation.z
|
||||
};
|
||||
hand.palm.add(finger.group);
|
||||
|
||||
let parent = finger.group;
|
||||
for (let s = 0; s < segLens.length; s++) {
|
||||
const segGroup = new THREE.Group();
|
||||
|
||||
const jGeom = new THREE.SphereGeometry(fingerWidth * 0.6, 8, 8);
|
||||
const joint = new THREE.Mesh(jGeom, jointMaterial);
|
||||
segGroup.add(joint);
|
||||
|
||||
const segGeom = new THREE.BoxGeometry(fingerWidth, fingerHeight, segLens[s]);
|
||||
const seg = new THREE.Mesh(segGeom, fingerMaterial);
|
||||
seg.position.z = -segLens[s] / 2;
|
||||
segGroup.add(seg);
|
||||
|
||||
parent.add(segGroup);
|
||||
|
||||
finger.segments.push(segGroup);
|
||||
finger.joints.push(joint);
|
||||
|
||||
if (s < segLens.length - 1) {
|
||||
const connector = new THREE.Group();
|
||||
connector.position.z = -segLens[s];
|
||||
segGroup.add(connector);
|
||||
parent = connector;
|
||||
}
|
||||
}
|
||||
|
||||
hand.fingers.push(finger);
|
||||
}
|
||||
|
||||
addFingerLabels();
|
||||
addHandLabel();
|
||||
}
|
||||
|
||||
function addFingerLabels() {
|
||||
const names = ['Thumb','Index','Middle','Ring','Pinky'];
|
||||
for (let i = 0; i < hand.fingers.length; i++) {
|
||||
const finger = hand.fingers[i];
|
||||
const canvas = document.createElement('canvas');
|
||||
const ctx = canvas.getContext('2d');
|
||||
canvas.width = 128; canvas.height = 32;
|
||||
ctx.fillStyle = '#ffffff'; ctx.fillRect(0,0,canvas.width,canvas.height);
|
||||
ctx.font = 'bold 16px Arial';
|
||||
ctx.fillStyle = '#000000';
|
||||
ctx.textAlign = 'center'; ctx.textBaseline = 'middle';
|
||||
ctx.fillText(names[i], canvas.width/2, canvas.height/2);
|
||||
|
||||
const texture = new THREE.CanvasTexture(canvas);
|
||||
const geom = new THREE.PlaneGeometry(2, 0.5);
|
||||
const mat = new THREE.MeshBasicMaterial({ map:texture, transparent:true, side:THREE.DoubleSide });
|
||||
const label = new THREE.Mesh(geom, mat);
|
||||
label.position.set(0, -1.5, -2);
|
||||
label.rotation.x = Math.PI / 2;
|
||||
finger.group.add(label);
|
||||
}
|
||||
}
|
||||
|
||||
function addHandLabel() {
|
||||
const canvas = document.createElement('canvas');
|
||||
const ctx = canvas.getContext('2d');
|
||||
canvas.width = 256; canvas.height = 64;
|
||||
ctx.fillStyle = '#ffffff'; ctx.fillRect(0,0,canvas.width,canvas.height);
|
||||
ctx.font = 'bold 24px Arial';
|
||||
ctx.fillStyle = '#000000';
|
||||
ctx.textAlign = 'center'; ctx.textBaseline = 'middle';
|
||||
ctx.fillText('RIGHT HAND (VERTICAL)', canvas.width/2, canvas.height/2);
|
||||
|
||||
const texture = new THREE.CanvasTexture(canvas);
|
||||
const geom = new THREE.PlaneGeometry(7, 1.75);
|
||||
const mat = new THREE.MeshBasicMaterial({ map:texture, transparent:true, side:THREE.DoubleSide });
|
||||
const label = new THREE.Mesh(geom, mat);
|
||||
label.position.set(0, -2, 0);
|
||||
label.rotation.x = Math.PI / 2;
|
||||
scene.add(label);
|
||||
}
|
||||
|
||||
function updateHandModel() {
|
||||
for (let i = 0; i < MAX_JOINTS; i++) {
|
||||
const info = fingerJointMap[i];
|
||||
if (!info) continue;
|
||||
const { finger, joint, type, min, max, angleMin, angleMax } = info;
|
||||
const raw = jointValues[i];
|
||||
const f = hand.fingers[finger];
|
||||
if (!f) continue;
|
||||
|
||||
const center = (min + max) / 2;
|
||||
let angle = 0;
|
||||
|
||||
if (type.includes('ABDUCTION')) {
|
||||
// symmetric around neutral
|
||||
const k = clamp((raw - center) / ((max - min) / 2), -1, 1);
|
||||
angle = angleMin + (k + 1) * 0.5 * (angleMax - angleMin);
|
||||
|
||||
const base = f.group.userData.baseRot || {x:0,y:0,z:0};
|
||||
if (finger === 0 && joint === 0) {
|
||||
// Thumb: abduction about Z (toward/away from palm)
|
||||
f.group.rotation.z = base.z + angle;
|
||||
} else {
|
||||
// Other fingers: side-to-side about Y
|
||||
f.group.rotation.y = base.y + angle;
|
||||
}
|
||||
} else if (type.includes('FLEXION')) {
|
||||
const isThumb = finger === 0;
|
||||
const isMCP = type === 'MCP_FLEXION';
|
||||
const isPIP = type === 'PIP_FLEXION';
|
||||
const positiveOnly = (isThumb && (type === 'MCP_FLEXION' || type === 'IP_FLEXION')) || (!isThumb && isPIP);
|
||||
|
||||
if (positiveOnly) {
|
||||
const t = raw <= center ? 0 : invLerp(center, max, raw); // 0..1
|
||||
angle = angleMin + t * (angleMax - angleMin); // 0..+limit
|
||||
} else {
|
||||
const k = clamp((raw - center) / ((max - min) / 2), -1, 1);
|
||||
angle = angleMin + (k + 1) * 0.5 * (angleMax - angleMin);
|
||||
}
|
||||
|
||||
if (isMCP) {
|
||||
// MCP flexion applies to the finger base group (same as abduction)
|
||||
const base = f.group.userData.baseRot || {x:0,y:0,z:0};
|
||||
f.group.rotation.x = base.x + angle;
|
||||
} else if (f.segments[joint]) {
|
||||
// PIP/DIP/IP flexion applies to individual segments
|
||||
f.segments[joint].rotation.x = angle;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------- Render Loop --------------------
|
||||
function onWindowResize() {
|
||||
camera.aspect = canvasContainer.clientWidth / canvasContainer.clientHeight;
|
||||
camera.updateProjectionMatrix();
|
||||
renderer.setSize(canvasContainer.clientWidth, canvasContainer.clientHeight);
|
||||
}
|
||||
|
||||
function animate() {
|
||||
requestAnimationFrame(animate);
|
||||
controls.update();
|
||||
renderer.render(scene, camera);
|
||||
}
|
||||
|
||||
// -------------------- Misc UI --------------------
|
||||
function addLogMessage(msg) {
|
||||
const el = document.createElement('div');
|
||||
el.textContent = msg;
|
||||
logContainer.appendChild(el);
|
||||
logContainer.scrollTop = logContainer.scrollHeight;
|
||||
while (logContainer.children.length > 100) {
|
||||
logContainer.removeChild(logContainer.firstChild);
|
||||
}
|
||||
}
|
||||
|
||||
// Camera view controls
|
||||
frontViewBtn?.addEventListener('click', () => { camera.position.set(0, 0, 20); camera.lookAt(0,0,0); controls.update(); });
|
||||
sideViewBtn?.addEventListener('click', () => { camera.position.set(20, 0, 0); camera.lookAt(0,0,0); controls.update(); });
|
||||
topViewBtn?.addEventListener('click', () => { camera.position.set(0, 20, 0); camera.lookAt(0,0,0); controls.update(); });
|
||||
resetViewBtn?.addEventListener('click', () => { camera.position.set(10,10,10); camera.lookAt(0,0,0); controls.update(); });
|
||||
|
||||
// Serial connect buttons
|
||||
connectButton?.addEventListener('click', connectToDevice);
|
||||
disconnectButton?.addEventListener('click', disconnectFromDevice);
|
||||
|
||||
// Web Serial support check
|
||||
if (!navigator.serial) {
|
||||
statusIndicator.textContent = 'Status: Web Serial API not supported in this browser';
|
||||
connectButton.disabled = true;
|
||||
addLogMessage('ERROR: Web Serial API is not supported in this browser. Try Chrome or Edge.');
|
||||
}
|
||||
|
||||
// -------------------- Boot --------------------
|
||||
initThreeJS();
|
||||
initializeJointElements();
|
||||
|
||||
// -------------------- Styles (inline) --------------------
|
||||
const styleElement = document.createElement('style');
|
||||
styleElement.textContent = `
|
||||
.joint-info { border-bottom: 1px solid #eee; padding: 8px 0; }
|
||||
.joint-name { font-weight: 600; margin-bottom: 4px; }
|
||||
.joint-value { font-size: 12px; color: #333; margin-bottom: 4px; }
|
||||
.bar-container { width: 100%; height: 8px; background: #ddd; border-radius: 4px; overflow: hidden; }
|
||||
.bar { height: 100%; width: 0%; background: #4caf50; }
|
||||
.joint-slider { width: 100%; margin: 6px 0; }
|
||||
.invert-toggle { display: inline-flex; align-items: center; gap: 6px; margin-top: 4px; font-size: 12px; color: #555; }
|
||||
.limits-row { display: flex; align-items: center; gap: 6px; margin-top: 6px; flex-wrap: wrap; }
|
||||
.limit-label { font-size: 11px; color: #666; }
|
||||
.limit-num { width: 60px; }
|
||||
.calib-row { display: flex; align-items: center; gap: 8px; margin-top: 4px; }
|
||||
.calib-btn { padding: 2px 6px; font-size: 11px; background: #f44336; color: white; border: none; border-radius: 3px; cursor: pointer; }
|
||||
.calib-btn:hover { background: #d32f2f; }
|
||||
.calib-status { font-size: 11px; color: #666; }
|
||||
.status.connected { color: #0a0; }
|
||||
.status.disconnected { color: #a00; }
|
||||
`;
|
||||
document.head.appendChild(styleElement);
|
||||
Reference in New Issue
Block a user