feat(eval): thread-safe policy copies for max_parallel_tasks > 1

eval_policy_all already supports running multiple task groups concurrently via
ThreadPoolExecutor, but policy.reset() was not thread-safe: all threads shared
the same policy object and its mutable state (action queues, temporal buffers).

Fix: each thread receives a shallow copy of the policy. copy.copy() creates a
new Python object whose _parameters dict is a shared reference — same tensor
storage, zero extra VRAM — while reset() rebinds per-episode state to fresh
objects per thread.

Caveat: ACT with temporal_ensemble_coeff is not safe with this approach (its
reset() mutates a shared sub-object). Keep max_parallel_tasks=1 for that config.

For MetaWorld (50 tasks, no temporal ensembling), max_parallel_tasks=4 raises
GPU utilization from ~20% to ~60-80% with no additional VRAM cost.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.gitignore

@@ -173,5 +173,7 @@ outputs/
 
 # Dev folders
 .cache/*
+*.stl
+*.urdf
 *.xml
 *.part

docs/source/_toctree.yml

@@ -17,12 +17,12 @@
     title: Train RL in Simulation
   - local: multi_gpu_training
     title: Multi GPU training
+  - local: hil_data_collection
+    title: Human In the Loop Data Collection
   - local: peft_training
     title: Training with PEFT (e.g., LoRA)
   - local: rename_map
     title: Using Rename Map and Empty Cameras
-  - local: umi_pi0_relative_ee
-    title: UMI Data with pi0 Relative EE Actions
   title: "Tutorials"
 - sections:
   - local: lerobot-dataset-v3
@@ -71,13 +71,17 @@
     title: Environments from the Hub
   - local: envhub_leisaac
     title: Control & Train Robots in Sim (LeIsaac)
+  title: "Simulation"
+- sections:
+  - local: adding_benchmarks
+    title: Adding a New Benchmark
+  - local: libero
+    title: LIBERO
+  - local: metaworld
+    title: Meta-World
   - local: envhub_isaaclab_arena
     title: NVIDIA IsaacLab Arena Environments
-  - local: libero
-    title: Using Libero
-  - local: metaworld
-    title: Using MetaWorld
-  title: "Simulation"
+  title: "Benchmarks"
 - sections:
   - local: introduction_processors
     title: Introduction to Robot Processors

docs/source/action_representations.mdx

@@ -202,22 +202,11 @@ Here is how the different processors compose. Each arrow is a processor step, an
                     └─────────────────────────────────────────┘
 
                     ┌─────────────────────────────────────────┐
-   State Derivation │   Action column  ────→  State + Action  │
-                    │   DeriveStateFromActionStep (pre only)  │
-                    │   (UMI-style: state from action chunk)  │
-                    └─────────────────────────────────────────┘
-
-                    ┌─────────────────────────────────────────┐
-   Action Repr.     │   Absolute  ←────→  Relative            │
+   Representation   │   Absolute  ←────→  Relative            │
                     │   RelativeActionsProcessorStep (pre)    │
                     │   AbsoluteActionsProcessorStep (post)   │
                     └─────────────────────────────────────────┘
 
-                    ┌─────────────────────────────────────────┐
-   State Repr.      │   Absolute  ────→  Relative             │
-                    │   RelativeStateProcessorStep (pre only) │
-                    └─────────────────────────────────────────┘
-
                     ┌─────────────────────────────────────────┐
    Normalization    │   Raw  ←────→  Normalized               │
                     │   NormalizerProcessorStep (pre)         │
@@ -227,10 +216,6 @@ Here is how the different processors compose. Each arrow is a processor step, an
 
 A typical training preprocessor might chain: `raw absolute joint actions → relative → normalize`. A typical inference postprocessor: `unnormalize → absolute → (optionally IK to joints)`.
 
-With UMI-style relative proprioception (`use_relative_state=True`), the preprocessor also converts observation.state to offsets from the current timestep via `RelativeStateProcessorStep` before normalization. This is a pre-processing-only step (state is an input, not an output).
-
-With `derive_state_from_action=True`, the preprocessor first runs `DeriveStateFromActionStep` to extract a 2-step state from the extended action chunk. This enables full UMI-style training without a separate `observation.state` column. See the [UMI pi0 guide](umi_pi0_relative_ee) for details.
-
 ## References
 
 - [Universal Manipulation Interface (UMI)](https://arxiv.org/abs/2402.10329) - Chi et al., 2024. Defines the relative trajectory action representation and compares it with absolute and delta actions.

docs/source/adding_benchmarks.mdx

@@ -0,0 +1,322 @@
+# Adding a New Benchmark
+
+This guide walks you through adding a new simulation benchmark to LeRobot. Follow the steps in order and use the existing benchmarks as templates.
+
+A benchmark in LeRobot is a set of [Gymnasium](https://gymnasium.farama.org/) environments that wrap a third-party simulator (like LIBERO or Meta-World) behind a standard `gym.Env` interface. The `lerobot-eval` CLI then runs evaluation uniformly across all benchmarks.
+
+## Existing benchmarks at a glance
+
+Before diving in, here is what is already integrated:
+
+| Benchmark      | Env file            | Config class       | Tasks               | Action dim   | Processor                    |
+| -------------- | ------------------- | ------------------ | ------------------- | ------------ | ---------------------------- |
+| LIBERO         | `envs/libero.py`    | `LiberoEnv`        | 130 across 5 suites | 7            | `LiberoProcessorStep`        |
+| Meta-World     | `envs/metaworld.py` | `MetaworldEnv`     | 50 (MT50)           | 4            | None                         |
+| IsaacLab Arena | Hub-hosted          | `IsaaclabArenaEnv` | Configurable        | Configurable | `IsaaclabArenaProcessorStep` |
+
+Use `src/lerobot/envs/libero.py` and `src/lerobot/envs/metaworld.py` as reference implementations.
+
+## How it all fits together
+
+### Data flow
+
+During evaluation, data moves through four stages:
+
+```
+1. gym.Env  ──→  raw observations (numpy dicts)
+
+2. Preprocessing  ──→  standard LeRobot keys + task description
+   (preprocess_observation in envs/utils.py, env.call("task_description"))
+
+3. Processors  ──→  env-specific then policy-specific transforms
+   (env_preprocessor, policy_preprocessor)
+
+4. Policy  ──→  select_action()  ──→  action tensor
+   then reverse: policy_postprocessor → env_postprocessor → numpy action → env.step()
+```
+
+Most benchmarks only need to care about stage 1 (producing observations in the right format) and optionally stage 3 (if env-specific transforms are needed).
+
+### Environment structure
+
+`make_env()` returns a nested dict of vectorized environments:
+
+```python
+dict[str, dict[int, gym.vector.VectorEnv]]
+#    ^suite       ^task_id
+```
+
+A single-task env (e.g. PushT) looks like `{"pusht": {0: vec_env}}`.
+A multi-task benchmark (e.g. LIBERO) looks like `{"libero_spatial": {0: vec0, 1: vec1, ...}, ...}`.
+
+### How evaluation runs
+
+All benchmarks are evaluated the same way by `lerobot-eval`:
+
+1. `make_env()` builds the nested `{suite: {task_id: VectorEnv}}` dict.
+2. `eval_policy_all()` iterates over every suite and task.
+3. For each task, it runs `n_episodes` rollouts via `rollout()`.
+4. Results are aggregated hierarchically: episode, task, suite, overall.
+5. Metrics include `pc_success` (success rate), `avg_sum_reward`, and `avg_max_reward`.
+
+The critical piece: your env must return `info["is_success"]` on every `step()` call. This is how the eval loop knows whether a task was completed.
+
+## What your environment must provide
+
+LeRobot does not enforce a strict observation schema. Instead it relies on a set of conventions that all benchmarks follow.
+
+### Env attributes
+
+Your `gym.Env` must set these attributes:
+
+| Attribute            | Type  | Why                                                  |
+| -------------------- | ----- | ---------------------------------------------------- |
+| `_max_episode_steps` | `int` | `rollout()` uses this to cap episode length          |
+| `task_description`   | `str` | Passed to VLA policies as a language instruction     |
+| `task`               | `str` | Fallback identifier if `task_description` is not set |
+
+### Success reporting
+
+Your `step()` and `reset()` must include `"is_success"` in the `info` dict:
+
+```python
+info = {"is_success": True}   # or False
+return observation, reward, terminated, truncated, info
+```
+
+### Observations
+
+The simplest approach is to map your simulator's outputs to the standard keys that `preprocess_observation()` already understands. Do this inside your `gym.Env` (e.g. in a `_format_raw_obs()` helper):
+
+| Your env should output    | LeRobot maps it to         | What it is                            |
+| ------------------------- | -------------------------- | ------------------------------------- |
+| `"pixels"` (single array) | `observation.image`        | Single camera image, HWC uint8        |
+| `"pixels"` (dict)         | `observation.images.<cam>` | Multiple cameras, each HWC uint8      |
+| `"agent_pos"`             | `observation.state`        | Proprioceptive state vector           |
+| `"environment_state"`     | `observation.env_state`    | Full environment state (e.g. PushT)   |
+| `"robot_state"`           | `observation.robot_state`  | Nested robot state dict (e.g. LIBERO) |
+
+If your simulator uses different key names, you have two options:
+
+1. **Recommended:** Rename them to the standard keys inside your `gym.Env` wrapper.
+2. **Alternative:** Write an env processor to transform observations after `preprocess_observation()` runs (see step 4 below).
+
+### Actions
+
+Actions are continuous numpy arrays in a `gym.spaces.Box`. The dimensionality depends on your benchmark (7 for LIBERO, 4 for Meta-World, etc.). Policies adapt to different action dimensions through their `input_features` / `output_features` config.
+
+### Feature declaration
+
+Each `EnvConfig` subclass declares two dicts that tell the policy what to expect:
+
+- `features` — maps feature names to `PolicyFeature(type, shape)` (e.g. action dim, image shape).
+- `features_map` — maps raw observation keys to LeRobot convention keys (e.g. `"agent_pos"` to `"observation.state"`).
+
+## Step by step
+
+<Tip>
+  At minimum, you need two files: a **gym.Env wrapper** and an **EnvConfig
+  subclass** with a `create_envs()` override. Everything else is optional or
+  documentation. No changes to `factory.py` are needed.
+</Tip>
+
+### Checklist
+
+| File                                     | Required | Why                                                          |
+| ---------------------------------------- | -------- | ------------------------------------------------------------ |
+| `src/lerobot/envs/<benchmark>.py`        | Yes      | Wraps the simulator as a standard gym.Env                    |
+| `src/lerobot/envs/configs.py`            | Yes      | Registers your benchmark and its `create_envs()` for the CLI |
+| `src/lerobot/processor/env_processor.py` | Optional | Custom observation/action transforms                         |
+| `src/lerobot/envs/utils.py`              | Optional | Only if you need new raw observation keys                    |
+| `pyproject.toml`                         | Yes      | Declares benchmark-specific dependencies                     |
+| `docs/source/<benchmark>.mdx`            | Yes      | User-facing documentation page                               |
+| `docs/source/_toctree.yml`               | Yes      | Adds your page to the docs sidebar                           |
+
+### 1. The gym.Env wrapper (`src/lerobot/envs/<benchmark>.py`)
+
+Create a `gym.Env` subclass that wraps the third-party simulator:
+
+```python
+class MyBenchmarkEnv(gym.Env):
+    metadata = {"render_modes": ["rgb_array"], "render_fps": <fps>}
+
+    def __init__(self, task_suite, task_id, ...):
+        super().__init__()
+        self.task = <task_name_string>
+        self.task_description = <natural_language_instruction>
+        self._max_episode_steps = <max_steps>
+        self.observation_space = spaces.Dict({...})
+        self.action_space = spaces.Box(low=..., high=..., shape=(...,), dtype=np.float32)
+
+    def reset(self, seed=None, **kwargs):
+        ...  # return (observation, info) — info must contain {"is_success": False}
+
+    def step(self, action: np.ndarray):
+        ...  # return (obs, reward, terminated, truncated, info) — info must contain {"is_success": <bool>}
+
+    def render(self):
+        ...  # return RGB image as numpy array
+
+    def close(self):
+        ...
+```
+
+**GPU-based simulators (e.g. MuJoCo with EGL rendering):** If your simulator allocates GPU/EGL contexts during `__init__`, defer that allocation to a `_ensure_env()` helper called on first `reset()`/`step()`. This avoids inheriting stale GPU handles when `AsyncVectorEnv` spawns worker processes. See `LiberoEnv._ensure_env()` for the pattern.
+
+Also provide a factory function that returns the nested dict structure:
+
+```python
+def create_mybenchmark_envs(
+    task: str,
+    n_envs: int,
+    gym_kwargs: dict | None = None,
+    env_cls: type | None = None,
+) -> dict[str, dict[int, Any]]:
+    """Create {suite_name: {task_id: VectorEnv}} for MyBenchmark."""
+    ...
+```
+
+See `create_libero_envs()` (multi-suite, multi-task) and `create_metaworld_envs()` (difficulty-grouped tasks) for reference.
+
+### 2. The config (`src/lerobot/envs/configs.py`)
+
+Register a config dataclass so users can select your benchmark with `--env.type=<name>`. Each config owns its environment creation and processor logic via two methods:
+
+- **`create_envs(n_envs, use_async_envs)`** — Returns `{suite: {task_id: VectorEnv}}`. The base class default uses `gym.make()` for single-task envs. Multi-task benchmarks override this.
+- **`get_env_processors()`** — Returns `(preprocessor, postprocessor)`. The base class default returns identity (no-op) pipelines. Override if your benchmark needs observation/action transforms.
+
+```python
+@EnvConfig.register_subclass("<benchmark_name>")
+@dataclass
+class MyBenchmarkEnvConfig(EnvConfig):
+    task: str = "<default_task>"
+    fps: int = <fps>
+    obs_type: str = "pixels_agent_pos"
+
+    features: dict[str, PolicyFeature] = field(default_factory=lambda: {
+        ACTION: PolicyFeature(type=FeatureType.ACTION, shape=(<action_dim>,)),
+    })
+    features_map: dict[str, str] = field(default_factory=lambda: {
+        ACTION: ACTION,
+        "agent_pos": OBS_STATE,
+        "pixels": OBS_IMAGE,
+    })
+
+    def __post_init__(self):
+        ...  # populate features based on obs_type
+
+    @property
+    def gym_kwargs(self) -> dict:
+        return {"obs_type": self.obs_type, "render_mode": self.render_mode}
+
+    def create_envs(self, n_envs: int, use_async_envs: bool = True):
+        """Override for multi-task benchmarks or custom env creation."""
+        from lerobot.envs.<benchmark> import create_<benchmark>_envs
+        return create_<benchmark>_envs(task=self.task, n_envs=n_envs, ...)
+
+    def get_env_processors(self):
+        """Override if your benchmark needs observation/action transforms."""
+        from lerobot.processor.pipeline import PolicyProcessorPipeline
+        from lerobot.processor.env_processor import MyBenchmarkProcessorStep
+        return (
+            PolicyProcessorPipeline(steps=[MyBenchmarkProcessorStep()]),
+            PolicyProcessorPipeline(steps=[]),
+        )
+```
+
+Key points:
+
+- The `register_subclass` name is what users pass on the CLI (`--env.type=<name>`).
+- `features` tells the policy what the environment produces.
+- `features_map` maps raw observation keys to LeRobot convention keys.
+- **No changes to `factory.py` needed** — the factory delegates to `cfg.create_envs()` and `cfg.get_env_processors()` automatically.
+
+### 3. Env processor (optional — `src/lerobot/processor/env_processor.py`)
+
+Only needed if your benchmark requires observation transforms beyond what `preprocess_observation()` handles (e.g. image flipping, coordinate conversion). Define the processor step here and return it from `get_env_processors()` in your config (see step 2):
+
+```python
+@dataclass
+@ProcessorStepRegistry.register(name="<benchmark>_processor")
+class MyBenchmarkProcessorStep(ObservationProcessorStep):
+    def _process_observation(self, observation):
+        processed = observation.copy()
+        # your transforms here
+        return processed
+
+    def transform_features(self, features):
+        return features  # update if shapes change
+
+    def observation(self, observation):
+        return self._process_observation(observation)
+```
+
+See `LiberoProcessorStep` for a full example (image rotation, quaternion-to-axis-angle conversion).
+
+### 4. Dependencies (`pyproject.toml`)
+
+Add a new optional-dependency group:
+
+```toml
+mybenchmark = ["my-benchmark-pkg==1.2.3", "lerobot[scipy-dep]"]
+```
+
+Pinning rules:
+
+- **Always pin** benchmark packages to exact versions for reproducibility (e.g. `metaworld==3.0.0`).
+- **Add platform markers** when needed (e.g. `; sys_platform == 'linux'`).
+- **Pin fragile transitive deps** if known (e.g. `gymnasium==1.1.0` for Meta-World).
+- **Document constraints** in your benchmark doc page.
+
+Users install with:
+
+```bash
+pip install -e ".[mybenchmark]"
+```
+
+### 5. Documentation (`docs/source/<benchmark>.mdx`)
+
+Write a user-facing page following the template in the next section. See `docs/source/libero.mdx` and `docs/source/metaworld.mdx` for full examples.
+
+### 6. Table of contents (`docs/source/_toctree.yml`)
+
+Add your benchmark to the "Benchmarks" section:
+
+```yaml
+- sections:
+    - local: libero
+      title: LIBERO
+    - local: metaworld
+      title: Meta-World
+    - local: envhub_isaaclab_arena
+      title: NVIDIA IsaacLab Arena Environments
+    - local: <your_benchmark>
+      title: <Your Benchmark Name>
+  title: "Benchmarks"
+```
+
+## Verifying your integration
+
+After completing the steps above, confirm that everything works:
+
+1. **Install** — `pip install -e ".[mybenchmark]"` and verify the dependency group installs cleanly.
+2. **Smoke test env creation** — call `make_env()` with your config in Python, check that the returned dict has the expected `{suite: {task_id: VectorEnv}}` shape, and that `reset()` returns observations with the right keys.
+3. **Run a full eval** — `lerobot-eval --env.type=<name> --env.task=<task> --eval.n_episodes=1 --eval.batch_size=1 --policy.path=<any_compatible_policy>` to exercise the full pipeline end-to-end.
+4. **Check success detection** — verify that `info["is_success"]` flips to `True` when the task is actually completed. This is what the eval loop uses to compute success rates.
+
+## Writing a benchmark doc page
+
+Each benchmark `.mdx` page should include:
+
+- **Title and description** — 1-2 paragraphs on what the benchmark tests and why it matters.
+- **Links** — paper, GitHub repo, project website (if available).
+- **Overview image or GIF.**
+- **Available tasks** — table of task suites with counts and brief descriptions.
+- **Installation** — `pip install -e ".[<benchmark>]"` plus any extra steps (env vars, system packages).
+- **Evaluation** — recommended `lerobot-eval` command with `n_episodes` and `batch_size` for reproducible results. Include single-task and multi-task examples if applicable.
+- **Policy inputs and outputs** — observation keys with shapes, action space description.
+- **Recommended evaluation episodes** — how many episodes per task is standard.
+- **Training** — example `lerobot-train` command.
+- **Reproducing published results** — link to pretrained model, eval command, results table (if available).
+
+See `docs/source/libero.mdx` and `docs/source/metaworld.mdx` for complete examples.

docs/source/env_processor.mdx

@@ -151,7 +151,7 @@ observation = {
 
 ### Factory Function
 
-The `make_env_pre_post_processors` function follows the same pattern as `make_pre_post_processors` for policies:
+The `make_env_pre_post_processors` function delegates to `env_cfg.get_env_processors()`:
 
 ```python
 from lerobot.envs.factory import make_env_pre_post_processors
@@ -159,47 +159,31 @@ from lerobot.envs.configs import LiberoEnv, PushtEnv
 
 # For LIBERO: Returns LiberoProcessorStep in preprocessor
 libero_cfg = LiberoEnv(task="libero_spatial", camera_name=["agentview"])
-env_preprocessor, env_postprocessor = make_env_pre_post_processors(libero_cfg)
+env_preprocessor, env_postprocessor = make_env_pre_post_processors(libero_cfg, policy_cfg)
 
 # For other environments: Returns identity processors (no-op)
 pusht_cfg = PushtEnv()
-env_preprocessor, env_postprocessor = make_env_pre_post_processors(pusht_cfg)
+env_preprocessor, env_postprocessor = make_env_pre_post_processors(pusht_cfg, policy_cfg)
 ```
 
-### Implementation in `envs/factory.py`
+### How It Works
+
+Each `EnvConfig` subclass can override `get_env_processors()` to return benchmark-specific
+processor pipelines. The base class returns identity (no-op) processors by default.
 
 ```python
-def make_env_pre_post_processors(
-    env_cfg: EnvConfig,
-) -> tuple[
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-]:
-    """
-    Create preprocessor and postprocessor pipelines for environment observations.
-
-    Args:
-        env_cfg: The configuration of the environment.
-
-    Returns:
-        A tuple containing:
-            - preprocessor: Pipeline that processes environment observations
-            - postprocessor: Pipeline that processes environment outputs
-    """
-    # For LIBERO environments, add the LiberoProcessorStep to preprocessor
-    if isinstance(env_cfg, LiberoEnv) or "libero" in env_cfg.type:
-        preprocessor = PolicyProcessorPipeline(steps=[LiberoProcessorStep()])
-    else:
-        # For all other environments, return an identity preprocessor
-        preprocessor = PolicyProcessorPipeline(steps=[])
-
-    # Postprocessor is currently identity for all environments
-    # Future: Could add environment-specific action transformations
-    postprocessor = PolicyProcessorPipeline(steps=[])
-
-    return preprocessor, postprocessor
+# In your EnvConfig subclass:
+def get_env_processors(self):
+    from lerobot.processor.pipeline import PolicyProcessorPipeline
+    return (
+        PolicyProcessorPipeline(steps=[MyProcessorStep()]),
+        PolicyProcessorPipeline(steps=[]),
+    )
 ```
 
+The factory function `make_env_pre_post_processors` simply delegates to this method,
+with a special case for `XVLAConfig` policies which override the env processors entirely.
+
 ### Integration in Evaluation
 
 In `lerobot_eval.py`, the environment processors are created once and used throughout:

docs/source/hil_data_collection.mdx

@@ -0,0 +1,269 @@
+# Human-In-the-Loop Data Collection
+
+Human-In-the-Loop (HIL) data collection lets you improve a trained policy by deploying it on a real robot while a human operator monitors and intervenes when needed. The intervention data (recovery movements and corrections) is recorded alongside autonomous segments, producing a richer training dataset that teaches the policy how to handle failures.
+
+---
+
+## Why Human-In-the-Loop?
+
+Standard behavioral cloning trains policies on successful demonstrations only. During deployment, small errors can compound and push the robot into states never seen during training (distribution shift). HIL data collection addresses this by:
+
+- Running the trained policy on the real robot
+- Having a human intervene when the robot is about to fail
+- Recording the human's recovery and correction as training data
+- Fine-tuning the policy on the combined dataset
+
+This produces a policy that not only knows how to perform the task, but also how to recover when things go wrong.
+
+---
+
+## How It Works
+
+During a HIL session, the human operator follows this loop within each episode:
+
+1. **Watch** the policy run autonomously
+2. **Pause** when failure is imminent, the robot holds its position
+3. **Take control** and teleoperate the robot back to a good state (recovery), then correct the behavior
+4. **Return control to the policy**, the policy resumes autonomous execution
+5. Repeat steps 2–4 as many times as needed during the episode
+6. **End the episode** when the task is complete, save and move on to the next rollout
+
+Both autonomous and human-controlled segments are recorded. The policy and human can alternate control multiple times within a single episode, and the episode continues from the current state after each handoff (no reset required just because intervention happened). This captures autonomous execution, recovery, and correction in one continuous trajectory. After collection, the combined dataset (original demonstrations + HIL data) is used to fine-tune the policy.
+
+This process can be repeated iteratively: deploy, collect, fine-tune, repeat. Each round targets the current policy's failure modes.
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│  Policy v0 (trained on demos)                                           │
+│       ↓                                                                 │
+│  HIL Collection (target current failure modes) → Fine-tune → Policy v1  │
+│       ↓                                                                 │
+│  HIL Collection (target new failure modes) → Fine-tune → Policy v2      │
+│       ↓                                                                 │
+│  ... (repeat until satisfactory performance)                            │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Hardware Requirements
+
+### Teleoperator Requirements
+
+The `examples/hil` HIL scripts require **teleoperators with active motors** that can:
+
+- Enable/disable torque programmatically
+- Move to target positions (to mirror the robot state when pausing)
+
+**Compatible teleoperators in the current `examples/hil` scripts:**
+
+- `openarm_mini` - OpenArm Mini
+- `so_leader` - SO100 / SO101 leader arm
+
+> [!IMPORTANT]
+> The provided `examples/hil` commands default to `bi_openarm_follower` + `openarm_mini`.
+> `so_follower` + `so_leader` configs are also registered and can be used via CLI flags.
+
+---
+
+## Script
+
+A single script handles both synchronous and RTC-based inference. Toggle RTC with `--rtc.enabled=true`:
+
+| Mode                     | Flag                 | Models                |
+| ------------------------ | -------------------- | --------------------- |
+| Standard (default)       | _(no flag needed)_   | ACT, Diffusion Policy |
+| Real-Time Chunking (RTC) | `--rtc.enabled=true` | Pi0, Pi0.5, SmolVLA   |
+
+---
+
+## Step-by-Step Guide
+
+### Step 1: Pre-train a Base Policy
+
+First, train a policy on your demonstration dataset:
+
+```bash
+python src/lerobot/scripts/lerobot_train.py \
+    --dataset.repo_id=your-username/demo-dataset \
+    --policy.type=pi0 \
+    --output_dir=outputs/pretrain \
+    --batch_size=32 \
+    --steps=50000
+```
+
+### Step 2: Collect HIL Data
+
+**Standard inference (ACT, Diffusion Policy):**
+
+```bash
+python examples/hil/hil_data_collection.py \
+    --robot.type=bi_openarm_follower \
+    --robot.left_arm_config.port=can1 \
+    --robot.left_arm_config.side=left \
+    --robot.right_arm_config.port=can0 \
+    --robot.right_arm_config.side=right \
+    --robot.cameras='{left_wrist: {type: opencv, index_or_path: "/dev/video0", width: 1280, height: 720, fps: 30}, right_wrist: {type: opencv, index_or_path: "/dev/video4", width: 1280, height: 720, fps: 30}, base: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}}' \
+    --teleop.type=openarm_mini \
+    --teleop.port_left=/dev/ttyACM0 \
+    --teleop.port_right=/dev/ttyACM1 \
+    --policy.path=outputs/pretrain/checkpoints/last/pretrained_model \
+    --dataset.repo_id=your-username/hil-dataset \
+    --dataset.single_task="Fold the T-shirt properly" \
+    --dataset.fps=30 \
+    --dataset.episode_time_s=1000 \
+    --dataset.num_episodes=50 \
+    --interpolation_multiplier=2
+```
+
+**With RTC for large models (Pi0, Pi0.5, SmolVLA):**
+
+For models with high inference latency, enable RTC for smooth execution:
+
+```bash
+python examples/hil/hil_data_collection.py \
+    --rtc.enabled=true \
+    --rtc.execution_horizon=20 \
+    --rtc.max_guidance_weight=5.0 \
+    --rtc.prefix_attention_schedule=LINEAR \
+    --robot.type=bi_openarm_follower \
+    --robot.left_arm_config.port=can1 \
+    --robot.left_arm_config.side=left \
+    --robot.right_arm_config.port=can0 \
+    --robot.right_arm_config.side=right \
+    --robot.cameras='{left_wrist: {type: opencv, index_or_path: "/dev/video0", width: 1280, height: 720, fps: 30}, right_wrist: {type: opencv, index_or_path: "/dev/video4", width: 1280, height: 720, fps: 30}, base: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}}' \
+    --teleop.type=openarm_mini \
+    --teleop.port_left=/dev/ttyACM0 \
+    --teleop.port_right=/dev/ttyACM1 \
+    --policy.path=outputs/pretrain/checkpoints/last/pretrained_model \
+    --dataset.repo_id=your-username/hil-rtc-dataset \
+    --dataset.single_task="Fold the T-shirt properly" \
+    --dataset.fps=30 \
+    --dataset.episode_time_s=1000 \
+    --dataset.num_episodes=50 \
+    --interpolation_multiplier=3
+```
+
+**Controls (Conceptual):**
+
+The interaction model is:
+
+- **Pause input**: pause autonomous policy execution
+- **Takeover input**: transfer control to the human operator and record intervention data
+- **Return-to-policy input**: hand control back to the policy and continue the same episode
+- **Episode control inputs**: save/re-record/stop/reset as needed
+
+Exact key/pedal bindings can differ across scripts and hardware integrations. Use each script's printed controls as the source of truth for the concrete mapping on your setup.
+
+**The HIL Protocol:**
+
+1. Watch the policy run autonomously (teleop is idle/free)
+2. When you see imminent failure, trigger the **pause input**
+   - Policy stops
+   - Teleoperator moves to match robot position (torque enabled)
+   - No frames recorded during pause
+3. Trigger the **takeover input** to take control
+   - Teleoperator torque disabled, free to move
+   - **Recovery**: Teleoperate the robot back to a good state
+   - **Correction**: Correct the behavior
+   - All movements are recorded
+4. Trigger the **return-to-policy input**
+   - Policy resumes autonomous execution from the current state
+   - You can intervene again at any time (repeat steps 2–4)
+5. End and save the episode when the task is complete (or episode time limit is reached)
+6. **Reset**: Teleop moves to robot position, you can move the robot to the starting position
+7. Start the next episode
+
+**Foot Pedal Setup (Linux):**
+
+If using a USB foot pedal (PCsensor FootSwitch), ensure access:
+
+```bash
+sudo setfacl -m u:$USER:rw /dev/input/by-id/usb-PCsensor_FootSwitch-event-kbd
+```
+
+### Step 3: Fine-tune the Policy
+
+Fine-tune on the **combined** dataset (`demo-dataset` + `hil-dataset` merged together):
+
+```bash
+python src/lerobot/scripts/lerobot_train.py \
+    --dataset.repo_id=your-username/hil-dataset \
+    --policy.type=pi0 \
+    --policy.pretrained_path=outputs/pretrain/checkpoints/last/pretrained_model \
+    --output_dir=outputs/hil_finetune \
+    --steps=20000
+```
+
+Then deploy the fine-tuned policy and repeat from Step 2 to target its remaining failure modes.
+
+---
+
+## Tips for Effective HIL Collection
+
+### When to Intervene
+
+Intervene when you see:
+
+- Robot about to make an irreversible mistake
+- Robot hesitating or showing uncertain behavior
+- Robot deviating from the expected trajectory
+
+### Recovery: Teleoperating Back to a Good State
+
+During recovery, teleoperate the robot back to a state where:
+
+- The robot is in a familiar, in-distribution configuration
+- The current subtask can still be completed
+- The recovery trajectory itself is informative training data
+
+### Quality of Corrections
+
+During correction:
+
+- Provide **confident, clean** trajectories
+- Complete the current subtask fully
+- Don't overcorrect or add unnecessary movements
+
+---
+
+## Related Work
+
+This HIL data collection approach builds on ideas from interactive imitation learning:
+
+- **DAgger** (Ross et al., 2011) introduced the core idea: instead of only training on expert demonstrations, query the expert for corrections on states the _learner_ visits. This breaks the compounding-error cycle of standard behavioral cloning by iteratively collecting on-policy data.
+
+- **HG-DAgger** (Kelly et al., 2019) made this practical for robotics: a human expert monitors the robot and only intervenes when needed, rather than labeling every state. The gating between autonomous and human control is exactly the pause → takeover → return-to-policy loop used in the scripts here.
+
+- **RaC** (Hu et al., 2025) scales this loop to long-horizon tasks by explicitly decomposing interventions into **recovery** (teleoperating back to a good state) and **correction** (demonstrating the right behavior from there). This decomposition is the protocol followed by the HIL scripts in `examples/hil`.
+
+- **π0.6/RECAP** (Physical Intelligence, 2025) applies the same iterative collect-and-finetune loop at scale with VLA models, showing that even large pretrained policies benefit substantially from targeted human corrections on their own failure modes. π0.6 is trained using RECAP.
+
+```bibtex
+@article{ross2011dagger,
+  title={A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning},
+  author={Ross, Stéphane and Gordon, Geoffrey and Bagnell, Drew},
+  journal={Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics},
+  year={2011}
+}
+
+@article{kelly2019hgdagger,
+  title={HG-DAgger: Interactive Imitation Learning with Human Experts},
+  author={Kelly, Michael and Sidrane, Chelsea and Driggs-Campbell, Katherine and Kochenderfer, Mykel J},
+  journal={arXiv preprint arXiv:1810.02890},
+  year={2019}
+}
+
+@article{hu2025rac,
+  title={RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction},
+  author={Hu, Zheyuan and Wu, Robyn and Enock, Naveen and Li, Jasmine and Kadakia, Riya and Erickson, Zackory and Kumar, Aviral},
+  journal={arXiv preprint arXiv:2509.07953},
+  year={2025}
+}
+
+@article{pi2025recap,
+  title={π0.6: a VLA That Learns From Experience},
+  author={Physical Intelligence},
+  year={2025}
+}
+```

docs/source/libero.mdx

@@ -1,36 +1,61 @@
 # LIBERO
 
-**LIBERO** is a benchmark designed to study **lifelong robot learning**. The idea is that robots won’t just be pretrained once in a factory, they’ll need to keep learning and adapting with their human users over time. This ongoing adaptation is called **lifelong learning in decision making (LLDM)**, and it’s a key step toward building robots that become truly personalized helpers.
+LIBERO is a benchmark designed to study **lifelong robot learning** — the idea that robots need to keep learning and adapting with their users over time, not just be pretrained once. It provides a set of standardized manipulation tasks that focus on **knowledge transfer**: how well a robot can apply what it has already learned to new situations. By evaluating on LIBERO, different algorithms can be compared fairly and researchers can build on each other's work.
 
-- 📄 [LIBERO paper](https://arxiv.org/abs/2306.03310)
-- 💻 [Original LIBERO repo](https://github.com/Lifelong-Robot-Learning/LIBERO)
-
-To make progress on this challenge, LIBERO provides a set of standardized tasks that focus on **knowledge transfer**: how well a robot can apply what it has already learned to new situations. By evaluating on LIBERO, different algorithms can be compared fairly and researchers can build on each other’s work.
-
-LIBERO includes **five task suites**:
-
-- **LIBERO-Spatial (`libero_spatial`)** – tasks that require reasoning about spatial relations.
-- **LIBERO-Object (`libero_object`)** – tasks centered on manipulating different objects.
-- **LIBERO-Goal (`libero_goal`)** – goal-conditioned tasks where the robot must adapt to changing targets.
-- **LIBERO-90 (`libero_90`)** – 90 short-horizon tasks from the LIBERO-100 collection.
-- **LIBERO-Long (`libero_10`)** – 10 long-horizon tasks from the LIBERO-100 collection.
-
-Together, these suites cover **130 tasks**, ranging from simple object manipulations to complex multi-step scenarios. LIBERO is meant to grow over time, and to serve as a shared benchmark where the community can test and improve lifelong learning algorithms.
+- Paper: [Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://arxiv.org/abs/2306.03310)
+- GitHub: [Lifelong-Robot-Learning/LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO)
+- Project website: [libero-project.github.io](https://libero-project.github.io)
 
 ![An overview of the LIBERO benchmark](https://libero-project.github.io/assets/img/libero/fig1.png)
 
-## Evaluating with LIBERO
+## Available tasks
 
-At **LeRobot**, we ported [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) into our framework and used it mainly to **evaluate [SmolVLA](https://huggingface.co/docs/lerobot/en/smolvla)**, our lightweight Vision-Language-Action model.
+LIBERO includes **five task suites** covering **130 tasks**, ranging from simple object manipulations to complex multi-step scenarios:
 
-LIBERO is now part of our **multi-eval supported simulation**, meaning you can benchmark your policies either on a **single suite of tasks** or across **multiple suites at once** with just a flag.
+| Suite          | CLI name         | Tasks | Description                                        |
+| -------------- | ---------------- | ----- | -------------------------------------------------- |
+| LIBERO-Spatial | `libero_spatial` | 10    | Tasks requiring reasoning about spatial relations  |
+| LIBERO-Object  | `libero_object`  | 10    | Tasks centered on manipulating different objects   |
+| LIBERO-Goal    | `libero_goal`    | 10    | Goal-conditioned tasks with changing targets       |
+| LIBERO-90      | `libero_90`      | 90    | Short-horizon tasks from the LIBERO-100 collection |
+| LIBERO-Long    | `libero_10`      | 10    | Long-horizon tasks from the LIBERO-100 collection  |
 
-To Install LIBERO, after following LeRobot official instructions, just do:
-`pip install -e ".[libero]"`
+## Installation
+
+After following the LeRobot installation instructions:
+
+```bash
+pip install -e ".[libero]"
+```
+
+<Tip>
+LIBERO requires Linux (`sys_platform == 'linux'`). LeRobot uses MuJoCo for simulation — set the rendering backend before training or evaluation:
+
+```bash
+export MUJOCO_GL=egl  # for headless servers (HPC, cloud)
+```
+
+</Tip>
+
+## Evaluation
+
+### Default evaluation (recommended)
+
+Evaluate across the four standard suites (10 episodes per task):
+
+```bash
+lerobot-eval \
+  --policy.path="your-policy-id" \
+  --env.type=libero \
+  --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
+  --eval.batch_size=1 \
+  --eval.n_episodes=10 \
+  --env.max_parallel_tasks=1
+```
 
 ### Single-suite evaluation
 
-Evaluate a policy on one LIBERO suite:
+Evaluate on one LIBERO suite:
 
 ```bash
 lerobot-eval \
@@ -42,15 +67,13 @@ lerobot-eval \
 ```
 
 - `--env.task` picks the suite (`libero_object`, `libero_spatial`, etc.).
-- `--env.task_ids` picks task ids to run (`[0]`, `[1,2,3]`, etc.). Omit this flag (or set it to `null`) to run all tasks in the suite.
+- `--env.task_ids` restricts to specific task indices (`[0]`, `[1,2,3]`, etc.). Omit to run all tasks in the suite.
 - `--eval.batch_size` controls how many environments run in parallel.
-- `--eval.n_episodes` sets how many episodes to run in total.
-
----
+- `--eval.n_episodes` sets how many episodes to run per task.
 
 ### Multi-suite evaluation
 
-Benchmark a policy across multiple suites at once:
+Benchmark a policy across multiple suites at once by passing a comma-separated list:
 
 ```bash
 lerobot-eval \
@@ -61,50 +84,49 @@ lerobot-eval \
   --eval.n_episodes=2
 ```
 
-- Pass a comma-separated list to `--env.task` for multi-suite evaluation.
+### Control mode
 
-### Control Mode
+LIBERO supports two control modes — `relative` (default) and `absolute`. Different VLA checkpoints are trained with different action parameterizations, so make sure the mode matches your policy:
 
-LIBERO now supports two control modes: relative and absolute. This matters because different VLA checkpoints are trained with different mode of action to output hence control parameterizations.
-You can switch them with: `env.control_mode = "relative"` and `env.control_mode = "absolute"`
+```bash
+--env.control_mode=relative   # or "absolute"
+```
 
 ### Policy inputs and outputs
 
-When using LIBERO through LeRobot, policies interact with the environment via **observations** and **actions**:
+**Observations:**
 
-- **Observations**
-  - `observation.state` – proprioceptive features (agent state).
-  - `observation.images.image` – main camera view (`agentview_image`).
-  - `observation.images.image2` – wrist camera view (`robot0_eye_in_hand_image`).
+- `observation.state` — 8-dim proprioceptive features (eef position, axis-angle orientation, gripper qpos)
+- `observation.images.image` — main camera view (`agentview_image`), HWC uint8
+- `observation.images.image2` — wrist camera view (`robot0_eye_in_hand_image`), HWC uint8
 
-  ⚠️ **Note:** LeRobot enforces the `.images.*` prefix for any multi-modal visual features. Always ensure that your policy config `input_features` use the same naming keys, and that your dataset metadata keys follow this convention during evaluation.
-  If your data contains different keys, you must rename the observations to match what the policy expects, since naming keys are encoded inside the normalization statistics layer.
-  This will be fixed with the upcoming Pipeline PR.
+<Tip warning={true}>
+  LeRobot enforces the `.images.*` prefix for visual features. Ensure your
+  policy config `input_features` use the same naming keys, and that your dataset
+  metadata keys follow this convention. If your data contains different keys,
+  you must rename the observations to match what the policy expects, since
+  naming keys are encoded inside the normalization statistics layer.
+</Tip>
 
-- **Actions**
-  - Continuous control values in a `Box(-1, 1, shape=(7,))` space.
+**Actions:**
 
-We also provide a notebook for quick testing:
-Training with LIBERO
+- Continuous control in `Box(-1, 1, shape=(7,))` — 6D end-effector delta + 1D gripper
 
-## Training with LIBERO
+### Recommended evaluation episodes
 
-When training on LIBERO tasks, make sure your dataset parquet and metadata keys follow the LeRobot convention.
+For reproducible benchmarking, use **10 episodes per task** across all four standard suites (Spatial, Object, Goal, Long). This gives 400 total episodes and matches the protocol used for published results.
 
-The environment expects:
+## Training
 
-- `observation.state` → 8-dim agent state
-- `observation.images.image` → main camera (`agentview_image`)
-- `observation.images.image2` → wrist camera (`robot0_eye_in_hand_image`)
+### Dataset
 
-⚠️ Cleaning the dataset upfront is **cleaner and more efficient** than remapping keys inside the code.
-To avoid potential mismatches and key errors, we provide a **preprocessed LIBERO dataset** that is fully compatible with the current LeRobot codebase and requires no additional manipulation:
-👉 [HuggingFaceVLA/libero](https://huggingface.co/datasets/HuggingFaceVLA/libero)
+We provide a preprocessed LIBERO dataset fully compatible with LeRobot:
 
-For reference, here is the **original dataset** published by Physical Intelligence:
-👉 [physical-intelligence/libero](https://huggingface.co/datasets/physical-intelligence/libero)
+- [HuggingFaceVLA/libero](https://huggingface.co/datasets/HuggingFaceVLA/libero)
 
----
+For reference, the original dataset published by Physical Intelligence:
+
+- [physical-intelligence/libero](https://huggingface.co/datasets/physical-intelligence/libero)
 
 ### Example training command
 
@@ -121,52 +143,39 @@ lerobot-train \
   --batch_size=4 \
   --eval.batch_size=1 \
   --eval.n_episodes=1 \
-  --eval_freq=1000 \
+  --eval_freq=1000
 ```
 
----
+## Reproducing published results
 
-### Note on rendering
+We reproduce the results of Pi0.5 on the LIBERO benchmark. We take the Physical Intelligence LIBERO base model (`pi05_libero`) and finetune for an additional 6k steps in bfloat16, with batch size of 256 on 8 H100 GPUs using the [HuggingFace LIBERO dataset](https://huggingface.co/datasets/HuggingFaceVLA/libero).
 
-LeRobot uses MuJoCo for simulation. You need to set the rendering backend before training or evaluation:
+The finetuned model: [lerobot/pi05_libero_finetuned](https://huggingface.co/lerobot/pi05_libero_finetuned)
 
-- `export MUJOCO_GL=egl` → for headless servers (e.g. HPC, cloud)
-
-## Reproducing π₀.₅ results
-
-We reproduce the results of π₀.₅ on the LIBERO benchmark using the LeRobot implementation. We take the Physical Intelligence LIBERO base model (`pi05_libero`) and finetune for an additional 6k steps in bfloat16, with batch size of 256 on 8 H100 GPUs using the [HuggingFace LIBERO dataset](https://huggingface.co/datasets/HuggingFaceVLA/libero).
-
-The finetuned model can be found here:
-
-- **π₀.₅ LIBERO**: [lerobot/pi05_libero_finetuned](https://huggingface.co/lerobot/pi05_libero_finetuned)
-
-We then evaluate the finetuned model using the LeRobot LIBERO implementation, by running the following command:
+### Evaluation command
 
 ```bash
 lerobot-eval \
-  --output_dir=/logs/ \
+  --output_dir=./eval_logs/ \
   --env.type=libero \
   --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
   --eval.batch_size=1 \
   --eval.n_episodes=10 \
   --policy.path=pi05_libero_finetuned \
   --policy.n_action_steps=10 \
-  --output_dir=./eval_logs/ \
   --env.max_parallel_tasks=1
 ```
 
-**Note:** We set `n_action_steps=10`, similar to the original OpenPI implementation.
+We set `n_action_steps=10`, matching the original OpenPI implementation.
 
 ### Results
 
-We obtain the following results on the LIBERO benchmark:
+| Model               | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average  |
+| ------------------- | -------------- | ------------- | ----------- | --------- | -------- |
+| **Pi0.5 (LeRobot)** | 97.0           | 99.0          | 98.0        | 96.0      | **97.5** |
 
-| Model    | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average  |
-| -------- | -------------- | ------------- | ----------- | --------- | -------- |
-| **π₀.₅** | 97.0           | 99.0          | 98.0        | 96.0      | **97.5** |
+These results are consistent with the [original results](https://github.com/Physical-Intelligence/openpi/tree/main/examples/libero#results) reported by Physical Intelligence:
 
-These results are consistent with the original [results](https://github.com/Physical-Intelligence/openpi/tree/main/examples/libero#results) reported by Physical Intelligence:
-
-| Model    | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average   |
-| -------- | -------------- | ------------- | ----------- | --------- | --------- |
-| **π₀.₅** | 98.8           | 98.2          | 98.0        | 92.4      | **96.85** |
+| Model              | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average   |
+| ------------------ | -------------- | ------------- | ----------- | --------- | --------- |
+| **Pi0.5 (OpenPI)** | 98.8           | 98.2          | 98.0        | 92.4      | **96.85** |

docs/source/metaworld.mdx

@@ -1,32 +1,111 @@
 # Meta-World
 
-Meta-World is a well-designed, open-source simulation benchmark for multi-task and meta reinforcement learning in continuous-control robotic manipulation. It gives researchers a shared, realistic playground to test whether algorithms can _learn many different tasks_ and _generalize quickly to new ones_ — two central challenges for real-world robotics.
+Meta-World is an open-source simulation benchmark for **multi-task and meta reinforcement learning** in continuous-control robotic manipulation. It bundles 50 diverse manipulation tasks using everyday objects and a common tabletop Sawyer arm, providing a standardized playground to test whether algorithms can learn many different tasks and generalize quickly to new ones.
 
-- 📄 [MetaWorld paper](https://arxiv.org/pdf/1910.10897)
-- 💻 [Original MetaWorld repo](https://github.com/Farama-Foundation/Metaworld)
+- Paper: [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://arxiv.org/abs/1910.10897)
+- GitHub: [Farama-Foundation/Metaworld](https://github.com/Farama-Foundation/Metaworld)
+- Project website: [metaworld.farama.org](https://metaworld.farama.org)
 
 ![MetaWorld MT10 demo](https://meta-world.github.io/figures/ml45.gif)
 
-## Why Meta-World matters
+## Available tasks
 
-- **Diverse, realistic tasks.** Meta-World bundles a large suite of simulated manipulation tasks (50 in the MT50 suite) using everyday objects and a common tabletop Sawyer arm. This diversity exposes algorithms to a wide variety of dynamics, contacts and goal specifications while keeping a consistent control and observation structure.
-- **Focus on generalization and multi-task learning.** By evaluating across task distributions that share structure but differ in goals and objects, Meta-World reveals whether an agent truly learns transferable skills rather than overfitting to a narrow task.
-- **Standardized evaluation protocol.** It provides clear evaluation modes and difficulty splits, so different methods can be compared fairly across easy, medium, hard and very-hard regimes.
-- **Empirical insight.** Past evaluations on Meta-World show impressive progress on some fronts, but also highlight that current multi-task and meta-RL methods still struggle with large, diverse task sets. That gap points to important research directions.
+Meta-World provides 50 tasks organized into difficulty groups. In LeRobot, you can evaluate on individual tasks, difficulty groups, or the full MT50 suite:
 
-## What it enables in LeRobot
+| Group      | CLI name             | Tasks | Description                                            |
+| ---------- | -------------------- | ----- | ------------------------------------------------------ |
+| Easy       | `easy`               | 28    | Tasks with simple dynamics and single-step goals       |
+| Medium     | `medium`             | 11    | Tasks requiring multi-step reasoning                   |
+| Hard       | `hard`               | 6     | Tasks with complex contacts and precise manipulation   |
+| Very Hard  | `very_hard`          | 5     | The most challenging tasks in the suite                |
+| MT50 (all) | Comma-separated list | 50    | All 50 tasks — the most challenging multi-task setting |
 
-In LeRobot, you can evaluate any policy or vision-language-action (VLA) model on Meta-World tasks and get a clear success-rate measure. The integration is designed to be straightforward:
+You can also pass individual task names directly (e.g., `assembly-v3`, `dial-turn-v3`).
 
-- We provide a LeRobot-ready dataset for Meta-World (MT50) on the HF Hub: `https://huggingface.co/datasets/lerobot/metaworld_mt50`.
-  - This dataset is formatted for the MT50 evaluation that uses all 50 tasks (the most challenging multi-task setting).
-  - MT50 gives the policy a one-hot task vector and uses fixed object/goal positions for consistency.
+We provide a LeRobot-ready dataset for Meta-World MT50 on the HF Hub: [lerobot/metaworld_mt50](https://huggingface.co/datasets/lerobot/metaworld_mt50). This dataset is formatted for the MT50 evaluation that uses all 50 tasks with fixed object/goal positions and one-hot task vectors for consistency.
 
-- Task descriptions and the exact keys required for evaluation are available in the repo/dataset — use these to ensure your policy outputs the right success signals.
+## Installation
 
-## Quick start, train a SmolVLA policy on Meta-World
+After following the LeRobot installation instructions:
 
-Example command to train a SmolVLA policy on a subset of tasks:
+```bash
+pip install -e ".[metaworld]"
+```
+
+<Tip warning={true}>
+If you encounter an `AssertionError: ['human', 'rgb_array', 'depth_array']` when running Meta-World environments, this is a mismatch between Meta-World and your Gymnasium version. Fix it with:
+
+```bash
+pip install "gymnasium==1.1.0"
+```
+
+</Tip>
+
+## Evaluation
+
+### Default evaluation (recommended)
+
+Evaluate on the medium difficulty split (a good balance of coverage and compute):
+
+```bash
+lerobot-eval \
+  --policy.path="your-policy-id" \
+  --env.type=metaworld \
+  --env.task=medium \
+  --eval.batch_size=1 \
+  --eval.n_episodes=10
+```
+
+### Single-task evaluation
+
+Evaluate on a specific task:
+
+```bash
+lerobot-eval \
+  --policy.path="your-policy-id" \
+  --env.type=metaworld \
+  --env.task=assembly-v3 \
+  --eval.batch_size=1 \
+  --eval.n_episodes=10
+```
+
+### Multi-task evaluation
+
+Evaluate across multiple tasks or difficulty groups:
+
+```bash
+lerobot-eval \
+  --policy.path="your-policy-id" \
+  --env.type=metaworld \
+  --env.task=assembly-v3,dial-turn-v3,handle-press-side-v3 \
+  --eval.batch_size=1 \
+  --eval.n_episodes=10
+```
+
+- `--env.task` accepts explicit task lists (comma-separated) or difficulty groups (e.g., `easy`, `medium`, `hard`, `very_hard`).
+- `--eval.batch_size` controls how many environments run in parallel.
+- `--eval.n_episodes` sets how many episodes to run per task.
+
+### Policy inputs and outputs
+
+**Observations:**
+
+- `observation.image` — single camera view (`corner2`), 480x480 HWC uint8
+- `observation.state` — 4-dim proprioceptive state (end-effector position + gripper)
+
+**Actions:**
+
+- Continuous control in `Box(-1, 1, shape=(4,))` — 3D end-effector delta + 1D gripper
+
+### Recommended evaluation episodes
+
+For reproducible benchmarking, use **10 episodes per task**. For the full MT50 suite this gives 500 total episodes. If you care about generalization, run on the full MT50 — it is intentionally challenging and reveals strengths/weaknesses better than a few narrow tasks.
+
+## Training
+
+### Example training command
+
+Train a SmolVLA policy on a subset of Meta-World tasks:
 
 ```bash
 lerobot-train \
@@ -44,37 +123,8 @@ lerobot-train \
   --eval_freq=1000
 ```
 
-Notes:
-
-- `--env.task` accepts explicit task lists (comma separated) or difficulty groups (e.g., `env.task="hard"`).
-- Adjust `batch_size`, `steps`, and `eval_freq` to match your compute budget.
-- **Gymnasium Assertion Error**: if you encounter an error like
-  `AssertionError: ['human', 'rgb_array', 'depth_array']` when running MetaWorld environments, this comes from a mismatch between MetaWorld and your Gymnasium version.
-  We recommend using:
-
-```bash
-  pip install "gymnasium==1.1.0"
-```
-
-to ensure proper compatibility.
-
-## Quick start — evaluate a trained policy
-
-To evaluate a trained policy on the Meta-World medium difficulty split:
-
-```bash
-lerobot-eval \
-  --policy.path="your-policy-id" \
-  --env.type=metaworld \
-  --env.task=medium \
-  --eval.batch_size=1 \
-  --eval.n_episodes=2
-```
-
-This will run episodes and return per-task success rates using the standard Meta-World evaluation keys.
-
 ## Practical tips
 
-- If you care about generalization, run on the full MT50 suite — it’s intentionally challenging and reveals strengths/weaknesses better than a few narrow tasks.
-- Use the one-hot task conditioning for multi-task training (MT10 / MT50 conventions) so policies have explicit task context.
+- Use the one-hot task conditioning for multi-task training (MT10/MT50 conventions) so policies have explicit task context.
 - Inspect the dataset task descriptions and the `info["is_success"]` keys when writing post-processing or logging so your success metrics line up with the benchmark.
+- Adjust `batch_size`, `steps`, and `eval_freq` to match your compute budget.

docs/source/umi_pi0_relative_ee.mdx

@@ -1,227 +0,0 @@
-# UMI Data with pi0 Relative EE Actions
-
-This guide explains how to train a pi0 policy with UMI-style relative end-effector (EE) actions and deploy it on a real OpenArm robot.
-
-**What we will do:**
-
-1. Prepare the dataset (EE pose + gripper in the action column).
-2. Recompute statistics for relative actions.
-3. Train pi0 with `derive_state_from_action=true`.
-4. Evaluate the trained policy on a real robot.
-
-## Background
-
-[UMI (Universal Manipulation Interface)](https://umi-gripper.github.io) collects manipulation data with hand-held grippers, recovering 6-DoF EE poses via SLAM. The key insight from UMI (Chi et al., 2024) is that the action space must include **both EE trajectory and gripper width**, and actions should be expressed as **relative trajectories** (offsets from the current pose).
-
-### Dataset layout
-
-The dataset should have this structure:
-
-| Feature                   | Shape     | Content                                                  |
-| ------------------------- | --------- | -------------------------------------------------------- |
-| `observation.images.cam0` | `[3,H,W]` | Wrist camera image                                       |
-| `action`                  | `[8]`     | `[x, y, z, ax, ay, az, proximal, distal]` (EE + gripper) |
-
-No separate `observation.pose` or `observation.joints` columns are needed — the model derives its proprioception state directly from the action column (`derive_state_from_action=true`).
-
-### Why relative actions?
-
-With relative actions, each action in a chunk is an **offset from the current state** rather than an absolute target:
-
-```
-relative_action[i] = absolute_action[t + i] − state[t]
-```
-
-UMI ablations show this is critical: absolute actions achieve only 25% success vs 100% for relative trajectory on the cup arrangement task. Compared to delta actions (each step relative to the previous), relative trajectory avoids error accumulation. See the [Action Representations](action_representations) guide for details.
-
-### `derive_state_from_action`
-
-When `derive_state_from_action=true`, pi0 derives `observation.state` from the action column during training — no separate state column needed. Under the hood:
-
-- `action_delta_indices` extends to `[-1, 0, 1, ..., chunk_size-1]` (one extra leading timestep).
-- `DeriveStateFromActionStep` extracts `[action[t-1], action[t]]` as a 2-step state and strips the extra timestep from the action chunk.
-- `RelativeActionsProcessorStep` converts actions to offsets from `state[t]`.
-- `RelativeStateProcessorStep` converts the 2-step state to relative proprioception (velocity + zeros) and flattens.
-
-This implies `use_relative_state=true` and `state_obs_steps=2`.
-
-During **inference**, `DeriveStateFromActionStep` is a no-op — state comes from the robot via forward kinematics. `RelativeStateProcessorStep` buffers the previous state and applies the same conversion automatically.
-
-## Step 1: Recompute Stats
-
-After preparing the dataset with EE pose in the action column, recompute statistics with `derive_state_from_action=true`. This computes relative action and state stats so the normalizer sees offset distributions:
-
-```bash
-lerobot-edit-dataset \
-    --repo-id=glannuzel/grabette-dataset \
-    --operation=recompute_stats \
-    --operation.relative_action=true \
-    --operation.relative_exclude_joints='["proximal", "distal"]' \
-    --operation.derive_state_from_action=true \
-    --operation.chunk_size=30 \
-    --push_to_hub=true
-```
-
-| Flag                            | Purpose                                                                         |
-| ------------------------------- | ------------------------------------------------------------------------------- |
-| `relative_action=true`          | Compute stats on `action − state` (relative actions)                            |
-| `relative_exclude_joints`       | Keep gripper dims absolute (they don't benefit from relative encoding)          |
-| `derive_state_from_action=true` | Derive state from action column (implies `relative_state`, `state_obs_steps=2`) |
-| `chunk_size=30`                 | Must match training chunk size                                                  |
-
-## Step 2: Train
-
-```bash
-#!/bin/bash
-set -euo pipefail
-
-export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:${LD_LIBRARY_PATH:-}
-
-DATASET="glannuzel/grabette-dataset"
-NUM_PROCESSES=8
-
-echo "=== Training pi0 on $DATASET (UMI relative EE, ${NUM_PROCESSES} GPUs) ==="
-accelerate launch --multi_gpu --num_processes=$NUM_PROCESSES \
-    -m lerobot.scripts.lerobot_train \
-    --dataset.repo_id="$DATASET" \
-    --dataset.video_backend=pyav \
-    --policy.type=pi0 \
-    --policy.pretrained_path=lerobot/pi0_base \
-    --policy.repo_id=pepijn/grabette-umi-pi0 \
-    --policy.chunk_size=30 \
-    --policy.n_action_steps=30 \
-    --policy.derive_state_from_action=true \
-    --use_relative_actions=true \
-    --policy.relative_exclude_joints='["proximal", "distal"]' \
-    --batch_size=32 \
-    --steps=5000 \
-    --policy.scheduler_decay_steps=5000 \
-    --policy.dtype=bfloat16 \
-    --policy.compile_model=false \
-    --policy.gradient_checkpointing=true \
-    --policy.device=cuda \
-    --output_dir=/fsx/pepijn/outputs/grabette-umi \
-    --job_name=grabette-umi-v2 \
-    --wandb.enable=true \
-    --wandb.disable_artifact=true \
-    --wandb.project=grabette-umi \
-    --log_freq=100 \
-    --save_freq=5000
-```
-
-Key flags:
-
-| Flag                            | Purpose                                                                |
-| ------------------------------- | ---------------------------------------------------------------------- |
-| `derive_state_from_action=true` | Derive proprioception from action column (full UMI mode)               |
-| `use_relative_actions=true`     | Actions are offsets from current state                                 |
-| `relative_exclude_joints`       | `["proximal", "distal"]` — gripper stays absolute, EE pose is relative |
-| `chunk_size=30`                 | Action horizon: 30 steps (~0.65s at 46 FPS)                            |
-| `n_action_steps=30`             | Execute full chunk before replanning                                   |
-
-Note: `derive_state_from_action=true` automatically implies `use_relative_state=true` and `state_obs_steps=2`. No `rename_map` is needed since there are no separate observation columns to rename.
-
-## Step 3: Evaluate
-
-The evaluation script in `examples/umi_pi0_relative_ee/evaluate.py` runs inference on a real OpenArm robot:
-
-```bash
-python examples/umi_pi0_relative_ee/evaluate.py
-```
-
-Edit `HF_MODEL_ID`, camera index, and robot configuration at the top of the file.
-
-### How inference works
-
-At inference, the training dataset has no `observation.state` — it was derived from actions. The evaluate script provides `observation.state` from the robot via forward kinematics:
-
-1. **Robot → FK** — Arm joint positions → EE pose `[x,y,z,ax,ay,az]`, gripper → `[proximal, distal]`. Combined into `observation.state` (8D).
-2. **Preprocessor** (loaded from checkpoint) — `DeriveStateFromActionStep` is a no-op. `RelativeStateProcessorStep` buffers previous state, stacks `[prev, current]`, subtracts current → velocity info. `RelativeActionsProcessorStep` caches state. `NormalizerProcessorStep` normalizes.
-3. **pi0 inference** — Predicts normalized relative action chunk (30 steps).
-4. **Postprocessor** — `UnnormalizerProcessorStep` unnormalizes, `AbsoluteActionsProcessorStep` adds cached state → absolute EE targets.
-5. **IK → Robot** — Absolute `[x,y,z,ax,ay,az]` → arm joint targets with full 6-DOF IK (orientation weight = 1.0). `[proximal, distal]` → direct gripper position commands.
-
-### Latency compensation
-
-Set `LATENCY_SKIP_STEPS` to skip the first few predicted action steps, compensating for system latency:
-
-```python
-LATENCY_SKIP_STEPS = 7  # ceil(total_latency_ms / (1000 / FPS))
-```
-
-At 46 FPS (~22ms/step) with ~150ms total latency: `ceil(150/22) ≈ 7`. Start with 0 for a safe first test.
-
-## Replay Viewer
-
-Visualize any dataset episode in a browser-based 3D viewer before running on hardware. The viewer shows the EE trajectory overlaid on the OpenArm URDF model.
-
-### Quick start
-
-```bash
-python examples/umi_pi0_relative_ee/replay.py
-```
-
-### Options
-
-| Flag        | Default                      | Description                          |
-| ----------- | ---------------------------- | ------------------------------------ |
-| `--repo-id` | `glannuzel/grabette-dataset` | HuggingFace dataset repo to load     |
-| `--episode` | `0`                          | Episode index to replay              |
-| `--port`    | `8765`                       | HTTP server port                     |
-| `--force`   | off                          | Re-extract trajectory even if cached |
-
-### Viewer controls
-
-The panel in the top-left corner shows live EE coordinates and gripper state. Transport controls:
-
-- **Play / Pause** — toggle automatic playback.
-- **Step buttons** (◀ ▶) — advance or rewind one frame.
-- **Reset** (⟳) — jump to frame 0.
-- **Scrubber** — drag to seek.
-- **Speed selector** — 0.25× to 4× playback speed.
-
-### Color legend
-
-| Color              | Meaning                                       |
-| ------------------ | --------------------------------------------- |
-| Red sphere         | Current EE position                           |
-| Yellow trail       | Past trajectory                               |
-| Dark trail         | Future trajectory                             |
-| Orange ring + axes | URDF `ee_target` frame (zero-joint reference) |
-
-## How the Pieces Fit Together
-
-```
-Training (derive_state_from_action=true):
-  DataLoader loads action: [B, 31, 8]  (chunk_size=30 + 1 leading)
-      → DeriveStateFromActionStep
-          state  = action[:, :2, :]     → [B, 2, 8]
-          action = action[:, 1:, :]     → [B, 30, 8]
-      → RelativeActionsProcessorStep    (action -= state[:, -1, :])
-      → RelativeStateProcessorStep      (state offsets from current, flatten → [B, 16])
-      → NormalizerProcessorStep         → pi0 model
-
-Inference:
-  arm joints → FK → observation.state [8D: x,y,z,ax,ay,az,prox,dist]
-                        ↓
-                DeriveStateFromActionStep (no-op)
-                        ↓
-                RelativeActionsProcessorStep (caches state)
-                        ↓
-                RelativeStateProcessorStep (buffers prev, stacks, subtracts, flattens)
-                        ↓
-                NormalizerProcessorStep → pi0 model → relative action chunk [30, 8]
-                        ↓
-                UnnormalizerProcessorStep
-                        ↓
-                AbsoluteActionsProcessorStep (+ cached state → absolute EE)
-                        ↓
-                IK → joint targets → robot
-```
-
-## References
-
-- [UMI: Universal Manipulation Interface](https://umi-gripper.github.io) — Chi et al., 2024. Defines relative trajectory actions.
-- [Action Representations](action_representations) — LeRobot guide comparing absolute, relative, and delta actions.
-- [pi0 documentation](pi0) — Full pi0 configuration including `use_relative_actions`.
-- [`examples/so100_to_so100_EE/`](https://github.com/huggingface/lerobot/tree/main/examples/so100_to_so100_EE) — EE-space evaluation example this builds on.

examples/dataset/create_progress_videos.py

@@ -0,0 +1,680 @@
+#!/usr/bin/env python
+
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Create MP4 (or GIF) videos with sarm_progress overlay for specified episodes.
+
+Downloads datasets from HuggingFace, seeks directly into the episode segment
+of the source video, draws a progress line on each frame, and writes the result.
+
+Usage:
+    python examples/dataset/create_progress_videos.py \
+        --repo-id lerobot-data-collection/level2_final_quality3 \
+        --episode 1100
+
+    python examples/dataset/create_progress_videos.py \
+        --repo-id lerobot-data-collection/level2_final_quality3 \
+        --episode 1100 \
+        --camera-key observation.images.top \
+        --output-dir ./my_videos \
+        --gif
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import subprocess
+from pathlib import Path
+
+import cv2
+import numpy as np
+import pandas as pd
+from huggingface_hub import snapshot_download
+
+GRAPH_Y_TOP_FRAC = 0.01
+GRAPH_Y_BOT_FRAC = 0.99
+LINE_THICKNESS = 3
+SHADOW_THICKNESS = 6
+REF_ALPHA = 0.45
+FILL_ALPHA = 0.55
+SCORE_FONT_SCALE = 0.8
+TASK_FONT_SCALE = 0.55
+
+
+def download_episode_metadata(repo_id: str, episode: int) -> Path:
+    """Download only the metadata and sarm_progress files for a dataset.
+
+    Args:
+        repo_id: HuggingFace dataset repository ID.
+        episode: Episode index (used for logging only; all meta is fetched).
+
+    Returns:
+        Local cache path for the downloaded snapshot.
+    """
+    logging.info("[1/4] Downloading metadata for %s (episode %d) ...", repo_id, episode)
+    local_path = Path(
+        snapshot_download(
+            repo_id=repo_id,
+            repo_type="dataset",
+            allow_patterns=["meta/**", "sarm_progress.parquet"],
+            ignore_patterns=["*.mp4"],
+        )
+    )
+    return local_path
+
+
+def load_episode_meta(local_path: Path, episode: int, camera_key: str | None) -> dict:
+    """Read info.json and episode parquet to resolve fps, video path, and timestamps.
+
+    Args:
+        local_path: Local cache directory containing meta/.
+        episode: Episode index to look up.
+        camera_key: Camera observation key (e.g. "observation.images.base").
+            If None, the first available video key is used.
+
+    Returns:
+        Dict with keys: fps, camera, video_rel, chunk_index, file_index,
+        from_ts, to_ts, task_name.
+    """
+    info = json.loads((local_path / "meta" / "info.json").read_text())
+    fps = info["fps"]
+    features = info["features"]
+
+    video_keys = [k for k, v in features.items() if v.get("dtype") == "video"]
+    if not video_keys:
+        raise RuntimeError("No video keys found in dataset features")
+
+    if camera_key is not None:
+        if camera_key not in video_keys:
+            raise RuntimeError(f"camera_key='{camera_key}' not found. Available: {video_keys}")
+        selected_camera = camera_key
+    else:
+        selected_camera = video_keys[0]
+    logging.info("   fps=%d  camera='%s'  all_cams=%s", fps, selected_camera, video_keys)
+
+    episode_rows = []
+    for parquet_file in sorted((local_path / "meta" / "episodes").glob("**/*.parquet")):
+        episode_rows.append(pd.read_parquet(parquet_file))
+    episode_df = pd.concat(episode_rows, ignore_index=True)
+    row = episode_df[episode_df["episode_index"] == episode]
+    if row.empty:
+        raise RuntimeError(f"Episode {episode} not found in episode metadata")
+    row = row.iloc[0]
+
+    chunk_col = f"videos/{selected_camera}/chunk_index"
+    file_col = f"videos/{selected_camera}/file_index"
+    ts_from_col = f"videos/{selected_camera}/from_timestamp"
+    ts_to_col = f"videos/{selected_camera}/to_timestamp"
+
+    if chunk_col not in row.index:
+        chunk_col = f"{selected_camera}/chunk_index"
+        file_col = f"{selected_camera}/file_index"
+        ts_from_col = f"{selected_camera}/from_timestamp"
+        ts_to_col = f"{selected_camera}/to_timestamp"
+    if chunk_col not in row.index:
+        raise RuntimeError(
+            f"Cannot find video metadata columns for {selected_camera}.\nAvailable: {list(row.index)}"
+        )
+
+    chunk_index = int(row[chunk_col])
+    file_index = int(row[file_col])
+    from_timestamp = float(row[ts_from_col])
+    to_timestamp = float(row[ts_to_col])
+
+    video_template = info.get(
+        "video_path", "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4"
+    )
+    video_rel = video_template.format(
+        video_key=selected_camera,
+        chunk_index=chunk_index,
+        file_index=file_index,
+    )
+
+    task_name = _resolve_task_name(row, local_path)
+
+    return {
+        "fps": fps,
+        "camera": selected_camera,
+        "video_rel": video_rel,
+        "chunk_index": chunk_index,
+        "file_index": file_index,
+        "from_ts": from_timestamp,
+        "to_ts": to_timestamp,
+        "task_name": task_name,
+    }
+
+
+def _resolve_task_name(row: pd.Series, local_path: Path) -> str:
+    """Best-effort extraction of the task name for an episode row.
+
+    Args:
+        row: Single-episode row from the episodes parquet.
+        local_path: Dataset cache root.
+
+    Returns:
+        Task name string, or empty string if unavailable.
+    """
+    try:
+        if "tasks" in row.index and row["tasks"] is not None:
+            tasks_val = row["tasks"]
+            if isinstance(tasks_val, (list, tuple, np.ndarray)) and len(tasks_val) > 0:
+                return str(tasks_val[0])
+            return str(tasks_val).strip("[]'")
+
+        tasks_parquet = local_path / "meta" / "tasks.parquet"
+        if tasks_parquet.exists():
+            tasks_df = pd.read_parquet(tasks_parquet)
+            task_idx = int(row.get("task_index", 0)) if "task_index" in row.index else 0
+            match = tasks_df[tasks_df["task_index"] == task_idx]
+            if not match.empty:
+                return str(match.index[0])
+    except Exception as exc:
+        logging.warning("Could not load task name: %s", exc)
+    return ""
+
+
+def download_video_file(repo_id: str, local_path: Path, video_rel: str) -> Path:
+    """Download the specific video file if not already cached.
+
+    Args:
+        repo_id: HuggingFace dataset repository ID.
+        local_path: Local cache directory.
+        video_rel: Relative path to the video file within the dataset.
+
+    Returns:
+        Absolute path to the downloaded video file.
+    """
+    video_path = local_path / video_rel
+    if video_path.exists():
+        logging.info("   Video already cached: %s", video_path)
+        return video_path
+    logging.info("[2/4] Downloading video file %s ...", video_rel)
+    snapshot_download(
+        repo_id=repo_id,
+        repo_type="dataset",
+        local_dir=str(local_path),
+        allow_patterns=[video_rel],
+    )
+    if not video_path.exists():
+        raise RuntimeError(f"Video not found after download: {video_path}")
+    return video_path
+
+
+def load_progress_data(local_path: Path, episode: int) -> np.ndarray | None:
+    """Load sarm_progress values for an episode.
+
+    Args:
+        local_path: Dataset cache root.
+        episode: Episode index.
+
+    Returns:
+        Sorted (N, 2) array of (frame_index, progress), or None if unavailable.
+    """
+    parquet_path = local_path / "sarm_progress.parquet"
+    if not parquet_path.exists():
+        logging.warning("sarm_progress.parquet not found")
+        return None
+    df = pd.read_parquet(parquet_path)
+    logging.info("   sarm_progress.parquet columns: %s", list(df.columns))
+    episode_df = df[df["episode_index"] == episode].copy()
+    if episode_df.empty:
+        logging.warning("No sarm_progress rows for episode %d", episode)
+        return None
+    episode_df = episode_df.sort_values("frame_index")
+
+    if "progress_dense" in episode_df.columns and episode_df["progress_dense"].notna().any():
+        progress_column = "progress_dense"
+    elif "progress_sparse" in episode_df.columns:
+        progress_column = "progress_sparse"
+    else:
+        progress_columns = [c for c in episode_df.columns if "progress" in c.lower()]
+        if not progress_columns:
+            return None
+        progress_column = progress_columns[0]
+
+    logging.info("   Using progress column: '%s'", progress_column)
+    return episode_df[["frame_index", progress_column]].rename(columns={progress_column: "progress"}).values
+
+
+def _precompute_pixel_coords(
+    progress_data: np.ndarray,
+    num_frames: int,
+    frame_width: int,
+    frame_height: int,
+) -> np.ndarray:
+    """Map progress samples to pixel coordinates for overlay drawing.
+
+    Args:
+        progress_data: (N, 2) array of (frame_index, progress).
+        num_frames: Total number of video frames.
+        frame_width: Video width in pixels.
+        frame_height: Video height in pixels.
+
+    Returns:
+        (N, 2) array of (x, y) pixel coordinates.
+    """
+    frame_indices = progress_data[:, 0].astype(float)
+    progress_values = np.clip(progress_data[:, 1].astype(float), 0.0, 1.0)
+
+    y_top = int(frame_height * GRAPH_Y_TOP_FRAC)
+    y_bot = int(frame_height * GRAPH_Y_BOT_FRAC)
+    graph_height = y_bot - y_top
+
+    x_coords = (frame_indices / (num_frames - 1) * (frame_width - 1)).astype(int)
+    y_coords = (y_bot - progress_values * graph_height).astype(int)
+
+    return np.stack([x_coords, y_coords], axis=1)
+
+
+def _progress_color(normalized_position: float) -> tuple[int, int, int]:
+    """Interpolate BGR color from red to green based on position in [0, 1].
+
+    Args:
+        normalized_position: Value in [0, 1] indicating how far along the episode.
+
+    Returns:
+        BGR color tuple.
+    """
+    red = int(255 * (1.0 - normalized_position))
+    green = int(255 * normalized_position)
+    return (0, green, red)
+
+
+def _prerender_fill_polygon(
+    pixel_coords: np.ndarray,
+    frame_width: int,
+    frame_height: int,
+) -> np.ndarray:
+    """Pre-render the grey fill polygon under the progress curve as a BGRA image.
+
+    Args:
+        pixel_coords: (N, 2) array of (x, y) pixel coordinates.
+        frame_width: Video width in pixels.
+        frame_height: Video height in pixels.
+
+    Returns:
+        BGRA image array of shape (frame_height, frame_width, 4).
+    """
+    y_bot = int(frame_height * GRAPH_Y_BOT_FRAC)
+    fill_image = np.zeros((frame_height, frame_width, 4), dtype=np.uint8)
+    polygon = np.concatenate(
+        [
+            pixel_coords,
+            [[pixel_coords[-1][0], y_bot], [pixel_coords[0][0], y_bot]],
+        ],
+        axis=0,
+    ).astype(np.int32)
+    cv2.fillPoly(fill_image, [polygon], color=(128, 128, 128, int(255 * FILL_ALPHA)))
+    return fill_image
+
+
+def _alpha_composite_region(base: np.ndarray, overlay_bgra: np.ndarray, x_limit: int) -> None:
+    """Blend BGRA overlay onto BGR base in-place, up to x_limit columns.
+
+    Args:
+        base: BGR frame to draw on (modified in-place).
+        overlay_bgra: BGRA overlay image.
+        x_limit: Only blend columns [0, x_limit).
+    """
+    if x_limit <= 0:
+        return
+    region_base = base[:, :x_limit]
+    region_overlay = overlay_bgra[:, :x_limit]
+    alpha = region_overlay[:, :, 3:4].astype(np.float32) / 255.0
+    region_base[:] = np.clip(
+        region_overlay[:, :, :3].astype(np.float32) * alpha + region_base.astype(np.float32) * (1.0 - alpha),
+        0,
+        255,
+    ).astype(np.uint8)
+
+
+def _draw_text_outlined(
+    frame: np.ndarray,
+    text: str,
+    position: tuple[int, int],
+    font_scale: float,
+    thickness: int = 1,
+) -> None:
+    """Draw white text with a dark outline for readability on any background.
+
+    Args:
+        frame: BGR image to draw on (modified in-place).
+        text: String to render.
+        position: (x, y) bottom-left corner of the text.
+        font_scale: OpenCV font scale.
+        thickness: Text stroke thickness.
+    """
+    font = cv2.FONT_HERSHEY_SIMPLEX
+    cv2.putText(frame, text, position, font, font_scale, (0, 0, 0), thickness + 2, cv2.LINE_AA)
+    cv2.putText(frame, text, position, font, font_scale, (255, 255, 255), thickness, cv2.LINE_AA)
+
+
+def composite_progress_video(
+    video_path: Path,
+    from_timestamp: float,
+    to_timestamp: float,
+    progress_data: np.ndarray,
+    output_path: Path,
+    fps: float,
+    task_name: str = "",
+) -> Path:
+    """Read episode frames by seeking into the source video, draw progress overlay, write output.
+
+    Uses cv2.CAP_PROP_POS_MSEC to seek directly into the source video,
+    eliminating the need for an intermediate clip file.
+
+    Args:
+        video_path: Path to the full source video file.
+        from_timestamp: Start timestamp of the episode in seconds.
+        to_timestamp: End timestamp of the episode in seconds.
+        progress_data: (N, 2) array of (frame_index, progress).
+        output_path: Path to write the output MP4.
+        fps: Frames per second for the output video.
+        task_name: Optional task name to display at the top of the video.
+
+    Returns:
+        Path to the written output file (MP4).
+    """
+    capture = cv2.VideoCapture(str(video_path))
+    try:
+        capture.set(cv2.CAP_PROP_POS_MSEC, from_timestamp * 1000)
+
+        frame_width = int(capture.get(cv2.CAP_PROP_FRAME_WIDTH))
+        frame_height = int(capture.get(cv2.CAP_PROP_FRAME_HEIGHT))
+        duration_seconds = to_timestamp - from_timestamp
+        num_frames = int(round(duration_seconds * fps))
+
+        logging.info(
+            "   Video: %dx%d, %d frames @ %.1f fps (%.2fs)",
+            frame_width,
+            frame_height,
+            num_frames,
+            fps,
+            duration_seconds,
+        )
+
+        pixel_coords = _precompute_pixel_coords(progress_data, num_frames, frame_width, frame_height)
+        y_ref = int(frame_height * GRAPH_Y_TOP_FRAC)
+
+        fill_image = _prerender_fill_polygon(pixel_coords, frame_width, frame_height)
+
+        ref_line_image = np.zeros((frame_height, frame_width, 4), dtype=np.uint8)
+        cv2.line(
+            ref_line_image,
+            (0, y_ref),
+            (frame_width - 1, y_ref),
+            (200, 200, 200, int(255 * REF_ALPHA)),
+            1,
+            cv2.LINE_AA,
+        )
+
+        frame_indices = progress_data[:, 0].astype(int)
+        progress_values = progress_data[:, 1].astype(float)
+
+        logging.info("[3/4] Compositing %d frames ...", num_frames)
+        fourcc = cv2.VideoWriter_fourcc(*"mp4v")
+        writer = cv2.VideoWriter(str(output_path), fourcc, fps, (frame_width, frame_height))
+
+        for frame_idx in range(num_frames):
+            ret, frame = capture.read()
+            if not ret:
+                break
+
+            drawn_count = int(np.searchsorted(frame_indices, frame_idx, side="right"))
+            x_current = (
+                int(pixel_coords[min(drawn_count, len(pixel_coords)) - 1][0]) + 1 if drawn_count > 0 else 0
+            )
+
+            _alpha_composite_region(frame, ref_line_image, frame_width)
+            _alpha_composite_region(frame, fill_image, x_current)
+
+            if drawn_count >= 2:
+                time_position = (drawn_count - 1) / max(len(progress_values) - 1, 1)
+                line_color = _progress_color(time_position)
+                points = pixel_coords[:drawn_count].reshape(-1, 1, 2).astype(np.int32)
+                cv2.polylines(
+                    frame,
+                    [points],
+                    isClosed=False,
+                    color=(255, 255, 255),
+                    thickness=SHADOW_THICKNESS,
+                    lineType=cv2.LINE_AA,
+                )
+                cv2.polylines(
+                    frame,
+                    [points],
+                    isClosed=False,
+                    color=line_color,
+                    thickness=LINE_THICKNESS,
+                    lineType=cv2.LINE_AA,
+                )
+
+            if drawn_count > 0:
+                score = float(progress_values[min(drawn_count, len(progress_values)) - 1])
+                score_text = f"{score:.2f}"
+                (text_width, _), _ = cv2.getTextSize(
+                    score_text, cv2.FONT_HERSHEY_SIMPLEX, SCORE_FONT_SCALE, 2
+                )
+                score_x = frame_width - text_width - 12
+                score_y = frame_height - 12
+                time_position = (drawn_count - 1) / max(len(progress_values) - 1, 1)
+                score_color = _progress_color(time_position)
+                cv2.putText(
+                    frame,
+                    score_text,
+                    (score_x, score_y),
+                    cv2.FONT_HERSHEY_SIMPLEX,
+                    SCORE_FONT_SCALE,
+                    (0, 0, 0),
+                    4,
+                    cv2.LINE_AA,
+                )
+                cv2.putText(
+                    frame,
+                    score_text,
+                    (score_x, score_y),
+                    cv2.FONT_HERSHEY_SIMPLEX,
+                    SCORE_FONT_SCALE,
+                    score_color,
+                    2,
+                    cv2.LINE_AA,
+                )
+
+            if task_name:
+                (text_width, _), _ = cv2.getTextSize(task_name, cv2.FONT_HERSHEY_SIMPLEX, TASK_FONT_SCALE, 1)
+                task_x = max((frame_width - text_width) // 2, 4)
+                _draw_text_outlined(frame, task_name, (task_x, 22), TASK_FONT_SCALE)
+
+            writer.write(frame)
+            if frame_idx % 100 == 0:
+                logging.info("   Frame %d/%d ...", frame_idx, num_frames)
+
+        writer.release()
+    finally:
+        capture.release()
+
+    logging.info("   MP4 written: %s", output_path)
+    return output_path
+
+
+def convert_mp4_to_gif(mp4_path: Path) -> Path:
+    """Convert an MP4 to an optimized GIF using ffmpeg palette generation.
+
+    Args:
+        mp4_path: Path to the source MP4 file.
+
+    Returns:
+        Path to the generated GIF file.
+    """
+    capture = cv2.VideoCapture(str(mp4_path))
+    frame_width = int(capture.get(cv2.CAP_PROP_FRAME_WIDTH))
+    capture.release()
+
+    gif_path = mp4_path.with_suffix(".gif")
+    palette_path = mp4_path.parent / "_palette.png"
+
+    logging.info("[4/4] Converting to GIF ...")
+    result_palette = subprocess.run(  # nosec B607
+        [
+            "ffmpeg",
+            "-y",
+            "-i",
+            str(mp4_path),
+            "-vf",
+            f"fps=10,scale={frame_width}:-1:flags=lanczos,palettegen=max_colors=128:stats_mode=diff",
+            "-update",
+            "1",
+            str(palette_path),
+        ],
+        capture_output=True,
+        text=True,
+    )
+    if result_palette.returncode != 0:
+        logging.warning("palettegen failed:\n%s", result_palette.stderr[-500:])
+
+    result_gif = subprocess.run(  # nosec B607
+        [
+            "ffmpeg",
+            "-y",
+            "-i",
+            str(mp4_path),
+            "-i",
+            str(palette_path),
+            "-filter_complex",
+            f"fps=10,scale={frame_width}:-1:flags=lanczos[v];[v][1:v]paletteuse=dither=bayer:bayer_scale=3",
+            str(gif_path),
+        ],
+        capture_output=True,
+        text=True,
+    )
+    if result_gif.returncode != 0:
+        logging.warning("GIF encode failed:\n%s", result_gif.stderr[-500:])
+
+    palette_path.unlink(missing_ok=True)
+    logging.info("   GIF written: %s", gif_path)
+    return gif_path
+
+
+def process_dataset(
+    repo_id: str,
+    episode: int,
+    camera_key: str | None,
+    output_dir: Path,
+    create_gif: bool = False,
+) -> Path | None:
+    """Full pipeline: download, extract metadata, composite progress, write output.
+
+    Args:
+        repo_id: HuggingFace dataset repository ID.
+        episode: Episode index.
+        camera_key: Camera key to use, or None for auto-selection.
+        output_dir: Directory to write output files.
+        create_gif: If True, also generate a GIF from the MP4.
+
+    Returns:
+        Path to the final output file, or None on failure.
+    """
+    safe_name = repo_id.replace("/", "_")
+    logging.info("Processing: %s  |  episode %d", repo_id, episode)
+
+    local_path = download_episode_metadata(repo_id, episode)
+    logging.info("   Local cache: %s", local_path)
+
+    episode_meta = load_episode_meta(local_path, episode, camera_key)
+    logging.info("   Episode meta: %s", episode_meta)
+
+    video_path = download_video_file(repo_id, local_path, episode_meta["video_rel"])
+
+    progress_data = load_progress_data(local_path, episode)
+    if progress_data is None:
+        logging.error("Could not load sarm_progress data. Skipping overlay.")
+        return None
+
+    logging.info("   Progress frames: %d", len(progress_data))
+
+    output_path = output_dir / f"{safe_name}_ep{episode}_progress.mp4"
+    final_path = composite_progress_video(
+        video_path=video_path,
+        from_timestamp=episode_meta["from_ts"],
+        to_timestamp=episode_meta["to_ts"],
+        progress_data=progress_data,
+        output_path=output_path,
+        fps=episode_meta["fps"],
+        task_name=episode_meta.get("task_name", ""),
+    )
+
+    if create_gif:
+        final_path = convert_mp4_to_gif(final_path)
+
+    logging.info("Done: %s", final_path)
+    return final_path
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Create MP4/GIF videos with sarm_progress overlay for dataset episodes."
+    )
+    parser.add_argument(
+        "--repo-id",
+        type=str,
+        required=True,
+        help="HuggingFace dataset repository ID (e.g. 'lerobot-data-collection/level2_final_quality3').",
+    )
+    parser.add_argument(
+        "--episode",
+        type=int,
+        required=True,
+        help="Episode index to visualize.",
+    )
+    parser.add_argument(
+        "--camera-key",
+        type=str,
+        default=None,
+        help="Camera observation key (e.g. 'observation.images.base'). Auto-selects first camera if omitted.",
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=Path,
+        default=Path("progress_videos"),
+        help="Directory to write output files (default: ./progress_videos).",
+    )
+    parser.add_argument(
+        "--gif",
+        action="store_true",
+        help="Also generate a GIF from the MP4 output.",
+    )
+    args = parser.parse_args()
+
+    logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
+
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+
+    result = process_dataset(
+        repo_id=args.repo_id,
+        episode=args.episode,
+        camera_key=args.camera_key,
+        output_dir=args.output_dir,
+        create_gif=args.gif,
+    )
+
+    if result:
+        logging.info("Output: %s", result)
+
+
+if __name__ == "__main__":
+    main()

examples/hil/hil_utils.py

@@ -0,0 +1,228 @@
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Shared utilities for Human-in-the-Loop data collection scripts."""
+
+import logging
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from lerobot.processor import (
+    IdentityProcessorStep,
+    RobotAction,
+    RobotObservation,
+    RobotProcessorPipeline,
+)
+from lerobot.processor.converters import (
+    observation_to_transition,
+    robot_action_observation_to_transition,
+    transition_to_observation,
+    transition_to_robot_action,
+)
+from lerobot.robots import Robot
+from lerobot.teleoperators import Teleoperator
+from lerobot.utils.control_utils import is_headless
+from lerobot.utils.robot_utils import precise_sleep
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class HILDatasetConfig:
+    repo_id: str
+    single_task: str
+    root: str | Path | None = None
+    fps: int = 30
+    episode_time_s: float = 120
+    num_episodes: int = 50
+    video: bool = True
+    push_to_hub: bool = True
+    private: bool = False
+    tags: list[str] | None = None
+    num_image_writer_processes: int = 0
+    num_image_writer_threads_per_camera: int = 4
+    video_encoding_batch_size: int = 1
+    vcodec: str = "auto"
+    streaming_encoding: bool = True
+    encoder_queue_maxsize: int = 30
+    encoder_threads: int | None = None
+    rename_map: dict[str, str] = field(default_factory=dict)
+
+
+def teleop_has_motor_control(teleop: Teleoperator) -> bool:
+    """Check if teleoperator has motor control capabilities."""
+    return all(hasattr(teleop, attr) for attr in ("enable_torque", "disable_torque", "write_goal_positions"))
+
+
+def teleop_disable_torque(teleop: Teleoperator) -> None:
+    """Disable teleop torque if supported."""
+    if hasattr(teleop, "disable_torque"):
+        teleop.disable_torque()
+
+
+def teleop_enable_torque(teleop: Teleoperator) -> None:
+    """Enable teleop torque if supported."""
+    if hasattr(teleop, "enable_torque"):
+        teleop.enable_torque()
+
+
+def teleop_smooth_move_to(teleop: Teleoperator, target_pos: dict, duration_s: float = 2.0, fps: int = 50):
+    """Smoothly move teleop to target position if motor control is available."""
+    if not teleop_has_motor_control(teleop):
+        logger.warning("Teleop does not support motor control - cannot mirror robot position")
+        return
+
+    teleop_enable_torque(teleop)
+    current = teleop.get_action()
+    steps = max(int(duration_s * fps), 1)
+
+    for step in range(steps + 1):
+        t = step / steps
+        interp = {}
+        for k in current:
+            if k in target_pos:
+                interp[k] = current[k] * (1 - t) + target_pos[k] * t
+            else:
+                interp[k] = current[k]
+        teleop.write_goal_positions(interp)
+        time.sleep(1 / fps)
+
+
+def init_keyboard_listener():
+    """Initialize keyboard listener with HIL controls."""
+    events = {
+        "exit_early": False,
+        "rerecord_episode": False,
+        "stop_recording": False,
+        "policy_paused": False,
+        "correction_active": False,
+        "resume_policy": False,
+        "in_reset": False,
+        "start_next_episode": False,
+    }
+
+    if is_headless():
+        logger.warning("Headless environment - keyboard controls unavailable")
+        return None, events
+
+    from pynput import keyboard
+
+    def on_press(key):
+        try:
+            if events["in_reset"]:
+                if key in [keyboard.Key.space, keyboard.Key.right]:
+                    logger.info("[HIL] Starting next episode...")
+                    events["start_next_episode"] = True
+                elif hasattr(key, "char") and key.char == "c":
+                    events["start_next_episode"] = True
+                elif key == keyboard.Key.esc:
+                    logger.info("[HIL] ESC - Stop recording, pushing to hub...")
+                    events["stop_recording"] = True
+                    events["start_next_episode"] = True
+            else:
+                if key == keyboard.Key.space:
+                    if not events["policy_paused"] and not events["correction_active"]:
+                        logger.info("[HIL] PAUSED - Press 'c' to take control or 'p' to resume policy")
+                        events["policy_paused"] = True
+                elif hasattr(key, "char") and key.char == "c":
+                    if events["policy_paused"] and not events["correction_active"]:
+                        logger.info("[HIL] Taking control...")
+                        events["start_next_episode"] = True
+                elif hasattr(key, "char") and key.char == "p":
+                    if events["policy_paused"] or events["correction_active"]:
+                        logger.info("[HIL] Resuming policy...")
+                        events["resume_policy"] = True
+                elif key == keyboard.Key.right:
+                    logger.info("[HIL] End episode")
+                    events["exit_early"] = True
+                elif key == keyboard.Key.left:
+                    logger.info("[HIL] Re-record episode")
+                    events["rerecord_episode"] = True
+                    events["exit_early"] = True
+                elif key == keyboard.Key.esc:
+                    logger.info("[HIL] ESC - Stop recording...")
+                    events["stop_recording"] = True
+                    events["exit_early"] = True
+        except Exception as e:
+            logger.info(f"Key error: {e}")
+
+    listener = keyboard.Listener(on_press=on_press)
+    listener.start()
+    return listener, events
+
+
+def make_identity_processors():
+    """Create identity processors for recording."""
+    teleop_proc = RobotProcessorPipeline[tuple[RobotAction, RobotObservation], RobotAction](
+        steps=[IdentityProcessorStep()],
+        to_transition=robot_action_observation_to_transition,
+        to_output=transition_to_robot_action,
+    )
+    obs_proc = RobotProcessorPipeline[RobotObservation, RobotObservation](
+        steps=[IdentityProcessorStep()],
+        to_transition=observation_to_transition,
+        to_output=transition_to_observation,
+    )
+    return teleop_proc, obs_proc
+
+
+def reset_loop(robot: Robot, teleop: Teleoperator, events: dict, fps: int):
+    """Reset period where human repositions environment."""
+    logger.info("[HIL] RESET")
+
+    events["in_reset"] = True
+    events["start_next_episode"] = False
+
+    obs = robot.get_observation()
+    robot_pos = {k: v for k, v in obs.items() if k.endswith(".pos") and k in robot.observation_features}
+    teleop_smooth_move_to(teleop, robot_pos, duration_s=2.0, fps=50)
+
+    logger.info("Press any key to enable teleoperation")
+    while not events["start_next_episode"] and not events["stop_recording"]:
+        precise_sleep(0.05)
+
+    if events["stop_recording"]:
+        return
+
+    events["start_next_episode"] = False
+    teleop_disable_torque(teleop)
+    logger.info("Teleop enabled - press any key to start episode")
+
+    while not events["start_next_episode"] and not events["stop_recording"]:
+        loop_start = time.perf_counter()
+        action = teleop.get_action()
+        robot.send_action(action)
+        precise_sleep(1 / fps - (time.perf_counter() - loop_start))
+
+    events["in_reset"] = False
+    events["start_next_episode"] = False
+    events["exit_early"] = False
+    events["policy_paused"] = False
+    events["correction_active"] = False
+    events["resume_policy"] = False
+
+
+def print_controls(rtc: bool = False):
+    """Print control instructions."""
+    mode = "Human-in-the-Loop Data Collection" + (" (RTC)" if rtc else "")
+    logger.info(
+        "%s\n  Controls:\n"
+        "    SPACE  - Pause policy\n"
+        "    c      - Take control\n"
+        "    p      - Resume policy after pause/correction\n"
+        "    →      - End episode\n"
+        "    ESC    - Stop and push to hub",
+        mode,
+    )

examples/rtc/eval_with_real_robot.py

@@ -69,15 +69,20 @@ Usage:
         --policy.path=lerobot-data-collection/folding_final \
         --robot.type=bi_openarm_follower \
         --robot.cameras='{left_wrist: {type: opencv, index_or_path: "/dev/video4", width: 1280, height: 720, fps: 30}, base: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}, right_wrist: {type: opencv, index_or_path: "/dev/video0", width: 1280, height: 720, fps: 30}}' \
-        --robot.left_arm_config.port=can1 \
+        --robot.left_arm_config.port=can0 \
         --robot.left_arm_config.side=left \
         --robot.left_arm_config.can_interface=socketcan \
-        --robot.right_arm_config.port=can0 \
+        --robot.left_arm_config.disable_torque_on_disconnect=true \
+        --robot.left_arm_config.max_relative_target=8.0 \
+        --robot.right_arm_config.port=can1 \
         --robot.right_arm_config.side=right \
         --robot.right_arm_config.can_interface=socketcan \
+        --robot.right_arm_config.disable_torque_on_disconnect=true \
+        --robot.right_arm_config.max_relative_target=8.0 \
         --task="Fold the T-shirt properly" \
         --fps=30 \
         --duration=2000 \
+        --interpolation_multiplier=3 \
         --rtc.enabled=true \
         --rtc.execution_horizon=20 \
         --rtc.max_guidance_weight=5.0 \
@@ -104,9 +109,7 @@ from lerobot.configs.policies import PreTrainedConfig
 from lerobot.configs.types import RTCAttentionSchedule
 from lerobot.datasets.feature_utils import build_dataset_frame, hw_to_dataset_features
 from lerobot.policies.factory import get_policy_class, make_pre_post_processors
-from lerobot.policies.rtc.action_queue import ActionQueue
-from lerobot.policies.rtc.configuration_rtc import RTCConfig
-from lerobot.policies.rtc.latency_tracker import LatencyTracker
+from lerobot.policies.rtc import ActionInterpolator, ActionQueue, LatencyTracker, RTCConfig
 from lerobot.processor import (
     NormalizerProcessorStep,
     RelativeActionsProcessorStep,
@@ -181,6 +184,7 @@ class RTCDemoConfig(HubMixin):
     # Demo parameters
     duration: float = 30.0  # Duration to run the demo (seconds)
     fps: float = 10.0  # Action execution frequency (Hz)
+    interpolation_multiplier: int = 1  # Control rate multiplier (1=off, 2=2x, 3=3x)
 
     # Compute device
     device: str | None = None  # Device to run on (cuda, cpu, auto)
@@ -461,20 +465,23 @@ def actor_control(
         action_keys = [k for k in robot.action_features() if k.endswith(".pos")]
 
         action_count = 0
-        action_interval = 1.0 / cfg.fps
+        interpolator = ActionInterpolator(multiplier=cfg.interpolation_multiplier)
+        action_interval = interpolator.get_control_interval(cfg.fps)
 
         while not shutdown_event.is_set():
             start_time = time.perf_counter()
 
-            # Try to get an action from the queue with timeout
-            action = action_queue.get()
+            if interpolator.needs_new_action():
+                new_action = action_queue.get()
+                if new_action is not None:
+                    interpolator.add(new_action.cpu())
 
+            action = interpolator.get()
             if action is not None:
                 action = action.cpu()
                 action_dict = {key: action[i].item() for i, key in enumerate(action_keys)}
                 action_processed = robot_action_processor((action_dict, None))
                 robot.send_action(action_processed)
-
                 action_count += 1
 
             dt_s = time.perf_counter() - start_time

examples/umi_pi0_relative_ee/evaluate.py

@@ -1,297 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-Inference script for a pi0 model trained with UMI-style relative EE actions
-on an OpenArm robot (single right arm, one wrist camera).
-
-Training dataset layout:
-  observation.images.cam0  [3, 720, 960]
-  action                   [x, y, z, ax, ay, az, proximal, distal]  (shape 8)
-
-The model uses ``derive_state_from_action=true``, so observation.state is
-derived from the action column during training.  At inference the state must
-be provided by the robot — this script uses FK to compute the current EE
-pose and gripper position, which it exposes as ``observation.state``.
-
-Pipeline:
-  1. Read arm joints from robot → FK → observation.state [x,y,z,ax,ay,az,prox,dist]
-  2. Read camera image → observation.images.cam0
-  3. pi0 preprocessor (loaded from checkpoint):
-     - DeriveStateFromActionStep: no-op at inference (state from robot)
-     - RelativeActionsProcessorStep: caches current state
-     - RelativeStateProcessorStep: buffers prev state, stacks [prev,cur],
-       subtracts current → velocity info, flattens
-     - NormalizerProcessorStep: normalizes
-  4. pi0 predicts relative action chunk (30 steps)
-  5. pi0 postprocessor: unnormalize, add cached state → absolute EE
-  6. IK: absolute EE [x,y,z,ax,ay,az] → arm joint targets
-  7. Gripper [proximal, distal] → gripper motor targets
-  8. Send to robot
-
-Usage:
-    python evaluate.py
-"""
-
-from __future__ import annotations
-
-import numpy as np
-from scipy.spatial.transform import Rotation
-
-from lerobot.cameras.opencv.configuration_opencv import OpenCVCameraConfig
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-from lerobot.model.kinematics import RobotKinematics
-from lerobot.policies.factory import make_pre_post_processors
-from lerobot.policies.pi0.modeling_pi0 import PI0Policy
-from lerobot.processor import RelativeStateProcessorStep
-from lerobot.robots.openarm_follower import OpenArmFollower, OpenArmFollowerConfig
-from lerobot.scripts.lerobot_record import record_loop
-from lerobot.types import RobotAction, RobotObservation
-from lerobot.utils.control_utils import init_keyboard_listener
-from lerobot.utils.utils import log_say
-from lerobot.utils.visualization_utils import init_rerun
-
-# ---------------------------------------------------------------------------
-# Configuration — adapt these to your setup
-# ---------------------------------------------------------------------------
-
-FPS = 46
-EPISODE_TIME_SEC = 60
-TASK_DESCRIPTION = "red cube"
-
-HF_MODEL_ID = "pepijn223/grabette-umi-pi0"
-
-# Latency compensation: skip this many predicted action steps to account for
-# camera + inference + execution latency.  Formula: ceil(total_ms / (1000/FPS)).
-# At 46 FPS (~22ms/step) with ~150ms total latency: ceil(150/22) ≈ 7.
-# Start with 0 for a safe first test, then increase to match measured latency.
-LATENCY_SKIP_STEPS = 0
-
-URDF_PATH = "src/lerobot/robots/openarm_follower/urdf/openarm_bimanual_pybullet.urdf"
-URDF_EE_FRAME = "openarm_right_ee_target"
-
-IK_POSITION_WEIGHT = 1.0
-IK_ORIENTATION_WEIGHT = 1.0
-
-# ---------------------------------------------------------------------------
-# Dataset features for inference
-#
-# The training dataset has only observation.images.cam0 and action.
-# observation.state is derived from action during training
-# (derive_state_from_action=true) but must be supplied by the robot at
-# inference.  We define it here so build_dataset_frame can map FK output
-# to the right feature.
-# ---------------------------------------------------------------------------
-
-DATASET_FEATURES: dict = {
-    "observation.state": {
-        "dtype": "float32",
-        "shape": [8],
-        "names": ["x", "y", "z", "ax", "ay", "az", "proximal", "distal"],
-    },
-    "observation.images.cam0": {
-        "dtype": "video",
-        "shape": [3, 720, 960],
-        "names": ["channels", "height", "width"],
-        "info": {
-            "video.height": 720,
-            "video.width": 960,
-            "video.codec": "h264",
-            "video.pix_fmt": "yuv420p",
-            "video.is_depth_map": False,
-            "video.fps": FPS,
-            "video.channels": 3,
-            "has_audio": False,
-        },
-    },
-    "action": {
-        "dtype": "float32",
-        "shape": [8],
-        "names": ["x", "y", "z", "ax", "ay", "az", "proximal", "distal"],
-    },
-    "timestamp": {"dtype": "float32", "shape": [1], "names": None},
-    "frame_index": {"dtype": "int64", "shape": [1], "names": None},
-    "episode_index": {"dtype": "int64", "shape": [1], "names": None},
-    "index": {"dtype": "int64", "shape": [1], "names": None},
-    "task_index": {"dtype": "int64", "shape": [1], "names": None},
-}
-
-
-# ---------------------------------------------------------------------------
-# FK / IK callables
-# ---------------------------------------------------------------------------
-
-
-class JointsToEE:
-    """FK: raw robot observation → flat dict matching observation.state names.
-
-    Arm joint positions → EE pose [x,y,z,ax,ay,az] via forward kinematics.
-    Gripper motor positions → [proximal, distal].
-    Camera images pass through unchanged.
-    """
-
-    def __init__(self, kinematics: RobotKinematics, arm_motor_names: list[str]):
-        self.kin = kinematics
-        self.arm = arm_motor_names
-
-    def __call__(self, obs: RobotObservation) -> RobotObservation:
-        q = np.array([float(obs[f"{m}.pos"]) for m in self.arm])
-        t = self.kin.forward_kinematics(q)
-        rot = Rotation.from_matrix(t[:3, :3]).as_rotvec()
-
-        out: dict = {
-            "x": float(t[0, 3]),
-            "y": float(t[1, 3]),
-            "z": float(t[2, 3]),
-            "ax": float(rot[0]),
-            "ay": float(rot[1]),
-            "az": float(rot[2]),
-            "proximal": float(obs["proximal.pos"]),
-            "distal": float(obs["distal.pos"]),
-        }
-        for k, v in obs.items():
-            if not k.endswith((".pos", ".vel", ".torque")):
-                out[k] = v
-        return out
-
-
-class EEToJoints:
-    """IK: policy action dict → motor position dict for the robot.
-
-    Reads [x,y,z,ax,ay,az] from the action, runs IK for arm joint targets.
-    Passes [proximal, distal] as direct gripper position commands.
-    """
-
-    def __init__(
-        self,
-        kinematics: RobotKinematics,
-        arm_motor_names: list[str],
-        position_weight: float = 1.0,
-        orientation_weight: float = 1.0,
-    ):
-        self.kin = kinematics
-        self.arm = arm_motor_names
-        self.pw = position_weight
-        self.ow = orientation_weight
-        self.q_curr: np.ndarray | None = None
-
-    def __call__(self, args: tuple[RobotAction, RobotObservation]) -> RobotAction:
-        action, obs = args
-
-        q_raw = np.array([float(obs[f"{m}.pos"]) for m in self.arm])
-        if self.q_curr is None:
-            self.q_curr = q_raw
-
-        t_des = np.eye(4)
-        t_des[:3, :3] = Rotation.from_rotvec([action["ax"], action["ay"], action["az"]]).as_matrix()
-        t_des[:3, 3] = [action["x"], action["y"], action["z"]]
-
-        q_target = self.kin.inverse_kinematics(
-            self.q_curr, t_des, position_weight=self.pw, orientation_weight=self.ow
-        )
-        self.q_curr = q_target
-
-        out: dict = {f"{m}.pos": float(q_target[i]) for i, m in enumerate(self.arm)}
-        out["proximal.pos"] = float(action["proximal"])
-        out["distal.pos"] = float(action["distal"])
-        return out
-
-
-# ---------------------------------------------------------------------------
-# Main
-# ---------------------------------------------------------------------------
-
-
-def main():
-    camera_config = {
-        "cam0": OpenCVCameraConfig(index_or_path=0, width=960, height=720, fps=FPS),
-    }
-    robot_config = OpenArmFollowerConfig(
-        port="can0",
-        id="right_openarm",
-        side="right",
-        cameras=camera_config,
-        max_relative_target=8.0,
-        gripper_port="/dev/ttyUSB0",
-    )
-    robot = OpenArmFollower(robot_config)
-
-    policy = PI0Policy.from_pretrained(HF_MODEL_ID)
-    policy.config.latency_skip_steps = LATENCY_SKIP_STEPS
-
-    arm_motor_names = list(robot.bus.motors.keys())
-
-    kinematics = RobotKinematics(
-        urdf_path=URDF_PATH,
-        target_frame_name=URDF_EE_FRAME,
-        joint_names=arm_motor_names,
-    )
-
-    fk = JointsToEE(kinematics, arm_motor_names)
-    ik = EEToJoints(kinematics, arm_motor_names, IK_POSITION_WEIGHT, IK_ORIENTATION_WEIGHT)
-
-    dataset = LeRobotDataset.create(
-        repo_id="tmp/openarm_eval_scratch",
-        fps=FPS,
-        features=DATASET_FEATURES,
-        robot_type=robot.name,
-        use_videos=True,
-        image_writer_threads=4,
-    )
-
-    preprocessor, postprocessor = make_pre_post_processors(
-        policy_cfg=policy,
-        pretrained_path=HF_MODEL_ID,
-        dataset_stats=dataset.meta.stats,
-        preprocessor_overrides={"device_processor": {"device": str(policy.config.device)}},
-    )
-
-    relative_state_steps = [s for s in preprocessor.steps if isinstance(s, RelativeStateProcessorStep)]
-
-    robot.connect()
-
-    listener, events = init_keyboard_listener()
-    init_rerun(session_name="openarm_umi_pi0_relative_ee_evaluate")
-
-    try:
-        if not robot.is_connected:
-            raise ValueError("Robot is not connected!")
-
-        log_say("Starting policy execution")
-        for step in relative_state_steps:
-            step.reset()
-
-        record_loop(
-            robot=robot,
-            events=events,
-            fps=FPS,
-            policy=policy,
-            preprocessor=preprocessor,
-            postprocessor=postprocessor,
-            dataset=dataset,
-            control_time_s=EPISODE_TIME_SEC,
-            single_task=TASK_DESCRIPTION,
-            display_data=True,
-            robot_action_processor=ik,
-            robot_observation_processor=fk,
-        )
-    finally:
-        robot.disconnect()
-        listener.stop()
-
-
-if __name__ == "__main__":
-    main()

examples/umi_pi0_relative_ee/replay.py

@@ -1,113 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-Replay a dataset episode in EE frame using a browser-based URDF viewer.
-
-Extracts ``observation.pose`` from the dataset, saves a trajectory JSON file,
-then launches a local HTTP server and opens the replay viewer.  The trajectory
-is re-centered so frame 0 starts at the OpenArm ``openarm_right_ee_target``
-EE tip (zero-joint pose).
-
-Usage:
-    python replay.py
-    python replay.py --episode 3 --repo-id myuser/mydata
-"""
-
-from __future__ import annotations
-
-import argparse
-import http.server
-import json
-import os
-import threading
-import webbrowser
-from pathlib import Path
-
-VIEWER_DIR = Path(__file__).resolve().parents[2] / "src/lerobot/robots/openarm_follower/urdf"
-TRAJECTORY_FILENAME = "trajectory_ep0.json"
-
-
-def extract_trajectory(repo_id: str, episode: int, output_path: Path) -> dict:
-    from lerobot.datasets.lerobot_dataset import LeRobotDataset
-
-    dataset = LeRobotDataset(repo_id, episodes=[episode])
-    poses = dataset.select_columns("observation.pose")
-    actions = dataset.select_columns("action")
-
-    frames = []
-    for i in range(dataset.num_frames):
-        p = poses[i]["observation.pose"]
-        a = actions[i]["action"]
-        frames.append(
-            {
-                "x": float(p[0]),
-                "y": float(p[1]),
-                "z": float(p[2]),
-                "ax": float(p[3]),
-                "ay": float(p[4]),
-                "az": float(p[5]),
-                "proximal": float(a[0]),
-                "distal": float(a[1]),
-            }
-        )
-    payload = {"fps": dataset.fps, "num_frames": dataset.num_frames, "frames": frames}
-    with open(output_path, "w") as f:
-        json.dump(payload, f)
-    print(f"Extracted {dataset.num_frames} frames at {dataset.fps} FPS → {output_path}")
-    return payload
-
-
-# ---------------------------------------------------------------------------
-# Viewer mode
-# ---------------------------------------------------------------------------
-
-
-def serve_and_open(directory: Path, port: int = 8765):
-    os.chdir(directory)
-    handler = http.server.SimpleHTTPRequestHandler
-    httpd = http.server.HTTPServer(("", port), handler)
-    url = f"http://localhost:{port}/replay_viewer.html"
-    print(f"Serving at {url}")
-    threading.Thread(target=lambda: webbrowser.open(url), daemon=True).start()
-    try:
-        httpd.serve_forever()
-    except KeyboardInterrupt:
-        print("\nServer stopped.")
-        httpd.server_close()
-
-
-def run_viewer(args):
-    trajectory_path = VIEWER_DIR / TRAJECTORY_FILENAME
-    if not trajectory_path.exists() or args.force:
-        extract_trajectory(args.repo_id, args.episode, trajectory_path)
-    else:
-        print(f"Using cached trajectory at {trajectory_path}  (pass --force to re-extract)")
-    serve_and_open(VIEWER_DIR, args.port)
-
-
-def main():
-    parser = argparse.ArgumentParser(description="Replay a dataset episode in EE frame (URDF viewer)")
-    parser.add_argument("--repo-id", default="glannuzel/grabette-dataset")
-    parser.add_argument("--episode", type=int, default=0)
-    parser.add_argument("--port", type=int, default=8765)
-    parser.add_argument("--force", action="store_true", help="Re-extract trajectory even if cached")
-    args = parser.parse_args()
-    run_viewer(args)
-
-
-if __name__ == "__main__":
-    main()

pyproject.toml

@@ -220,6 +220,8 @@ lerobot-replay="lerobot.scripts.lerobot_replay:main"
 lerobot-setup-motors="lerobot.scripts.lerobot_setup_motors:main"
 lerobot-teleoperate="lerobot.scripts.lerobot_teleoperate:main"
 lerobot-eval="lerobot.scripts.lerobot_eval:main"
+lerobot-eval-parallel="lerobot.scripts.lerobot_eval_parallel:main"
+lerobot-eval-autotune="lerobot.scripts.lerobot_eval_autotune:main"
 lerobot-train="lerobot.scripts.lerobot_train:main"
 lerobot-train-tokenizer="lerobot.scripts.lerobot_train_tokenizer:main"
 lerobot-dataset-viz="lerobot.scripts.lerobot_dataset_viz:main"
@@ -306,8 +308,7 @@ default.extend-ignore-identifiers-re = [
     "thw",
     "inpt",
     "ROBOTIS",
-    "OT_VALUE",
-    "metalness",
+    "OT_VALUE"
 ]
 
 # TODO: Uncomment when ready to use

src/lerobot/configs/default.py

@@ -67,7 +67,13 @@ class EvalConfig:
     # `batch_size` specifies the number of environments to use in a gym.vector.VectorEnv.
     batch_size: int = 50
     # `use_async_envs` specifies whether to use asynchronous environments (multiprocessing).
-    use_async_envs: bool = False
+    # Defaults to True; automatically downgraded to SyncVectorEnv when batch_size=1.
+    use_async_envs: bool = True
+    # Sharding: split n_episodes across independent processes.
+    # shard_id=0, num_shards=1 is the default (no sharding, existing behaviour).
+    # Set via lerobot_eval_parallel or manually: --eval.shard_id=K --eval.num_shards=N
+    shard_id: int = 0
+    num_shards: int = 1
 
     def __post_init__(self) -> None:
         if self.batch_size > self.n_episodes:
@@ -79,6 +85,12 @@ class EvalConfig:
                 f"to increase the number of episodes to match the batch size (e.g. `eval.n_episodes={self.batch_size}`), "
                 f"or lower the batch size (e.g. `eval.batch_size={self.n_episodes}`)."
             )
+        if self.num_shards < 1:
+            raise ValueError(f"`num_shards` must be >= 1, got {self.num_shards}")
+        if not (0 <= self.shard_id < self.num_shards):
+            raise ValueError(
+                f"`shard_id` must be in [0, num_shards), got shard_id={self.shard_id}, num_shards={self.num_shards}"
+            )
 
 
 @dataclass

src/lerobot/configs/policies.py

@@ -115,17 +115,6 @@ class PreTrainedConfig(draccus.ChoiceRegistry, HubMixin, abc.ABC):  # type: igno
     def reward_delta_indices(self) -> list | None:  # type: ignore[type-arg]    #TODO: No implementation
         raise NotImplementedError
 
-    @property
-    def state_delta_indices(self) -> list | None:  # type: ignore[type-arg]
-        """Delta indices specifically for observation.state.
-
-        When not None, overrides ``observation_delta_indices`` for the
-        ``observation.state`` key only. Useful for loading state history
-        (e.g. ``[-1, 0]`` for UMI-style relative proprioception) without
-        also loading multiple image timesteps.
-        """
-        return None
-
     @abc.abstractmethod
     def get_optimizer_preset(self) -> OptimizerConfig:
         raise NotImplementedError

src/lerobot/datasets/compute_stats.py

@@ -767,94 +767,3 @@ def compute_relative_action_stats(
     )
 
     return stats
-
-
-def compute_relative_state_stats(
-    hf_dataset,
-    features: dict,
-    state_obs_steps: int = 2,
-    exclude_joints: list[str] | None = None,
-    source_key: str = OBS_STATE,
-) -> dict[str, np.ndarray]:
-    """Compute normalization statistics for observation.state after relative conversion.
-
-    For UMI-style relative proprioception with ``state_obs_steps`` timesteps,
-    each state observation becomes a stack of offsets from the current timestep:
-    ``state[t-k] - state[t]`` for k in ``range(state_obs_steps-1, -1, -1)``.
-
-    The stats are computed over the flattened ``[state_obs_steps * state_dim]``
-    vector that the model actually sees after ``prepare_state`` flattening.
-
-    Args:
-        hf_dataset: The HuggingFace dataset with the source column and
-            "episode_index" columns.
-        features: Dataset feature metadata.
-        state_obs_steps: Number of observation timesteps (must be >= 2).
-        exclude_joints: State dimension names to keep absolute.
-        source_key: Column to read data from. Defaults to "observation.state".
-            When ``derive_state_from_action=True``, pass ``ACTION`` to read
-            from the action column instead.
-
-    Returns:
-        Statistics dict with keys "mean", "std", "min", "max", "q01", …, "q99".
-    """
-    from lerobot.processor.relative_action_processor import RelativeStateProcessorStep
-
-    if exclude_joints is None:
-        exclude_joints = []
-
-    state_dim = features[source_key]["shape"][0]
-    state_names = features.get(source_key, {}).get("names")
-    mask_step = RelativeStateProcessorStep(
-        enabled=True,
-        exclude_joints=exclude_joints,
-        state_names=state_names,
-    )
-    relative_mask = np.array(mask_step._build_mask(state_dim), dtype=np.float32)
-
-    logging.info(f"Loading data from '{source_key}' for relative state stats...")
-    all_states = np.array(hf_dataset[source_key], dtype=np.float32)
-    episode_indices = np.array(hf_dataset["episode_index"])
-
-    # Build all valid windows of length state_obs_steps within each episode
-    n = len(all_states)
-    if n < state_obs_steps:
-        raise ValueError(f"Dataset has {n} frames but state_obs_steps={state_obs_steps}")
-
-    max_start = n - state_obs_steps
-    starts = np.arange(max_start + 1)
-    valid = episode_indices[starts] == episode_indices[starts + state_obs_steps - 1]
-    valid_starts = starts[valid]
-
-    if len(valid_starts) == 0:
-        raise RuntimeError("No valid state windows found within single episodes")
-
-    offsets = np.arange(state_obs_steps)
-    mask_dim = len(relative_mask)
-
-    running_stats = RunningQuantileStats()
-
-    batch_size = 50_000
-    for i in range(0, len(valid_starts), batch_size):
-        batch_starts = valid_starts[i : i + batch_size]
-        frame_idx = batch_starts[:, None] + offsets[None, :]  # [N, state_obs_steps]
-        windows = all_states[frame_idx].copy()  # [N, state_obs_steps, state_dim]
-
-        # Subtract current (last) timestep from all timesteps for masked dims
-        current = windows[:, -1:, :]  # [N, 1, state_dim]
-        windows[:, :, :mask_dim] -= current[:, :, :mask_dim] * relative_mask[None, None, :]
-
-        # Flatten to [N, state_obs_steps * state_dim] (same as prepare_state)
-        flattened = windows.reshape(len(batch_starts), -1)
-        running_stats.update(flattened)
-
-    stats = running_stats.get_statistics()
-
-    excluded_dims = int(mask_dim - relative_mask.sum())
-    logging.info(
-        f"Relative state stats ({len(valid_starts)} windows, obs_steps={state_obs_steps}): "
-        f"relative_dims={int(relative_mask.sum())}/{mask_dim} (excluded={excluded_dims}), "
-        f"mean={np.abs(stats['mean']).mean():.4f}, std={stats['std'].mean():.4f}"
-    )
-
-    return stats

src/lerobot/datasets/dataset_tools.py

@@ -41,7 +41,6 @@ from lerobot.datasets.compute_stats import (
     aggregate_stats,
     compute_episode_stats,
     compute_relative_action_stats,
-    compute_relative_state_stats,
 )
 from lerobot.datasets.dataset_metadata import LeRobotDatasetMetadata
 from lerobot.datasets.io_utils import (
@@ -1545,10 +1544,6 @@ def recompute_stats(
     relative_exclude_joints: list[str] | None = None,
     chunk_size: int = 50,
     num_workers: int = 0,
-    relative_state: bool = False,
-    relative_exclude_state_joints: list[str] | None = None,
-    state_obs_steps: int = 2,
-    derive_state_from_action: bool = False,
 ) -> LeRobotDataset:
     """Recompute stats.json from scratch by iterating all episodes.
 
@@ -1566,22 +1561,10 @@ def recompute_stats(
             ``policy.chunk_size``. Only used when ``relative_action=True``.
         num_workers: Number of parallel threads for relative action stats computation.
             Values ≤1 mean single-threaded. Only used when ``relative_action=True``.
-        relative_state: If True, compute observation.state stats in relative space
-            (multi-timestep offsets from current). This matches the normalization
-            the model sees during training with ``use_relative_state=True``.
-        relative_exclude_state_joints: State dim names to exclude from relative conversion.
-        state_obs_steps: Number of observation timesteps for relative state stats.
-            Should match ``policy.state_obs_steps``. Only used when ``relative_state=True``.
-        derive_state_from_action: If True, compute relative state stats from the
-            action column instead of observation.state. Implies ``relative_state=True``
-            and ``state_obs_steps=2``.
 
     Returns:
         The same dataset with updated stats.
     """
-    if derive_state_from_action:
-        relative_state = True
-        state_obs_steps = 2
     features = dataset.meta.features
     meta_keys = {"index", "episode_index", "task_index", "frame_index", "timestamp"}
     numeric_features = {
@@ -1613,20 +1596,6 @@ def recompute_stats(
         )
         features_to_compute.pop(ACTION, None)
 
-    # When relative_state is enabled, compute state stats over the flattened
-    # multi-timestep relative representation (matching what the model sees).
-    relative_state_stats = None
-    if relative_state and (OBS_STATE in features or derive_state_from_action):
-        source_key = ACTION if derive_state_from_action else OBS_STATE
-        relative_state_stats = compute_relative_state_stats(
-            hf_dataset=dataset.hf_dataset,
-            features=features,
-            state_obs_steps=state_obs_steps,
-            exclude_joints=relative_exclude_state_joints,
-            source_key=source_key,
-        )
-        features_to_compute.pop(OBS_STATE, None)
-
     logging.info(f"Recomputing stats for features: {list(features_to_compute.keys())}")
 
     data_dir = dataset.root / DATA_DIR
@@ -1663,9 +1632,6 @@ def recompute_stats(
     if relative_action_stats is not None:
         new_stats[ACTION] = relative_action_stats
 
-    if relative_state_stats is not None:
-        new_stats[OBS_STATE] = relative_state_stats
-
     # Merge: keep existing stats for features we didn't recompute
     if dataset.meta.stats:
         for key, value in dataset.meta.stats.items():

src/lerobot/datasets/factory.py

@@ -25,7 +25,7 @@ from lerobot.datasets.lerobot_dataset import LeRobotDataset
 from lerobot.datasets.multi_dataset import MultiLeRobotDataset
 from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset
 from lerobot.datasets.transforms import ImageTransforms
-from lerobot.utils.constants import ACTION, OBS_PREFIX, OBS_STATE, REWARD
+from lerobot.utils.constants import ACTION, OBS_PREFIX, REWARD
 
 IMAGENET_STATS = {
     "mean": [[[0.485]], [[0.456]], [[0.406]]],  # (c,1,1)
@@ -52,15 +52,12 @@ def resolve_delta_timestamps(
             returns `None` if the resulting dict is empty.
     """
     delta_timestamps = {}
-    state_delta = getattr(cfg, "state_delta_indices", None)
     for key in ds_meta.features:
         if key == REWARD and cfg.reward_delta_indices is not None:
             delta_timestamps[key] = [i / ds_meta.fps for i in cfg.reward_delta_indices]
         if key == ACTION and cfg.action_delta_indices is not None:
             delta_timestamps[key] = [i / ds_meta.fps for i in cfg.action_delta_indices]
-        if key == OBS_STATE and state_delta is not None:
-            delta_timestamps[key] = [i / ds_meta.fps for i in state_delta]
-        elif key.startswith(OBS_PREFIX) and cfg.observation_delta_indices is not None:
+        if key.startswith(OBS_PREFIX) and cfg.observation_delta_indices is not None:
             delta_timestamps[key] = [i / ds_meta.fps for i in cfg.observation_delta_indices]
 
     if len(delta_timestamps) == 0:

src/lerobot/envs/configs.py

@@ -12,11 +12,16 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import annotations
+
 import abc
+import importlib
 from dataclasses import dataclass, field, fields
 from typing import Any
 
 import draccus
+import gymnasium as gym
+from gymnasium.envs.registration import registry as gym_registry
 
 from lerobot.configs.types import FeatureType, PolicyFeature
 from lerobot.robots import RobotConfig
@@ -39,6 +44,13 @@ from lerobot.utils.constants import (
 )
 
 
+def _make_vec_env_cls(use_async: bool, n_envs: int):
+    """Return the right VectorEnv constructor."""
+    if use_async and n_envs > 1:
+        return gym.vector.AsyncVectorEnv
+    return gym.vector.SyncVectorEnv
+
+
 @dataclass
 class EnvConfig(draccus.ChoiceRegistry, abc.ABC):
     task: str | None = None
@@ -67,6 +79,50 @@ class EnvConfig(draccus.ChoiceRegistry, abc.ABC):
     def gym_kwargs(self) -> dict:
         raise NotImplementedError()
 
+    def create_envs(
+        self,
+        n_envs: int,
+        use_async_envs: bool = True,
+    ) -> dict[str, dict[int, gym.vector.VectorEnv]]:
+        """Create {suite: {task_id: VectorEnv}}.
+
+        Default: single-task env via gym.make(). Multi-task benchmarks override.
+        AsyncVectorEnv is the default for n_envs > 1; auto-downgraded to Sync for n_envs=1.
+        """
+        env_cls = gym.vector.AsyncVectorEnv if (use_async_envs and n_envs > 1) else gym.vector.SyncVectorEnv
+
+        if self.gym_id not in gym_registry:
+            print(f"gym id '{self.gym_id}' not found, attempting to import '{self.package_name}'...")
+            try:
+                importlib.import_module(self.package_name)
+            except ModuleNotFoundError as e:
+                raise ModuleNotFoundError(
+                    f"Package '{self.package_name}' required for env '{self.type}' not found. "
+                    f"Please install it or check PYTHONPATH."
+                ) from e
+
+            if self.gym_id not in gym_registry:
+                raise gym.error.NameNotFound(
+                    f"Environment '{self.gym_id}' not registered even after importing '{self.package_name}'."
+                )
+
+        def _make_one():
+            return gym.make(self.gym_id, disable_env_checker=self.disable_env_checker, **self.gym_kwargs)
+
+        try:
+            from gymnasium.vector import AutoresetMode
+
+            vec = env_cls([_make_one for _ in range(n_envs)], autoreset_mode=AutoresetMode.SAME_STEP)
+        except ImportError:
+            vec = env_cls([_make_one for _ in range(n_envs)])
+        return {self.type: {0: vec}}
+
+    def get_env_processors(self):
+        """Return (preprocessor, postprocessor) for this env. Default: identity."""
+        from lerobot.processor.pipeline import PolicyProcessorPipeline
+
+        return PolicyProcessorPipeline(steps=[]), PolicyProcessorPipeline(steps=[])
+
 
 @dataclass
 class HubEnvConfig(EnvConfig):
@@ -338,6 +394,12 @@ class LiberoEnv(EnvConfig):
         else:
             raise ValueError(f"Unsupported obs_type: {self.obs_type}")
 
+        if self.camera_name_mapping is not None:
+            mapped_agentview = self.camera_name_mapping.get("agentview_image", "image")
+            mapped_eye_in_hand = self.camera_name_mapping.get("robot0_eye_in_hand_image", "image2")
+            self.features_map[LIBERO_KEY_PIXELS_AGENTVIEW] = f"{OBS_IMAGES}.{mapped_agentview}"
+            self.features_map[LIBERO_KEY_PIXELS_EYE_IN_HAND] = f"{OBS_IMAGES}.{mapped_eye_in_hand}"
+
     @property
     def gym_kwargs(self) -> dict:
         kwargs: dict[str, Any] = {"obs_type": self.obs_type, "render_mode": self.render_mode}
@@ -345,6 +407,33 @@ class LiberoEnv(EnvConfig):
             kwargs["task_ids"] = self.task_ids
         return kwargs
 
+    def create_envs(self, n_envs: int, use_async_envs: bool = True):
+        from lerobot.envs.libero import create_libero_envs
+
+        if self.task is None:
+            raise ValueError("LiberoEnv requires a task to be specified")
+        env_cls = _make_vec_env_cls(use_async_envs, n_envs)
+        return create_libero_envs(
+            task=self.task,
+            n_envs=n_envs,
+            camera_name=self.camera_name,
+            init_states=self.init_states,
+            gym_kwargs=self.gym_kwargs,
+            env_cls=env_cls,
+            control_mode=self.control_mode,
+            episode_length=self.episode_length,
+            camera_name_mapping=self.camera_name_mapping,
+        )
+
+    def get_env_processors(self):
+        from lerobot.processor.env_processor import LiberoProcessorStep
+        from lerobot.processor.pipeline import PolicyProcessorPipeline
+
+        return (
+            PolicyProcessorPipeline(steps=[LiberoProcessorStep()]),
+            PolicyProcessorPipeline(steps=[]),
+        )
+
 
 @EnvConfig.register_subclass("metaworld")
 @dataclass
@@ -387,6 +476,19 @@ class MetaworldEnv(EnvConfig):
             "render_mode": self.render_mode,
         }
 
+    def create_envs(self, n_envs: int, use_async_envs: bool = True):
+        from lerobot.envs.metaworld import create_metaworld_envs
+
+        if self.task is None:
+            raise ValueError("MetaWorld requires a task to be specified")
+        env_cls = _make_vec_env_cls(use_async_envs, n_envs)
+        return create_metaworld_envs(
+            task=self.task,
+            n_envs=n_envs,
+            gym_kwargs=self.gym_kwargs,
+            env_cls=env_cls,
+        )
+
 
 @EnvConfig.register_subclass("isaaclab_arena")
 @dataclass
@@ -454,3 +556,18 @@ class IsaaclabArenaEnv(HubEnvConfig):
     @property
     def gym_kwargs(self) -> dict:
         return {}
+
+    def get_env_processors(self):
+        from lerobot.processor.env_processor import IsaaclabArenaProcessorStep
+        from lerobot.processor.pipeline import PolicyProcessorPipeline
+
+        state_keys = tuple(k.strip() for k in (self.state_keys or "").split(",") if k.strip())
+        camera_keys = tuple(k.strip() for k in (self.camera_keys or "").split(",") if k.strip())
+        if not state_keys and not camera_keys:
+            raise ValueError("At least one of state_keys or camera_keys must be specified.")
+        return (
+            PolicyProcessorPipeline(
+                steps=[IsaaclabArenaProcessorStep(state_keys=state_keys, camera_keys=camera_keys)]
+            ),
+            PolicyProcessorPipeline(steps=[]),
+        )

src/lerobot/envs/factory.py

@@ -13,96 +13,52 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import importlib
+from __future__ import annotations
+
 from typing import Any
 
 import gymnasium as gym
-from gymnasium.envs.registration import registry as gym_registry
 
-from lerobot.configs.policies import PreTrainedConfig
-from lerobot.envs.configs import AlohaEnv, EnvConfig, HubEnvConfig, IsaaclabArenaEnv, LiberoEnv, PushtEnv
+from lerobot.envs.configs import EnvConfig, HubEnvConfig
 from lerobot.envs.utils import _call_make_env, _download_hub_file, _import_hub_module, _normalize_hub_result
-from lerobot.policies.xvla.configuration_xvla import XVLAConfig
-from lerobot.processor import ProcessorStep
-from lerobot.processor.env_processor import IsaaclabArenaProcessorStep, LiberoProcessorStep
-from lerobot.processor.pipeline import PolicyProcessorPipeline
 
 
 def make_env_config(env_type: str, **kwargs) -> EnvConfig:
-    if env_type == "aloha":
-        return AlohaEnv(**kwargs)
-    elif env_type == "pusht":
-        return PushtEnv(**kwargs)
-    elif env_type == "libero":
-        return LiberoEnv(**kwargs)
-    else:
-        raise ValueError(f"Policy type '{env_type}' is not available.")
+    try:
+        cls = EnvConfig.get_choice_class(env_type)
+    except KeyError as err:
+        raise ValueError(
+            f"Environment type '{env_type}' is not registered. "
+            f"Available: {list(EnvConfig.get_known_choices().keys())}"
+        ) from err
+    return cls(**kwargs)
 
 
 def make_env_pre_post_processors(
     env_cfg: EnvConfig,
-    policy_cfg: PreTrainedConfig,
-) -> tuple[
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-]:
+    policy_cfg: Any,
+) -> tuple[Any, Any]:
     """
     Create preprocessor and postprocessor pipelines for environment observations.
 
-    This function creates processor pipelines that transform raw environment
-    observations and actions. By default, it returns identity processors that do nothing.
-    For specific environments like LIBERO, it adds environment-specific processing steps.
-
-    Args:
-        env_cfg: The configuration of the environment.
-
-    Returns:
-        A tuple containing:
-            - preprocessor: Pipeline that processes environment observations
-            - postprocessor: Pipeline that processes environment outputs (currently identity)
+    Returns a tuple of (preprocessor, postprocessor). By default, delegates to
+    ``env_cfg.get_env_processors()``.  The XVLAConfig policy-specific override
+    stays here because it depends on the *policy* config, not the env config.
     """
-    # Preprocessor and Postprocessor steps are Identity for most environments
-    preprocessor_steps: list[ProcessorStep] = []
-    postprocessor_steps: list[ProcessorStep] = []
+    from lerobot.policies.xvla.configuration_xvla import XVLAConfig
+
     if isinstance(policy_cfg, XVLAConfig):
         from lerobot.policies.xvla.processor_xvla import make_xvla_libero_pre_post_processors
 
         return make_xvla_libero_pre_post_processors()
 
-    # For LIBERO environments, add the LiberoProcessorStep to preprocessor
-    if isinstance(env_cfg, LiberoEnv) or "libero" in env_cfg.type:
-        preprocessor_steps.append(LiberoProcessorStep())
-
-    # For Isaaclab Arena environments, add the IsaaclabArenaProcessorStep
-    if isinstance(env_cfg, IsaaclabArenaEnv) or "isaaclab_arena" in env_cfg.type:
-        # Parse comma-separated keys (handle None for state-based policies)
-        if env_cfg.state_keys:
-            state_keys = tuple(k.strip() for k in env_cfg.state_keys.split(",") if k.strip())
-        else:
-            state_keys = ()
-        if env_cfg.camera_keys:
-            camera_keys = tuple(k.strip() for k in env_cfg.camera_keys.split(",") if k.strip())
-        else:
-            camera_keys = ()
-        if not state_keys and not camera_keys:
-            raise ValueError("At least one of state_keys or camera_keys must be specified.")
-        preprocessor_steps.append(
-            IsaaclabArenaProcessorStep(
-                state_keys=state_keys,
-                camera_keys=camera_keys,
-            )
-        )
-
-    preprocessor = PolicyProcessorPipeline(steps=preprocessor_steps)
-    postprocessor = PolicyProcessorPipeline(steps=postprocessor_steps)
-
-    return preprocessor, postprocessor
+    return env_cfg.get_env_processors()
 
 
 def make_env(
     cfg: EnvConfig | str,
     n_envs: int = 1,
-    use_async_envs: bool = False,
+    use_async_envs: bool = True,
     hub_cache_dir: str | None = None,
     trust_remote_code: bool = False,
 ) -> dict[str, dict[int, gym.vector.VectorEnv]]:
@@ -163,57 +119,4 @@ def make_env(
     if n_envs < 1:
         raise ValueError("`n_envs` must be at least 1")
 
-    env_cls = gym.vector.AsyncVectorEnv if use_async_envs else gym.vector.SyncVectorEnv
-
-    if "libero" in cfg.type:
-        from lerobot.envs.libero import create_libero_envs
-
-        if cfg.task is None:
-            raise ValueError("LiberoEnv requires a task to be specified")
-
-        return create_libero_envs(
-            task=cfg.task,
-            n_envs=n_envs,
-            camera_name=cfg.camera_name,
-            init_states=cfg.init_states,
-            gym_kwargs=cfg.gym_kwargs,
-            env_cls=env_cls,
-            control_mode=cfg.control_mode,
-            episode_length=cfg.episode_length,
-        )
-    elif "metaworld" in cfg.type:
-        from lerobot.envs.metaworld import create_metaworld_envs
-
-        if cfg.task is None:
-            raise ValueError("MetaWorld requires a task to be specified")
-
-        return create_metaworld_envs(
-            task=cfg.task,
-            n_envs=n_envs,
-            gym_kwargs=cfg.gym_kwargs,
-            env_cls=env_cls,
-        )
-
-    if cfg.gym_id not in gym_registry:
-        print(f"gym id '{cfg.gym_id}' not found, attempting to import '{cfg.package_name}'...")
-        try:
-            importlib.import_module(cfg.package_name)
-        except ModuleNotFoundError as e:
-            raise ModuleNotFoundError(
-                f"Package '{cfg.package_name}' required for env '{cfg.type}' not found. "
-                f"Please install it or check PYTHONPATH."
-            ) from e
-
-        if cfg.gym_id not in gym_registry:
-            raise gym.error.NameNotFound(
-                f"Environment '{cfg.gym_id}' not registered even after importing '{cfg.package_name}'."
-            )
-
-    def _make_one():
-        return gym.make(cfg.gym_id, disable_env_checker=cfg.disable_env_checker, **(cfg.gym_kwargs or {}))
-
-    vec = env_cls([_make_one for _ in range(n_envs)], autoreset_mode=gym.vector.AutoresetMode.SAME_STEP)
-
-    # normalize to {suite: {task_id: vec_env}} for consistency
-    suite_name = cfg.type  # e.g., "pusht", "aloha"
-    return {suite_name: {0: vec}}
+    return cfg.create_envs(n_envs=n_envs, use_async_envs=use_async_envs)

src/lerobot/envs/libero.py

@@ -150,7 +150,17 @@ class LiberoEnv(gym.Env):
 
         self.init_state_id = self.episode_index  # tie each sub-env to a fixed init state
 
-        self._env = self._make_envs_task(task_suite, self.task_id)
+        # Extract task metadata without allocating GPU resources (safe before fork).
+        task = task_suite.get_task(task_id)
+        self.task = task.name
+        self.task_description = task.language
+        self._task_bddl_file = os.path.join(
+            get_libero_path("bddl_files"), task.problem_folder, task.bddl_file
+        )
+        self._env: OffScreenRenderEnv | None = (
+            None  # deferred — created on first reset() inside the worker subprocess
+        )
+
         default_steps = 500
         self._max_episode_steps = (
             TASK_SUITE_MAX_STEPS.get(task_suite_name, default_steps)
@@ -221,28 +231,33 @@ class LiberoEnv(gym.Env):
             low=ACTION_LOW, high=ACTION_HIGH, shape=(ACTION_DIM,), dtype=np.float32
         )
 
+    def _ensure_env(self) -> None:
+        """Create the underlying OffScreenRenderEnv on first use.
+
+        Called inside the worker subprocess after fork(), so each worker gets
+        its own clean EGL context rather than inheriting a stale one from the
+        parent process (which causes EGL_BAD_CONTEXT crashes with AsyncVectorEnv).
+        """
+        if self._env is not None:
+            return
+        env = OffScreenRenderEnv(
+            bddl_file_name=self._task_bddl_file,
+            camera_heights=self.observation_height,
+            camera_widths=self.observation_width,
+        )
+        env.reset()
+        self._env = env
+
     def render(self):
+        self._ensure_env()
         raw_obs = self._env.env._get_observations()
-        image = self._format_raw_obs(raw_obs)["pixels"]["image"]
+        pixels = self._format_raw_obs(raw_obs)["pixels"]
+        image = next(iter(pixels.values()))
         image = image[::-1, ::-1]  # flip both H and W for visualization
         return image
 
-    def _make_envs_task(self, task_suite: Any, task_id: int = 0):
-        task = task_suite.get_task(task_id)
-        self.task = task.name
-        self.task_description = task.language
-        task_bddl_file = os.path.join(get_libero_path("bddl_files"), task.problem_folder, task.bddl_file)
-
-        env_args = {
-            "bddl_file_name": task_bddl_file,
-            "camera_heights": self.observation_height,
-            "camera_widths": self.observation_width,
-        }
-        env = OffScreenRenderEnv(**env_args)
-        env.reset()
-        return env
-
     def _format_raw_obs(self, raw_obs: RobotObservation) -> RobotObservation:
+        assert self._env is not None, "_format_raw_obs called before _ensure_env()"
         images = {}
         for camera_name in self.camera_name:
             image = raw_obs[camera_name]
@@ -294,6 +309,7 @@ class LiberoEnv(gym.Env):
         )
 
     def reset(self, seed=None, **kwargs):
+        self._ensure_env()
         super().reset(seed=seed)
         self._env.seed(seed)
         raw_obs = self._env.reset()
@@ -320,6 +336,8 @@ class LiberoEnv(gym.Env):
         return observation, info
 
     def step(self, action: np.ndarray) -> tuple[RobotObservation, float, bool, bool, dict[str, Any]]:
+        self._ensure_env()
+        assert self._env is not None
         if action.ndim != 1:
             raise ValueError(
                 f"Expected action to be 1-D (shape (action_dim,)), "
@@ -339,18 +357,13 @@ class LiberoEnv(gym.Env):
         )
         observation = self._format_raw_obs(raw_obs)
         if terminated:
-            info["final_info"] = {
-                "task": self.task,
-                "task_id": self.task_id,
-                "done": bool(done),
-                "is_success": bool(is_success),
-            }
             self.reset()
         truncated = False
         return observation, reward, terminated, truncated, info
 
     def close(self):
-        self._env.close()
+        if self._env is not None:
+            self._env.close()
 
 
 def _make_env_fns(
@@ -364,6 +377,7 @@ def _make_env_fns(
     init_states: bool,
     gym_kwargs: Mapping[str, Any],
     control_mode: str,
+    camera_name_mapping: dict[str, str] | None = None,
 ) -> list[Callable[[], LiberoEnv]]:
     """Build n_envs factory callables for a single (suite, task_id)."""
 
@@ -379,6 +393,7 @@ def _make_env_fns(
             episode_index=episode_index,
             n_envs=n_envs,
             control_mode=control_mode,
+            camera_name_mapping=camera_name_mapping,
             **local_kwargs,
         )
 
@@ -388,6 +403,57 @@ def _make_env_fns(
     return fns
 
 
+class _LazyAsyncVectorEnv:
+    """Wrapper that defers AsyncVectorEnv creation until first use.
+
+    Creating all tasks' AsyncVectorEnvs upfront spawns N_tasks × n_envs worker
+    processes, all of which allocate EGL/GPU resources immediately. Since tasks
+    are evaluated sequentially, only one task's workers need to be alive at a
+    time. This wrapper stores the factory functions and creates the real
+    AsyncVectorEnv on first reset(), keeping peak process count = n_envs.
+    """
+
+    def __init__(self, env_fns: list[Callable]):
+        self._env_fns = env_fns
+        self._env: gym.vector.AsyncVectorEnv | None = None
+        self.num_envs = len(env_fns)
+        # Instantiate one env to expose spaces (no GPU — _ensure_env is lazy).
+        tmp = env_fns[0]()
+        self.observation_space = tmp.observation_space
+        self.action_space = tmp.action_space
+        self.single_observation_space = tmp.observation_space
+        self.single_action_space = tmp.action_space
+        tmp.close()
+
+    def _ensure(self):
+        if self._env is None:
+            self._env = gym.vector.AsyncVectorEnv(self._env_fns, context="forkserver")
+
+    def reset(self, **kwargs):
+        self._ensure()
+        return self._env.reset(**kwargs)
+
+    def step(self, actions):
+        self._ensure()
+        return self._env.step(actions)
+
+    def call(self, name, *args, **kwargs):
+        self._ensure()
+        return self._env.call(name, *args, **kwargs)
+
+    def get_attr(self, name):
+        self._ensure()
+        return self._env.get_attr(name)
+
+    def close(self):
+        if self._env is not None:
+            self._env.close()
+            self._env = None
+
+    def __del__(self):
+        self.close()
+
+
 # ---- Main API ----------------------------------------------------------------
 
 
@@ -400,6 +466,7 @@ def create_libero_envs(
     env_cls: Callable[[Sequence[Callable[[], Any]]], Any] | None = None,
     control_mode: str = "relative",
     episode_length: int | None = None,
+    camera_name_mapping: dict[str, str] | None = None,
 ) -> dict[str, dict[int, Any]]:
     """
     Create vectorized LIBERO environments with a consistent return shape.
@@ -430,6 +497,8 @@ def create_libero_envs(
     if task_ids_filter is not None:
         print(f"Restricting to task_ids={task_ids_filter}")
 
+    is_async = env_cls is gym.vector.AsyncVectorEnv
+
     out: dict[str, dict[int, Any]] = defaultdict(dict)
     for suite_name in suite_names:
         suite = _get_suite(suite_name)
@@ -449,9 +518,12 @@ def create_libero_envs(
                 init_states=init_states,
                 gym_kwargs=gym_kwargs,
                 control_mode=control_mode,
+                camera_name_mapping=camera_name_mapping,
             )
-            out[suite_name][tid] = env_cls(fns)
+            if is_async:
+                out[suite_name][tid] = _LazyAsyncVectorEnv(fns)
+            else:
+                out[suite_name][tid] = env_cls(fns)
             print(f"Built vec env | suite={suite_name} | task_id={tid} | n_envs={n_envs}")
 
-    # return plain dicts for predictability
     return {suite: dict(task_map) for suite, task_map in out.items()}

src/lerobot/envs/metaworld.py

@@ -97,8 +97,9 @@ class MetaworldEnv(gym.Env):
         self.visualization_height = visualization_height
         self.camera_name = camera_name
 
-        self._env = self._make_envs_task(self.task)
-        self._max_episode_steps = self._env.max_path_length
+        self._env_name = self.task  # already stripped of "metaworld-" prefix above
+        self._env = None  # deferred — created on first reset() inside the worker subprocess
+        self._max_episode_steps = 500  # MT1 environments always have max_path_length=500
         self.task_description = TASK_DESCRIPTIONS[self.task]
 
         self.expert_policy = TASK_POLICY_MAPPING[self.task]()
@@ -136,6 +137,24 @@ class MetaworldEnv(gym.Env):
 
         self.action_space = spaces.Box(low=-1, high=1, shape=(ACTION_DIM,), dtype=np.float32)
 
+    def _ensure_env(self) -> None:
+        """Create the underlying MetaWorld env on first use.
+
+        Called inside the worker subprocess after fork(), so each worker gets
+        its own clean rendering context rather than inheriting a stale one from
+        the parent process (which causes crashes with AsyncVectorEnv).
+        """
+        if self._env is not None:
+            return
+        mt1 = metaworld.MT1(self._env_name, seed=42)
+        env = mt1.train_classes[self._env_name](render_mode="rgb_array", camera_name=self.camera_name)
+        env.set_task(mt1.train_tasks[0])
+        if self.camera_name == "corner2":
+            env.model.cam_pos[2] = [0.75, 0.075, 0.7]
+        env.reset()
+        env._freeze_rand_vec = False  # otherwise no randomization
+        self._env = env
+
     def render(self) -> np.ndarray:
         """
         Render the current environment frame.
@@ -143,26 +162,13 @@ class MetaworldEnv(gym.Env):
         Returns:
             np.ndarray: The rendered RGB image from the environment.
         """
+        self._ensure_env()
         image = self._env.render()
         if self.camera_name == "corner2":
             # Images from this camera are flipped — correct them
             image = np.flip(image, (0, 1))
         return image
 
-    def _make_envs_task(self, env_name: str):
-        mt1 = metaworld.MT1(env_name, seed=42)
-        env = mt1.train_classes[env_name](render_mode="rgb_array", camera_name=self.camera_name)
-        env.set_task(mt1.train_tasks[0])
-        if self.camera_name == "corner2":
-            env.model.cam_pos[2] = [
-                0.75,
-                0.075,
-                0.7,
-            ]  # corner2 position, similar to https://arxiv.org/pdf/2206.14244
-        env.reset()
-        env._freeze_rand_vec = False  # otherwise no randomization
-        return env
-
     def _format_raw_obs(self, raw_obs: np.ndarray) -> RobotObservation:
         image = None
         if self._env is not None:
@@ -209,6 +215,7 @@ class MetaworldEnv(gym.Env):
             observation (RobotObservation): The initial formatted observation.
             info (Dict[str, Any]): Additional info about the reset state.
         """
+        self._ensure_env()
         super().reset(seed=seed)
 
         raw_obs, info = self._env.reset(seed=seed)
@@ -232,6 +239,7 @@ class MetaworldEnv(gym.Env):
             truncated (bool): Whether the episode was truncated due to a time limit.
             info (Dict[str, Any]): Additional environment info.
         """
+        self._ensure_env()
         if action.ndim != 1:
             raise ValueError(
                 f"Expected action to be 1-D (shape (action_dim,)), "
@@ -263,7 +271,8 @@ class MetaworldEnv(gym.Env):
         return observation, reward, terminated, truncated, info
 
     def close(self):
-        self._env.close()
+        if self._env is not None:
+            self._env.close()
 
 
 # ---- Main API ----------------------------------------------------------------

src/lerobot/envs/utils.py

@@ -130,56 +130,51 @@ def env_to_policy_features(env_cfg: EnvConfig) -> dict[str, PolicyFeature]:
     return policy_features
 
 
-def are_all_envs_same_type(env: gym.vector.VectorEnv) -> bool:
-    first_type = type(env.envs[0])  # Get type of first env
-    return all(type(e) is first_type for e in env.envs)  # Fast type check
+def _get_sub_env_attr(env: gym.vector.VectorEnv, attr: str, index: int = 0):
+    """Retrieve an attribute from a sub-environment, works for both Sync and Async."""
+    try:
+        return env.get_attr(attr)[index]
+    except (AttributeError, Exception):
+        return None
+
+
+def _sub_env_has_attr(env: gym.vector.VectorEnv, attr: str) -> bool:
+    try:
+        env.get_attr(attr)
+        return True
+    except (AttributeError, Exception):
+        return False
 
 
 def check_env_attributes_and_types(env: gym.vector.VectorEnv) -> None:
     with warnings.catch_warnings():
-        warnings.simplefilter("once", UserWarning)  # Apply filter only in this function
+        warnings.simplefilter("once", UserWarning)
 
-        if not (hasattr(env.envs[0], "task_description") and hasattr(env.envs[0], "task")):
+        if not (_sub_env_has_attr(env, "task_description") and _sub_env_has_attr(env, "task")):
             warnings.warn(
                 "The environment does not have 'task_description' and 'task'. Some policies require these features.",
                 UserWarning,
                 stacklevel=2,
             )
-        if not are_all_envs_same_type(env):
-            warnings.warn(
-                "The environments have different types. Make sure you infer the right task from each environment. Empty task will be passed instead.",
-                UserWarning,
-                stacklevel=2,
-            )
 
 
 def add_envs_task(env: gym.vector.VectorEnv, observation: RobotObservation) -> RobotObservation:
     """Adds task feature to the observation dict with respect to the first environment attribute."""
-    if hasattr(env.envs[0], "task_description"):
-        task_result = env.call("task_description")
+    if _sub_env_has_attr(env, "task_description"):
+        task_result = list(env.call("task_description"))
 
-        if isinstance(task_result, tuple):
-            task_result = list(task_result)
-
-        if not isinstance(task_result, list):
-            raise TypeError(f"Expected task_description to return a list, got {type(task_result)}")
         if not all(isinstance(item, str) for item in task_result):
             raise TypeError("All items in task_description result must be strings")
 
         observation["task"] = task_result
-    elif hasattr(env.envs[0], "task"):
-        task_result = env.call("task")
+    elif _sub_env_has_attr(env, "task"):
+        task_result = list(env.call("task"))
 
-        if isinstance(task_result, tuple):
-            task_result = list(task_result)
-
-        if not isinstance(task_result, list):
-            raise TypeError(f"Expected task to return a list, got {type(task_result)}")
         if not all(isinstance(item, str) for item in task_result):
             raise TypeError("All items in task result must be strings")
 
         observation["task"] = task_result
-    else:  #  For envs without language instructions, e.g. aloha transfer cube and etc.
+    else:
         num_envs = observation[list(observation.keys())[0]].shape[0]
         observation["task"] = ["" for _ in range(num_envs)]
     return observation

src/lerobot/policies/pi0/configuration_pi0.py

@@ -57,28 +57,6 @@ class PI0Config(PreTrainedConfig):
     # Populated at runtime from dataset metadata by make_policy.
     action_feature_names: list[str] | None = None
 
-    # Relative state (UMI-style relative proprioception): converts multi-timestep
-    # observation.state to offsets from the current timestep, providing velocity info.
-    # Requires state_obs_steps >= 2. The flattened multi-timestep state is padded to
-    # max_state_dim, so ensure state_obs_steps * state_dim <= max_state_dim.
-    use_relative_state: bool = False
-    state_obs_steps: int = 1
-    relative_exclude_state_joints: list[str] = field(default_factory=list)
-    # Populated at runtime from dataset metadata by make_policy.
-    state_feature_names: list[str] | None = None
-
-    # Derive observation.state from the action column (UMI-style).
-    # When True, action_delta_indices loads one extra leading timestep [-1, 0, ..., chunk_size-1],
-    # DeriveStateFromActionStep extracts [action[t-1], action[t]] as a 2-step state,
-    # and strips the extra timestep from the action chunk.
-    # Implies use_relative_state=True and state_obs_steps=2.
-    derive_state_from_action: bool = False
-
-    # Latency compensation: skip this many steps from the start of each predicted
-    # action chunk during inference. E.g. at 10Hz with ~200ms total latency,
-    # latency_skip_steps=2 compensates for the delay.
-    latency_skip_steps: int = 0
-
     # Real-Time Chunking (RTC) configuration
     rtc_config: RTCConfig | None = None
 
@@ -128,10 +106,6 @@ class PI0Config(PreTrainedConfig):
     def __post_init__(self):
         super().__post_init__()
 
-        if self.derive_state_from_action:
-            self.use_relative_state = True
-            self.state_obs_steps = 2
-
         # Validate configuration
         if self.n_action_steps > self.chunk_size:
             raise ValueError(
@@ -147,13 +121,6 @@ class PI0Config(PreTrainedConfig):
         if self.dtype not in ["bfloat16", "float32"]:
             raise ValueError(f"Invalid dtype: {self.dtype}")
 
-        if self.use_relative_state and self.state_obs_steps < 2:
-            raise ValueError(
-                "use_relative_state requires state_obs_steps >= 2 "
-                f"(got {self.state_obs_steps}). Set state_obs_steps=2 for "
-                "UMI-style relative proprioception."
-            )
-
     def validate_features(self) -> None:
         """Validate and set up input/output features."""
         for i in range(self.empty_cameras):
@@ -199,16 +166,8 @@ class PI0Config(PreTrainedConfig):
     def observation_delta_indices(self) -> None:
         return None
 
-    @property
-    def state_delta_indices(self) -> list[int] | None:
-        if self.state_obs_steps >= 2:
-            return list(range(-(self.state_obs_steps - 1), 1))
-        return None
-
     @property
     def action_delta_indices(self) -> list:
-        if self.derive_state_from_action:
-            return [-1] + list(range(self.chunk_size))
         return list(range(self.chunk_size))
 
     @property

src/lerobot/policies/pi0/modeling_pi0.py

@@ -1230,11 +1230,8 @@ class PI0Policy(PreTrainedPolicy):
         return images, img_masks
 
     def prepare_state(self, batch):
-        """Flatten multi-timestep state and pad to max_state_dim."""
-        state = batch[OBS_STATE]
-        if state.ndim == 3:
-            state = state.flatten(start_dim=1)
-        state = pad_vector(state, self.config.max_state_dim)
+        """Pad state"""
+        state = pad_vector(batch[OBS_STATE], self.config.max_state_dim)
         return state
 
     def prepare_action(self, batch):
@@ -1253,8 +1250,7 @@ class PI0Policy(PreTrainedPolicy):
 
         # Action queue logic for n_action_steps > 1
         if len(self._action_queue) == 0:
-            skip = self.config.latency_skip_steps
-            actions = self.predict_action_chunk(batch)[:, skip : skip + self.config.n_action_steps]
+            actions = self.predict_action_chunk(batch)[:, : self.config.n_action_steps]
             # Transpose to get shape (n_action_steps, batch_size, action_dim)
             self._action_queue.extend(actions.transpose(0, 1))
 

src/lerobot/policies/pi0/processor_pi0.py

@@ -24,7 +24,6 @@ from lerobot.processor import (
     AbsoluteActionsProcessorStep,
     AddBatchDimensionProcessorStep,
     ComplementaryDataProcessorStep,
-    DeriveStateFromActionStep,
     DeviceProcessorStep,
     NormalizerProcessorStep,
     PolicyAction,
@@ -32,7 +31,6 @@ from lerobot.processor import (
     ProcessorStep,
     ProcessorStepRegistry,
     RelativeActionsProcessorStep,
-    RelativeStateProcessorStep,
     RenameObservationsProcessorStep,
     TokenizerProcessorStep,
     UnnormalizerProcessorStep,
@@ -130,25 +128,13 @@ def make_pi0_pre_post_processors(
         A tuple containing the configured pre-processor and post-processor pipelines.
     """
 
-    derive_state_step = DeriveStateFromActionStep(
-        enabled=getattr(config, "derive_state_from_action", False),
-    )
-
     relative_step = RelativeActionsProcessorStep(
         enabled=config.use_relative_actions,
         exclude_joints=getattr(config, "relative_exclude_joints", []),
         action_names=getattr(config, "action_feature_names", None),
     )
 
-    relative_state_step = RelativeStateProcessorStep(
-        enabled=getattr(config, "use_relative_state", False),
-        exclude_joints=getattr(config, "relative_exclude_state_joints", []),
-        state_names=getattr(config, "state_feature_names", None),
-    )
-
-    # Order: DeriveStateFromAction extracts state from the extended action chunk,
-    # then relative_action uses current state[t] for subtraction,
-    # then relative_state converts the multi-timestep state to offsets.
+    # OpenPI order: raw → relative → normalize → model → unnormalize → absolute
     input_steps: list[ProcessorStep] = [
         RenameObservationsProcessorStep(rename_map={}),  # To mimic the same processor as pretrained one
         AddBatchDimensionProcessorStep(),
@@ -160,9 +146,7 @@ def make_pi0_pre_post_processors(
             padding="max_length",
         ),
         DeviceProcessorStep(device=config.device),
-        derive_state_step,
         relative_step,
-        relative_state_step,
         NormalizerProcessorStep(
             features={**config.input_features, **config.output_features},
             norm_map=config.normalization_mapping,

src/lerobot/policies/pi0_fast/processor_pi0_fast.py

@@ -136,7 +136,7 @@ def make_pi0_fast_pre_post_processors(
     # Pi0Fast order: relative → normalize → tokenize → model → unnormalize → absolute
     # This matches pi0/pi0.5: RelativeActionsProcessorStep runs first on raw absolute actions,
     # caching the raw state. NormalizerProcessorStep then normalizes the raw relative actions,
-    # so the normalizer (and action tokenizer) sees delta values, relative stats are required.
+    # so the normalizer (and action tokenizer) sees delta values — relative stats are required.
     # NOTE: RelativeActionsProcessorStep only modifies the action in the transition; it reads
     # state from the observation but does not change it. NormalizerProcessorStep still runs
     # before Pi0FastPrepareStateAndLanguageTokenizerProcessorStep, so the state tokenizer

src/lerobot/policies/rtc/__init__.py

@@ -0,0 +1,29 @@
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Real-Time Chunking (RTC) utilities for action-chunking policies."""
+
+from lerobot.policies.rtc.action_interpolator import ActionInterpolator
+from lerobot.policies.rtc.action_queue import ActionQueue
+from lerobot.policies.rtc.configuration_rtc import RTCConfig
+from lerobot.policies.rtc.latency_tracker import LatencyTracker
+from lerobot.policies.rtc.modeling_rtc import RTCProcessor
+
+__all__ = [
+    "ActionInterpolator",
+    "ActionQueue",
+    "LatencyTracker",
+    "RTCConfig",
+    "RTCProcessor",
+]

src/lerobot/policies/rtc/action_interpolator.py

@@ -0,0 +1,116 @@
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Action interpolation for smoother robot control.
+
+Provides configurable Nx control rate by interpolating between consecutive actions.
+Useful with RTC and action-chunking policies to reduce jerkiness.
+"""
+
+from torch import Tensor
+
+
+class ActionInterpolator:
+    """Interpolates between consecutive actions for smoother control.
+
+    When enabled with multiplier N, produces N actions per policy action
+    by linearly interpolating between the previous and current action.
+
+    Example with multiplier=3:
+        prev_action -> [1/3 interpolated, 2/3 interpolated, current_action]
+
+    This effectively multiplies the control rate for smoother motion.
+
+    Usage:
+        interpolator = ActionInterpolator(multiplier=2)  # 2x control rate
+
+        # In control loop:
+        if interpolator.needs_new_action():
+            new_action = queue.get()
+            if new_action:
+                interpolator.add(new_action.cpu())
+
+        action = interpolator.get()
+        if action:
+            robot.send_action(action)
+    """
+
+    def __init__(self, multiplier: int = 1):
+        """Initialize the interpolator.
+
+        Args:
+            multiplier: Control rate multiplier (1 = no interpolation, 2 = 2x, 3 = 3x, etc.)
+        """
+        if multiplier < 1:
+            raise ValueError(f"multiplier must be >= 1, got {multiplier}")
+        self.multiplier = multiplier
+        self._prev: Tensor | None = None
+        self._buffer: list[Tensor] = []
+        self._idx = 0
+
+    @property
+    def enabled(self) -> bool:
+        """Whether interpolation is active (multiplier > 1)."""
+        return self.multiplier > 1
+
+    def reset(self):
+        """Reset interpolation state (call between episodes)."""
+        self._prev = None
+        self._buffer = []
+        self._idx = 0
+
+    def needs_new_action(self) -> bool:
+        """Check if a new action is needed from the queue."""
+        return self._idx >= len(self._buffer)
+
+    def add(self, action: Tensor) -> None:
+        """Add a new action and compute interpolated sequence.
+
+        Args:
+            action: New action tensor from policy/queue (already on CPU).
+        """
+        if self.multiplier > 1 and self._prev is not None:
+            self._buffer = []
+            for i in range(1, self.multiplier + 1):
+                t = i / self.multiplier
+                interp = self._prev + t * (action - self._prev)
+                self._buffer.append(interp)
+        else:
+            # First step: no previous action yet, so run at base FPS without interpolation.
+            self._buffer = [action.clone()]
+        self._prev = action.clone()
+        self._idx = 0
+
+    def get(self) -> Tensor | None:
+        """Get the next interpolated action.
+
+        Returns:
+            Next action tensor, or None if buffer is exhausted.
+        """
+        if self._idx >= len(self._buffer):
+            return None
+        action = self._buffer[self._idx]
+        self._idx += 1
+        return action
+
+    def get_control_interval(self, fps: float) -> float:
+        """Get the control interval based on interpolation multiplier.
+
+        Args:
+            fps: Base frames per second.
+
+        Returns:
+            Control interval in seconds (divided by multiplier).
+        """
+        return 1.0 / (fps * self.multiplier)

src/lerobot/policies/rtc/action_queue.py

@@ -79,6 +79,13 @@ class ActionQueue:
             self.last_index += 1
             return action.clone()
 
+    def clear(self) -> None:
+        """Clear queued actions and reset consumption index."""
+        with self.lock:
+            self.queue = None
+            self.original_queue = None
+            self.last_index = 0
+
     def qsize(self) -> int:
         """Get the number of remaining actions in the queue.
 
@@ -123,14 +130,26 @@ class ActionQueue:
         with self.lock:
             if self.original_queue is None:
                 return None
-            return self.original_queue[self.last_index :]
+            return self.original_queue[self.last_index :].clone()
+
+    def get_processed_left_over(self) -> Tensor | None:
+        """Get leftover processed actions (the actions currently executed by the robot).
+
+        Returns:
+            Tensor | None: Remaining processed actions (remaining_steps, action_dim),
+                or None if no processed queue exists.
+        """
+        with self.lock:
+            if self.queue is None:
+                return None
+            return self.queue[self.last_index :].clone()
 
     def merge(
         self,
         original_actions: Tensor,
         processed_actions: Tensor,
         real_delay: int,
-        action_index_before_inference: int | None = 0,
+        action_index_before_inference: int | None = None,
     ):
         """Merge new actions into the queue.
 
@@ -145,10 +164,10 @@ class ActionQueue:
             action_index_before_inference: Index before inference started, for validation.
         """
         with self.lock:
-            self._check_delays(real_delay, action_index_before_inference)
+            delay = self._check_and_resolve_delays(real_delay, action_index_before_inference)
 
             if self.cfg.enabled:
-                self._replace_actions_queue(original_actions, processed_actions, real_delay)
+                self._replace_actions_queue(original_actions, processed_actions, delay)
                 return
 
             self._append_actions_queue(original_actions, processed_actions)
@@ -164,12 +183,13 @@ class ActionQueue:
             processed_actions: Post-processed actions for robot.
             real_delay: Number of time steps to skip due to inference delay.
         """
-        self.original_queue = original_actions[real_delay:].clone()
-        self.queue = processed_actions[real_delay:].clone()
+        clamped_delay = max(0, min(real_delay, len(original_actions), len(processed_actions)))
+        self.original_queue = original_actions[clamped_delay:].clone()
+        self.queue = processed_actions[clamped_delay:].clone()
 
         logger.debug(f"original_actions shape: {self.original_queue.shape}")
         logger.debug(f"processed_actions shape: {self.queue.shape}")
-        logger.debug(f"real_delay: {real_delay}")
+        logger.debug(f"real_delay: {real_delay}, clamped_delay: {clamped_delay}")
 
         self.last_index = 0
 
@@ -196,7 +216,9 @@ class ActionQueue:
 
         self.last_index = 0
 
-    def _check_delays(self, real_delay: int, action_index_before_inference: int | None = None):
+    def _check_and_resolve_delays(
+        self, real_delay: int, action_index_before_inference: int | None = None
+    ) -> int:
         """Validate that computed delays match expectations.
 
         Compares the delay computed from inference latency with the actual
@@ -205,15 +227,20 @@ class ActionQueue:
         Args:
             real_delay: Delay computed from inference latency.
             action_index_before_inference: Action index when inference started.
-        """
-        if action_index_before_inference is None:
-            return
 
-        indexes_diff = self.last_index - action_index_before_inference
-        if indexes_diff != real_delay:
-            # Let's check that action index difference (real delay calculated based on action queue)
-            # is the same as delay calculated based on inference latency
-            logger.warning(
-                f"[ACTION_QUEUE] Indexes diff is not equal to real delay. "
-                f"Indexes diff: {indexes_diff}, real delay: {real_delay}"
-            )
+        Returns:
+            int: Delay to use.
+        """
+        effective_delay = max(0, real_delay)
+
+        if action_index_before_inference is not None:
+            indexes_diff = max(0, self.last_index - action_index_before_inference)
+            if indexes_diff != real_delay:
+                logger.warning(
+                    "Indexes diff is not equal to real delay. indexes_diff=%d, real_delay=%d",
+                    indexes_diff,
+                    real_delay,
+                )
+                return real_delay
+
+        return effective_delay

src/lerobot/processor/__init__.py

@@ -77,12 +77,9 @@ from .policy_robot_bridge import (
 )
 from .relative_action_processor import (
     AbsoluteActionsProcessorStep,
-    DeriveStateFromActionStep,
     RelativeActionsProcessorStep,
-    RelativeStateProcessorStep,
     to_absolute_actions,
     to_relative_actions,
-    to_relative_state,
 )
 from .rename_processor import RenameObservationsProcessorStep
 from .tokenizer_processor import ActionTokenizerProcessorStep, TokenizerProcessorStep
@@ -110,9 +107,7 @@ __all__ = [
     "make_default_robot_action_processor",
     "make_default_robot_observation_processor",
     "AbsoluteActionsProcessorStep",
-    "DeriveStateFromActionStep",
     "RelativeActionsProcessorStep",
-    "RelativeStateProcessorStep",
     "MapDeltaActionToRobotActionStep",
     "MapTensorToDeltaActionDictStep",
     "NormalizerProcessorStep",
@@ -144,7 +139,6 @@ __all__ = [
     "TruncatedProcessorStep",
     "to_absolute_actions",
     "to_relative_actions",
-    "to_relative_state",
     "UnnormalizerProcessorStep",
     "VanillaObservationProcessorStep",
 ]

src/lerobot/processor/relative_action_processor.py

@@ -30,13 +30,10 @@ from .pipeline import ProcessorStep, ProcessorStepRegistry
 __all__ = [
     "MapDeltaActionToRobotActionStep",
     "MapTensorToDeltaActionDictStep",
-    "DeriveStateFromActionStep",
     "RelativeActionsProcessorStep",
     "AbsoluteActionsProcessorStep",
-    "RelativeStateProcessorStep",
     "to_relative_actions",
     "to_absolute_actions",
-    "to_relative_state",
 ]
 
 
@@ -84,41 +81,6 @@ def to_absolute_actions(actions: Tensor, state: Tensor, mask: Sequence[bool]) ->
     return actions
 
 
-@ProcessorStepRegistry.register("derive_state_from_action_processor")
-@dataclass
-class DeriveStateFromActionStep(ProcessorStep):
-    """Derives 2-step observation.state from the action chunk (UMI-style).
-
-    Expects action with one extra leading timestep: [B, chunk_size+1, D]
-    from action_delta_indices = [-1, 0, 1, ..., chunk_size-1].
-    Extracts [action[t-1], action[t]] as state and strips the extra timestep.
-    No-op during inference (state comes from robot).
-    """
-
-    enabled: bool = False
-
-    def __call__(self, transition: EnvTransition) -> EnvTransition:
-        if not self.enabled:
-            return transition
-        action = transition.get(TransitionKey.ACTION)
-        if action is None or action.ndim < 3:
-            return transition
-        new_transition = transition.copy()
-        new_obs = dict(new_transition.get(TransitionKey.OBSERVATION, {}))
-        new_obs[OBS_STATE] = action[..., :2, :]
-        new_transition[TransitionKey.ACTION] = action[..., 1:, :]
-        new_transition[TransitionKey.OBSERVATION] = new_obs
-        return new_transition
-
-    def get_config(self) -> dict[str, Any]:
-        return {"enabled": self.enabled}
-
-    def transform_features(
-        self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
-    ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
-        return features
-
-
 @ProcessorStepRegistry.register("delta_actions_processor")
 @dataclass
 class RelativeActionsProcessorStep(ProcessorStep):
@@ -162,14 +124,7 @@ class RelativeActionsProcessorStep(ProcessorStep):
 
     def __call__(self, transition: EnvTransition) -> EnvTransition:
         observation = transition.get(TransitionKey.OBSERVATION, {})
-        raw_state = observation.get(OBS_STATE) if observation else None
-
-        # When state_delta_indices loads multi-timestep state [B, n_obs, D],
-        # use only the current (last) timestep for relative action conversion.
-        if raw_state is not None:
-            state = raw_state[..., -1, :] if raw_state.ndim >= 3 else raw_state
-        else:
-            state = None
+        state = observation.get(OBS_STATE) if observation else None
 
         # Always cache state for the paired AbsoluteActionsProcessorStep
         if state is not None:
@@ -200,120 +155,6 @@ class RelativeActionsProcessorStep(ProcessorStep):
         return features
 
 
-def to_relative_state(state: Tensor, mask: Sequence[bool]) -> Tensor:
-    """Convert multi-timestep absolute state to relative (offset from current timestep).
-
-    Each timestep becomes: ``state[..., t, :] - state[..., -1, :]`` for masked dims.
-    The last (current) timestep becomes zeros for masked dims.
-
-    Args:
-        state: (..., n_obs, state_dim) — last timestep is the reference (current).
-        mask: Which dims to convert. Can be shorter than state_dim.
-    """
-    mask_t = torch.tensor(mask, dtype=state.dtype, device=state.device)
-    dims = mask_t.shape[0]
-    current = state[..., -1:, :]  # (..., 1, state_dim)
-    state = state.clone()
-    state[..., :dims] -= current[..., :dims] * mask_t
-    return state
-
-
-@ProcessorStepRegistry.register("relative_state_processor")
-@dataclass
-class RelativeStateProcessorStep(ProcessorStep):
-    """Converts observation.state to relative (offset from current timestep).
-
-    UMI-style relative proprioception: each state timestep is expressed as
-    an offset from the current EE pose, providing velocity information.
-
-    During training (multi-timestep input from ``state_delta_indices``):
-        ``state[..., t, :] -= state[..., -1, :]`` — subtract current from all.
-
-    During inference (single timestep): buffers the previous state and stacks
-    ``[previous, current]`` before applying the relative conversion, producing
-    the same ``[n_obs, D]`` shape the model expects.
-
-    Attributes:
-        enabled: Whether to apply the relative conversion.
-        exclude_joints: Joint/dim names to keep absolute.
-        state_names: State dimension names from dataset metadata.
-    """
-
-    enabled: bool = False
-    exclude_joints: list[str] = field(default_factory=list)
-    state_names: list[str] | None = None
-    _previous_state: torch.Tensor | None = field(default=None, init=False, repr=False)
-
-    def _build_mask(self, state_dim: int) -> list[bool]:
-        if not self.exclude_joints or self.state_names is None:
-            return [True] * state_dim
-
-        exclude_tokens = [str(name).lower() for name in self.exclude_joints if name]
-        if not exclude_tokens:
-            return [True] * state_dim
-
-        mask = []
-        for name in self.state_names[:state_dim]:
-            state_name = str(name).lower()
-            is_excluded = any(token == state_name or token in state_name for token in exclude_tokens)
-            mask.append(not is_excluded)
-
-        if len(mask) < state_dim:
-            mask.extend([True] * (state_dim - len(mask)))
-
-        return mask
-
-    def __call__(self, transition: EnvTransition) -> EnvTransition:
-        if not self.enabled:
-            return transition
-
-        observation = transition.get(TransitionKey.OBSERVATION, {})
-        state = observation.get(OBS_STATE) if observation else None
-
-        if state is None:
-            return transition
-
-        new_transition = transition.copy()
-        new_obs = dict(new_transition.get(TransitionKey.OBSERVATION, {}))
-        mask = self._build_mask(state.shape[-1])
-
-        if state.ndim >= 3:
-            # [B, n_obs, D] — multi-timestep (training with state_delta_indices)
-            relative = to_relative_state(state, mask)
-            new_obs[OBS_STATE] = relative.flatten(start_dim=-2)  # [B, n_obs*D]
-        elif state.ndim == 2:
-            # [B, D] — single timestep (inference): buffer previous and stack
-            current = state
-            if self._previous_state is None:
-                self._previous_state = current.clone()
-            prev = self._previous_state
-            if prev.device != current.device or prev.dtype != current.dtype:
-                prev = prev.to(device=current.device, dtype=current.dtype)
-            stacked = torch.stack([prev, current], dim=-2)  # [B, 2, D]
-            relative = to_relative_state(stacked, mask)
-            new_obs[OBS_STATE] = relative.flatten(start_dim=-2)  # [B, 2*D]
-            self._previous_state = current.clone()
-
-        new_transition[TransitionKey.OBSERVATION] = new_obs
-        return new_transition
-
-    def reset(self) -> None:
-        """Reset the state buffer. Call at episode boundaries during inference."""
-        self._previous_state = None
-
-    def get_config(self) -> dict[str, Any]:
-        return {
-            "enabled": self.enabled,
-            "exclude_joints": self.exclude_joints,
-            "state_names": self.state_names,
-        }
-
-    def transform_features(
-        self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
-    ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
-        return features
-
-
 @ProcessorStepRegistry.register("absolute_actions_processor")
 @dataclass
 class AbsoluteActionsProcessorStep(ProcessorStep):

src/lerobot/processor/tokenizer_processor.py

@@ -136,8 +136,8 @@ class TokenizerProcessorStep(ObservationProcessorStep):
         # Standardize to a list of strings for the tokenizer
         if isinstance(task, str):
             return [task]
-        elif isinstance(task, list) and all(isinstance(t, str) for t in task):
-            return task
+        elif isinstance(task, (list, tuple)) and all(isinstance(t, str) for t in task):
+            return list(task)
 
         return None
 

src/lerobot/robots/bi_openarm_follower/bi_openarm_follower.py

@@ -62,8 +62,6 @@ class BiOpenArmFollower(Robot):
             can_bitrate=config.left_arm_config.can_bitrate,
             can_data_bitrate=config.left_arm_config.can_data_bitrate,
             motor_config=config.left_arm_config.motor_config,
-            gripper_port=config.left_arm_config.gripper_port,
-            gripper_motor_ids=config.left_arm_config.gripper_motor_ids,
             position_kd=config.left_arm_config.position_kd,
             position_kp=config.left_arm_config.position_kp,
             joint_limits=config.left_arm_config.joint_limits,
@@ -82,8 +80,6 @@ class BiOpenArmFollower(Robot):
             can_bitrate=config.right_arm_config.can_bitrate,
             can_data_bitrate=config.right_arm_config.can_data_bitrate,
             motor_config=config.right_arm_config.motor_config,
-            gripper_port=config.right_arm_config.gripper_port,
-            gripper_motor_ids=config.right_arm_config.gripper_motor_ids,
             position_kd=config.right_arm_config.position_kd,
             position_kp=config.right_arm_config.position_kp,
             joint_limits=config.right_arm_config.joint_limits,
@@ -100,9 +96,11 @@ class BiOpenArmFollower(Robot):
         left_arm_motors_ft = self.left_arm._motors_ft
         right_arm_motors_ft = self.right_arm._motors_ft
 
+        # Right first, then left — matches the teleoperator (OpenArmMini) ordering
+        # and the dataset feature names recorded during data collection.
         return {
-            **{f"left_{k}": v for k, v in left_arm_motors_ft.items()},
             **{f"right_{k}": v for k, v in right_arm_motors_ft.items()},
+            **{f"left_{k}": v for k, v in left_arm_motors_ft.items()},
         }
 
     @property
@@ -154,14 +152,16 @@ class BiOpenArmFollower(Robot):
         left_cam_keys = set(self.left_arm.cameras.keys())
         right_cam_keys = set(self.right_arm.cameras.keys())
 
-        left_obs = self.left_arm.get_observation()
-        for key, value in left_obs.items():
-            obs_dict[key if key in left_cam_keys else f"left_{key}"] = value
-
+        # Right first, then left — matches the teleoperator (OpenArmMini) ordering
+        # and the dataset feature names recorded during data collection.
         right_obs = self.right_arm.get_observation()
         for key, value in right_obs.items():
             obs_dict[key if key in right_cam_keys else f"right_{key}"] = value
 
+        left_obs = self.left_arm.get_observation()
+        for key, value in left_obs.items():
+            obs_dict[key if key in left_cam_keys else f"left_{key}"] = value
+
         return obs_dict
 
     @check_if_not_connected
@@ -187,7 +187,7 @@ class BiOpenArmFollower(Robot):
         prefixed_sent_action_left = {f"left_{key}": value for key, value in sent_action_left.items()}
         prefixed_sent_action_right = {f"right_{key}": value for key, value in sent_action_right.items()}
 
-        return {**prefixed_sent_action_left, **prefixed_sent_action_right}
+        return {**prefixed_sent_action_right, **prefixed_sent_action_left}
 
     @check_if_not_connected
     def disconnect(self):

src/lerobot/robots/bi_openarm_follower/config_bi_openarm_follower.py

@@ -23,10 +23,12 @@ from ..config import RobotConfig
 
 
 @RobotConfig.register_subclass("bi_openarm_follower")
-@dataclass
+@dataclass(kw_only=True)
 class BiOpenArmFollowerConfig(RobotConfig):
     """Configuration class for Bi OpenArm Follower robots."""
 
+    id: str | None = "bi_openarm_follower"
+
     left_arm_config: OpenArmFollowerConfigBase
     right_arm_config: OpenArmFollowerConfigBase
 

src/lerobot/robots/openarm_follower/config_openarm_follower.py

@@ -28,8 +28,7 @@ LEFT_DEFAULT_JOINTS_LIMITS: dict[str, tuple[float, float]] = {
     "joint_5": (-85.0, 85.0),
     "joint_6": (-40.0, 40.0),
     "joint_7": (-80.0, 80.0),
-    "proximal": (0.0, 100.0),
-    "distal": (0.0, 100.0),
+    "gripper": (-65.0, 0.0),
 }
 
 RIGHT_DEFAULT_JOINTS_LIMITS: dict[str, tuple[float, float]] = {
@@ -40,8 +39,7 @@ RIGHT_DEFAULT_JOINTS_LIMITS: dict[str, tuple[float, float]] = {
     "joint_5": (-85.0, 85.0),
     "joint_6": (-40.0, 40.0),
     "joint_7": (-80.0, 80.0),
-    "proximal": (0.0, 100.0),
-    "distal": (0.0, 100.0),
+    "gripper": (-65.0, 0.0),
 }
 
 
@@ -75,8 +73,13 @@ class OpenArmFollowerConfigBase:
     # Camera configurations
     cameras: dict[str, CameraConfig] = field(default_factory=dict)
 
-    # Arm motor configuration (7 DOF, Damiao on CAN bus)
+    # Motor configuration for OpenArms (7 DOF per arm)
     # Maps motor names to (send_can_id, recv_can_id, motor_type)
+    # Based on: https://docs.openarm.dev/software/setup/configure-test
+    # OpenArms uses 4 types of motors:
+    # - DM8009 (DM-J8009P-2EC) for shoulders (high torque)
+    # - DM4340P and DM4340 for shoulder rotation and elbow
+    # - DM4310 (DM-J4310-2EC V1.1) for wrist and gripper
     motor_config: dict[str, tuple[int, int, str]] = field(
         default_factory=lambda: {
             "joint_1": (0x01, 0x11, "dm8009"),  # J1 - Shoulder pan (DM8009)
@@ -86,18 +89,19 @@ class OpenArmFollowerConfigBase:
             "joint_5": (0x05, 0x15, "dm4310"),  # J5 - Wrist roll (DM4310)
             "joint_6": (0x06, 0x16, "dm4310"),  # J6 - Wrist pitch (DM4310)
             "joint_7": (0x07, 0x17, "dm4310"),  # J7 - Wrist rotation (DM4310)
+            "gripper": (0x08, 0x18, "dm4310"),  # J8 - Gripper (DM4310)
         }
     )
 
-    # UMI-style gripper (Feetech STS3215 on serial bus)
-    gripper_port: str = "/dev/ttyUSB0"
-    gripper_motor_ids: dict[str, int] = field(default_factory=lambda: {"proximal": 1, "distal": 2})
+    # MIT control parameters for position control (used in send_action)
+    # List of 8 values: [joint_1, joint_2, joint_3, joint_4, joint_5, joint_6, joint_7, gripper]
+    position_kp: list[float] = field(
+        default_factory=lambda: [240.0, 240.0, 240.0, 240.0, 24.0, 31.0, 25.0, 25.0]
+    )
+    position_kd: list[float] = field(default_factory=lambda: [5.0, 5.0, 3.0, 5.0, 0.3, 0.3, 0.3, 0.3])
 
-    # MIT control parameters for the 7 arm joints
-    position_kp: list[float] = field(default_factory=lambda: [240.0, 240.0, 240.0, 240.0, 24.0, 31.0, 25.0])
-    position_kd: list[float] = field(default_factory=lambda: [5.0, 5.0, 3.0, 5.0, 0.3, 0.3, 0.3])
-
-    # Joint limits. Can be overridden via CLI or by setting config.side to 'left' or 'right'.
+    # Values for joint limits. Can be overridden via CLI (for custom values) or by setting config.side to either 'left' or 'right'.
+    # If config.side is left set to None and no CLI values are passed, the default joint limit values are small for safety.
     joint_limits: dict[str, tuple[float, float]] = field(
         default_factory=lambda: {
             "joint_1": (-5.0, 5.0),
@@ -107,8 +111,7 @@ class OpenArmFollowerConfigBase:
             "joint_5": (-5.0, 5.0),
             "joint_6": (-5.0, 5.0),
             "joint_7": (-5.0, 5.0),
-            "proximal": (0.0, 100.0),
-            "distal": (0.0, 100.0),
+            "gripper": (-5.0, 0.0),
         }
     )
 

src/lerobot/robots/openarm_follower/openarm_follower.py

@@ -22,7 +22,6 @@ from typing import Any
 from lerobot.cameras.utils import make_cameras_from_configs
 from lerobot.motors import Motor, MotorCalibration, MotorNormMode
 from lerobot.motors.damiao import DamiaoMotorsBus
-from lerobot.motors.feetech import FeetechMotorsBus, OperatingMode
 from lerobot.types import RobotAction, RobotObservation
 from lerobot.utils.decorators import check_if_already_connected, check_if_not_connected
 
@@ -39,7 +38,8 @@ logger = logging.getLogger(__name__)
 
 class OpenArmFollower(Robot):
     """
-    OpenArms Follower Robot: 7 DOF Damiao arm (CAN) + UMI-style Feetech gripper (serial).
+    OpenArms Follower Robot which uses CAN bus communication to control 7 DOF arm with a gripper.
+    The arm uses Damiao motors in MIT control mode.
     """
 
     config_class = OpenArmFollowerConfig
@@ -49,17 +49,19 @@ class OpenArmFollower(Robot):
         super().__init__(config)
         self.config = config
 
-        # Arm motors (Damiao on CAN bus)
-        arm_motors: dict[str, Motor] = {}
+        # Arm motors
+        motors: dict[str, Motor] = {}
         for motor_name, (send_id, recv_id, motor_type_str) in config.motor_config.items():
-            motor = Motor(send_id, motor_type_str, MotorNormMode.DEGREES)
+            motor = Motor(
+                send_id, motor_type_str, MotorNormMode.DEGREES
+            )  # Always use degrees for Damiao motors
             motor.recv_id = recv_id
             motor.motor_type_str = motor_type_str
-            arm_motors[motor_name] = motor
+            motors[motor_name] = motor
 
         self.bus = DamiaoMotorsBus(
             port=self.config.port,
-            motors=arm_motors,
+            motors=motors,
             calibration=self.calibration,
             can_interface=self.config.can_interface,
             use_can_fd=self.config.use_can_fd,
@@ -67,17 +69,6 @@ class OpenArmFollower(Robot):
             data_bitrate=self.config.can_data_bitrate if self.config.use_can_fd else None,
         )
 
-        # Gripper motors (Feetech STS3215 on serial bus)
-        gripper_motors: dict[str, Motor] = {
-            name: Motor(motor_id, "sts3215", MotorNormMode.RANGE_0_100)
-            for name, motor_id in config.gripper_motor_ids.items()
-        }
-        self.gripper_bus = FeetechMotorsBus(
-            port=config.gripper_port,
-            motors=gripper_motors,
-            calibration=self.calibration,
-        )
-
         if config.side is not None:
             if config.side == "left":
                 config.joint_limits = LEFT_DEFAULT_JOINTS_LIMITS
@@ -93,6 +84,7 @@ class OpenArmFollower(Robot):
             )
         logger.info(f"Values used for joint limits: {config.joint_limits}.")
 
+        # Initialize cameras
         self.cameras = make_cameras_from_configs(config.cameras)
 
     @property
@@ -101,10 +93,8 @@ class OpenArmFollower(Robot):
         features: dict[str, type] = {}
         for motor in self.bus.motors:
             features[f"{motor}.pos"] = float
-            features[f"{motor}.vel"] = float
-            features[f"{motor}.torque"] = float
-        for motor in self.gripper_bus.motors:
-            features[f"{motor}.pos"] = float
+            features[f"{motor}.vel"] = float  # Add this
+            features[f"{motor}.torque"] = float  # Add this
         return features
 
     @property
@@ -126,11 +116,8 @@ class OpenArmFollower(Robot):
 
     @property
     def is_connected(self) -> bool:
-        return (
-            self.bus.is_connected
-            and self.gripper_bus.is_connected
-            and all(cam.is_connected for cam in self.cameras.values())
-        )
+        """Check if robot is connected."""
+        return self.bus.is_connected and all(cam.is_connected for cam in self.cameras.values())
 
     @check_if_already_connected
     def connect(self, calibrate: bool = True) -> None:
@@ -140,12 +127,12 @@ class OpenArmFollower(Robot):
         We assume that at connection time, the arms are in a safe rest position,
         and torque can be safely disabled to run calibration if needed.
         """
+
+        # Connect to CAN bus
         logger.info(f"Connecting arm on {self.config.port}...")
         self.bus.connect()
 
-        logger.info(f"Connecting gripper on {self.config.gripper_port}...")
-        self.gripper_bus.connect()
-
+        # Run calibration if needed
         if not self.is_calibrated and calibrate:
             logger.info(
                 "Mismatch between calibration values in the motor and the calibration file or no calibration file found"
@@ -157,7 +144,7 @@ class OpenArmFollower(Robot):
 
         self.configure()
 
-        if self.bus.is_calibrated:
+        if self.is_calibrated:
             self.bus.set_zero_position()
 
         self.bus.enable_torque()
@@ -166,39 +153,47 @@ class OpenArmFollower(Robot):
 
     @property
     def is_calibrated(self) -> bool:
-        return self.bus.is_calibrated and self.gripper_bus.is_calibrated
+        """Check if robot is calibrated."""
+        return self.bus.is_calibrated
 
     def calibrate(self) -> None:
         """
-        Run calibration for both the Damiao arm and Feetech gripper.
+        Run calibration procedure for OpenArms robot.
 
-        Arm calibration: set zero position with arm hanging, ±90° default range.
-        Gripper calibration: SO100-style half-turn homing + range recording.
+        The calibration procedure:
+        1. Disable torque
+        2. Ask user to position arms in hanging position with grippers closed
+        3. Set this as zero position
+        4. Record range of motion for each joint
+        5. Save calibration
         """
         if self.calibration:
+            # Calibration file exists, ask user whether to use it or run new calibration
             user_input = input(
                 f"Press ENTER to use provided calibration file associated with the id {self.id}, or type 'c' and press ENTER to run calibration: "
             )
             if user_input.strip().lower() != "c":
                 logger.info(f"Writing calibration file associated with the id {self.id} to the motors")
                 self.bus.write_calibration(self.calibration)
-                self.gripper_bus.write_calibration(self.calibration)
                 return
 
         logger.info(f"\nRunning calibration for {self}")
-
-        # --- Arm calibration (Damiao) ---
         self.bus.disable_torque()
+
+        # Step 1: Set zero position
         input(
-            "\nCalibration: Set Zero Position\n"
+            "\nCalibration: Set Zero Position)\n"
             "Position the arm in the following configuration:\n"
             "  - Arm hanging straight down\n"
             "  - Gripper closed\n"
             "Press ENTER when ready..."
         )
+
+        # Set current position as zero for all motors
         self.bus.set_zero_position()
         logger.info("Arm zero position set.")
 
+        logger.info("Setting range: -90° to +90° for safety by default for all joints")
         for motor_name, motor in self.bus.motors.items():
             self.calibration[motor_name] = MotorCalibration(
                 id=motor.id,
@@ -207,52 +202,17 @@ class OpenArmFollower(Robot):
                 range_min=-90,
                 range_max=90,
             )
+
         self.bus.write_calibration(self.calibration)
-
-        # --- Gripper calibration (Feetech) ---
-        self.gripper_bus.disable_torque()
-        for motor in self.gripper_bus.motors:
-            self.gripper_bus.write("Operating_Mode", motor, OperatingMode.POSITION.value)
-
-        input("Move gripper to the middle of its range of motion and press ENTER....")
-        homing_offsets = self.gripper_bus.set_half_turn_homings()
-
-        gripper_motor_names = list(self.gripper_bus.motors.keys())
-        print(
-            f"Move gripper joints ({', '.join(gripper_motor_names)}) through their "
-            "entire ranges of motion.\nRecording positions. Press ENTER to stop..."
-        )
-        range_mins, range_maxes = self.gripper_bus.record_ranges_of_motion(gripper_motor_names)
-
-        for motor_name, m in self.gripper_bus.motors.items():
-            self.calibration[motor_name] = MotorCalibration(
-                id=m.id,
-                drive_mode=0,
-                homing_offset=homing_offsets[motor_name],
-                range_min=range_mins[motor_name],
-                range_max=range_maxes[motor_name],
-            )
-        self.gripper_bus.write_calibration(self.calibration)
-
         self._save_calibration()
         print(f"Calibration saved to {self.calibration_fpath}")
 
     def configure(self) -> None:
-        """Configure both arm (Damiao) and gripper (Feetech) motors."""
+        """Configure motors with appropriate settings."""
+        # TODO(Steven, Pepijn): Slightly different from what it is happening in the leader
         with self.bus.torque_disabled():
             self.bus.configure_motors()
 
-        with self.gripper_bus.torque_disabled():
-            self.gripper_bus.configure_motors()
-            for motor in self.gripper_bus.motors:
-                self.gripper_bus.write("Operating_Mode", motor, OperatingMode.POSITION.value)
-                self.gripper_bus.write("P_Coefficient", motor, 16)
-                self.gripper_bus.write("I_Coefficient", motor, 0)
-                self.gripper_bus.write("D_Coefficient", motor, 32)
-                self.gripper_bus.write("Max_Torque_Limit", motor, 500)
-                self.gripper_bus.write("Protection_Current", motor, 250)
-                self.gripper_bus.write("Overload_Torque", motor, 25)
-
     def setup_motors(self) -> None:
         raise NotImplementedError(
             "Motor ID configuration is typically done via manufacturer tools for CAN motors."
@@ -260,23 +220,25 @@ class OpenArmFollower(Robot):
 
     @check_if_not_connected
     def get_observation(self) -> RobotObservation:
-        """Read all motor states from arm (CAN) and gripper (serial), plus cameras."""
+        """
+        Get current observation from robot including position, velocity, and torque.
+
+        Reads all motor states (pos/vel/torque) in one CAN refresh cycle
+        instead of 3 separate reads.
+        """
         start = time.perf_counter()
+
         obs_dict: dict[str, Any] = {}
 
-        # Arm motors (Damiao) — pos/vel/torque in one CAN refresh cycle
         states = self.bus.sync_read_all_states()
+
         for motor in self.bus.motors:
             state = states.get(motor, {})
             obs_dict[f"{motor}.pos"] = state.get("position", 0.0)
             obs_dict[f"{motor}.vel"] = state.get("velocity", 0.0)
             obs_dict[f"{motor}.torque"] = state.get("torque", 0.0)
 
-        # Gripper motors (Feetech) — position only
-        gripper_positions = self.gripper_bus.sync_read("Present_Position")
-        for motor, val in gripper_positions.items():
-            obs_dict[f"{motor}.pos"] = val
-
+        # Capture images from cameras
         for cam_key, cam in self.cameras.items():
             start = time.perf_counter()
             obs_dict[cam_key] = cam.read_latest()
@@ -296,76 +258,86 @@ class OpenArmFollower(Robot):
         custom_kd: dict[str, float] | None = None,
     ) -> RobotAction:
         """
-        Send action command to robot. Arm joints go to Damiao CAN bus,
-        gripper joints go to Feetech serial bus.
+        Send action command to robot.
+
+        The action magnitude may be clipped based on safety limits.
 
         Args:
-            action: Dictionary with motor positions (e.g., "joint_1.pos", "proximal.pos")
-            custom_kp: Optional custom kp gains per arm motor
-            custom_kd: Optional custom kd gains per arm motor
+            action: Dictionary with motor positions (e.g., "joint_1.pos", "joint_2.pos")
+            custom_kp: Optional custom kp gains per motor (e.g., {"joint_1": 120.0, "joint_2": 150.0})
+            custom_kd: Optional custom kd gains per motor (e.g., {"joint_1": 1.5, "joint_2": 2.0})
 
         Returns:
             The action actually sent (potentially clipped)
         """
+
         goal_pos = {key.removesuffix(".pos"): val for key, val in action.items() if key.endswith(".pos")}
 
-        # Apply joint limit clipping
+        # Apply joint limit clipping to arm
         for motor_name, position in goal_pos.items():
             if motor_name in self.config.joint_limits:
                 min_limit, max_limit = self.config.joint_limits[motor_name]
                 clipped_position = max(min_limit, min(max_limit, position))
                 if clipped_position != position:
-                    logger.debug(f"Clipped {motor_name} from {position:.2f} to {clipped_position:.2f}")
+                    logger.debug(f"Clipped {motor_name} from {position:.2f}° to {clipped_position:.2f}°")
                 goal_pos[motor_name] = clipped_position
 
-        # Split into arm and gripper actions
-        arm_motors = set(self.bus.motors.keys())
-        gripper_motors = set(self.gripper_bus.motors.keys())
-        arm_goal = {k: v for k, v in goal_pos.items() if k in arm_motors}
-        gripper_goal = {k: v for k, v in goal_pos.items() if k in gripper_motors}
-
-        # Cap arm goal position when too far away from present position
-        if self.config.max_relative_target is not None and arm_goal:
+        # Cap goal position when too far away from present position.
+        # /!\ Slower fps expected due to reading from the follower.
+        if self.config.max_relative_target is not None:
             present_pos = self.bus.sync_read("Present_Position")
-            goal_present_pos = {key: (g_pos, present_pos[key]) for key, g_pos in arm_goal.items()}
-            arm_goal = ensure_safe_goal_position(goal_present_pos, self.config.max_relative_target)
+            goal_present_pos = {key: (g_pos, present_pos[key]) for key, g_pos in goal_pos.items()}
+            goal_pos = ensure_safe_goal_position(goal_present_pos, self.config.max_relative_target)
 
-        # Arm: batch MIT control (Damiao)
-        if arm_goal:
-            arm_motor_names = list(self.bus.motors.keys())
-            commands = {}
-            for motor_name, position_degrees in arm_goal.items():
-                idx = arm_motor_names.index(motor_name) if motor_name in arm_motor_names else 0
-                if custom_kp is not None and motor_name in custom_kp:
-                    kp = custom_kp[motor_name]
-                else:
-                    kp = (
-                        self.config.position_kp[idx]
-                        if isinstance(self.config.position_kp, list)
-                        else self.config.position_kp
-                    )
-                if custom_kd is not None and motor_name in custom_kd:
-                    kd = custom_kd[motor_name]
-                else:
-                    kd = (
-                        self.config.position_kd[idx]
-                        if isinstance(self.config.position_kd, list)
-                        else self.config.position_kd
-                    )
-                commands[motor_name] = (kp, kd, position_degrees, 0.0, 0.0)
-            self.bus._mit_control_batch(commands)
+        # TODO(Steven, Pepijn): Refactor writing
+        # Motor name to index mapping for gains
+        motor_index = {
+            "joint_1": 0,
+            "joint_2": 1,
+            "joint_3": 2,
+            "joint_4": 3,
+            "joint_5": 4,
+            "joint_6": 5,
+            "joint_7": 6,
+            "gripper": 7,
+        }
 
-        # Gripper: position control (Feetech)
-        if gripper_goal:
-            self.gripper_bus.sync_write("Goal_Position", gripper_goal)
+        # Use batch MIT control for arm (sends all commands, then collects responses)
+        commands = {}
+        for motor_name, position_degrees in goal_pos.items():
+            idx = motor_index.get(motor_name, 0)
+            # Use custom gains if provided, otherwise use config defaults
+            if custom_kp is not None and motor_name in custom_kp:
+                kp = custom_kp[motor_name]
+            else:
+                kp = (
+                    self.config.position_kp[idx]
+                    if isinstance(self.config.position_kp, list)
+                    else self.config.position_kp
+                )
+            if custom_kd is not None and motor_name in custom_kd:
+                kd = custom_kd[motor_name]
+            else:
+                kd = (
+                    self.config.position_kd[idx]
+                    if isinstance(self.config.position_kd, list)
+                    else self.config.position_kd
+                )
+            commands[motor_name] = (kp, kd, position_degrees, 0.0, 0.0)
+
+        self.bus._mit_control_batch(commands)
 
-        goal_pos.update(arm_goal)
         return {f"{motor}.pos": val for motor, val in goal_pos.items()}
 
     @check_if_not_connected
     def disconnect(self):
+        """Disconnect from robot."""
+
+        # Disconnect CAN bus
         self.bus.disconnect(self.config.disable_torque_on_disconnect)
-        self.gripper_bus.disconnect(self.config.disable_torque_on_disconnect)
+
+        # Disconnect cameras
         for cam in self.cameras.values():
             cam.disconnect()
+
         logger.info(f"{self} disconnected.")

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/collision/link0_symp.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:baf52578e1d9e6225f3818cae82b6074a0b948d3cef8e9a3e6dfafca78507590
-size 40284

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/collision/link1_symp.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:066113d13d5cc85098609003bc7ebb73c570015350877f5ed7162ef1b6601852
-size 17784

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/collision/link2_symp.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:382ab32e4ae0880e8a1512e7a6ca6ce1f478a6c125db4efa977429ffb1d6b02a
-size 13384

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/collision/link3_symp.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:00c908cefab152c00416a570a48bf9aafed1549085f19ff2d882dc3f355d9f59
-size 156984

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/collision/link4_symp.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b54883b8c7c96268a68a5879f95998a53ad0b0c4fe74325fad63a6caef669c73
-size 1139984

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/collision/link5_symp.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:678a2802906eff7b45a836d2f34a2d8e51def50b6599376968f888e05c72739e
-size 751484

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/collision/link6_symp.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:95529bec23733476dfdbbb266c7db0d25a473a568de73c8337a82440fe4a9ac3
-size 30084

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/collision/link7_symp.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:434f207f21f75f5f0bd604e390b8e5bc7b62b619265222846770e06b3f9b5cfb
-size 23884

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/visual/link0.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:c77bdc9419947088e1dfc452e29c6092cc7b02b239ff4f2f5be3d77e393af185
-size 4148234

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/visual/link1.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:2a182c7fd4d25226aff6d9e2dd9d1008a85fce7df8e16525722a5c5053f8b055
-size 5741534

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/visual/link2.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:de1b190a56c16dea14546fb8e22c86eccabc2ca5e054819630c1932592381745
-size 4543534

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/visual/link3.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8d47d75d0c47a65021708ba63cb73162bf7c3b1d14e9cfc70dfa47336034ac76
-size 4978834

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/visual/link4.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8ca8149f2ce8b1b102270ec0b1a4b75c3e6f98c09b084430a450adce808607e1
-size 4944684

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/visual/link5.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:30ea7abfdd3661b315f897bb82a5f34fd966357b358e890e155e928e931ea975
-size 6322984

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/visual/link6.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e49f91279f109baecb9ff54f5041eeb4514f757ba6daa65c3ea01fb1991967e4
-size 4818434

src/lerobot/robots/openarm_follower/urdf/meshes/arm/v10/visual/link7.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:a91593b67d2dec16d1dfb6f1305df3ddcd214cdef02e97d5aba30bc633e775b2
-size 5114784

src/lerobot/robots/openarm_follower/urdf/meshes/body/v10/collision/body_link0_symp.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:6c18bbf7e86b03e3faf802e61e8eb438b38dcbcf146d97cffe6e808c65e9a72a
-size 293284

src/lerobot/robots/openarm_follower/urdf/meshes/body/v10/visual/body_link0.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7ab0afa9a26bebbf5c6149894c41d96622caa649555ae7c5f0f2548f86148d91
-size 7955034

src/lerobot/robots/openarm_follower/urdf/meshes/ee/openarm_hand/collision/finger.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8e96e1314618cf434908f70df78f68dd2b049c03538964e8d41fc99abe41564d
-size 13284

src/lerobot/robots/openarm_follower/urdf/meshes/ee/openarm_hand/collision/hand.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8e5d373ebbd3fd001b506058644062ad71a68f1ced5ca5d5ed0f6de20137956b
-size 18284

src/lerobot/robots/openarm_follower/urdf/meshes/ee/openarm_hand/visual/finger.stl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:25c115c7c55a422f30ad11581dc21576dc8fc4e09e659890772d86fe82ec04d7
-size 432484

src/lerobot/robots/openarm_follower/urdf/openarm_bimanual_pybullet.urdf

@@ -1,630 +0,0 @@
-<?xml version='1.0' encoding='utf-8'?>
-<robot name="openarm">
-  <link name="world" />
-  <joint name="openarm_body_world_joint" type="fixed">
-    <parent link="world" />
-    <child link="openarm_body_link0" />
-    <origin rpy="0 0 0" xyz="0 0 0" />
-  </joint>
-  <link name="openarm_body_link0">
-    <visual name="openarm_body_link0_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/body/v10/visual/body_link0.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_body_link0_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/body/v10/collision/body_link0_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <mass value="13.89" />
-      <inertia ixx="1.653" ixy="0.0" ixz="0.0" iyy="1.653" iyz="0.0" izz="0.051" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_openarm_body_link0_joint" type="fixed">
-    <parent link="openarm_body_link0" />
-    <child link="openarm_left_link0" />
-    <origin rpy="-1.5708 0 0" xyz="0.0 0.031 0.698" />
-  </joint>
-  <link name="openarm_left_link0">
-    <visual name="openarm_left_link0_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link0.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link0_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link0_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0009483362816297526 -0.0001580207020448382 0.03076860287587199" />
-      <mass value="1.1432284943239561" />
-      <inertia ixx="0.001128" ixy="-4e-06" ixz="-3.3e-05" iyy="0.000962" iyz="-7e-06" izz="0.00147" />
-    </inertial>
-  </link>
-  <link name="openarm_left_link1">
-    <visual name="openarm_left_link1_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 0.0 -0.0625" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link1.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link1_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 0.0 -0.0625" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link1_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.0011467657911800769 -3.319987657026362e-05 0.05395284380736254" />
-      <mass value="1.1416684646202298" />
-      <inertia ixx="0.001567" ixy="-1e-06" ixz="-2.9e-05" iyy="0.001273" iyz="1e-06" izz="0.001016" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint1" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0 0.0 0.0625" />
-    <parent link="openarm_left_link0" />
-    <child link="openarm_left_link1" />
-    <axis xyz="0 0 1" />
-    <limit effort="40" lower="-3.490659" upper="1.3962629999999998" velocity="16.754666" />
-  </joint>
-  <link name="openarm_left_link2">
-    <visual name="openarm_left_link2_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0301 0.0 -0.1225" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link2.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link2_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0301 0.0 -0.1225" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link2_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.00839629182351943 2.0145102027597523e-08 0.03256649300522363" />
-      <mass value="0.2775092746011571" />
-      <inertia ixx="0.000359" ixy="1e-06" ixz="-0.000109" iyy="0.000376" iyz="1e-06" izz="0.000232" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint2" type="revolute">
-    <origin rpy="-1.57079632679 0 0" xyz="-0.0301 0.0 0.06" />
-    <parent link="openarm_left_link1" />
-    <child link="openarm_left_link2" />
-    <axis xyz="-1 0 0" />
-    <limit effort="40" lower="-3.3161253267948965" upper="0.17453267320510335" velocity="16.754666" />
-  </joint>
-  <link name="openarm_left_link3">
-    <visual name="openarm_left_link3_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.18875" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link3.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link3_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.18875" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link3_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.002104752099628911 -0.0005549085042607548 0.09047470545721961" />
-      <mass value="1.073863338202347" />
-      <inertia ixx="0.004372" ixy="1e-06" ixz="1.1e-05" iyy="0.004319" iyz="-3.6e-05" izz="0.000661" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint3" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0301 0.0 0.06625" />
-    <parent link="openarm_left_link2" />
-    <child link="openarm_left_link3" />
-    <axis xyz="0 0 1" />
-    <limit effort="27" lower="-1.570796" upper="1.570796" velocity="5.445426" />
-  </joint>
-  <link name="openarm_left_link4">
-    <visual name="openarm_left_link4_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0315 -0.3425" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link4.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link4_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0315 -0.3425" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link4_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0029006831074562967 -0.03030575826634669 0.06339637422196209" />
-      <mass value="0.6348534566833373" />
-      <inertia ixx="0.000623" ixy="-1e-06" ixz="-1.9e-05" iyy="0.000511" iyz="3.8e-05" izz="0.000334" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint4" type="revolute">
-    <origin rpy="0 0 0" xyz="-0.0 0.0315 0.15375" />
-    <parent link="openarm_left_link3" />
-    <child link="openarm_left_link4" />
-    <axis xyz="0 1 0" />
-    <limit effort="27" lower="0.0" upper="2.443461" velocity="5.445426" />
-  </joint>
-  <link name="openarm_left_link5">
-    <visual name="openarm_left_link5_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.438" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link5.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link5_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.438" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link5_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.003049665024221911 -0.0008866902457326625 0.043079803024980934" />
-      <mass value="0.6156588026168502" />
-      <inertia ixx="0.000423" ixy="-8e-06" ixz="6e-06" iyy="0.000445" iyz="-6e-06" izz="0.000324" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint5" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0 -0.0315 0.0955" />
-    <parent link="openarm_left_link4" />
-    <child link="openarm_left_link5" />
-    <axis xyz="0 0 1" />
-    <limit effort="7" lower="-1.570796" upper="1.570796" velocity="20.943946" />
-  </joint>
-  <link name="openarm_left_link6">
-    <visual name="openarm_left_link6_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0375 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link6.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link6_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0375 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link6_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.037136587005447405 -0.00033230528343419053 -9.498374522309838e-05" />
-      <mass value="0.475202773187987" />
-      <inertia ixx="0.000143" ixy="1e-06" ixz="1e-06" iyy="0.000157" iyz="1e-06" izz="0.000159" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint6" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0375 0.0 0.1205" />
-    <parent link="openarm_left_link5" />
-    <child link="openarm_left_link6" />
-    <axis xyz="1 0 0" />
-    <limit effort="7" lower="-0.785398" upper="0.785398" velocity="20.943946" />
-  </joint>
-  <link name="openarm_left_link7">
-    <visual name="openarm_left_link7_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link7.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link7_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link7_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="6.875510271106056e-05 -0.01266175250761268 0.06951945409987448" />
-      <mass value="0.4659771327380578" />
-      <inertia ixx="0.000639" ixy="1e-06" ixz="1e-06" iyy="0.000497" iyz="8.9e-05" izz="0.000342" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint7" type="revolute">
-    <origin rpy="0 0 0" xyz="-0.0375 0.0 0.0" />
-    <parent link="openarm_left_link6" />
-    <child link="openarm_left_link7" />
-    <axis xyz="0 -1 0" />
-    <limit effort="7" lower="-1.570796" upper="1.570796" velocity="20.943946" />
-  </joint>
-  <joint name="openarm_right_openarm_body_link0_joint" type="fixed">
-    <parent link="openarm_body_link0" />
-    <child link="openarm_right_link0" />
-    <origin rpy="1.5708 0 0" xyz="0.0 -0.031 0.698" />
-  </joint>
-  <link name="openarm_right_link0">
-    <visual name="openarm_right_link0_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link0.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link0_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link0_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0009483362816297526 0.0001580207020448382 0.03076860287587199" />
-      <mass value="1.1432284943239561" />
-      <inertia ixx="0.001128" ixy="-4e-06" ixz="-3.3e-05" iyy="0.000962" iyz="-7e-06" izz="0.00147" />
-    </inertial>
-  </link>
-  <link name="openarm_right_link1">
-    <visual name="openarm_right_link1_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 0.0 -0.0625" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link1.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link1_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 0.0 -0.0625" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link1_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.0011467657911800769 3.319987657026362e-05 0.05395284380736254" />
-      <mass value="1.1416684646202298" />
-      <inertia ixx="0.001567" ixy="-1e-06" ixz="-2.9e-05" iyy="0.001273" iyz="1e-06" izz="0.001016" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint1" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0 0.0 0.0625" />
-    <parent link="openarm_right_link0" />
-    <child link="openarm_right_link1" />
-    <axis xyz="0 0 1" />
-    <limit effort="40" lower="-1.396263" upper="3.490659" velocity="16.754666" />
-  </joint>
-  <link name="openarm_right_link2">
-    <visual name="openarm_right_link2_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0301 0.0 -0.1225" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link2.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link2_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0301 0.0 -0.1225" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link2_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.00839629182351943 -2.0145102027597523e-08 0.03256649300522363" />
-      <mass value="0.2775092746011571" />
-      <inertia ixx="0.000359" ixy="1e-06" ixz="-0.000109" iyy="0.000376" iyz="1e-06" izz="0.000232" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint2" type="revolute">
-    <origin rpy="1.57079632679 0 0" xyz="-0.0301 0.0 0.06" />
-    <parent link="openarm_right_link1" />
-    <child link="openarm_right_link2" />
-    <axis xyz="-1 0 0" />
-    <limit effort="40" lower="-0.17453267320510335" upper="3.3161253267948965" velocity="16.754666" />
-  </joint>
-  <link name="openarm_right_link3">
-    <visual name="openarm_right_link3_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.18875" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link3.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link3_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.18875" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link3_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.002104752099628911 0.0005549085042607548 0.09047470545721961" />
-      <mass value="1.073863338202347" />
-      <inertia ixx="0.004372" ixy="1e-06" ixz="1.1e-05" iyy="0.004319" iyz="-3.6e-05" izz="0.000661" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint3" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0301 0.0 0.06625" />
-    <parent link="openarm_right_link2" />
-    <child link="openarm_right_link3" />
-    <axis xyz="0 0 1" />
-    <limit effort="27" lower="-1.570796" upper="1.570796" velocity="5.445426" />
-  </joint>
-  <link name="openarm_right_link4">
-    <visual name="openarm_right_link4_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0315 -0.3425" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link4.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link4_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0315 -0.3425" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link4_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0029006831074562967 -0.03030575826634669 0.06339637422196209" />
-      <mass value="0.6348534566833373" />
-      <inertia ixx="0.000623" ixy="-1e-06" ixz="-1.9e-05" iyy="0.000511" iyz="3.8e-05" izz="0.000334" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint4" type="revolute">
-    <origin rpy="0 0 0" xyz="-0.0 0.0315 0.15375" />
-    <parent link="openarm_right_link3" />
-    <child link="openarm_right_link4" />
-    <axis xyz="0 1 0" />
-    <limit effort="27" lower="0.0" upper="2.443461" velocity="5.445426" />
-  </joint>
-  <link name="openarm_right_link5">
-    <visual name="openarm_right_link5_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.438" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link5.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link5_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.438" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link5_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.003049665024221911 0.0008866902457326625 0.043079803024980934" />
-      <mass value="0.6156588026168502" />
-      <inertia ixx="0.000423" ixy="-8e-06" ixz="6e-06" iyy="0.000445" iyz="-6e-06" izz="0.000324" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint5" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0 -0.0315 0.0955" />
-    <parent link="openarm_right_link4" />
-    <child link="openarm_right_link5" />
-    <axis xyz="0 0 1" />
-    <limit effort="7" lower="-1.570796" upper="1.570796" velocity="20.943946" />
-  </joint>
-  <link name="openarm_right_link6">
-    <visual name="openarm_right_link6_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0375 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link6.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link6_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0375 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link6_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.037136587005447405 0.00033230528343419053 -9.498374522309838e-05" />
-      <mass value="0.475202773187987" />
-      <inertia ixx="0.000143" ixy="1e-06" ixz="1e-06" iyy="0.000157" iyz="1e-06" izz="0.000159" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint6" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0375 0.0 0.1205" />
-    <parent link="openarm_right_link5" />
-    <child link="openarm_right_link6" />
-    <axis xyz="1 0 0" />
-    <limit effort="7" lower="-0.785398" upper="0.785398" velocity="20.943946" />
-  </joint>
-  <link name="openarm_right_link7">
-    <visual name="openarm_right_link7_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link7.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link7_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link7_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="6.875510271106056e-05 0.01266175250761268 0.06951945409987448" />
-      <mass value="0.4659771327380578" />
-      <inertia ixx="0.000639" ixy="1e-06" ixz="1e-06" iyy="0.000497" iyz="8.9e-05" izz="0.000342" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint7" type="revolute">
-    <origin rpy="0 0 0" xyz="-0.0375 0.0 0.0" />
-    <parent link="openarm_right_link6" />
-    <child link="openarm_right_link7" />
-    <axis xyz="0 1 0" />
-    <limit effort="7" lower="-1.570796" upper="1.570796" velocity="20.943946" />
-  </joint>
-  <link name="openarm_left_hand">
-    <visual name="openarm_left_hand_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 -0.6585" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/hand.dae" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_hand_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 -0.6585" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/hand.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0 0.002 0.03" />
-      <mass value="0.35" />
-      <inertia ixx="0.0002473" ixy="1e-06" ixz="1e-06" iyy="1.763e-05" iyz="1e-06" izz="0.0002521" />
-    </inertial>
-  </link>
-  <joint name="left_openarm_hand_joint" type="fixed">
-    <parent link="openarm_left_link7" />
-    <child link="openarm_left_hand" />
-    <origin rpy="0 0 0" xyz="0 -0.0 0.1001" />
-  </joint>
-  <link name="openarm_left_hand_tcp">
-    <inertial>
-      <origin xyz="0 0 0" rpy="0 0 0" />
-      <mass value="0.001" />
-      <inertia ixx="0.000001" ixy="0.0" ixz="0.0" iyy="0.000001" iyz="0.0" izz="0.000001" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_hand_tcp_joint" type="fixed">
-    <origin rpy="0 0 0" xyz="0 -0.0 0.08" />
-    <parent link="openarm_left_hand" />
-    <child link="openarm_left_hand_tcp" />
-  </joint>
-  <link name="openarm_left_left_finger">
-    <visual name="openarm_left_left_finger_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/finger.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_left_finger_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/finger.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0064528 0.01702 0.0219685" />
-      <mass value="0.03602545343277134" />
-      <inertia ixx="2.3749999999999997e-06" ixy="1e-06" ixz="1e-06" iyy="2.3749999999999997e-06" iyz="1e-06" izz="7.5e-07" />
-    </inertial>
-  </link>
-  <link name="openarm_left_right_finger">
-    <visual name="openarm_left_right_finger_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/finger.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_right_finger_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/finger.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0064528 -0.01702 0.0219685" />
-      <mass value="0.03602545343277134" />
-      <inertia ixx="2.3749999999999997e-06" ixy="1e-06" ixz="1e-06" iyy="2.3749999999999997e-06" iyz="1e-06" izz="7.5e-07" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_finger_joint1" type="prismatic">
-    <parent link="openarm_left_hand" />
-    <child link="openarm_left_right_finger" />
-    <origin rpy="0 0 0" xyz="0 -0.006 0.015" />
-    <axis xyz="0 -1 0" />
-    <limit effort="333" lower="0.0" upper="0.044" velocity="10.0" />
-  </joint>
-  <joint name="openarm_left_finger_joint2" type="prismatic">
-    <parent link="openarm_left_hand" />
-    <child link="openarm_left_left_finger" />
-    <origin rpy="0 0 0" xyz="0 0.006 0.015" />
-    <axis xyz="0 1 0" />
-    <limit effort="333" lower="0.0" upper="0.044" velocity="10.0" />
-    <mimic joint="openarm_left_finger_joint1" />
-  </joint>
-  <link name="openarm_right_hand">
-    <visual name="openarm_right_hand_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 -0.6585" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/hand.dae" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_hand_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 -0.6585" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/hand.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0 0.002 0.03" />
-      <mass value="0.35" />
-      <inertia ixx="0.0002473" ixy="1e-06" ixz="1e-06" iyy="1.763e-05" iyz="1e-06" izz="0.0002521" />
-    </inertial>
-  </link>
-  <link name="openarm_right_ee_target">
-    <inertial>
-      <origin xyz="0 0 0" rpy="0 0 0" />
-      <mass value="0.001" />
-      <inertia ixx="0.000001" ixy="0.0" ixz="0.0" iyy="0.000001" iyz="0.0" izz="0.000001" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_ee_target_joint" type="fixed">
-    <parent link="openarm_right_link7" />
-    <child link="openarm_right_ee_target" />
-    <origin rpy="0 0 0" xyz="0 0.0 0.07" />
-  </joint>
-  <joint name="right_openarm_hand_joint" type="fixed">
-    <parent link="openarm_right_link7" />
-    <child link="openarm_right_hand" />
-    <origin rpy="0 0 0" xyz="0 -0.0 0.1001" />
-  </joint>
-  <link name="openarm_right_hand_tcp">
-    <inertial>
-      <origin xyz="0 0 0" rpy="0 0 0" />
-      <mass value="0.001" />
-      <inertia ixx="0.000001" ixy="0.0" ixz="0.0" iyy="0.000001" iyz="0.0" izz="0.000001" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_hand_tcp_joint" type="fixed">
-    <origin rpy="0 0 0" xyz="0 -0.0 0.08" />
-    <parent link="openarm_right_hand" />
-    <child link="openarm_right_hand_tcp" />
-  </joint>
-  <link name="openarm_right_left_finger">
-    <visual name="openarm_right_left_finger_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/finger.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_left_finger_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/finger.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0064528 0.01702 0.0219685" />
-      <mass value="0.03602545343277134" />
-      <inertia ixx="2.3749999999999997e-06" ixy="1e-06" ixz="1e-06" iyy="2.3749999999999997e-06" iyz="1e-06" izz="7.5e-07" />
-    </inertial>
-  </link>
-  <link name="openarm_right_right_finger">
-    <visual name="openarm_right_right_finger_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/finger.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_right_finger_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/finger.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0064528 -0.01702 0.0219685" />
-      <mass value="0.03602545343277134" />
-      <inertia ixx="2.3749999999999997e-06" ixy="1e-06" ixz="1e-06" iyy="2.3749999999999997e-06" iyz="1e-06" izz="7.5e-07" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_finger_joint1" type="prismatic">
-    <parent link="openarm_right_hand" />
-    <child link="openarm_right_right_finger" />
-    <origin rpy="0 0 0" xyz="0 -0.006 0.015" />
-    <axis xyz="0 -1 0" />
-    <limit effort="333" lower="0.0" upper="0.044" velocity="10.0" />
-  </joint>
-  <joint name="openarm_right_finger_joint2" type="prismatic">
-    <parent link="openarm_right_hand" />
-    <child link="openarm_right_left_finger" />
-    <origin rpy="0 0 0" xyz="0 0.006 0.015" />
-    <axis xyz="0 1 0" />
-    <limit effort="333" lower="0.0" upper="0.044" velocity="10.0" />
-    <mimic joint="openarm_right_finger_joint1" />
-  </joint>
-</robot>

src/lerobot/robots/openarm_follower/urdf/replay_viewer.html

@@ -1,408 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="UTF-8" />
-<title>Dataset Replay — EE Frame Viewer</title>
-<style>
-  * { margin: 0; padding: 0; box-sizing: border-box; }
-  body { background: #0d1117; overflow: hidden; font-family: 'JetBrains Mono', monospace; color: #c9d1d9; }
-  canvas { display: block; }
-
-  #panel {
-    position: absolute; top: 14px; left: 14px;
-    background: rgba(13,17,23,0.92); border: 1px solid #30363d;
-    border-radius: 10px; padding: 16px 20px; z-index: 10;
-    width: 340px; backdrop-filter: blur(8px);
-  }
-  #panel h2 { font-size: 14px; color: #58a6ff; margin-bottom: 10px; letter-spacing: 0.5px; }
-
-  .row { display: flex; align-items: center; gap: 8px; margin: 6px 0; font-size: 12px; }
-  .row label { width: 70px; color: #8b949e; flex-shrink: 0; }
-  .row .val { color: #f0f6fc; font-variant-numeric: tabular-nums; }
-
-  #transport {
-    margin-top: 12px; display: flex; align-items: center; gap: 8px;
-  }
-  #transport button {
-    background: #21262d; color: #c9d1d9; border: 1px solid #30363d;
-    padding: 6px 14px; border-radius: 6px; cursor: pointer;
-    font-family: inherit; font-size: 12px; transition: background 0.15s;
-  }
-  #transport button:hover { background: #30363d; }
-  #transport button.active { background: #1f6feb; border-color: #1f6feb; color: #fff; }
-
-  #scrubber {
-    width: 100%; margin-top: 8px;
-    -webkit-appearance: none; appearance: none;
-    height: 6px; border-radius: 3px; background: #21262d; outline: none;
-  }
-  #scrubber::-webkit-slider-thumb {
-    -webkit-appearance: none; width: 14px; height: 14px;
-    border-radius: 50%; background: #58a6ff; cursor: pointer;
-  }
-
-  #speed-ctrl { margin-top: 6px; }
-  #speed-ctrl select {
-    background: #21262d; color: #c9d1d9; border: 1px solid #30363d;
-    padding: 4px 8px; border-radius: 4px; font-family: inherit; font-size: 11px;
-  }
-
-  #frame-counter {
-    font-size: 11px; color: #8b949e; margin-top: 6px;
-    font-variant-numeric: tabular-nums;
-  }
-
-  .legend { display: flex; align-items: center; gap: 6px; margin: 3px 0; font-size: 11px; }
-  .dot { width: 10px; height: 10px; border-radius: 50%; display: inline-block; }
-</style>
-<link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono:wght@400;600&display=swap" rel="stylesheet">
-</head>
-<body>
-
-<div id="panel">
-  <h2>DATASET REPLAY — EE FRAME</h2>
-  <div style="font-size:11px;color:#8b949e;margin-bottom:8px;">glannuzel/grabette-dataset · episode 0</div>
-
-  <div class="legend"><span class="dot" style="background:#ff6b6b"></span> EE target (dataset)</div>
-  <div class="legend"><span class="dot" style="background:#ffd43b"></span> Trajectory (past)</div>
-  <div class="legend"><span class="dot" style="background:#30363d"></span> Trajectory (future)</div>
-
-  <div class="row"><label>x</label><span class="val" id="v-x">—</span></div>
-  <div class="row"><label>y</label><span class="val" id="v-y">—</span></div>
-  <div class="row"><label>z</label><span class="val" id="v-z">—</span></div>
-  <div class="row"><label>ax</label><span class="val" id="v-ax">—</span></div>
-  <div class="row"><label>ay</label><span class="val" id="v-ay">—</span></div>
-  <div class="row"><label>az</label><span class="val" id="v-az">—</span></div>
-  <div class="row"><label>gripper</label><span class="val" id="v-grip">—</span></div>
-
-  <div id="transport">
-    <button id="btn-play" onclick="togglePlay()">▶ Play</button>
-    <button onclick="stepFrame(-1)">◀</button>
-    <button onclick="stepFrame(1)">▶</button>
-    <button onclick="resetPlay()">⟳</button>
-  </div>
-  <input type="range" id="scrubber" min="0" max="1" value="0" step="1" />
-  <div id="speed-ctrl">
-    <label style="font-size:11px;color:#8b949e;">Speed:</label>
-    <select id="speed-select" onchange="setSpeed(this.value)">
-      <option value="0.25">0.25×</option>
-      <option value="0.5">0.5×</option>
-      <option value="1" selected>1×</option>
-      <option value="2">2×</option>
-      <option value="4">4×</option>
-    </select>
-  </div>
-  <div id="frame-counter">Frame 0 / 0 · 0.00s</div>
-</div>
-
-<script type="importmap">
-{
-  "imports": {
-    "three": "https://cdn.jsdelivr.net/npm/three@0.169.0/build/three.module.js",
-    "three/examples/jsm/": "https://cdn.jsdelivr.net/npm/three@0.169.0/examples/jsm/"
-  }
-}
-</script>
-
-<script type="module">
-import * as THREE from 'three';
-import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
-import { STLLoader } from 'three/examples/jsm/loaders/STLLoader.js';
-
-let trajectory = null;
-let currentFrame = 0;
-let playing = false;
-let speed = 1.0;
-let lastTime = 0;
-let accumulator = 0;
-
-// Anchor: EE tip world position at zero-joint pose (in Y-up Three.js space)
-const eeAnchor = new THREE.Vector3();
-// Z-up → Y-up rotation (same as robotGroup): -90° around X
-const zUpToYUp = new THREE.Quaternion().setFromAxisAngle(new THREE.Vector3(1, 0, 0), -Math.PI / 2);
-
-const scene = new THREE.Scene();
-scene.background = new THREE.Color(0x0d1117);
-
-const camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 0.01, 100);
-
-const renderer = new THREE.WebGLRenderer({ antialias: true });
-renderer.setSize(window.innerWidth, window.innerHeight);
-renderer.setPixelRatio(window.devicePixelRatio);
-renderer.shadowMap.enabled = true;
-document.body.appendChild(renderer.domElement);
-
-const controls = new OrbitControls(camera, renderer.domElement);
-controls.enableDamping = true;
-controls.dampingFactor = 0.08;
-
-scene.add(new THREE.AmbientLight(0xffffff, 0.8));
-const dirLight = new THREE.DirectionalLight(0xffffff, 1.4);
-dirLight.position.set(2, 4, 3);
-scene.add(dirLight);
-scene.add(new THREE.DirectionalLight(0x8899cc, 0.6).translateX(-2).translateY(1).translateZ(-3));
-scene.add(new THREE.DirectionalLight(0xffffff, 0.5).translateY(-1).translateZ(2));
-
-const grid = new THREE.GridHelper(2, 20, 0x21262d, 0x161b22);
-scene.add(grid);
-scene.add(new THREE.AxesHelper(0.15));
-
-// EE marker
-const eeMarker = new THREE.Mesh(
-  new THREE.SphereGeometry(0.012, 20, 20),
-  new THREE.MeshStandardMaterial({ color: 0xff6b6b, emissive: 0xff6b6b, emissiveIntensity: 0.7 })
-);
-scene.add(eeMarker);
-eeMarker.add(new THREE.AxesHelper(0.06));
-
-// Trajectory lines
-const MAX_POINTS = 2000;
-const pastGeo = new THREE.BufferGeometry();
-pastGeo.setAttribute('position', new THREE.Float32BufferAttribute(new Float32Array(MAX_POINTS * 3), 3));
-const pastLine = new THREE.Line(pastGeo, new THREE.LineBasicMaterial({ color: 0xffd43b, linewidth: 2 }));
-scene.add(pastLine);
-
-const futureGeo = new THREE.BufferGeometry();
-futureGeo.setAttribute('position', new THREE.Float32BufferAttribute(new Float32Array(MAX_POINTS * 3), 3));
-const futureLine = new THREE.Line(futureGeo, new THREE.LineBasicMaterial({ color: 0x30363d, linewidth: 1 }));
-scene.add(futureLine);
-
-// URDF
-const stlLoader = new STLLoader();
-const robotGroup = new THREE.Group();
-// URDF is Z-up; Three.js is Y-up → rotate -90° around X
-robotGroup.rotation.x = -Math.PI / 2;
-scene.add(robotGroup);
-let urdfLinks = {};
-
-function rotvecToQuat(ax, ay, az) {
-  const angle = Math.sqrt(ax * ax + ay * ay + az * az);
-  if (angle < 1e-8) return new THREE.Quaternion();
-  return new THREE.Quaternion().setFromAxisAngle(
-    new THREE.Vector3(ax / angle, ay / angle, az / angle), angle
-  );
-}
-
-async function loadURDF() {
-  const resp = await fetch('./openarm_bimanual_pybullet.urdf');
-  const text = await resp.text();
-  const xml = new DOMParser().parseFromString(text, 'text/xml');
-
-  const links = {};
-
-  for (const linkEl of xml.querySelectorAll('link')) {
-    const name = linkEl.getAttribute('name');
-    const group = new THREE.Group();
-    group.name = name;
-
-    const visual = linkEl.querySelector('visual');
-    if (visual) {
-      const meshEl = visual.querySelector('mesh');
-      const originEl = visual.querySelector('origin');
-      if (meshEl) {
-        const filename = meshEl.getAttribute('filename');
-        const scaleStr = meshEl.getAttribute('scale');
-        const sc = scaleStr ? scaleStr.split(' ').map(Number) : [1, 1, 1];
-        let xyz = [0, 0, 0];
-        if (originEl && originEl.getAttribute('xyz'))
-          xyz = originEl.getAttribute('xyz').split(' ').map(Number);
-        if (filename.endsWith('.stl')) {
-          try {
-            const geo = await new Promise((res, rej) =>
-              stlLoader.load(filename, res, undefined, rej));
-            const mesh = new THREE.Mesh(geo, new THREE.MeshStandardMaterial({
-              color: 0x8899bb, metalness: 0.3, roughness: 0.5,
-            }));
-            mesh.scale.set(sc[0], sc[1], sc[2]);
-            mesh.position.set(xyz[0], xyz[1], xyz[2]);
-            group.add(mesh);
-          } catch (e) { /* skip missing mesh */ }
-        }
-      }
-    }
-    links[name] = group;
-  }
-
-  const rootLinks = new Set(Object.keys(links));
-
-  for (const jointEl of xml.querySelectorAll('joint')) {
-    const parentName = jointEl.querySelector('parent').getAttribute('link');
-    const childName = jointEl.querySelector('child').getAttribute('link');
-    rootLinks.delete(childName);
-
-    const originEl = jointEl.querySelector('origin');
-    let xyz = [0, 0, 0], rpy = [0, 0, 0];
-    if (originEl) {
-      if (originEl.getAttribute('xyz')) xyz = originEl.getAttribute('xyz').split(' ').map(Number);
-      if (originEl.getAttribute('rpy')) rpy = originEl.getAttribute('rpy').split(' ').map(Number);
-    }
-
-    const parent = links[parentName];
-    const child = links[childName];
-    if (!parent || !child) continue;
-
-    child.position.set(xyz[0], xyz[1], xyz[2]);
-    if (rpy[0] || rpy[1] || rpy[2])
-      child.rotation.set(rpy[0], rpy[1], rpy[2], 'XYZ');
-    parent.add(child);
-  }
-
-  for (const n of rootLinks)
-    if (links[n]) robotGroup.add(links[n]);
-
-  // EE target marker on the URDF
-  const eeTargetLink = links['openarm_right_ee_target'];
-  if (eeTargetLink) {
-    eeTargetLink.add(new THREE.Mesh(
-      new THREE.TorusGeometry(0.02, 0.002, 8, 32),
-      new THREE.MeshStandardMaterial({ color: 0xffaa00, emissive: 0xffaa00, emissiveIntensity: 0.5 })
-    ));
-    eeTargetLink.add(new THREE.AxesHelper(0.05));
-  }
-
-  urdfLinks = links;
-}
-
-async function loadTrajectory() {
-  const resp = await fetch('./trajectory_ep0.json');
-  trajectory = await resp.json();
-  document.getElementById('scrubber').max = trajectory.num_frames - 1;
-  document.getElementById('scrubber').value = 0;
-}
-
-function computeOffset() {
-  if (!trajectory || !urdfLinks['openarm_right_ee_target']) return;
-
-  robotGroup.updateMatrixWorld(true);
-  const eeLink = urdfLinks['openarm_right_ee_target'];
-  eeLink.getWorldPosition(eeAnchor);
-
-  controls.target.copy(eeAnchor);
-  camera.position.set(eeAnchor.x + 0.8, eeAnchor.y + 0.3, eeAnchor.z + 0.0);
-  controls.update();
-
-  updateFrame(0);
-}
-
-function mapFramePos(f) {
-  const f0 = trajectory.frames[0];
-  const delta = new THREE.Vector3(f.x - f0.x, f.y - f0.y, f.z - f0.z);
-  delta.applyQuaternion(zUpToYUp);
-  return delta.add(eeAnchor);
-}
-
-function updateFrame(idx) {
-  if (!trajectory) return;
-  currentFrame = Math.max(0, Math.min(idx, trajectory.num_frames - 1));
-
-  const f = trajectory.frames[currentFrame];
-  const pos = mapFramePos(f);
-
-  eeMarker.position.copy(pos);
-  // Orientation: rotate the dataset axis-angle into Y-up space
-  const q = rotvecToQuat(f.ax, f.ay, f.az);
-  eeMarker.quaternion.copy(zUpToYUp).multiply(q);
-
-  // Past trajectory
-  const pastArr = pastGeo.attributes.position.array;
-  let pi = 0;
-  for (let i = 0; i <= currentFrame && i < MAX_POINTS; i++) {
-    const p = mapFramePos(trajectory.frames[i]);
-    pastArr[pi++] = p.x; pastArr[pi++] = p.y; pastArr[pi++] = p.z;
-  }
-  pastGeo.setDrawRange(0, Math.min(currentFrame + 1, MAX_POINTS));
-  pastGeo.attributes.position.needsUpdate = true;
-
-  // Future trajectory
-  const futArr = futureGeo.attributes.position.array;
-  let fi = 0;
-  for (let i = currentFrame; i < trajectory.num_frames && (i - currentFrame) < MAX_POINTS; i++) {
-    const p = mapFramePos(trajectory.frames[i]);
-    futArr[fi++] = p.x; futArr[fi++] = p.y; futArr[fi++] = p.z;
-  }
-  futureGeo.setDrawRange(0, Math.min(trajectory.num_frames - currentFrame, MAX_POINTS));
-  futureGeo.attributes.position.needsUpdate = true;
-
-  // UI
-  document.getElementById('v-x').textContent = pos.x.toFixed(4);
-  document.getElementById('v-y').textContent = pos.y.toFixed(4);
-  document.getElementById('v-z').textContent = pos.z.toFixed(4);
-  document.getElementById('v-ax').textContent = f.ax.toFixed(4);
-  document.getElementById('v-ay').textContent = f.ay.toFixed(4);
-  document.getElementById('v-az').textContent = f.az.toFixed(4);
-  document.getElementById('v-grip').textContent =
-    `p=${f.proximal.toFixed(2)} d=${f.distal.toFixed(2)}`;
-
-  document.getElementById('scrubber').value = currentFrame;
-  const timeS = (currentFrame / trajectory.fps).toFixed(2);
-  document.getElementById('frame-counter').textContent =
-    `Frame ${currentFrame} / ${trajectory.num_frames - 1} · ${timeS}s`;
-}
-
-// Playback controls
-window.togglePlay = function() {
-  playing = !playing;
-  const btn = document.getElementById('btn-play');
-  btn.textContent = playing ? '⏸ Pause' : '▶ Play';
-  btn.classList.toggle('active', playing);
-  if (playing) { lastTime = performance.now(); accumulator = 0; }
-};
-
-window.stepFrame = function(delta) {
-  playing = false;
-  document.getElementById('btn-play').textContent = '▶ Play';
-  document.getElementById('btn-play').classList.remove('active');
-  updateFrame(currentFrame + delta);
-};
-
-window.resetPlay = function() {
-  playing = false;
-  document.getElementById('btn-play').textContent = '▶ Play';
-  document.getElementById('btn-play').classList.remove('active');
-  updateFrame(0);
-};
-
-window.setSpeed = function(v) { speed = parseFloat(v); };
-
-document.getElementById('scrubber').addEventListener('input', (e) => {
-  updateFrame(parseInt(e.target.value));
-});
-
-window.addEventListener('resize', () => {
-  camera.aspect = window.innerWidth / window.innerHeight;
-  camera.updateProjectionMatrix();
-  renderer.setSize(window.innerWidth, window.innerHeight);
-});
-
-function animate(now) {
-  requestAnimationFrame(animate);
-  controls.update();
-
-  if (playing && trajectory) {
-    const dt = (now - lastTime) / 1000;
-    lastTime = now;
-    accumulator += dt * speed;
-    const frameDuration = 1.0 / trajectory.fps;
-    while (accumulator >= frameDuration) {
-      accumulator -= frameDuration;
-      if (currentFrame < trajectory.num_frames - 1) {
-        updateFrame(currentFrame + 1);
-      } else {
-        playing = false;
-        document.getElementById('btn-play').textContent = '▶ Play';
-        document.getElementById('btn-play').classList.remove('active');
-        break;
-      }
-    }
-  }
-
-  renderer.render(scene, camera);
-}
-requestAnimationFrame(animate);
-
-Promise.all([loadURDF(), loadTrajectory()])
-  .then(() => computeOffset())
-  .catch(err => console.error(err));
-</script>
-</body>
-</html>

src/lerobot/robots/openarm_follower/urdf/viewer.html

@@ -1,311 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="UTF-8" />
-<title>OpenArm URDF Viewer</title>
-<style>
-  * { margin: 0; padding: 0; box-sizing: border-box; }
-  body { background: #1a1a2e; overflow: hidden; font-family: 'IBM Plex Mono', monospace; }
-  canvas { display: block; }
-  #info {
-    position: absolute; top: 16px; left: 16px;
-    color: #e0e0e0; font-size: 13px; line-height: 1.6;
-    background: rgba(0,0,0,0.7); padding: 14px 18px; border-radius: 8px;
-    border: 1px solid #333; max-width: 340px; z-index: 10;
-  }
-  #info h2 { font-size: 15px; color: #fff; margin-bottom: 8px; }
-  .legend { display: flex; align-items: center; gap: 8px; margin: 4px 0; }
-  .dot { width: 12px; height: 12px; border-radius: 50%; display: inline-block; flex-shrink: 0; }
-  .dot-red { background: #ff4444; }
-  .dot-green { background: #44ff44; }
-  .dot-blue { background: #4488ff; }
-  #frame-select { margin-top: 10px; }
-  #frame-select button {
-    background: #333; color: #e0e0e0; border: 1px solid #555;
-    padding: 6px 10px; margin: 2px; border-radius: 4px; cursor: pointer;
-    font-family: inherit; font-size: 12px;
-  }
-  #frame-select button:hover { background: #555; }
-  #frame-select button.active { background: #4488ff; color: #fff; border-color: #4488ff; }
-  #status { margin-top: 8px; font-size: 11px; color: #888; }
-</style>
-<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;600&display=swap" rel="stylesheet">
-</head>
-<body>
-
-<div id="info">
-  <h2>OpenArm Right Arm — EE Frame Options</h2>
-  <div class="legend"><span class="dot dot-red"></span> openarm_right_link7 (wrist output)</div>
-  <div class="legend"><span class="dot" style="background:#ffaa00"></span> openarm_right_ee_target (+5cm)</div>
-  <div class="legend"><span class="dot dot-green"></span> openarm_right_hand (+10cm)</div>
-  <div class="legend"><span class="dot dot-blue"></span> openarm_right_hand_tcp (+18cm)</div>
-  <div id="frame-select">
-    <button onclick="focusFrame('link7')" class="active">link7</button>
-    <button onclick="focusFrame('ee_target')">ee_target</button>
-    <button onclick="focusFrame('hand')">hand</button>
-    <button onclick="focusFrame('tcp')">hand_tcp</button>
-  </div>
-  <div id="status">Loading URDF...</div>
-  <p style="margin-top:8px;font-size:11px;color:#888;">Drag to orbit · Scroll to zoom · Right-drag to pan</p>
-</div>
-
-<script type="importmap">
-{
-  "imports": {
-    "three": "https://cdn.jsdelivr.net/npm/three@0.169.0/build/three.module.js",
-    "three/examples/jsm/": "https://cdn.jsdelivr.net/npm/three@0.169.0/examples/jsm/"
-  }
-}
-</script>
-
-<script type="module">
-import * as THREE from 'three';
-import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
-import { STLLoader } from 'three/examples/jsm/loaders/STLLoader.js';
-
-const statusEl = document.getElementById('status');
-
-const scene = new THREE.Scene();
-scene.background = new THREE.Color(0x1a1a2e);
-
-const camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 0.01, 100);
-camera.position.set(0.8, 1.0, 1.8);
-
-const renderer = new THREE.WebGLRenderer({ antialias: true });
-renderer.setSize(window.innerWidth, window.innerHeight);
-renderer.setPixelRatio(window.devicePixelRatio);
-renderer.shadowMap.enabled = true;
-document.body.appendChild(renderer.domElement);
-
-const controls = new OrbitControls(camera, renderer.domElement);
-controls.target.set(0, 0, 0.9);
-controls.enableDamping = true;
-controls.dampingFactor = 0.08;
-controls.update();
-
-// Lighting
-scene.add(new THREE.AmbientLight(0xffffff, 0.6));
-const dirLight = new THREE.DirectionalLight(0xffffff, 1.2);
-dirLight.position.set(3, 5, 4);
-scene.add(dirLight);
-scene.add(new THREE.DirectionalLight(0x8888ff, 0.4).translateX(-2).translateY(1).translateZ(-3));
-
-// Ground grid
-scene.add(new THREE.GridHelper(4, 40, 0x333355, 0x222244));
-scene.add(new THREE.AxesHelper(0.3));
-
-// Parse URDF manually — build the kinematic tree and load STL meshes
-const stlLoader = new STLLoader();
-const robotGroup = new THREE.Group();
-scene.add(robotGroup);
-
-async function loadURDF() {
-  const resp = await fetch('./openarm_bimanual_pybullet.urdf');
-  const text = await resp.text();
-  const parser = new DOMParser();
-  const xml = parser.parseFromString(text, 'text/xml');
-
-  // Parse links and joints
-  const links = {};
-  const joints = [];
-
-  for (const linkEl of xml.querySelectorAll('link')) {
-    const name = linkEl.getAttribute('name');
-    const group = new THREE.Group();
-    group.name = name;
-
-    // Try to load visual mesh
-    const visual = linkEl.querySelector('visual');
-    if (visual) {
-      const meshEl = visual.querySelector('mesh');
-      const originEl = visual.querySelector('origin');
-      if (meshEl) {
-        const filename = meshEl.getAttribute('filename');
-        const scaleStr = meshEl.getAttribute('scale');
-        const scale = scaleStr ? scaleStr.split(' ').map(Number) : [1, 1, 1];
-
-        let xyz = [0, 0, 0];
-        if (originEl && originEl.getAttribute('xyz')) {
-          xyz = originEl.getAttribute('xyz').split(' ').map(Number);
-        }
-
-        if (filename.endsWith('.stl')) {
-          try {
-            const geo = await new Promise((resolve, reject) => {
-              stlLoader.load(filename, resolve, undefined, reject);
-            });
-            const mat = new THREE.MeshStandardMaterial({
-              color: 0x6688aa,
-              metalness: 0.3,
-              roughness: 0.6,
-              transparent: true,
-              opacity: 0.7,
-            });
-            const mesh = new THREE.Mesh(geo, mat);
-            mesh.scale.set(scale[0], scale[1], scale[2]);
-            mesh.position.set(xyz[0], xyz[1], xyz[2]);
-            group.add(mesh);
-          } catch (e) {
-            // Mesh file not found, skip
-          }
-        }
-      }
-    }
-
-    links[name] = group;
-  }
-
-  // Parse joints and build hierarchy
-  for (const jointEl of xml.querySelectorAll('joint')) {
-    const name = jointEl.getAttribute('name');
-    const type = jointEl.getAttribute('type');
-    const parentName = jointEl.querySelector('parent').getAttribute('link');
-    const childName = jointEl.querySelector('child').getAttribute('link');
-    const originEl = jointEl.querySelector('origin');
-
-    let xyz = [0, 0, 0];
-    let rpy = [0, 0, 0];
-    if (originEl) {
-      if (originEl.getAttribute('xyz')) xyz = originEl.getAttribute('xyz').split(' ').map(Number);
-      if (originEl.getAttribute('rpy')) rpy = originEl.getAttribute('rpy').split(' ').map(Number);
-    }
-
-    joints.push({ name, type, parentName, childName, xyz, rpy });
-  }
-
-  // Build tree
-  const rootLinks = new Set(Object.keys(links));
-  for (const j of joints) {
-    rootLinks.delete(j.childName);
-  }
-
-  for (const j of joints) {
-    const parent = links[j.parentName];
-    const child = links[j.childName];
-    if (parent && child) {
-      child.position.set(j.xyz[0], j.xyz[1], j.xyz[2]);
-      if (j.rpy[0] || j.rpy[1] || j.rpy[2]) {
-        child.rotation.set(j.rpy[0], j.rpy[1], j.rpy[2], 'XYZ');
-      }
-      parent.add(child);
-    }
-  }
-
-  // Add root links to scene
-  for (const name of rootLinks) {
-    if (links[name]) robotGroup.add(links[name]);
-  }
-
-  // Place markers at the three EE frame candidates
-  const targets = {
-    link7: 'openarm_right_link7',
-    ee_target: 'openarm_right_ee_target',
-    hand: 'openarm_right_hand',
-    tcp: 'openarm_right_hand_tcp',
-  };
-  const colors = { link7: 0xff4444, ee_target: 0xffaa00, hand: 0x44ff44, tcp: 0x4488ff };
-  const labels = { link7: 'link7', ee_target: 'ee_target', hand: 'hand', tcp: 'hand_tcp' };
-
-  for (const [key, linkName] of Object.entries(targets)) {
-    const link = links[linkName];
-    if (!link) continue;
-
-    // Marker sphere
-    const sphere = new THREE.Mesh(
-      new THREE.SphereGeometry(0.018, 16, 16),
-      new THREE.MeshStandardMaterial({ color: colors[key], emissive: colors[key], emissiveIntensity: 0.6 })
-    );
-    link.add(sphere);
-
-    // Ring around sphere for visibility
-    const ring = new THREE.Mesh(
-      new THREE.TorusGeometry(0.03, 0.003, 8, 32),
-      new THREE.MeshStandardMaterial({ color: colors[key], emissive: colors[key], emissiveIntensity: 0.4 })
-    );
-    link.add(ring);
-
-    // Axes helper
-    link.add(new THREE.AxesHelper(0.08));
-
-    // Sprite label
-    const canvas = document.createElement('canvas');
-    canvas.width = 512; canvas.height = 80;
-    const ctx = canvas.getContext('2d');
-    ctx.font = 'bold 36px IBM Plex Mono, monospace';
-    ctx.fillStyle = '#' + colors[key].toString(16).padStart(6, '0');
-    ctx.fillText(labels[key], 4, 50);
-    const tex = new THREE.CanvasTexture(canvas);
-    const sprite = new THREE.Sprite(new THREE.SpriteMaterial({ map: tex, depthTest: false }));
-    sprite.scale.set(0.3, 0.05, 1);
-    sprite.position.set(0.06, 0.0, 0.03);
-    link.add(sprite);
-  }
-
-  // Dashed lines between markers (in world space)
-  robotGroup.updateMatrixWorld(true);
-  const positions = {};
-  for (const [key, linkName] of Object.entries(targets)) {
-    const link = links[linkName];
-    if (link) {
-      const wp = new THREE.Vector3();
-      link.getWorldPosition(wp);
-      positions[key] = wp;
-    }
-  }
-
-  function addDashedLine(from, to) {
-    const geo = new THREE.BufferGeometry().setFromPoints([from, to]);
-    const mat = new THREE.LineDashedMaterial({ color: 0xaaaaaa, dashSize: 0.012, gapSize: 0.008 });
-    const line = new THREE.Line(geo, mat);
-    line.computeLineDistances();
-    scene.add(line);
-  }
-  if (positions.link7 && positions.hand) addDashedLine(positions.link7, positions.hand);
-  if (positions.hand && positions.tcp) addDashedLine(positions.hand, positions.tcp);
-
-  // Store for focus buttons
-  window._framePositions = positions;
-  window._links = links;
-  window._targets = targets;
-
-  // Focus on the hand area
-  if (positions.hand) {
-    controls.target.copy(positions.hand);
-    camera.position.set(positions.hand.x + 0.5, positions.hand.y + 0.4, positions.hand.z + 0.5);
-    controls.update();
-  }
-
-  const meshCount = robotGroup.children.length;
-  statusEl.textContent = `Loaded. Right arm chain visible with ${Object.keys(links).length} links.`;
-}
-
-window.focusFrame = function(key) {
-  const pos = window._framePositions?.[key];
-  if (!pos) return;
-  controls.target.copy(pos);
-  camera.position.set(pos.x + 0.35, pos.y + 0.25, pos.z + 0.35);
-  controls.update();
-  document.querySelectorAll('#frame-select button').forEach(b => b.classList.remove('active'));
-  event.target.classList.add('active');
-};
-
-window.addEventListener('resize', () => {
-  camera.aspect = window.innerWidth / window.innerHeight;
-  camera.updateProjectionMatrix();
-  renderer.setSize(window.innerWidth, window.innerHeight);
-});
-
-function animate() {
-  requestAnimationFrame(animate);
-  controls.update();
-  renderer.render(scene, camera);
-}
-animate();
-
-loadURDF().catch(err => {
-  statusEl.textContent = `Error: ${err.message}`;
-  console.error(err);
-});
-</script>
-</body>
-</html>

src/lerobot/robots/so_follower/robot_kinematic_processor.py

@@ -255,16 +255,19 @@ class InverseKinematicsEEToJoints(RobotActionProcessorStep):
     """
     Computes desired joint positions from a target end-effector pose using inverse kinematics (IK).
 
+    This step translates a Cartesian command (position and orientation of the end-effector) into
+    the corresponding joint-space commands for each motor.
+
     Attributes:
         kinematics: The robot's kinematic model for inverse kinematics.
-        motor_names: Arm joint names for IK computation.
-        gripper_names: Gripper joint name(s). ee.gripper_pos is written to all of them.
+        motor_names: A list of motor names for which to compute joint positions.
+        q_curr: Internal state storing the last joint positions, used as an initial guess for the IK solver.
         initial_guess_current_joints: If True, use the robot's current joint state as the IK guess.
+            If False, use the solution from the previous step.
     """
 
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
     q_curr: np.ndarray | None = field(default=None, init=False, repr=False)
     initial_guess_current_joints: bool = True
 
@@ -275,73 +278,63 @@ class InverseKinematicsEEToJoints(RobotActionProcessorStep):
         wx = action.pop("ee.wx")
         wy = action.pop("ee.wy")
         wz = action.pop("ee.wz")
+        gripper_pos = action.pop("ee.gripper_pos")
 
-        ee_keys = [x, y, z, wx, wy, wz]
-
-        if self.gripper_names:
-            gripper_pos = action.pop("ee.gripper_pos")
-            ee_keys.append(gripper_pos)
-        if None in ee_keys:
-            raise ValueError("Missing required end-effector pose components in action")
+        if None in (x, y, z, wx, wy, wz, gripper_pos):
+            raise ValueError(
+                "Missing required end-effector pose components: ee.x, ee.y, ee.z, ee.wx, ee.wy, ee.wz, ee.gripper_pos must all be present in action"
+            )
 
         observation = self.transition.get(TransitionKey.OBSERVATION).copy()
         if observation is None:
-            raise ValueError("Joints observation is required for computing robot kinematics")
+            raise ValueError("Joints observation is require for computing robot kinematics")
 
         q_raw = np.array(
-            [
-                float(v)
-                for k, v in observation.items()
-                if isinstance(k, str) and k.endswith(".pos") and k.removesuffix(".pos") in self.motor_names
-            ],
+            [float(v) for k, v in observation.items() if isinstance(k, str) and k.endswith(".pos")],
             dtype=float,
         )
+        if q_raw is None:
+            raise ValueError("Joints observation is require for computing robot kinematics")
 
-        if self.initial_guess_current_joints:
+        if self.initial_guess_current_joints:  # Use current joints as initial guess
             self.q_curr = q_raw
-        else:
+        else:  # Use previous ik solution as initial guess
             if self.q_curr is None:
                 self.q_curr = q_raw
 
+        # Build desired 4x4 transform from pos + rotvec (twist)
         t_des = np.eye(4, dtype=float)
         t_des[:3, :3] = Rotation.from_rotvec([wx, wy, wz]).as_matrix()
         t_des[:3, 3] = [x, y, z]
 
+        # Compute inverse kinematics
         q_target = self.kinematics.inverse_kinematics(self.q_curr, t_des)
         self.q_curr = q_target
 
+        # TODO: This is sentitive to order of motor_names = q_target mapping
         for i, name in enumerate(self.motor_names):
-            action[f"{name}.pos"] = float(q_target[i])
+            if name != "gripper":
+                action[f"{name}.pos"] = float(q_target[i])
+            else:
+                action["gripper.pos"] = float(gripper_pos)
 
-        if self.gripper_names:
-            for gname in self.gripper_names:
-                action[f"{gname}.pos"] = float(gripper_pos)
-
-        # When gripper_names is empty, gripper keys (e.g. proximal.pos, distal.pos)
-        # are already in the action dict as absolute positions — left untouched.
         return action
 
     def transform_features(
         self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
     ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
-        ee_feats = ["x", "y", "z", "wx", "wy", "wz"]
-        if self.gripper_names:
-            ee_feats.append("gripper_pos")
-        for feat in ee_feats:
+        for feat in ["x", "y", "z", "wx", "wy", "wz", "gripper_pos"]:
             features[PipelineFeatureType.ACTION].pop(f"ee.{feat}", None)
 
         for name in self.motor_names:
             features[PipelineFeatureType.ACTION][f"{name}.pos"] = PolicyFeature(
                 type=FeatureType.ACTION, shape=(1,)
             )
-        for name in self.gripper_names:
-            features[PipelineFeatureType.ACTION][f"{name}.pos"] = PolicyFeature(
-                type=FeatureType.ACTION, shape=(1,)
-            )
 
         return features
 
     def reset(self):
+        """Resets the initial guess for the IK solver."""
         self.q_curr = None
 
 
@@ -409,39 +402,24 @@ class GripperVelocityToJoint(RobotActionProcessorStep):
 
 
 def compute_forward_kinematics_joints_to_ee(
-    joints: dict[str, Any],
-    kinematics: RobotKinematics,
-    motor_names: list[str],
-    gripper_names: list[str] | None = None,
+    joints: dict[str, Any], kinematics: RobotKinematics, motor_names: list[str]
 ) -> dict[str, Any]:
-    if gripper_names is None:
-        gripper_names = ["gripper"]
-
     motor_joint_values = [joints[f"{n}.pos"] for n in motor_names]
 
     q = np.array(motor_joint_values, dtype=float)
     t = kinematics.forward_kinematics(q)
     pos = t[:3, 3]
     tw = Rotation.from_matrix(t[:3, :3]).as_rotvec()
-
+    gripper_pos = joints["gripper.pos"]
     for n in motor_names:
         joints.pop(f"{n}.pos")
-
     joints["ee.x"] = float(pos[0])
     joints["ee.y"] = float(pos[1])
     joints["ee.z"] = float(pos[2])
     joints["ee.wx"] = float(tw[0])
     joints["ee.wy"] = float(tw[1])
     joints["ee.wz"] = float(tw[2])
-
-    # When gripper_names is non-empty, fold them into ee.gripper_pos (e.g. SO100).
-    # When empty, gripper joints pass through as-is (absolute position control).
-    if gripper_names:
-        gripper_pos = joints[f"{gripper_names[0]}.pos"]
-        for n in gripper_names:
-            joints.pop(f"{n}.pos", None)
-        joints["ee.gripper_pos"] = float(gripper_pos)
-
+    joints["ee.gripper_pos"] = float(gripper_pos)
     return joints
 
 
@@ -451,33 +429,27 @@ class ForwardKinematicsJointsToEEObservation(ObservationProcessorStep):
     """
     Computes the end-effector pose from joint positions using forward kinematics (FK).
 
+    This step is typically used to add the robot's Cartesian pose to the observation space,
+    which can be useful for visualization or as an input to a policy.
+
     Attributes:
         kinematics: The robot's kinematic model.
-        motor_names: Arm joint names used for FK computation.
-        gripper_names: Gripper joint name(s) to fold into ee.gripper_pos.
-            Empty list means gripper joints pass through as absolute positions.
     """
 
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
 
     def observation(self, observation: RobotObservation) -> RobotObservation:
-        return compute_forward_kinematics_joints_to_ee(
-            observation, self.kinematics, self.motor_names, self.gripper_names
-        )
+        return compute_forward_kinematics_joints_to_ee(observation, self.kinematics, self.motor_names)
 
     def transform_features(
         self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
     ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
+        # We only use the ee pose in the dataset, so we don't need the joint positions
         for n in self.motor_names:
             features[PipelineFeatureType.OBSERVATION].pop(f"{n}.pos", None)
-        ee_keys = ["x", "y", "z", "wx", "wy", "wz"]
-        if self.gripper_names:
-            for n in self.gripper_names:
-                features[PipelineFeatureType.OBSERVATION].pop(f"{n}.pos", None)
-            ee_keys.append("gripper_pos")
-        for k in ee_keys:
+        # We specify the dataset features of this step that we want to be stored in the dataset
+        for k in ["x", "y", "z", "wx", "wy", "wz", "gripper_pos"]:
             features[PipelineFeatureType.OBSERVATION][f"ee.{k}"] = PolicyFeature(
                 type=FeatureType.STATE, shape=(1,)
             )
@@ -490,33 +462,27 @@ class ForwardKinematicsJointsToEEAction(RobotActionProcessorStep):
     """
     Computes the end-effector pose from joint positions using forward kinematics (FK).
 
+    This step is typically used to add the robot's Cartesian pose to the observation space,
+    which can be useful for visualization or as an input to a policy.
+
     Attributes:
         kinematics: The robot's kinematic model.
-        motor_names: Arm joint names used for FK computation.
-        gripper_names: Gripper joint name(s) to fold into ee.gripper_pos.
-            Empty list means gripper joints pass through as absolute positions.
     """
 
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
 
     def action(self, action: RobotAction) -> RobotAction:
-        return compute_forward_kinematics_joints_to_ee(
-            action, self.kinematics, self.motor_names, self.gripper_names
-        )
+        return compute_forward_kinematics_joints_to_ee(action, self.kinematics, self.motor_names)
 
     def transform_features(
         self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
     ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
+        # We only use the ee pose in the dataset, so we don't need the joint positions
         for n in self.motor_names:
             features[PipelineFeatureType.ACTION].pop(f"{n}.pos", None)
-        ee_keys = ["x", "y", "z", "wx", "wy", "wz"]
-        if self.gripper_names:
-            for n in self.gripper_names:
-                features[PipelineFeatureType.ACTION].pop(f"{n}.pos", None)
-            ee_keys.append("gripper_pos")
-        for k in ee_keys:
+        # We specify the dataset features of this step that we want to be stored in the dataset
+        for k in ["x", "y", "z", "wx", "wy", "wz", "gripper_pos"]:
             features[PipelineFeatureType.ACTION][f"ee.{k}"] = PolicyFeature(
                 type=FeatureType.STATE, shape=(1,)
             )
@@ -528,14 +494,13 @@ class ForwardKinematicsJointsToEEAction(RobotActionProcessorStep):
 class ForwardKinematicsJointsToEE(ProcessorStep):
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
 
     def __post_init__(self):
         self.joints_to_ee_action_processor = ForwardKinematicsJointsToEEAction(
-            kinematics=self.kinematics, motor_names=self.motor_names, gripper_names=self.gripper_names
+            kinematics=self.kinematics, motor_names=self.motor_names
         )
         self.joints_to_ee_observation_processor = ForwardKinematicsJointsToEEObservation(
-            kinematics=self.kinematics, motor_names=self.motor_names, gripper_names=self.gripper_names
+            kinematics=self.kinematics, motor_names=self.motor_names
         )
 
     def __call__(self, transition: EnvTransition) -> EnvTransition:
@@ -559,13 +524,13 @@ class ForwardKinematicsJointsToEE(ProcessorStep):
 @dataclass
 class InverseKinematicsRLStep(ProcessorStep):
     """
-    IK step for the RL pipeline. Same logic as InverseKinematicsEEToJoints but
-    operates on EnvTransition directly and stores the IK solution.
+    Computes desired joint positions from a target end-effector pose using inverse kinematics (IK).
+
+    This is modified from the InverseKinematicsEEToJoints step to be used in the RL pipeline.
     """
 
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
     q_curr: np.ndarray | None = field(default=None, init=False, repr=False)
     initial_guess_current_joints: bool = True
 
@@ -573,7 +538,7 @@ class InverseKinematicsRLStep(ProcessorStep):
         new_transition = dict(transition)
         action = new_transition.get(TransitionKey.ACTION)
         if action is None:
-            raise ValueError("Action is required for InverseKinematicsRLStep")
+            raise ValueError("Action is required for InverseKinematicsEEToJoints")
         action = dict(action)
 
         x = action.pop("ee.x")
@@ -582,46 +547,45 @@ class InverseKinematicsRLStep(ProcessorStep):
         wx = action.pop("ee.wx")
         wy = action.pop("ee.wy")
         wz = action.pop("ee.wz")
+        gripper_pos = action.pop("ee.gripper_pos")
 
-        ee_keys = [x, y, z, wx, wy, wz]
-        if self.gripper_names:
-            gripper_pos = action.pop("ee.gripper_pos")
-            ee_keys.append(gripper_pos)
-        if None in ee_keys:
-            raise ValueError("Missing required end-effector pose components in action")
+        if None in (x, y, z, wx, wy, wz, gripper_pos):
+            raise ValueError(
+                "Missing required end-effector pose components: ee.x, ee.y, ee.z, ee.wx, ee.wy, ee.wz, ee.gripper_pos must all be present in action"
+            )
 
         observation = new_transition.get(TransitionKey.OBSERVATION).copy()
         if observation is None:
-            raise ValueError("Joints observation is required for computing robot kinematics")
+            raise ValueError("Joints observation is require for computing robot kinematics")
 
         q_raw = np.array(
-            [
-                float(v)
-                for k, v in observation.items()
-                if isinstance(k, str) and k.endswith(".pos") and k.removesuffix(".pos") in self.motor_names
-            ],
+            [float(v) for k, v in observation.items() if isinstance(k, str) and k.endswith(".pos")],
             dtype=float,
         )
+        if q_raw is None:
+            raise ValueError("Joints observation is require for computing robot kinematics")
 
-        if self.initial_guess_current_joints:
+        if self.initial_guess_current_joints:  # Use current joints as initial guess
             self.q_curr = q_raw
-        else:
+        else:  # Use previous ik solution as initial guess
             if self.q_curr is None:
                 self.q_curr = q_raw
 
+        # Build desired 4x4 transform from pos + rotvec (twist)
         t_des = np.eye(4, dtype=float)
         t_des[:3, :3] = Rotation.from_rotvec([wx, wy, wz]).as_matrix()
         t_des[:3, 3] = [x, y, z]
 
+        # Compute inverse kinematics
         q_target = self.kinematics.inverse_kinematics(self.q_curr, t_des)
         self.q_curr = q_target
 
+        # TODO: This is sentitive to order of motor_names = q_target mapping
         for i, name in enumerate(self.motor_names):
-            action[f"{name}.pos"] = float(q_target[i])
-
-        if self.gripper_names:
-            for gname in self.gripper_names:
-                action[f"{gname}.pos"] = float(gripper_pos)
+            if name != "gripper":
+                action[f"{name}.pos"] = float(q_target[i])
+            else:
+                action["gripper.pos"] = float(gripper_pos)
 
         new_transition[TransitionKey.ACTION] = action
         complementary_data = new_transition.get(TransitionKey.COMPLEMENTARY_DATA, {})
@@ -632,22 +596,16 @@ class InverseKinematicsRLStep(ProcessorStep):
     def transform_features(
         self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
     ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
-        ee_feats = ["x", "y", "z", "wx", "wy", "wz"]
-        if self.gripper_names:
-            ee_feats.append("gripper_pos")
-        for feat in ee_feats:
+        for feat in ["x", "y", "z", "wx", "wy", "wz", "gripper_pos"]:
             features[PipelineFeatureType.ACTION].pop(f"ee.{feat}", None)
 
         for name in self.motor_names:
             features[PipelineFeatureType.ACTION][f"{name}.pos"] = PolicyFeature(
                 type=FeatureType.ACTION, shape=(1,)
             )
-        for name in self.gripper_names:
-            features[PipelineFeatureType.ACTION][f"{name}.pos"] = PolicyFeature(
-                type=FeatureType.ACTION, shape=(1,)
-            )
 
         return features
 
     def reset(self):
+        """Resets the initial guess for the IK solver."""
         self.q_curr = None

src/lerobot/scripts/lerobot_edit_dataset.py

@@ -254,10 +254,6 @@ class RecomputeStatsConfig(OperationConfig):
     relative_exclude_joints: list[str] | None = None
     chunk_size: int = 50
     num_workers: int = 0
-    relative_state: bool = False
-    relative_exclude_state_joints: list[str] | None = None
-    state_obs_steps: int = 2
-    derive_state_from_action: bool = False
 
 
 @OperationConfig.register_subclass("info")
@@ -567,14 +563,6 @@ def handle_recompute_stats(cfg: EditDatasetConfig) -> None:
             f"Relative action stats enabled (chunk_size={cfg.operation.chunk_size}, "
             f"exclude_joints={cfg.operation.relative_exclude_joints})"
         )
-    if cfg.operation.relative_state:
-        logging.info(
-            f"Relative state stats enabled (state_obs_steps={cfg.operation.state_obs_steps}, "
-            f"exclude_state_joints={cfg.operation.relative_exclude_state_joints})"
-        )
-
-    if cfg.operation.derive_state_from_action:
-        logging.info("Derive state from action enabled (implies relative_state=True, state_obs_steps=2)")
 
     recompute_stats(
         dataset,
@@ -583,10 +571,6 @@ def handle_recompute_stats(cfg: EditDatasetConfig) -> None:
         relative_exclude_joints=cfg.operation.relative_exclude_joints,
         chunk_size=cfg.operation.chunk_size,
         num_workers=cfg.operation.num_workers,
-        relative_state=cfg.operation.relative_state,
-        relative_exclude_state_joints=cfg.operation.relative_exclude_state_joints,
-        state_obs_steps=cfg.operation.state_obs_steps,
-        derive_state_from_action=cfg.operation.derive_state_from_action,
     )
 
     logging.info(f"Stats written to {dataset.root}")

src/lerobot/scripts/lerobot_eval.py

@@ -47,8 +47,10 @@ You can learn about the CLI options for this script in the `EvalPipelineConfig`
 """
 
 import concurrent.futures as cf
+import copy
 import json
 import logging
+import math
 import threading
 import time
 from collections import defaultdict
@@ -56,7 +58,6 @@ from collections.abc import Callable
 from contextlib import nullcontext
 from copy import deepcopy
 from dataclasses import asdict
-from functools import partial
 from pathlib import Path
 from pprint import pformat
 from typing import Any, TypedDict
@@ -73,7 +74,6 @@ from lerobot.configs import parser
 from lerobot.configs.eval import EvalPipelineConfig
 from lerobot.envs.factory import make_env, make_env_pre_post_processors
 from lerobot.envs.utils import (
-    add_envs_task,
     check_env_attributes_and_types,
     close_envs,
     preprocess_observation,
@@ -93,6 +93,14 @@ from lerobot.utils.utils import (
 )
 
 
+def _shard_episodes(n_episodes: int, shard_id: int, num_shards: int) -> list[int]:
+    """Return the episode indices assigned to this shard (round-robin distribution).
+
+    Example: _shard_episodes(10, 1, 4) -> [1, 5, 9]
+    """
+    return list(range(shard_id, n_episodes, num_shards))
+
+
 def rollout(
     env: gym.vector.VectorEnv,
     policy: PreTrainedPolicy,
@@ -166,9 +174,15 @@ def rollout(
         if return_observations:
             all_observations.append(deepcopy(observation))
 
-        # Infer "task" from attributes of environments.
-        # TODO: works with SyncVectorEnv but not AsyncVectorEnv
-        observation = add_envs_task(env, observation)
+        # Infer "task" from sub-environments (prefer natural language description).
+        # env.call() works with both SyncVectorEnv and AsyncVectorEnv.
+        try:
+            observation["task"] = list(env.call("task_description"))
+        except Exception:
+            try:
+                observation["task"] = list(env.call("task"))
+            except Exception:
+                observation["task"] = [""] * env.num_envs
 
         # Apply environment-specific preprocessing (e.g., LiberoProcessorStep for LIBERO)
         observation = env_preprocessor(observation)
@@ -193,14 +207,13 @@ def rollout(
 
         # VectorEnv stores is_success in `info["final_info"][env_index]["is_success"]`. "final_info" isn't
         # available if none of the envs finished.
-        if "final_info" in info:
-            final_info = info["final_info"]
-            if not isinstance(final_info, dict):
-                raise RuntimeError(
-                    "Unsupported `final_info` format: expected dict (Gymnasium >= 1.0). "
-                    "You're likely using an older version of gymnasium (< 1.0). Please upgrade."
-                )
-            successes = final_info["is_success"].tolist()
+        if "final_info" in info and isinstance(info["final_info"], dict):
+            successes = info["final_info"]["is_success"].tolist()
+        elif "is_success" in info:
+            is_success = info["is_success"]
+            successes = (
+                is_success.tolist() if hasattr(is_success, "tolist") else [bool(is_success)] * env.num_envs
+            )
         else:
             successes = [False] * env.num_envs
 
@@ -549,6 +562,14 @@ def eval_main(cfg: EvalPipelineConfig):
     # Create environment-specific preprocessor and postprocessor (e.g., for LIBERO environments)
     env_preprocessor, env_postprocessor = make_env_pre_post_processors(env_cfg=cfg.env, policy_cfg=cfg.policy)
 
+    # Sharding: each shard runs a subset of n_episodes with non-overlapping seeds.
+    shard_id = cfg.eval.shard_id
+    num_shards = cfg.eval.num_shards
+    episodes_for_shard = _shard_episodes(cfg.eval.n_episodes, shard_id, num_shards)
+    n_per_shard = len(episodes_for_shard)
+    # Shift the seed so each shard gets a different, non-overlapping seed range.
+    shard_seed = (cfg.seed or 0) + shard_id * math.ceil(cfg.eval.n_episodes / num_shards)
+
     with torch.no_grad(), torch.autocast(device_type=device.type) if cfg.policy.use_amp else nullcontext():
         info = eval_policy_all(
             envs=envs,
@@ -557,10 +578,10 @@ def eval_main(cfg: EvalPipelineConfig):
             env_postprocessor=env_postprocessor,
             preprocessor=preprocessor,
             postprocessor=postprocessor,
-            n_episodes=cfg.eval.n_episodes,
+            n_episodes=n_per_shard,
             max_episodes_rendered=10,
             videos_dir=Path(cfg.output_dir) / "videos",
-            start_seed=cfg.seed,
+            start_seed=shard_seed,
             max_parallel_tasks=cfg.env.max_parallel_tasks,
         )
         print("Overall Aggregated Metrics:")
@@ -573,8 +594,13 @@ def eval_main(cfg: EvalPipelineConfig):
     # Close all vec envs
     close_envs(envs)
 
-    # Save info
-    with open(Path(cfg.output_dir) / "eval_info.json", "w") as f:
+    # Save info — use shard-specific filename when running in parallel mode.
+    if num_shards > 1:
+        out_path = Path(cfg.output_dir) / f"shard_{shard_id}_of_{num_shards}.json"
+    else:
+        out_path = Path(cfg.output_dir) / "eval_info.json"
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    with open(out_path, "w") as f:
         json.dump(info, f, indent=2)
 
     logging.info("End of eval")
@@ -734,39 +760,58 @@ def eval_policy_all(
             group_acc[group]["video_paths"].extend(paths)
             overall["video_paths"].extend(paths)
 
+    def _make_thread_policy(p: PreTrainedPolicy) -> PreTrainedPolicy:
+        """Shallow copy sharing weight tensors, with independent per-thread state.
+
+        copy.copy() gives a new Python object whose _parameters dict is a shared
+        reference (same tensor storage, zero extra VRAM). reset() then rebinds
+        mutable state (action queues etc.) to fresh per-thread objects.
+
+        Note: does NOT work for ACT with temporal_ensemble_coeff — that policy's
+        reset() mutates a shared sub-object. Use max_parallel_tasks=1 for that config.
+        """
+        thread_p = copy.copy(p)
+        thread_p.reset()
+        return thread_p
+
     # Choose runner (sequential vs threaded)
-    task_runner = partial(
-        run_one,
-        policy=policy,
-        env_preprocessor=env_preprocessor,
-        env_postprocessor=env_postprocessor,
-        preprocessor=preprocessor,
-        postprocessor=postprocessor,
-        n_episodes=n_episodes,
-        max_episodes_rendered=max_episodes_rendered,
-        videos_dir=videos_dir,
-        return_episode_data=return_episode_data,
-        start_seed=start_seed,
-    )
+    _runner_kwargs = {
+        "env_preprocessor": env_preprocessor,
+        "env_postprocessor": env_postprocessor,
+        "preprocessor": preprocessor,
+        "postprocessor": postprocessor,
+        "n_episodes": n_episodes,
+        "max_episodes_rendered": max_episodes_rendered,
+        "videos_dir": videos_dir,
+        "return_episode_data": return_episode_data,
+        "start_seed": start_seed,
+    }
 
     if max_parallel_tasks <= 1:
-        # sequential path (single accumulator path on the main thread)
-        # NOTE: keeping a single-threaded accumulator avoids concurrent list appends or locks
         for task_group, task_id, env in tasks:
-            tg, tid, metrics = task_runner(task_group, task_id, env)
-            _accumulate_to(tg, metrics)
-            per_task_infos.append({"task_group": tg, "task_id": tid, "metrics": metrics})
+            try:
+                tg, tid, metrics = run_one(task_group, task_id, env, policy=policy, **_runner_kwargs)
+                _accumulate_to(tg, metrics)
+                per_task_infos.append({"task_group": tg, "task_id": tid, "metrics": metrics})
+            finally:
+                env.close()
     else:
-        # threaded path: submit all tasks, consume completions on main thread and accumulate there
+        # threaded path: each thread gets a shallow policy copy (shared weights, independent state)
         with cf.ThreadPoolExecutor(max_workers=max_parallel_tasks) as executor:
             fut2meta = {}
             for task_group, task_id, env in tasks:
-                fut = executor.submit(task_runner, task_group, task_id, env)
-                fut2meta[fut] = (task_group, task_id)
+                fut = executor.submit(
+                    run_one, task_group, task_id, env, policy=_make_thread_policy(policy), **_runner_kwargs
+                )
+                fut2meta[fut] = (task_group, task_id, env)
             for fut in cf.as_completed(fut2meta):
-                tg, tid, metrics = fut.result()
-                _accumulate_to(tg, metrics)
-                per_task_infos.append({"task_group": tg, "task_id": tid, "metrics": metrics})
+                tg, tid, env = fut2meta[fut]
+                try:
+                    tg, tid, metrics = fut.result()
+                    _accumulate_to(tg, metrics)
+                    per_task_infos.append({"task_group": tg, "task_id": tid, "metrics": metrics})
+                finally:
+                    env.close()
 
     # compute aggregated metrics helper (robust to lists/scalars)
     def _agg_from_list(xs):

src/lerobot/scripts/lerobot_eval_autotune.py

@@ -0,0 +1,249 @@
+#!/usr/bin/env python
+
+# Copyright 2024 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Probe hardware and recommend optimal lerobot-eval-parallel flags.
+
+Run standalone:
+    lerobot-eval-autotune --policy.path=lerobot/smolvla_libero --env.type=libero
+
+Or called programmatically from lerobot_eval_parallel when --num-shards auto.
+
+Steps:
+    1. Probe GPU VRAM and CPU core count.
+    2. Measure model VRAM footprint (load policy, delta of cuda.memory_allocated).
+    3. Compute max shards limited by VRAM (85% of total).
+    4. Probe env step time (optional, skipped when skip_timing=True).
+    5. Probe inference time (optional, skipped when skip_timing=True).
+    6. Derive num_shards = min(vram_limit, saturation_shards).
+    7. Choose MUJOCO_GL (egl vs osmesa) based on remaining VRAM headroom.
+    8. Compute batch_size = max(4, min(floor(cpu_cores * 0.8 / num_shards), 64)).
+    9. Print paste-ready command.
+"""
+
+import math
+import os
+import sys
+import time
+from dataclasses import dataclass
+
+
+@dataclass
+class AutotuneRecommendation:
+    num_shards: int
+    batch_size: int
+    mujoco_gl: str
+    use_amp: bool
+    # Probed values
+    gpu_name: str
+    vram_gb: float
+    cpu_cores: int
+    model_gb: float
+    env_step_ms: float | None
+    infer_ms: float | None
+
+
+_DEFAULT_ENV_STEP_MS = 22.0  # LIBERO on GPU, typical value
+_DEFAULT_INFER_MS = 5.0  # SmolVLA fp16 on H100
+
+
+def _probe_gpu() -> tuple[str, float]:
+    """Return (gpu_name, vram_gb). Falls back to CPU sentinel on non-CUDA systems."""
+    try:
+        import torch
+
+        if not torch.cuda.is_available():
+            return "CPU (no CUDA)", 0.0
+        props = torch.cuda.get_device_properties(0)
+        return props.name, props.total_memory / (1024**3)
+    except Exception:
+        return "unknown", 0.0
+
+
+def _probe_model_gb(passthrough: list[str]) -> float:
+    """Load the policy (from --policy.path) and measure VRAM delta. Returns GB."""
+    # Extract policy path from passthrough args
+    policy_path = None
+    for tok in passthrough:
+        if tok.startswith("policy.path="):
+            policy_path = tok.split("=", 1)[1]
+            break
+        if tok.startswith("--policy.path="):
+            policy_path = tok.split("=", 1)[1]
+            break
+    if policy_path is None:
+        return 0.0
+
+    try:
+        import torch
+
+        from lerobot.policies.factory import make_policy
+        from lerobot.policies.pretrained import PreTrainedConfig
+
+        if not torch.cuda.is_available():
+            return 0.0
+        torch.cuda.synchronize()
+        before = torch.cuda.memory_allocated(0)
+        cfg = PreTrainedConfig.from_pretrained(policy_path)
+        cfg.pretrained_path = policy_path  # type: ignore[assignment]
+        policy = make_policy(cfg=cfg)
+        policy.eval()
+        torch.cuda.synchronize()
+        after = torch.cuda.memory_allocated(0)
+        del policy
+        torch.cuda.empty_cache()
+        return (after - before) / (1024**3)
+    except Exception as e:
+        print(f"[autotune] could not measure model VRAM: {e}", file=sys.stderr)
+        return 0.0
+
+
+def _probe_env_step_ms(passthrough: list[str], batch_size: int = 8, n_steps: int = 30) -> float | None:
+    """Run a short env warmup and return median step latency in ms. Returns None on failure."""
+    try:
+        import numpy as np
+
+        from lerobot.envs.factory import make_env
+
+        # Parse env config from passthrough using lerobot's own parser
+        env_type = None
+        for tok in passthrough:
+            if tok.startswith("env.type=") or tok.startswith("--env.type="):
+                env_type = tok.split("=", 1)[1]
+                break
+        if env_type is None:
+            return None
+
+        # Minimal env config
+        from lerobot.envs.factory import make_env_config
+
+        env_cfg = make_env_config(env_type)
+        envs = make_env(env_cfg, n_envs=batch_size, use_async_envs=(batch_size > 1))
+        # Get first vec env
+        first_suite = next(iter(envs.values()))
+        env = next(iter(first_suite.values()))
+
+        env.reset()
+        dummy_action = np.zeros((batch_size, env.single_action_space.shape[0]))
+        timings = []
+        for _ in range(n_steps):
+            t0 = time.perf_counter()
+            env.step(dummy_action)
+            timings.append((time.perf_counter() - t0) * 1000)
+        env.close()
+        return float(np.median(timings))
+    except Exception as e:
+        print(f"[autotune] env step probe failed: {e}", file=sys.stderr)
+        return None
+
+
+def probe_and_recommend(
+    passthrough: list[str],
+    skip_timing: bool = False,
+) -> AutotuneRecommendation:
+    """Probe hardware + model and return the recommended configuration."""
+    gpu_name, vram_gb = _probe_gpu()
+    cpu_cores = os.cpu_count() or 4
+
+    # Model footprint
+    model_gb = _probe_model_gb(passthrough)
+    if model_gb == 0.0:
+        # Unknown model: assume a conservative 14 GB (SmolVLA fp16) as placeholder
+        model_gb = 14.0
+        print("[autotune] model size unknown, assuming 14 GB (SmolVLA fp16)", file=sys.stderr)
+
+    # Max shards from VRAM (leave 15% headroom for activations + env frames)
+    max_shards_vram = max(1, math.floor(vram_gb * 0.85 / model_gb)) if vram_gb > 0 else 1
+
+    # Timing probes
+    env_step_ms: float | None = None
+    infer_ms: float | None = None
+    if not skip_timing:
+        env_step_ms = _probe_env_step_ms(passthrough)
+        # Inference time: assume ~infer = env_step / saturation_factor heuristic
+        # Full probe would require loading policy — skip for now to stay fast.
+        infer_ms = _DEFAULT_INFER_MS
+
+    # Number of shards to saturate GPU: ceil(env_step / infer)
+    _step = env_step_ms or _DEFAULT_ENV_STEP_MS
+    _infer = infer_ms or _DEFAULT_INFER_MS
+    saturation_shards = max(1, math.ceil(_step / _infer))
+
+    num_shards = min(max_shards_vram, saturation_shards)
+
+    # Rendering mode: EGL if all model copies + env frame buffers fit in VRAM
+    env_vram_per_shard_gb = 0.01  # ~10 MB overhead per env batch
+    total_with_egl = num_shards * (model_gb + env_vram_per_shard_gb)
+    mujoco_gl = "egl" if (vram_gb == 0 or total_with_egl < vram_gb * 0.85) else "osmesa"
+
+    # Batch size: fill CPU cores evenly across shards
+    batch_size = max(4, min(math.floor(cpu_cores * 0.8 / num_shards), 64))
+
+    # Recommend AMP when model is large (saves ~50% VRAM)
+    use_amp = model_gb > 8.0
+
+    return AutotuneRecommendation(
+        num_shards=num_shards,
+        batch_size=batch_size,
+        mujoco_gl=mujoco_gl,
+        use_amp=use_amp,
+        gpu_name=gpu_name,
+        vram_gb=vram_gb,
+        cpu_cores=cpu_cores,
+        model_gb=model_gb,
+        env_step_ms=env_step_ms,
+        infer_ms=infer_ms,
+    )
+
+
+def main(argv: list[str] | None = None) -> None:
+    passthrough = argv if argv is not None else sys.argv[1:]
+
+    rec = probe_and_recommend(passthrough)
+
+    env_step_str = (
+        f"{rec.env_step_ms:.0f}ms" if rec.env_step_ms else f"~{_DEFAULT_ENV_STEP_MS:.0f}ms (estimated)"
+    )
+    infer_str = f"{rec.infer_ms:.0f}ms" if rec.infer_ms else f"~{_DEFAULT_INFER_MS:.0f}ms (estimated)"
+
+    print()
+    print(
+        f"GPU: {rec.gpu_name}  |  VRAM: {rec.vram_gb:.1f} GB  |  CPU cores: {rec.cpu_cores}  |  Model: {rec.model_gb:.1f} GB"
+    )
+    print()
+    print(f"  env_step_ms: {env_step_str}  |  infer_ms: {infer_str}")
+    print()
+    print(f"  num_shards:  {rec.num_shards}")
+    print(f"  batch_size:  {rec.batch_size}")
+    print(f"  MUJOCO_GL:   {rec.mujoco_gl}")
+    if rec.use_amp:
+        print("  use_amp:     true  (recommended — halves VRAM, faster matmuls)")
+    print()
+
+    # Build paste-ready command
+    flags = [f"--num-shards {rec.num_shards}", f"eval.batch_size={rec.batch_size}"]
+    if rec.use_amp:
+        flags.append("policy.use_amp=true")
+    flags_str = " \\\n    ".join(flags)
+    passthrough_str = " \\\n    ".join(passthrough) if passthrough else "[your flags]"
+
+    print("  Paste-ready command:")
+    print(f"  MUJOCO_GL={rec.mujoco_gl} lerobot-eval-parallel \\")
+    print(f"    {flags_str} \\")
+    print(f"    {passthrough_str}")
+    print()
+
+
+if __name__ == "__main__":
+    main()

src/lerobot/scripts/lerobot_eval_parallel.py

@@ -0,0 +1,185 @@
+#!/usr/bin/env python
+
+# Copyright 2024 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run lerobot-eval across N independent subprocesses (shards) for maximum GPU utilization.
+
+Each shard handles a disjoint subset of episodes and writes its own JSON results file.
+Results are merged and printed when all shards complete.
+
+Usage:
+    lerobot-eval-parallel --num-shards 4 [any lerobot-eval flags]
+    lerobot-eval-parallel --num-shards auto [any lerobot-eval flags]
+    lerobot-eval-parallel --num-shards auto --render-device cpu [any lerobot-eval flags]
+
+--num-shards auto:
+    Calls lerobot-eval-autotune to probe hardware and determine the optimal number of shards.
+
+--render-device gpu|cpu|auto:
+    Controls MUJOCO_GL env var. 'gpu' -> EGL (faster, ~3ms/frame, ~200KB VRAM/env).
+    'cpu' -> osmesa (slower, ~12ms/frame, 0 VRAM). 'auto' picks based on VRAM headroom.
+    Default: auto.
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+
+def _parse_known(argv: list[str]) -> tuple[argparse.Namespace, list[str]]:
+    p = argparse.ArgumentParser(add_help=False)
+    p.add_argument("--num-shards", default="1")
+    p.add_argument("--render-device", choices=["gpu", "cpu", "auto"], default="auto")
+    p.add_argument("--output-dir", default=None)
+    return p.parse_known_args(argv)
+
+
+def _resolve_num_shards(num_shards_str: str, passthrough: list[str]) -> int:
+    if num_shards_str == "auto":
+        from lerobot.scripts.lerobot_eval_autotune import probe_and_recommend
+
+        rec = probe_and_recommend(passthrough)
+        print(
+            f"[autotune] recommended num_shards={rec.num_shards}, batch_size={rec.batch_size}, MUJOCO_GL={rec.mujoco_gl}"
+        )
+        return rec.num_shards
+    return int(num_shards_str)
+
+
+def _resolve_mujoco_gl(render_device: str, num_shards: int, passthrough: list[str]) -> str:
+    if render_device == "gpu":
+        return "egl"
+    if render_device == "cpu":
+        return "osmesa"
+    # auto: use EGL for single shard; for multiple shards check VRAM headroom
+    if num_shards == 1:
+        return "egl"
+    try:
+        from lerobot.scripts.lerobot_eval_autotune import probe_and_recommend
+
+        rec = probe_and_recommend(passthrough, skip_timing=True)
+        return rec.mujoco_gl
+    except Exception:
+        # Conservative fallback: osmesa avoids EGL VRAM contention
+        return "osmesa"
+
+
+def _extract_output_dir(passthrough: list[str]) -> str | None:
+    for tok in passthrough:
+        if tok.startswith("--output-dir="):
+            return tok.split("=", 1)[1]
+        if tok == "--output-dir":
+            idx = passthrough.index(tok)
+            if idx + 1 < len(passthrough):
+                return passthrough[idx + 1]
+    return None
+
+
+def _merge_shards(output_dir: str, num_shards: int) -> dict:
+    """Merge per-shard JSON files into a single result dict and write eval_info.json."""
+    all_per_task: list[dict] = []
+    per_group: dict[str, dict] = {}
+
+    for k in range(num_shards):
+        shard_path = Path(output_dir) / f"shard_{k}_of_{num_shards}.json"
+        if not shard_path.exists():
+            print(f"[warning] shard file not found: {shard_path}", file=sys.stderr)
+            continue
+        with open(shard_path) as f:
+            shard = json.load(f)
+        all_per_task.extend(shard.get("per_task", []))
+        for group, metrics in shard.get("per_group", {}).items():
+            if group not in per_group:
+                per_group[group] = {"sum_rewards": [], "max_rewards": [], "successes": []}
+            for key in ("sum_rewards", "max_rewards", "successes"):
+                # metrics may store aggregates; reconstruct lists if possible
+                per_group[group][key].extend(metrics.get(key, []))
+
+    # Re-aggregate
+    import numpy as np
+
+    def _nanmean(xs: list) -> float:
+        return float(np.nanmean(xs)) if xs else float("nan")
+
+    groups_out = {}
+    all_sr, all_mr, all_succ = [], [], []
+    for group, acc in per_group.items():
+        groups_out[group] = {
+            "avg_sum_reward": _nanmean(acc["sum_rewards"]),
+            "avg_max_reward": _nanmean(acc["max_rewards"]),
+            "pc_success": _nanmean(acc["successes"]) * 100 if acc["successes"] else float("nan"),
+            "n_episodes": len(acc["sum_rewards"]),
+        }
+        all_sr.extend(acc["sum_rewards"])
+        all_mr.extend(acc["max_rewards"])
+        all_succ.extend(acc["successes"])
+
+    overall = {
+        "avg_sum_reward": _nanmean(all_sr),
+        "avg_max_reward": _nanmean(all_mr),
+        "pc_success": _nanmean(all_succ) * 100 if all_succ else float("nan"),
+        "n_episodes": len(all_sr),
+    }
+
+    merged = {"per_task": all_per_task, "per_group": groups_out, "overall": overall}
+    out_path = Path(output_dir) / "eval_info.json"
+    with open(out_path, "w") as f:
+        json.dump(merged, f, indent=2)
+    return merged
+
+
+def main(argv: list[str] | None = None) -> None:
+    args, passthrough = _parse_known(argv if argv is not None else sys.argv[1:])
+
+    num_shards = _resolve_num_shards(args.num_shards, passthrough)
+    mujoco_gl = _resolve_mujoco_gl(args.render_device, num_shards, passthrough)
+
+    output_dir = args.output_dir or _extract_output_dir(passthrough)
+
+    print(f"[lerobot-eval-parallel] launching {num_shards} shard(s), MUJOCO_GL={mujoco_gl}")
+
+    child_env = {**os.environ, "MUJOCO_GL": mujoco_gl, "OMP_NUM_THREADS": "1"}
+
+    procs = []
+    for k in range(num_shards):
+        cmd = [
+            sys.executable,
+            "-m",
+            "lerobot.scripts.lerobot_eval",
+            f"eval.shard_id={k}",
+            f"eval.num_shards={num_shards}",
+            *passthrough,
+        ]
+        if output_dir:
+            # Each shard shares the same output_dir; shard files are named shard_K_of_N.json
+            cmd.append(f"output_dir={output_dir}")
+        procs.append(subprocess.Popen(cmd, env=child_env))
+
+    return_codes = [p.wait() for p in procs]
+    if any(rc != 0 for rc in return_codes):
+        failed = [k for k, rc in enumerate(return_codes) if rc != 0]
+        print(f"[lerobot-eval-parallel] shards {failed} failed with non-zero exit codes.", file=sys.stderr)
+        sys.exit(1)
+
+    if output_dir and num_shards > 1:
+        merged = _merge_shards(output_dir, num_shards)
+        print("\n=== Merged Results ===")
+        print(json.dumps(merged["overall"], indent=2))
+
+
+if __name__ == "__main__":
+    main()

src/lerobot/scripts/lerobot_record.py

@@ -74,6 +74,8 @@ from pathlib import Path
 from pprint import pformat
 from typing import Any
 
+import torch
+
 from lerobot.cameras import (  # noqa: F401
     CameraConfig,  # noqa: F401
 )
@@ -90,6 +92,7 @@ from lerobot.datasets.pipeline_features import aggregate_pipeline_dataset_featur
 from lerobot.datasets.video_utils import VideoEncodingManager
 from lerobot.policies.factory import make_policy, make_pre_post_processors
 from lerobot.policies.pretrained import PreTrainedPolicy
+from lerobot.policies.rtc import ActionInterpolator
 from lerobot.policies.utils import make_robot_action
 from lerobot.processor import (
     PolicyAction,
@@ -226,6 +229,9 @@ class RecordConfig:
     play_sounds: bool = True
     # Resume recording on an existing dataset.
     resume: bool = False
+    # Action interpolation multiplier for smoother policy control (1=off, 2=2x, 3=3x)
+    # Only applies when using a policy (not teleop)
+    interpolation_multiplier: int = 1
 
     def __post_init__(self):
         # HACK: We parse again the cli args here to get the pretrained path if there was one.
@@ -298,6 +304,7 @@ def record_loop(
     control_time_s: int | None = None,
     single_task: str | None = None,
     display_data: bool = False,
+    interpolator: ActionInterpolator | None = None,
     display_compressed_images: bool = False,
 ):
     if dataset is not None and dataset.fps != fps:
@@ -334,6 +341,16 @@ def record_loop(
         preprocessor.reset()
         postprocessor.reset()
 
+    # Reset interpolator if provided
+    if interpolator is not None:
+        interpolator.reset()
+
+    # Calculate control interval based on interpolation
+    use_interpolation = interpolator is not None and interpolator.enabled and policy is not None
+    control_interval = interpolator.get_control_interval(fps) if interpolator else 1 / fps
+    # Pre-compute action key order outside the hot loop — it won't change mid-episode.
+    action_keys = sorted(robot.action_features) if use_interpolation else []
+
     no_action_count = 0
     timestamp = 0
     start_episode_t = time.perf_counter()
@@ -353,28 +370,67 @@ def record_loop(
         if policy is not None or dataset is not None:
             observation_frame = build_dataset_frame(dataset.features, obs_processed, prefix=OBS_STR)
 
+        # Track whether this iteration should be recorded to the dataset.
+        # Interpolated-only iterations send actions to the robot but don't record frames,
+        # keeping the dataset at the original fps while the robot moves at the higher rate.
+        is_record_frame = True
+
         # Get action from either policy or teleop
         if policy is not None and preprocessor is not None and postprocessor is not None:
-            action_values = predict_action(
-                observation=observation_frame,
-                policy=policy,
-                device=get_safe_torch_device(policy.config.device),
-                preprocessor=preprocessor,
-                postprocessor=postprocessor,
-                use_amp=policy.config.use_amp,
-                task=single_task,
-                robot_type=robot.robot_type,
-            )
+            # With interpolation: only call policy when interpolator needs new action
+            if use_interpolation:
+                ran_inference = False
 
-            act_processed_policy: RobotAction = make_robot_action(action_values, dataset.features)
+                if interpolator.needs_new_action():
+                    action_values = predict_action(
+                        observation=observation_frame,
+                        policy=policy,
+                        device=get_safe_torch_device(policy.config.device),
+                        preprocessor=preprocessor,
+                        postprocessor=postprocessor,
+                        use_amp=policy.config.use_amp,
+                        task=single_task,
+                        robot_type=robot.robot_type,
+                    )
+                    act_processed_policy = make_robot_action(action_values, dataset.features)
+                    robot_action_to_send = robot_action_processor((act_processed_policy, obs))
+
+                    action_tensor = torch.tensor([robot_action_to_send[k] for k in action_keys])
+                    interpolator.add(action_tensor)
+                    ran_inference = True
+
+                interp_action = interpolator.get()
+                if interp_action is not None:
+                    robot_action_to_send = {k: interp_action[i].item() for i, k in enumerate(action_keys)}
+                    action_values = robot_action_to_send
+                else:
+                    continue
+
+                is_record_frame = ran_inference
+            else:
+                action_values = predict_action(
+                    observation=observation_frame,
+                    policy=policy,
+                    device=get_safe_torch_device(policy.config.device),
+                    preprocessor=preprocessor,
+                    postprocessor=postprocessor,
+                    use_amp=policy.config.use_amp,
+                    task=single_task,
+                    robot_type=robot.robot_type,
+                )
+                act_processed_policy: RobotAction = make_robot_action(action_values, dataset.features)
+                # Applies a pipeline to the action, default is IdentityProcessor
+                robot_action_to_send = robot_action_processor((act_processed_policy, obs))
 
         elif policy is None and isinstance(teleop, Teleoperator):
+            act = teleop.get_action()
             if robot.name == "unitree_g1":
                 teleop.send_feedback(obs)
-            act = teleop.get_action()
 
             # Applies a pipeline to the raw teleop action, default is IdentityProcessor
             act_processed_teleop = teleop_action_processor((act, obs))
+            action_values = act_processed_teleop
+            robot_action_to_send = robot_action_processor((act_processed_teleop, obs))
 
         elif policy is None and isinstance(teleop, list):
             arm_action = teleop_arm.get_action()
@@ -383,6 +439,8 @@ def record_loop(
             base_action = robot._from_keyboard_to_base_action(keyboard_action)
             act = {**arm_action, **base_action} if len(base_action) > 0 else arm_action
             act_processed_teleop = teleop_action_processor((act, obs))
+            action_values = act_processed_teleop
+            robot_action_to_send = robot_action_processor((act_processed_teleop, obs))
         else:
             no_action_count += 1
             if no_action_count == 1 or no_action_count % 10 == 0:
@@ -393,22 +451,14 @@ def record_loop(
                 )
             continue
 
-        # Applies a pipeline to the action, default is IdentityProcessor
-        if policy is not None and act_processed_policy is not None:
-            action_values = act_processed_policy
-            robot_action_to_send = robot_action_processor((act_processed_policy, obs))
-        else:
-            action_values = act_processed_teleop
-            robot_action_to_send = robot_action_processor((act_processed_teleop, obs))
-
         # Send action to robot
         # Action can eventually be clipped using `max_relative_target`,
         # so action actually sent is saved in the dataset. action = postprocessor.process(action)
         # TODO(steven, pepijn, adil): we should use a pipeline step to clip the action, so the sent action is the action that we input to the robot.
         _sent_action = robot.send_action(robot_action_to_send)
 
-        # Write to dataset
-        if dataset is not None:
+        # Write to dataset (only on real policy frames, not interpolated-only iterations)
+        if dataset is not None and is_record_frame:
             action_frame = build_dataset_frame(dataset.features, action_values, prefix=ACTION)
             frame = {**observation_frame, **action_frame, "task": single_task}
             dataset.add_frame(frame)
@@ -420,7 +470,7 @@ def record_loop(
 
         dt_s = time.perf_counter() - start_loop_t
 
-        sleep_time_s: float = 1 / fps - dt_s
+        sleep_time_s: float = control_interval - dt_s
         if sleep_time_s < 0:
             logging.warning(
                 f"Record loop is running slower ({1 / dt_s:.1f} Hz) than the target FPS ({fps} Hz). Dataset frames might be dropped and robot control might be unstable. Common causes are: 1) Camera FPS not keeping up 2) Policy inference taking too long 3) CPU starvation"
@@ -506,6 +556,7 @@ def record(cfg: RecordConfig) -> LeRobotDataset:
         policy = None if cfg.policy is None else make_policy(cfg.policy, ds_meta=dataset.meta)
         preprocessor = None
         postprocessor = None
+        interpolator = None
         if cfg.policy is not None:
             preprocessor, postprocessor = make_pre_post_processors(
                 policy_cfg=cfg.policy,
@@ -516,6 +567,10 @@ def record(cfg: RecordConfig) -> LeRobotDataset:
                     "rename_observations_processor": {"rename_map": cfg.dataset.rename_map},
                 },
             )
+            # Create interpolator for smoother policy control
+            if cfg.interpolation_multiplier > 1:
+                interpolator = ActionInterpolator(multiplier=cfg.interpolation_multiplier)
+                logging.info(f"Action interpolation enabled: {cfg.interpolation_multiplier}x control rate")
 
         robot.connect()
         if teleop is not None:
@@ -547,6 +602,7 @@ def record(cfg: RecordConfig) -> LeRobotDataset:
                     control_time_s=cfg.dataset.episode_time_s,
                     single_task=cfg.dataset.single_task,
                     display_data=cfg.display_data,
+                    interpolator=interpolator,
                     display_compressed_images=display_compressed_images,
                 )
 

src/lerobot/teleoperators/openarm_mini/openarm_mini.py

@@ -32,9 +32,15 @@ from .config_openarm_mini import OpenArmMiniConfig
 logger = logging.getLogger(__name__)
 
 # Motors whose direction is inverted during readout
-RIGHT_MOTORS_TO_FLIP = ["joint_1", "joint_2", "joint_3", "joint_4", "joint_5"]
+RIGHT_MOTORS_TO_FLIP = ["joint_1", "joint_2", "joint_3", "joint_4", "joint_5", "joint_7"]
 LEFT_MOTORS_TO_FLIP = ["joint_1", "joint_3", "joint_4", "joint_5", "joint_6", "joint_7"]
 
+# Leader joint 6 maps to follower joint 7 and vice versa
+JOINT_REMAP = {"joint_6": "joint_7", "joint_7": "joint_6"}
+JOINT_REMAP_REVERSE = {"joint_7": "joint_6", "joint_6": "joint_7"}
+
+GRIPPER_TELEOP_TO_DEGREES = -0.65
+
 
 class OpenArmMini(Teleoperator):
     """
@@ -95,6 +101,8 @@ class OpenArmMini(Teleoperator):
 
     @property
     def action_features(self) -> dict[str, type]:
+        # Right first, then left — matches the robot (BiOpenArmFollower) ordering
+        # and the dataset feature names recorded during data collection.
         features: dict[str, type] = {}
         for motor in self.bus_right.motors:
             features[f"right_{motor}.pos"] = float
@@ -276,16 +284,70 @@ class OpenArmMini(Teleoperator):
         right_positions = self.bus_right.sync_read("Present_Position")
         left_positions = self.bus_left.sync_read("Present_Position")
 
+        # Right first, then left — matches the robot (BiOpenArmFollower) ordering
+        # and the dataset feature names recorded during data collection.
+        # Joint 6↔7 remap: leader joint_6 → follower joint_7 and vice versa.
         action: dict[str, Any] = {}
         for motor, val in right_positions.items():
-            action[f"right_{motor}.pos"] = -val if motor in RIGHT_MOTORS_TO_FLIP else val
+            target = JOINT_REMAP.get(motor, motor)
+            if motor == "gripper":
+                # Convert gripper from teleop 0-100 to openarms degrees: 0→0°, 100→-65°
+                action[f"right_{target}.pos"] = val * GRIPPER_TELEOP_TO_DEGREES
+            else:
+                action[f"right_{target}.pos"] = -val if motor in RIGHT_MOTORS_TO_FLIP else val
         for motor, val in left_positions.items():
-            action[f"left_{motor}.pos"] = -val if motor in LEFT_MOTORS_TO_FLIP else val
+            target = JOINT_REMAP.get(motor, motor)
+            if motor == "gripper":
+                action[f"left_{target}.pos"] = val * GRIPPER_TELEOP_TO_DEGREES
+            else:
+                action[f"left_{target}.pos"] = -val if motor in LEFT_MOTORS_TO_FLIP else val
 
         dt_ms = (time.perf_counter() - start) * 1e3
         logger.debug(f"{self} read action: {dt_ms:.1f}ms")
         return action
 
+    def enable_torque(self) -> None:
+        """Enable torque on both arms for position control."""
+        self.bus_right.enable_torque()
+        self.bus_left.enable_torque()
+
+    def disable_torque(self) -> None:
+        """Disable torque on both arms for free movement."""
+        self.bus_right.disable_torque()
+        self.bus_left.disable_torque()
+
+    def write_goal_positions(self, positions: dict[str, float]) -> None:
+        """Write goal positions to motors (inverse of get_action flip/gripper/remap logic)."""
+        right_goals: dict[str, float] = {}
+        left_goals: dict[str, float] = {}
+
+        for key, val in positions.items():
+            if not key.endswith(".pos"):
+                continue
+            motor_name = key.removesuffix(".pos")
+            if motor_name.startswith("right_"):
+                base = motor_name.removeprefix("right_")
+                # Reverse remap: follower joint_7 → leader joint_6 and vice versa
+                target = JOINT_REMAP_REVERSE.get(base, base)
+                if base == "gripper":
+                    # Convert robot degrees to teleop 0-100: 0°→0, -65°→100
+                    right_goals[target] = val / GRIPPER_TELEOP_TO_DEGREES
+                else:
+                    # Un-flip using the ORIGINAL motor name (target = leader motor)
+                    right_goals[target] = -val if target in RIGHT_MOTORS_TO_FLIP else val
+            elif motor_name.startswith("left_"):
+                base = motor_name.removeprefix("left_")
+                target = JOINT_REMAP_REVERSE.get(base, base)
+                if base == "gripper":
+                    left_goals[target] = val / GRIPPER_TELEOP_TO_DEGREES
+                else:
+                    left_goals[target] = -val if target in LEFT_MOTORS_TO_FLIP else val
+
+        if right_goals:
+            self.bus_right.sync_write("Goal_Position", right_goals)
+        if left_goals:
+            self.bus_left.sync_write("Goal_Position", left_goals)
+
     def send_feedback(self, feedback: dict[str, float]) -> None:
         raise NotImplementedError("Feedback is not yet implemented for OpenArm Mini.")
 

tests/envs/test_dispatch.py

@@ -0,0 +1,143 @@
+"""Tests for the benchmark dispatch refactor (create_envs / get_env_processors on EnvConfig)."""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass, field
+
+import gymnasium as gym
+import pytest
+from gymnasium.envs.registration import register, registry as gym_registry
+
+from lerobot.configs.types import PolicyFeature
+from lerobot.envs.configs import EnvConfig
+from lerobot.envs.factory import make_env, make_env_config, make_env_pre_post_processors
+
+logger = logging.getLogger(__name__)
+
+
+def test_registry_all_types():
+    """make_env_config should resolve every registered EnvConfig subclass via the registry."""
+    known = list(EnvConfig.get_known_choices().keys())
+    assert len(known) >= 6
+    for t in known:
+        cfg = make_env_config(t)
+        if not isinstance(cfg, EnvConfig):
+            continue
+        assert cfg.type == t
+
+
+def test_unknown_type():
+    with pytest.raises(ValueError, match="not registered"):
+        make_env_config("nonexistent")
+
+
+def test_identity_processors():
+    """Base class get_env_processors() returns identity pipelines."""
+    cfg = make_env_config("aloha")
+    pre, post = cfg.get_env_processors()
+    assert len(pre.steps) == 0 and len(post.steps) == 0
+
+
+def test_delegation():
+    """make_env() should call cfg.create_envs(), not use if/elif dispatch."""
+    sentinel = {"delegated": {0: "marker"}}
+    fake = type(
+        "Fake",
+        (),
+        {
+            "hub_path": None,
+            "create_envs": lambda self, n_envs, use_async_envs=False: sentinel,
+        },
+    )()
+    result = make_env(fake, n_envs=1)
+    assert result is sentinel
+
+
+def test_processors_delegation():
+    """make_env_pre_post_processors delegates to cfg.get_env_processors()."""
+    cfg = make_env_config("aloha")
+    pre, post = make_env_pre_post_processors(cfg, policy_cfg=None)
+    assert len(pre.steps) == 0
+
+
+def test_base_create_envs():
+    """Base class create_envs() should build a single-task VectorEnv via gym.make()."""
+    gym_id = "_dispatch_test/CartPole-v99"
+    if gym_id not in gym_registry:
+        register(id=gym_id, entry_point="gymnasium.envs.classic_control:CartPoleEnv")
+
+    @EnvConfig.register_subclass("_dispatch_base_test")
+    @dataclass
+    class _Env(EnvConfig):
+        task: str = "CartPole-v99"
+        fps: int = 10
+        features: dict[str, PolicyFeature] = field(default_factory=dict)
+
+        @property
+        def package_name(self):
+            return "_dispatch_test"
+
+        @property
+        def gym_id(self):
+            return gym_id
+
+        @property
+        def gym_kwargs(self):
+            return {}
+
+    try:
+        envs = _Env().create_envs(n_envs=2)
+        assert "_dispatch_base_test" in envs
+        env = envs["_dispatch_base_test"][0]
+        assert isinstance(env, gym.vector.VectorEnv)
+        assert env.num_envs == 2
+        env.close()
+    finally:
+        if gym_id in gym_registry:
+            del gym_registry[gym_id]
+
+
+def test_custom_create_envs_override():
+    """A custom EnvConfig subclass can override create_envs()."""
+    mock_vec = gym.vector.SyncVectorEnv([lambda: gym.make("CartPole-v1")])
+
+    @EnvConfig.register_subclass("_dispatch_custom_test")
+    @dataclass
+    class _Env(EnvConfig):
+        task: str = "x"
+        features: dict[str, PolicyFeature] = field(default_factory=dict)
+
+        @property
+        def gym_kwargs(self):
+            return {}
+
+        def create_envs(self, n_envs, use_async_envs=False):
+            return {"custom_suite": {0: mock_vec}}
+
+    try:
+        result = make_env(_Env(), n_envs=1)
+        assert "custom_suite" in result
+    finally:
+        mock_vec.close()
+
+
+def test_custom_get_env_processors_override():
+    """A custom EnvConfig subclass can override get_env_processors()."""
+    from lerobot.processor.pipeline import DataProcessorPipeline
+
+    @EnvConfig.register_subclass("_dispatch_proc_test")
+    @dataclass
+    class _Env(EnvConfig):
+        task: str = "x"
+        features: dict[str, PolicyFeature] = field(default_factory=dict)
+
+        @property
+        def gym_kwargs(self):
+            return {}
+
+        def get_env_processors(self):
+            return DataProcessorPipeline(steps=[]), DataProcessorPipeline(steps=[])
+
+    pre, post = _Env().get_env_processors()
+    assert isinstance(pre, DataProcessorPipeline)

tests/policies/rtc/test_action_interpolator.py

@@ -0,0 +1,559 @@
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for ActionInterpolator and its interaction with ActionQueue (RTC)."""
+
+import pytest
+import torch
+
+from lerobot.policies.rtc.action_interpolator import ActionInterpolator
+from lerobot.policies.rtc.action_queue import ActionQueue
+from lerobot.policies.rtc.configuration_rtc import RTCConfig
+
+# ====================== Fixtures ======================
+
+
+@pytest.fixture
+def interp2():
+    """Create an ActionInterpolator with multiplier=2."""
+    return ActionInterpolator(multiplier=2)
+
+
+@pytest.fixture
+def interp3():
+    """Create an ActionInterpolator with multiplier=3."""
+    return ActionInterpolator(multiplier=3)
+
+
+# ====================== Initialization Tests ======================
+
+
+def test_interpolator_multiplier_1_no_interpolation():
+    """Test multiplier=1 creates a disabled interpolator."""
+    interp = ActionInterpolator(multiplier=1)
+    assert interp.multiplier == 1
+    assert not interp.enabled
+
+
+def test_interpolator_multiplier_2_enabled():
+    """Test multiplier=2 creates an enabled interpolator."""
+    interp = ActionInterpolator(multiplier=2)
+    assert interp.multiplier == 2
+    assert interp.enabled
+
+
+def test_interpolator_multiplier_0_raises():
+    """Test multiplier=0 raises ValueError."""
+    with pytest.raises(ValueError, match="multiplier must be >= 1"):
+        ActionInterpolator(multiplier=0)
+
+
+def test_interpolator_negative_multiplier_raises():
+    """Test negative multiplier raises ValueError."""
+    with pytest.raises(ValueError, match="multiplier must be >= 1"):
+        ActionInterpolator(multiplier=-1)
+
+
+def test_interpolator_default_multiplier_is_1():
+    """Test default multiplier is 1 (disabled)."""
+    interp = ActionInterpolator()
+    assert interp.multiplier == 1
+    assert not interp.enabled
+
+
+# ====================== needs_new_action Tests ======================
+
+
+def test_needs_new_action_true_initially(interp2):
+    """Test needs_new_action() returns True before any action is added."""
+    assert interp2.needs_new_action()
+
+
+def test_needs_new_action_false_after_add(interp2):
+    """Test needs_new_action() returns False right after add()."""
+    interp2.add(torch.tensor([1.0, 2.0]))
+    assert not interp2.needs_new_action()
+
+
+def test_needs_new_action_true_after_buffer_exhausted(interp2):
+    """Test needs_new_action() returns True after consuming all buffered actions."""
+    interp2.add(torch.tensor([1.0, 2.0]))
+    interp2.get()
+    assert interp2.needs_new_action()
+
+
+def test_needs_new_action_true_after_all_interpolated_consumed(interp2):
+    """Test needs_new_action() tracks interpolated sub-steps correctly."""
+    interp2.add(torch.tensor([0.0, 0.0]))
+    interp2.get()
+    assert interp2.needs_new_action()
+
+    interp2.add(torch.tensor([2.0, 4.0]))
+    interp2.get()
+    assert not interp2.needs_new_action()
+    interp2.get()
+    assert interp2.needs_new_action()
+
+
+# ====================== Passthrough Tests (multiplier=1) ======================
+
+
+def test_passthrough_single_action_returned_as_is():
+    """Test multiplier=1 returns the action unchanged."""
+    interp = ActionInterpolator(multiplier=1)
+    action = torch.tensor([3.0, 5.0])
+    interp.add(action)
+
+    result = interp.get()
+    assert result is not None
+    torch.testing.assert_close(result, action)
+
+
+def test_passthrough_none_after_single_get():
+    """Test multiplier=1 returns None after consuming the single action."""
+    interp = ActionInterpolator(multiplier=1)
+    interp.add(torch.tensor([1.0]))
+    interp.get()
+    assert interp.get() is None
+
+
+def test_passthrough_sequential_actions():
+    """Test multiplier=1 passes through consecutive actions one at a time."""
+    interp = ActionInterpolator(multiplier=1)
+    for val in [1.0, 2.0, 3.0]:
+        action = torch.tensor([val])
+        interp.add(action)
+        result = interp.get()
+        torch.testing.assert_close(result, action)
+        assert interp.get() is None
+
+
+# ====================== Interpolation Tests (multiplier=2) ======================
+
+
+def test_interpolation_2x_first_action_no_interpolation(interp2):
+    """Test first action has no previous, so buffer is just [action]."""
+    interp2.add(torch.tensor([0.0, 0.0]))
+    result = interp2.get()
+    torch.testing.assert_close(result, torch.tensor([0.0, 0.0]))
+    assert interp2.get() is None
+
+
+def test_interpolation_2x_second_action_produces_two_steps(interp2):
+    """Test second action produces 2 interpolated sub-steps."""
+    interp2.add(torch.tensor([0.0, 0.0]))
+    interp2.get()
+
+    interp2.add(torch.tensor([2.0, 4.0]))
+    step1 = interp2.get()
+    step2 = interp2.get()
+
+    torch.testing.assert_close(step1, torch.tensor([1.0, 2.0]))
+    torch.testing.assert_close(step2, torch.tensor([2.0, 4.0]))
+    assert interp2.get() is None
+
+
+def test_interpolation_2x_three_consecutive_actions(interp2):
+    """Test interpolation across three consecutive actions."""
+    a0 = torch.tensor([0.0])
+    a1 = torch.tensor([4.0])
+    a2 = torch.tensor([10.0])
+
+    interp2.add(a0)
+    torch.testing.assert_close(interp2.get(), a0)
+
+    interp2.add(a1)
+    torch.testing.assert_close(interp2.get(), torch.tensor([2.0]))
+    torch.testing.assert_close(interp2.get(), torch.tensor([4.0]))
+
+    interp2.add(a2)
+    torch.testing.assert_close(interp2.get(), torch.tensor([7.0]))
+    torch.testing.assert_close(interp2.get(), torch.tensor([10.0]))
+
+
+# ====================== Interpolation Tests (multiplier=3) ======================
+
+
+def test_interpolation_3x_produces_three_steps(interp3):
+    """Test multiplier=3 produces 3 interpolated sub-steps."""
+    interp3.add(torch.tensor([0.0, 0.0]))
+    interp3.get()
+
+    interp3.add(torch.tensor([3.0, 6.0]))
+    s1 = interp3.get()
+    s2 = interp3.get()
+    s3 = interp3.get()
+
+    torch.testing.assert_close(s1, torch.tensor([1.0, 2.0]))
+    torch.testing.assert_close(s2, torch.tensor([2.0, 4.0]))
+    torch.testing.assert_close(s3, torch.tensor([3.0, 6.0]))
+    assert interp3.get() is None
+
+
+def test_interpolation_3x_last_step_equals_target(interp3):
+    """Test last interpolated step equals the target action exactly."""
+    interp3.add(torch.tensor([10.0]))
+    interp3.get()
+
+    target = torch.tensor([100.0])
+    interp3.add(target)
+    interp3.get()
+    interp3.get()
+    last = interp3.get()
+    torch.testing.assert_close(last, target)
+
+
+# ====================== Reset Tests ======================
+
+
+def test_reset_clears_buffer(interp2):
+    """Test reset() clears the action buffer."""
+    interp2.add(torch.tensor([1.0]))
+    interp2.reset()
+    assert interp2.needs_new_action()
+    assert interp2.get() is None
+
+
+def test_reset_clears_prev(interp2):
+    """Test after reset, next add produces single-element buffer (no prev)."""
+    interp2.add(torch.tensor([0.0]))
+    interp2.get()
+    interp2.add(torch.tensor([10.0]))
+    interp2.get()
+    interp2.get()
+
+    interp2.reset()
+    interp2.add(torch.tensor([5.0]))
+    result = interp2.get()
+    torch.testing.assert_close(result, torch.tensor([5.0]))
+    assert interp2.get() is None
+
+
+def test_reset_episode_boundary(interp2):
+    """Test reset between two simulated episodes."""
+    interp2.add(torch.tensor([0.0]))
+    interp2.get()
+    interp2.add(torch.tensor([10.0]))
+    interp2.get()
+    interp2.get()
+
+    interp2.reset()
+
+    interp2.add(torch.tensor([100.0]))
+    result = interp2.get()
+    torch.testing.assert_close(result, torch.tensor([100.0]))
+    assert interp2.get() is None
+
+
+# ====================== get_control_interval Tests ======================
+
+
+def test_control_interval_30fps_multiplier_1():
+    """Test control interval at 30fps with no interpolation."""
+    interp = ActionInterpolator(multiplier=1)
+    assert interp.get_control_interval(30.0) == pytest.approx(1.0 / 30.0)
+
+
+def test_control_interval_30fps_multiplier_2(interp2):
+    """Test control interval at 30fps with 2x interpolation."""
+    assert interp2.get_control_interval(30.0) == pytest.approx(1.0 / 60.0)
+
+
+def test_control_interval_30fps_multiplier_3(interp3):
+    """Test control interval at 30fps with 3x interpolation."""
+    assert interp3.get_control_interval(30.0) == pytest.approx(1.0 / 90.0)
+
+
+def test_control_interval_60fps_multiplier_2(interp2):
+    """Test control interval at 60fps with 2x interpolation."""
+    assert interp2.get_control_interval(60.0) == pytest.approx(1.0 / 120.0)
+
+
+# ====================== get() on Empty Tests ======================
+
+
+def test_get_returns_none_before_any_add():
+    """Test get() returns None when no action has been added."""
+    interp = ActionInterpolator(multiplier=2)
+    assert interp.get() is None
+
+
+def test_get_returns_none_after_reset(interp2):
+    """Test get() returns None after reset."""
+    interp2.add(torch.tensor([1.0]))
+    interp2.reset()
+    assert interp2.get() is None
+
+
+# ====================== Multi-Dimensional Action Tests ======================
+
+
+def test_6dof_interpolation(interp2):
+    """Test interpolation works correctly with 6-dimensional actions."""
+    prev = torch.zeros(6)
+    target = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0, 6.0])
+
+    interp2.add(prev)
+    interp2.get()
+
+    interp2.add(target)
+    mid = interp2.get()
+    end = interp2.get()
+
+    torch.testing.assert_close(mid, target / 2)
+    torch.testing.assert_close(end, target)
+
+
+# ====================== Simulated Control Loop Tests ======================
+
+
+def test_control_loop_produces_correct_action_count():
+    """Test N policy actions with multiplier M yields 1 + (N-1)*M robot commands."""
+    multiplier = 3
+    n_policy_actions = 5
+    interp = ActionInterpolator(multiplier=multiplier)
+
+    robot_commands = 0
+    for i in range(n_policy_actions):
+        action = torch.tensor([float(i)])
+        if interp.needs_new_action():
+            interp.add(action)
+        while True:
+            a = interp.get()
+            if a is None:
+                break
+            robot_commands += 1
+
+    expected = 1 + (n_policy_actions - 1) * multiplier
+    assert robot_commands == expected
+
+
+def test_control_loop_monotonic_increase():
+    """Test actions [0, 1, 2, 3] with multiplier=2 produce monotonically increasing values."""
+    interp = ActionInterpolator(multiplier=2)
+    all_values = []
+
+    for i in range(4):
+        interp.add(torch.tensor([float(i)]))
+        while True:
+            a = interp.get()
+            if a is None:
+                break
+            all_values.append(a.item())
+
+    for i in range(1, len(all_values)):
+        assert all_values[i] >= all_values[i - 1]
+
+
+# ====================== ActionQueue + ActionInterpolator Integration Tests ======================
+
+
+def _make_chunk(n_steps: int, action_dim: int = 2, offset: float = 0.0) -> torch.Tensor:
+    """Create a simple action chunk: each row is [offset + step_idx, offset + step_idx]."""
+    return torch.arange(n_steps, dtype=torch.float32).unsqueeze(1).expand(-1, action_dim) + offset
+
+
+def test_queue_interpolator_consumption_rate_matches_base_fps():
+    """Test queue.get() is called at base fps rate, not multiplied fps."""
+    cfg = RTCConfig(enabled=True, execution_horizon=10)
+    queue = ActionQueue(cfg)
+    interp = ActionInterpolator(multiplier=3)
+
+    chunk = _make_chunk(10)
+    queue.merge(chunk, chunk.clone(), real_delay=0)
+
+    queue_gets = 0
+    control_ticks = 0
+
+    while True:
+        if interp.needs_new_action():
+            if queue.empty():
+                break
+            action = queue.get()
+            if action is None:
+                break
+            interp.add(action)
+            queue_gets += 1
+
+        result = interp.get()
+        if result is not None:
+            control_ticks += 1
+
+    assert queue_gets == 10
+    assert control_ticks == 1 + 9 * 3
+
+
+def test_queue_interpolator_leftover_decreases_only_on_queue_get():
+    """Test get_left_over() shrinks only on queue.get(), not on interpolator sub-steps."""
+    cfg = RTCConfig(enabled=True, execution_horizon=10)
+    queue = ActionQueue(cfg)
+    interp = ActionInterpolator(multiplier=3)
+
+    chunk = _make_chunk(6)
+    queue.merge(chunk, chunk.clone(), real_delay=0)
+
+    assert interp.needs_new_action()
+    interp.add(queue.get())
+    leftover_after_first_get = queue.get_left_over()
+    assert leftover_after_first_get is not None
+    assert len(leftover_after_first_get) == 5
+
+    interp.get()
+    assert len(queue.get_left_over()) == 5
+
+    interp.add(queue.get())
+    assert len(queue.get_left_over()) == 4
+
+    for _ in range(3):
+        assert interp.get() is not None
+    assert len(queue.get_left_over()) == 4
+
+
+def test_queue_interpolator_processed_leftover_tracks_queue_index():
+    """Test get_processed_left_over() reflects queue's last_index, not interpolator state."""
+    cfg = RTCConfig(enabled=True, execution_horizon=10)
+    queue = ActionQueue(cfg)
+    interp = ActionInterpolator(multiplier=2)
+
+    original = _make_chunk(8, offset=0.0)
+    processed = _make_chunk(8, offset=100.0)
+    queue.merge(original, processed, real_delay=0)
+
+    left = queue.get_processed_left_over()
+    assert len(left) == 8
+
+    for _ in range(3):
+        if interp.needs_new_action():
+            action = queue.get()
+            if action is not None:
+                interp.add(action)
+        interp.get()
+
+    proc_left = queue.get_processed_left_over()
+    orig_left = queue.get_left_over()
+    assert proc_left is not None and orig_left is not None
+    assert len(proc_left) == len(orig_left)
+    assert proc_left[0, 0].item() >= 100.0
+    assert orig_left[0, 0].item() < 100.0
+
+
+def test_queue_interpolator_merge_resets_queue_but_interpolator_keeps_prev():
+    """Test queue merge doesn't affect interpolator's prev, enabling smooth transitions."""
+    cfg = RTCConfig(enabled=True, execution_horizon=10)
+    queue = ActionQueue(cfg)
+    interp = ActionInterpolator(multiplier=2)
+
+    chunk1 = torch.tensor([[0.0], [2.0], [4.0], [6.0], [8.0]])
+    queue.merge(chunk1, chunk1.clone(), real_delay=0)
+
+    consumed = []
+    for _ in range(5):
+        if interp.needs_new_action():
+            a = queue.get()
+            if a is not None:
+                interp.add(a)
+        r = interp.get()
+        if r is not None:
+            consumed.append(r.item())
+
+    assert interp.needs_new_action()
+    assert consumed[-1] == pytest.approx(4.0)
+
+    idx_before = queue.get_action_index()
+
+    chunk2 = torch.tensor([[10.0], [12.0], [14.0]])
+    queue.merge(chunk2, chunk2.clone(), real_delay=0, action_index_before_inference=idx_before)
+
+    first_action = queue.get()
+    assert first_action is not None
+    interp.add(first_action)
+    first_from_new = interp.get()
+    assert first_from_new is not None
+    assert first_from_new.item() == pytest.approx(7.0)
+
+
+def test_queue_interpolator_reset_does_not_affect_queue():
+    """Test interpolator reset leaves queue state untouched."""
+    cfg = RTCConfig(enabled=True, execution_horizon=10)
+    queue = ActionQueue(cfg)
+    interp = ActionInterpolator(multiplier=2)
+
+    chunk = _make_chunk(5)
+    queue.merge(chunk, chunk.clone(), real_delay=0)
+
+    interp.add(queue.get())
+    interp.get()
+    interp.add(queue.get())
+    interp.get()
+    interp.get()
+
+    assert queue.qsize() == 3
+
+    interp.reset()
+
+    assert queue.qsize() == 3
+    assert len(queue.get_left_over()) == 3
+
+    interp.add(queue.get())
+    result = interp.get()
+    assert result is not None
+    assert queue.qsize() == 2
+
+
+def test_queue_interpolator_no_interpolation_1_to_1():
+    """Test multiplier=1 produces exactly 1 robot command per queue.get()."""
+    cfg = RTCConfig(enabled=True, execution_horizon=10)
+    queue = ActionQueue(cfg)
+    interp = ActionInterpolator(multiplier=1)
+
+    chunk = _make_chunk(5)
+    queue.merge(chunk, chunk.clone(), real_delay=0)
+
+    robot_commands = 0
+    while not queue.empty():
+        if interp.needs_new_action():
+            action = queue.get()
+            if action is not None:
+                interp.add(action)
+        result = interp.get()
+        if result is not None:
+            robot_commands += 1
+
+    assert robot_commands == 5
+
+
+def test_queue_interpolator_delay_skips_stale_actions():
+    """Test merge with delay correctly skips stale actions for the interpolator."""
+    cfg = RTCConfig(enabled=True, execution_horizon=10)
+    queue = ActionQueue(cfg)
+    interp = ActionInterpolator(multiplier=2)
+
+    chunk1 = _make_chunk(10)
+    queue.merge(chunk1, chunk1.clone(), real_delay=0)
+
+    for _ in range(5):
+        if interp.needs_new_action():
+            a = queue.get()
+            if a is not None:
+                interp.add(a)
+        interp.get()
+
+    assert queue.get_action_index() == 3
+
+    chunk2 = _make_chunk(10, offset=100.0)
+    queue.merge(chunk2, chunk2.clone(), real_delay=3, action_index_before_inference=0)
+
+    first_action = queue.get()
+    assert first_action is not None
+    torch.testing.assert_close(first_action, torch.tensor([103.0, 103.0]))

tests/processor/test_tokenizer_processor.py

@@ -189,6 +189,30 @@ def test_list_of_strings_tokenization(mock_auto_tokenizer):
     assert attention_mask.shape == (2, 8)
 
 
+@require_package("transformers")
+@patch("lerobot.processor.tokenizer_processor.AutoTokenizer")
+def test_tuple_of_strings_tokenization(mock_auto_tokenizer):
+    """Test tokenization of a tuple of strings (returned by VectorEnv.call())."""
+    mock_tokenizer = MockTokenizer(vocab_size=100)
+    mock_auto_tokenizer.from_pretrained.return_value = mock_tokenizer
+
+    processor = TokenizerProcessorStep(tokenizer_name="test-tokenizer", max_length=8)
+
+    transition = create_transition(
+        observation={"state": torch.tensor([1.0, 2.0])},
+        action=torch.tensor([0.1, 0.2]),
+        complementary_data={"task": ("pick up cube", "place on table")},
+    )
+
+    result = processor(transition)
+
+    observation = result[TransitionKey.OBSERVATION]
+    tokens = observation[f"{OBS_LANGUAGE}.tokens"]
+    attention_mask = observation[f"{OBS_LANGUAGE}.attention_mask"]
+    assert tokens.shape == (2, 8)
+    assert attention_mask.shape == (2, 8)
+
+
 @require_package("transformers")
 @patch("lerobot.processor.tokenizer_processor.AutoTokenizer")
 def test_custom_keys(mock_auto_tokenizer):