feat(eval): thread-safe policy copies for max_parallel_tasks > 1

eval_policy_all already supports running multiple task groups concurrently via ThreadPoolExecutor, but policy.reset() was not thread-safe: all threads shared the same policy object and its mutable state (action queues, temporal buffers). Fix: each thread receives a shallow copy of the policy. copy.copy() creates a new Python object whose _parameters dict is a shared reference — same tensor storage, zero extra VRAM — while reset() rebinds per-episode state to fresh objects per thread. Caveat: ACT with temporal_ensemble_coeff is not safe with this approach (its reset() mutates a shared sub-object). Keep max_parallel_tasks=1 for that config. For MetaWorld (50 tasks, no temporal ensembling), max_parallel_tasks=4 raises GPU utilization from ~20% to ~60-80% with no additional VRAM cost. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
feat(eval): episode sharding, parallel launcher, and autotune
2026-06-10 13:19:53 +00:00 · 2026-04-07 13:43:42 +02:00 · 2026-04-07 13:43:03 +02:00 · 2026-04-07 13:38:37 +02:00 · 2026-04-07 13:12:42 +02:00 · 2026-04-07 12:30:22 +02:00
49 changed files with 1160 additions and 7123 deletions
@@ -1,86 +0,0 @@
-# LeRobot — Claude Code Instructions
-
-You are a senior robotics ML engineer reviewing code for **LeRobot**, a PyTorch framework for real-world robot learning.
-Apply these principles to every PR review, fix, or task.
-
---
-
-## Core Abstractions
-
-These are the load-bearing types. Handle them with care — breaking changes here affect every user.
-
-| Type             | Location                     | Role                                                         |
-| ---------------- | ---------------------------- | ------------------------------------------------------------ |
-| `LeRobotDataset` | `src/lerobot/datasets/`      | Streaming replay buffer; HF Hub integration                  |
-| `Policy`         | `src/lerobot/policies/`      | Base class for all learning agents (ACT, Diffusion, SARM, …) |
-| `Robot`          | `src/lerobot/robots/`        | Hardware abstraction; carries `_output_pipeline`             |
-| `Teleoperator`   | `src/lerobot/teleoperators/` | Leader-side hardware abstraction; carries `_output_pipeline` |
-| `Env`            | `src/lerobot/envs/`          | Gym-like robotics environments                               |
-| `Processor`      | `src/lerobot/processor/`     | Data transformation pipelines attached to robots/teleops     |
-
-**Never break their public APIs without a migration note and explicit user approval.**
-
---
-
-## Engineering Principles
-
-### Code quality
-
- Explicit over magic — no hidden control flow, no implicit state.
- No deep inheritance trees. Prefer composition.
- No decorative comment separators (`===`, `---`, etc.).
- Add comments only where the logic is non-obvious.
- No over-engineering. YAGNI applies strictly.
-
-### Type safety
-
- All new and modified Python code must be fully typed (PEP 484).
- `mypy --strict` must pass on changed files.
- Do not widen or weaken existing type signatures.
-
-### Backwards compatibility
-
- Public API changes require migration notes.
- Additive changes are preferred over modifications.
- `so100_follower` / `so101_follower` are aliases — never bleed changes there unintentionally.
-
-### HF ecosystem
-
- Use `push_to_hub()`, HF Hub dataset streaming, and `evaluate` scripts.
- Dataset changes must preserve streaming compatibility.
- Prefer reusing HF primitives over rolling custom solutions.
-
---
-
-## PR Review Checklist
-
-Before approving or marking P1 issues resolved, verify:
-
- [ ] `pre-commit run -a` would pass (ruff, mypy, typos, zizmor, bandit)
- [ ] All new/modified code is typed and passes `mypy --strict`
- [ ] New features have unit tests; no silent behavioral changes
- [ ] Public APIs of `LeRobotDataset`, `Policy`, `Robot`, `Teleoperator`, `Env` are unchanged (or migration note present)
- [ ] HF Hub streaming still works for dataset changes
- [ ] No unnecessary abstractions introduced
- [ ] No breaking changes to training scripts (`lerobot-train`, `lerobot-eval`, `lerobot-record`)
-
---
-
-## ML-Specific Checks
-
-Flag these as **P1** if found:
-
- **Data leakage**: train and val/test splits must be constructed before any normalization or augmentation that uses train statistics.
- **Loss function errors**: verify reduction mode (`mean` vs `sum`), correct masking, correct shape alignment.
- **Gradient flow**: new modules must have gradients flowing (check `requires_grad`, no detached tensors in the loss path by accident).
- **Distributed training**: operations on tensors must be DDP-safe; no in-place ops on parameters; batch norm needs `SyncBatchNorm` if used.
- **Memory leaks**: no accumulation of tensors outside the training loop; `optimizer.zero_grad()` called correctly.
-
---
-
-## What to Skip
-
- Don't flag style nitpicks on unchanged surrounding code.
- Don't propose refactors outside the PR's scope.
- Don't add docstrings or comments to code the PR didn't touch.
- Don't suggest speculative future features (YAGNI).
@@ -1,49 +0,0 @@
-name: Claude Code Review
-
-on:
-  pull_request:
-    types: [opened, synchronize, ready_for_review, reopened]
-
-jobs:
-  claude-review:
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      pull-requests: write
-      issues: read
-      id-token: write
-      actions: read
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-        with:
-          fetch-depth: 1
-          persist-credentials: false
-
-      - name: Run Claude Code Review
-        id: claude-review
-        uses: anthropics/claude-code-action@26ddc358fe3befff50c5ec2f80304c90c763f6f8 # v1
-        with:
-          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
-          use_sticky_comment: true
-          prompt: |
-            Read `.github/CLAUDE.md` for lerobot-specific conventions, then review this PR.
-            Provide structured, actionable feedback.
-
-            Focus areas (in priority order):
-            1. **Correctness**: Logic errors, off-by-ones, wrong tensor shapes, incorrect loss functions
-            2. **Type safety**: All new/modified Python code must pass `mypy --strict`; check for missing annotations
-            3. **Backwards compatibility**: Does this break `LeRobotDataset`, `Policy`, `Robot`, `Teleoperator`, `Env`, or `Processor` public APIs?
-            4. **Tests**: New features must have tests; no silent behavioral changes
-            5. **Code style**: Explicit over magic, no unnecessary abstractions, no decorative comments
-            6. **HF integration**: Dataset streaming, `push_to_hub`, HF Hub compatibility preserved?
-            7. **pre-commit**: Would `pre-commit run -a` pass? (ruff, mypy, typos, zizmor)
-
-            Format findings as P1 (must fix) / P2 (should fix) / P3 (nice to have).
-            Skip P3 if the PR is already high quality.
-          claude_args: '--model claude-opus-4-6'
-          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
-          # or https://code.claude.com/docs/en/cli-reference for available options
@@ -1,58 +0,0 @@
-name: Claude Code
-
-on:
-  issue_comment:
-    types: [created]
-  pull_request_review_comment:
-    types: [created]
-  issues:
-    types: [opened, assigned]
-  pull_request_review:
-    types: [submitted]
-
-jobs:
-  claude:
-    if: |
-      (github.event_name == 'issue_comment' &&
-       contains(github.event.comment.body, '@claude') &&
-       (github.event.comment.author_association == 'OWNER' || github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'COLLABORATOR')) ||
-      (github.event_name == 'pull_request_review_comment' &&
-       contains(github.event.comment.body, '@claude') &&
-       (github.event.comment.author_association == 'OWNER' || github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'COLLABORATOR')) ||
-      (github.event_name == 'pull_request_review' &&
-       contains(github.event.review.body, '@claude') &&
-       (github.event.review.author_association == 'OWNER' || github.event.review.author_association == 'MEMBER' || github.event.review.author_association == 'COLLABORATOR')) ||
-      (github.event_name == 'issues' &&
-       (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')) &&
-       (github.event.issue.author_association == 'OWNER' || github.event.issue.author_association == 'MEMBER' || github.event.issue.author_association == 'COLLABORATOR'))
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      pull-requests: write
-      issues: write
-      id-token: write
-      actions: read
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-        with:
-          fetch-depth: 1
-          persist-credentials: false
-
-      - name: Run Claude Code
-        id: claude
-        uses: anthropics/claude-code-action@26ddc358fe3befff50c5ec2f80304c90c763f6f8 # v1
-        with:
-          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
-          use_sticky_comment: true
-
-          # This is an optional setting that allows Claude to read CI results on PRs
-          additional_permissions: |
-            actions: read
-
-          claude_args: '--system-prompt "Read .github/CLAUDE.md for lerobot-specific conventions before responding."'
-          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
-          # or https://code.claude.com/docs/en/cli-reference for available options
@@ -33,7 +33,7 @@ jobs:
      github.event.workflow_run.event == 'pull_request' &&
      github.event.workflow_run.conclusion == 'success' &&
      github.repository == 'huggingface/lerobot'
-    uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@90b4ee2c10b81b5c1a6367c4e6fc9e2fb510a7e3  # main
+    uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@main
    with:
      package_name: lerobot
    secrets:
@@ -55,7 +55,7 @@ jobs:
      github.repository == 'huggingface/lerobot'
    permissions:
      contents: read
-    uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@90b4ee2c10b81b5c1a6367c4e6fc9e2fb510a7e3  # main
+    uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@main
    with:
      commit_sha: ${{ github.sha }}
      package: lerobot
@@ -78,7 +78,7 @@ jobs:
    permissions:
      contents: read
      pull-requests: write
-    uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@90b4ee2c10b81b5c1a6367c4e6fc9e2fb510a7e3  # main
+    uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@main
    with:
      commit_sha: ${{ github.event.pull_request.head.sha }}
      pr_number: ${{ github.event.number }}
@@ -27,7 +27,6 @@ on:
      - "tests/**"
      - ".github/workflows/**"
      - "pyproject.toml"
-      - "uv.lock"
      - "Makefile"
  push:
    branches:
@@ -37,7 +36,6 @@ on:
      - "tests/**"
      - ".github/workflows/**"
      - "pyproject.toml"
-      - "uv.lock"
      - "Makefile"

 permissions:
@@ -65,7 +63,7 @@ jobs:
      HF_LEROBOT_HOME: /mnt/cache/.cache/huggingface/lerobot
      HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+      - uses: actions/checkout@v6
        with:
          persist-credentials: false
          lfs: true
@@ -83,14 +81,14 @@ jobs:
          libusb-1.0-0-dev speech-dispatcher libgeos-dev portaudio19-dev

      - name: Setup uv and Python
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e  # v6
+        uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
        with:
          enable-cache: true
          version: ${{ env.UV_VERSION }}
          python-version: ${{ env.PYTHON_VERSION }}

      - name: Install lerobot with test extras
-        run: uv sync --locked --extra "test"
+        run: uv sync --extra "test"

      - name: Login to Hugging Face
        if: env.HF_USER_TOKEN != ''
@@ -29,7 +29,6 @@ on:
      - "tests/**"
      - ".github/workflows/**"
      - "pyproject.toml"
-      - "uv.lock"
      - "Makefile"

 permissions:
@@ -63,7 +62,7 @@ jobs:
      HF_LEROBOT_HOME: /mnt/cache/.cache/huggingface/lerobot
      HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+      - uses: actions/checkout@v6
        with:
          lfs: true
          persist-credentials: false
@@ -80,14 +79,14 @@ jobs:
          speech-dispatcher libgeos-dev portaudio19-dev

      - name: Setup uv and Python
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e  # v6
+        uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
        with:
          enable-cache: true
          version: ${{ env.UV_VERSION }}
          python-version: ${{ env.PYTHON_VERSION }}

      - name: Install lerobot with all extras
-        run: uv sync --locked --extra all # TODO(Steven): Make flash-attn optional
+        run: uv sync --extra all # TODO(Steven): Make flash-attn optional

      - name: Login to Hugging Face
        if: env.HF_USER_TOKEN != ''
@@ -137,21 +136,21 @@ jobs:
          sudo apt-get update
          sudo apt-get install git-lfs
          git lfs install
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+      - uses: actions/checkout@v6
        with:
          lfs: true
          persist-credentials: false
      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f  # v3
+        uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
        with:
          cache-binary: false
      - name: Login to Docker Hub
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9  # v3
+        uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
        with:
          username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
          password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
      - name: Build and push Docker image
-        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8  # v6
+        uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
        with:
          context: .
          file: ./docker/Dockerfile.internal
@@ -12,8 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-# This workflow handles Docker image publishing & testing.
-name: Docker Publish & Test
+# This workflow handles nightly testing & docker images publishing.
+name: Nightly
 permissions:
  contents: read

@@ -39,8 +39,8 @@ concurrency:

 jobs:
  # This job builds a CPU image for testing & distribution
-  build-docker-cpu:
-    name: Build CPU Docker
+  build-docker-cpu-nightly:
+    name: Build CPU Docker for Nightly
    runs-on:
      group: aws-general-8-plus
    if: github.repository == 'huggingface/lerobot'
@@ -74,8 +74,8 @@ jobs:
          tags: ${{ env.DOCKER_IMAGE_NAME_CPU }}

  # This job builds a GPU image for testing & distribution
-  build-docker-gpu:
-    name: Build GPU Docker
+  build-docker-gpu-nightly:
+    name: Build GPU Docker for Nightly
    runs-on:
      group: aws-general-8-plus
    if: github.repository == 'huggingface/lerobot'
@@ -109,9 +109,9 @@ jobs:
          tags: ${{ env.DOCKER_IMAGE_NAME_GPU }}

  # This job runs the E2E tests + pytest with all extras in the CPU image
-  cpu-tests:
-    name: CPU Tests
-    needs: [build-docker-cpu]
+  nightly-cpu-tests:
+    name: Nightly CPU Tests
+    needs: [build-docker-cpu-nightly]
    runs-on:
      group: aws-g6-4xlarge-plus
    env:
@@ -121,7 +121,7 @@ jobs:
      TRITON_CACHE_DIR: /home/user_lerobot/.cache/triton
      HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
    container:
-      image: ${{ needs.build-docker-cpu.outputs.image_tag }} # zizmor: ignore[unpinned-images]
+      image: ${{ needs.build-docker-cpu-nightly.outputs.image_tag }} # zizmor: ignore[unpinned-images]
      options: --shm-size "16gb"
      credentials:
        username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
@@ -142,9 +142,9 @@ jobs:
        run: make test-end-to-end

  # This job runs the E2E tests + pytest with all extras in the GPU image
-  gpu-tests:
-    name: GPU Tests
-    needs: [build-docker-gpu]
+  nightly-gpu-tests:
+    name: Nightly GPU Tests
+    needs: [build-docker-gpu-nightly]
    runs-on:
      group: aws-g6-4xlarge-plus
    env:
@@ -154,7 +154,7 @@ jobs:
      TRITON_CACHE_DIR: /home/user_lerobot/.cache/triton
      HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
    container:
-      image: ${{ needs.build-docker-gpu.outputs.image_tag }} # zizmor: ignore[unpinned-images]
+      image: ${{ needs.build-docker-gpu-nightly.outputs.image_tag }} # zizmor: ignore[unpinned-images]
      options: --gpus all --shm-size "16gb"
      credentials:
        username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
@@ -175,9 +175,9 @@ jobs:
        run: make test-end-to-end

  # This job runs multi-GPU training tests with 4 GPUs
-  multi-gpu-tests:
-    name: Multi-GPU Tests
-    needs: [build-docker-gpu]
+  nightly-multi-gpu-tests:
+    name: Nightly Multi-GPU Tests
+    needs: [build-docker-gpu-nightly]
    runs-on:
      group: aws-g4dn-12xlarge  # Instance with 4 GPUs
    env:
@@ -188,7 +188,7 @@ jobs:
      CUDA_VISIBLE_DEVICES: "0,1,2,3"
      HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
    container:
-      image: ${{ needs.build-docker-gpu.outputs.image_tag }} # zizmor: ignore[unpinned-images]
+      image: ${{ needs.build-docker-gpu-nightly.outputs.image_tag }} # zizmor: ignore[unpinned-images]
      options: --gpus all --shm-size "16gb"
      credentials:
        username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
@@ -43,16 +43,16 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+        uses: actions/checkout@v6
        with:
          persist-credentials: false

      - name: Set up Python
-        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6
+        uses: actions/setup-python@v6
        with:
          python-version: '3.12'

      - name: Run pre-commit hooks
-        uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd  # v3.0.1
+        uses: pre-commit/action@v3.0.1 # zizmor: ignore[unpinned-uses]
        with:
          extra_args: --all-files --show-diff-on-failure --color=always
@@ -38,12 +38,12 @@ jobs:

    steps:
      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+        uses: actions/checkout@v6
        with:
          persist-credentials: false

      - name: Set up Python
-        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6
+        uses: actions/setup-python@v6
        with:
          python-version: '3.12'

@@ -104,7 +104,7 @@ jobs:
      - name: Publish to TestPyPI for pre-releases
        # True for tags like 'v0.2.0-rc1'
        if: startsWith(github.ref, 'refs/tags/v') && contains(github.ref, '-')
-        uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e  # v1.13.0
+        uses: pypa/gh-action-pypi-publish@v1.13.0 # zizmor: ignore[unpinned-uses, use-trusted-publishing]
        with:
          repository-url: https://test.pypi.org/legacy/
          verbose: true
@@ -112,7 +112,7 @@ jobs:

      - name: Publish to PyPI
        if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref, '-')
-        uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e  # v1.13.0
+        uses: pypa/gh-action-pypi-publish@v1.13.0 # zizmor: ignore[unpinned-uses, use-trusted-publishing]
        with:
          verbose: true
          print-hash: true
@@ -127,7 +127,7 @@ jobs:
    env:
      MUJOCO_GL: egl
    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+      - uses: actions/checkout@v6
        with:
          lfs: true
          persist-credentials: false
@@ -137,7 +137,7 @@ jobs:
          git curl libglib2.0-0 libegl1-mesa-dev ffmpeg libusb-1.0-0-dev \
          speech-dispatcher libgeos-dev portaudio19-dev
      - name: Setup uv and Python
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e  # v6
+        uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
        with:
          enable-cache: true # zizmor: ignore[cache-poisoning]
          version: ${{ env.UV_VERSION }}
@@ -43,12 +43,12 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+        uses: actions/checkout@v6 # zizmor: ignore[unpinned-uses]
        with:
          fetch-depth: 0
          persist-credentials: false

      - name: Secret Scanning
-        uses: trufflesecurity/trufflehog@eafb8c5f6a06175141c27f17bcc17941853d0047  # v3.90.0
+        uses: trufflesecurity/trufflehog@v3.90.0  # zizmor: ignore[unpinned-uses]
        with:
          extra_args: --only-verified
@@ -12,81 +12,38 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-# This workflow tests the project against the latest upstream dependencies
-# (within pyproject.toml constraints) and opens a PR to update uv.lock
-# if the tests pass and the lockfile has changed.
-name: Latest Dependency Tests
+# This workflow handles full testing with unboud dependencies versions.
+name: Unbound Dependency Tests

 on:
  # Allows running this workflow manually from the Actions tab
  workflow_dispatch:

-  # Runs at 03:00 UTC
-  schedule:
-    - cron: "0 3 * * *"
+  # Run on the 1st and 15th of every month at 09:00 UTC
+  # schedule:
+  #  - cron: '0 2 1,15 * *'
+
+permissions:
+  contents: read

 # Sets up the environment variables
 env:
  UV_VERSION: "0.8.0"
  PYTHON_VERSION: "3.12"
-  DOCKER_IMAGE_NAME: huggingface/lerobot-gpu:latest-deps
+  DOCKER_IMAGE_NAME: huggingface/lerobot-gpu:unbound

-# Ensures that only the latest run is active, canceling older runs.
+# Ensures that only the latest action is built, canceling older runs.
 concurrency:
-  group: ${{ github.workflow }}
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
  cancel-in-progress: true

 jobs:

-  # This job upgrades the lockfile and checks if dependencies have changed
-  upgrade-lock:
-    name: Upgrade Lockfile
+  # This job runs the E2E tests + pytest with all unbound extras
+  full-tests:
+    name: Full Unbound Tests
    runs-on: ubuntu-latest
    if: github.repository == 'huggingface/lerobot'
-    permissions:
-      contents: read
-    outputs:
-      changed: ${{ steps.diff.outputs.changed }}
-    steps:
-      - uses: actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Setup uv and Python
-        uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
-        with:
-          version: ${{ env.UV_VERSION }}
-          python-version: ${{ env.PYTHON_VERSION }}
-
-      - name: Upgrade uv.lock
-        run: uv lock --upgrade
-
-      - name: Check for changes
-        id: diff
-        run: |
-          if git diff --quiet uv.lock; then
-            echo "changed=false" >> "$GITHUB_OUTPUT"
-            echo "uv.lock is up to date — no dependency changes."
-          else
-            echo "changed=true" >> "$GITHUB_OUTPUT"
-            echo "uv.lock has changed — running tests."
-          fi
-
-      - name: Upload updated lockfile
-        if: steps.diff.outputs.changed == 'true'
-        uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
-        with:
-          name: uv-lock
-          path: uv.lock
-
-  # This job runs the full test suite with the upgraded dependencies
-  cpu-tests:
-    name: CPU Tests (Latest Deps)
-    needs: [upgrade-lock]
-    if: needs.upgrade-lock.outputs.changed == 'true'
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
    env:
      MUJOCO_GL: egl
      HF_HOME: /mnt/cache/.cache/huggingface
@@ -98,11 +55,6 @@ jobs:
          lfs: true
          persist-credentials: false

-      - name: Download updated lockfile
-        uses: actions/download-artifact@v4 # zizmor: ignore[unpinned-uses]
-        with:
-          name: uv-lock
-
      # NOTE(Steven): Mount to `/mnt` to avoid the limited storage on `/home`. Consider cleaning default SDKs or using self-hosted runners for more space.
      # (As of 2024-06-10, the runner's `/home` has only 6.2 GB free—8% of its 72 GB total.)
      - name: Setup /mnt storage
@@ -121,32 +73,34 @@ jobs:
          version: ${{ env.UV_VERSION }}
          python-version: ${{ env.PYTHON_VERSION }}

-      - name: Install lerobot with all extras
-        run: uv sync --locked --extra all # TODO(Steven): Make flash-attn optional
+      - name: Unbound dependencies
+        run: |
+          sed -i 's/,[[:space:]]*<[0-9\.]*//g' pyproject.toml
+          echo "Dependencies unbound:" && cat pyproject.toml

+      - name: Install lerobot with all extras
+        run: uv sync --extra all # TODO(Steven): Make flash-attn optional
      - name: Login to Hugging Face
        if: env.HF_USER_TOKEN != ''
        run: |
          uv run hf auth login --token "$HF_USER_TOKEN" --add-to-git-credential
          uv run hf auth whoami
-
      - name: Run pytest (all extras)
-        run: uv run pytest tests -vv --maxfail=10
+        run: uv run pytest tests -vv

      - name: Run end-to-end tests
        run: uv run make test-end-to-end

-  # This job builds a GPU-enabled Docker image with the upgraded dependencies
+  # This job builds a GPU enabled image for testing
  build-and-push-docker:
    name: Build and Push Docker
-    needs: [upgrade-lock]
-    if: needs.upgrade-lock.outputs.changed == 'true'
-    permissions:
-      contents: read
    runs-on:
      group: aws-general-8-plus
+    if: github.repository == 'huggingface/lerobot'
    outputs:
      image_tag: ${{ env.DOCKER_IMAGE_NAME }}
+    env:
+      GITHUB_REF: ${{ github.ref }}
    steps:
      - name: Install Git LFS
        run: |
@@ -157,12 +111,6 @@ jobs:
        with:
          lfs: true
          persist-credentials: false
-
-      - name: Download updated lockfile
-        uses: actions/download-artifact@v4 # zizmor: ignore[unpinned-uses]
-        with:
-          name: uv-lock
-
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
        with:
@@ -179,13 +127,14 @@ jobs:
          file: ./docker/Dockerfile.internal
          push: true
          tags: ${{ env.DOCKER_IMAGE_NAME }}
+          build-args: |
+            UNBOUND_DEPS=true

-  # This job runs pytest with all extras on a GPU-enabled host
+  # This job runs pytest with all unbound extras in a GPU enabled host
+  # It runs everytime a test image is created
  gpu-tests:
-    name: GPU Tests (Latest Deps)
+    name: GPU Unbound Tests
    needs: [build-and-push-docker]
-    permissions:
-      contents: read
    runs-on:
      group: aws-g6-4xlarge-plus
    env:
@@ -210,69 +159,17 @@ jobs:
        run: |
          hf auth login --token "$HF_USER_TOKEN" --add-to-git-credential
          hf auth whoami
-      - name: Fix ptxas permissions
-        run: chmod +x /lerobot/.venv/lib/python3.12/site-packages/triton/backends/nvidia/bin/ptxas
      - name: Run pytest on GPU
-        run: pytest tests -vv --maxfail=10
+        run: pytest tests -vv
      - name: Run end-to-end tests
        run: make test-end-to-end

-  # This job creates or updates a PR with the upgraded lockfile
-  open-pr:
-    name: Open PR
-    needs: [cpu-tests, gpu-tests, upgrade-lock]
-    if: success() && needs.upgrade-lock.outputs.changed == 'true'
-    runs-on: ubuntu-latest
-    permissions:
-      contents: write
-      pull-requests: write
-    env:
-      GH_TOKEN: ${{ secrets.UPDATE_LOCK_TOKEN }}
-    steps:
-      - uses: actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Download updated lockfile
-        uses: actions/download-artifact@v4 # zizmor: ignore[unpinned-uses]
-        with:
-          name: uv-lock
-
-      - name: Create or update PR
-        run: |
-          set -euo pipefail
-          BRANCH="auto/update-uv-lock"
-
-          git config user.name "github-actions[bot]"
-          git config user.email "github-actions[bot]@users.noreply.github.com"
-          git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git"
-
-          git checkout -B "$BRANCH"
-          git add uv.lock
-          git commit -m "chore(dependencies): update uv.lock"
-          git push --force origin "$BRANCH"
-
-          # Create PR only if one doesn't already exist for this branch
-          EXISTING_PR=$(gh pr list --head "$BRANCH" --state open --json number --jq '.[0].number')
-          if [ -z "$EXISTING_PR" ]; then
-            gh pr create \
-              --title "chore(dependencies): update uv.lock" \
-              --body "Automated update of \`uv.lock\` after successful latest dependency tests (CPU + GPU).
-
-          This PR upgrades all dependencies to their latest versions within the ranges specified in \`pyproject.toml\`." \
-              --head "$BRANCH" \
-              --base main
-          else
-            echo "PR #$EXISTING_PR already exists, branch has been updated."
-          fi
-
-  # This job deletes the temporary Docker image after tests complete
-  cleanup-docker:
-    name: Cleanup Docker Image
+  # This job deletes the test image recently created
+  # It runs everytime after the gpu-tests have finished
+  delete-unbound-image:
+    name: Delete Unbound Image
    needs: [gpu-tests, build-and-push-docker]
    if: always() && needs.build-and-push-docker.result == 'success'
-    permissions:
-      contents: read
    runs-on: ubuntu-latest
    steps:
      - name: Get Docker Hub Token and Delete Image
@@ -283,7 +180,8 @@ jobs:
          IMAGE_FULL: ${{ needs.build-and-push-docker.outputs.image_tag }}
        run: |
          IMAGE_NAME=$(echo "$IMAGE_FULL" | cut -d':' -f1)
-          IMAGE_TAG=$(echo "$IMAGE_FULL" | cut -d':' -f2-)
+          IMAGE_TAG=$(echo "$IMAGE_FULL" | cut -d':' -f2)
+
          echo "Attempting to delete image: $IMAGE_NAME:$IMAGE_TAG"

          TOKEN=$(curl -s -H "Content-Type: application/json" \
@@ -25,6 +25,7 @@ node_modules/

 # Lock files
 poetry.lock
+uv.lock
 Pipfile.lock

 ### Build & Distribution ###
@@ -4,8 +4,7 @@

 <div align="center">

-[![Tests](https://github.com/huggingface/lerobot/actions/workflows/latest_deps_tests.yml/badge.svg?branch=main)](https://github.com/huggingface/lerobot/actions/workflows/latest_deps_tests.yml?query=branch%3Amain)
-[![Tests](https://github.com/huggingface/lerobot/actions/workflows/docker_publish.yml/badge.svg?branch=main)](https://github.com/huggingface/lerobot/actions/workflows/docker_publish.yml?query=branch%3Amain)
+[![Tests](https://github.com/huggingface/lerobot/actions/workflows/nightly.yml/badge.svg?branch=main)](https://github.com/huggingface/lerobot/actions/workflows/nightly.yml?query=branch%3Amain)
 [![Python versions](https://img.shields.io/pypi/pyversions/lerobot)](https://www.python.org/downloads/)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/huggingface/lerobot/blob/main/LICENSE)
 [![Status](https://img.shields.io/pypi/status/lerobot)](https://pypi.org/project/lerobot/)
@@ -73,10 +73,17 @@ ENV HOME=/home/user_lerobot \
 RUN uv venv --python python${PYTHON_VERSION}

 # Install Python dependencies for caching
-COPY --chown=user_lerobot:user_lerobot setup.py pyproject.toml uv.lock README.md MANIFEST.in ./
+COPY --chown=user_lerobot:user_lerobot setup.py pyproject.toml README.md MANIFEST.in ./
 COPY --chown=user_lerobot:user_lerobot src/ src/

-RUN uv sync --locked --extra all --no-cache
+ARG UNBOUND_DEPS=false
+
+RUN if [ "$UNBOUND_DEPS" = "true" ]; then \
+    sed -i 's/,[[:space:]]*<[0-9\.]*//g' pyproject.toml; \
+    echo "Dependencies unbound:" && cat pyproject.toml; \
+    fi
+
+RUN uv pip install --no-cache ".[all]"

 RUN chmod +x /lerobot/.venv/lib/python${PYTHON_VERSION}/site-packages/triton/backends/nvidia/bin/ptxas

@@ -61,10 +61,17 @@ ENV HOME=/home/user_lerobot \
 RUN uv venv

 # Install Python dependencies for caching
-COPY --chown=user_lerobot:user_lerobot setup.py pyproject.toml uv.lock README.md MANIFEST.in ./
+COPY --chown=user_lerobot:user_lerobot setup.py pyproject.toml README.md MANIFEST.in ./
 COPY --chown=user_lerobot:user_lerobot src/ src/

-RUN uv sync --locked --extra all --no-cache
+ARG UNBOUND_DEPS=false
+
+RUN if [ "$UNBOUND_DEPS" = "true" ]; then \
+    sed -i 's/,[[:space:]]*<[0-9\.]*//g' pyproject.toml; \
+    echo "Dependencies unbound:" && cat pyproject.toml; \
+    fi
+
+RUN uv pip install --no-cache ".[all]"

 # Copy the rest of the application code
 # Make sure to have the git-LFS files for testing
@@ -1,77 +0,0 @@
-# Docker
-
-This directory contains Dockerfiles for running LeRobot in containerized environments. Both images are **built nightly from `main`** and published to Docker Hub with the full environment pre-baked — no dependency setup required.
-
-## Pre-built Images
-
-```bash
-# CPU-only image (based on Dockerfile.user)
-docker pull huggingface/lerobot-cpu:latest
-
-# GPU image with CUDA support (based on Dockerfile.internal)
-docker pull huggingface/lerobot-gpu:latest
-```
-
-## Quick Start
-
-The fastest way to start training is to pull the GPU image and run `lerobot-train` directly. This is the same environment used for all of our CI, so it is a well-tested, batteries-included setup.
-
-```bash
-docker run -it --rm --gpus all --shm-size 16gb huggingface/lerobot-gpu:latest
-
-# inside the container:
-lerobot-train --policy.type=act --dataset.repo_id=lerobot/aloha_sim_transfer_cube_human
-```
-
-## Dockerfiles
-
-### `Dockerfile.user` (CPU)
-
-A lightweight image based on `python:3.12-slim`. Includes all Python dependencies and system libraries but does not include CUDA — there is no GPU support. Useful for exploring the codebase, running scripts, or working with robots, but not practical for training.
-
-### `Dockerfile.internal` (GPU)
-
-A CUDA-enabled image based on `nvidia/cuda`. This is the image for training — mostly used for internal interactions with the GPU cluster.
-
-## Usage
-
-### Running a pre-built image
-
-```bash
-# CPU
-docker run -it --rm huggingface/lerobot-cpu:latest
-
-# GPU
-docker run -it --rm --gpus all --shm-size 16gb huggingface/lerobot-gpu:latest
-```
-
-### Building locally
-
-From the repo root:
-
-```bash
-# CPU
-docker build -f docker/Dockerfile.user -t lerobot-user .
-docker run -it --rm lerobot-user
-
-# GPU
-docker build -f docker/Dockerfile.internal -t lerobot-internal .
-docker run -it --rm --gpus all --shm-size 16gb lerobot-internal
-```
-
-### Multi-GPU training
-
-To select specific GPUs, set `CUDA_VISIBLE_DEVICES` when launching the container:
-
-```bash
-# Use 4 GPUs
-docker run -it --rm --gpus all --shm-size 16gb \
-  -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
-  huggingface/lerobot-gpu:latest
-```
-
-### USB device access (e.g. robots, cameras)
-
-```bash
-docker run -it --device=/dev/ -v /dev/:/dev/ --rm huggingface/lerobot-cpu:latest
-```
@@ -26,7 +26,7 @@ During evaluation, data moves through four stages:
 1. gym.Env  ──→  raw observations (numpy dicts)

 2. Preprocessing  ──→  standard LeRobot keys + task description
-   (preprocess_observation, add_envs_task in envs/utils.py)
+   (preprocess_observation in envs/utils.py, env.call("task_description"))

 3. Processors  ──→  env-specific then policy-specific transforms
   (env_preprocessor, policy_preprocessor)
@@ -161,6 +161,8 @@ class MyBenchmarkEnv(gym.Env):
        ...
 ```

+**GPU-based simulators (e.g. MuJoCo with EGL rendering):** If your simulator allocates GPU/EGL contexts during `__init__`, defer that allocation to a `_ensure_env()` helper called on first `reset()`/`step()`. This avoids inheriting stale GPU handles when `AsyncVectorEnv` spawns worker processes. See `LiberoEnv._ensure_env()` for the pattern.
+
 Also provide a factory function that returns the nested dict structure:

 ```python
@@ -207,7 +209,7 @@ class MyBenchmarkEnvConfig(EnvConfig):
    def gym_kwargs(self) -> dict:
        return {"obs_type": self.obs_type, "render_mode": self.render_mode}

-    def create_envs(self, n_envs: int, use_async_envs: bool = False):
+    def create_envs(self, n_envs: int, use_async_envs: bool = True):
        """Override for multi-task benchmarks or custom env creation."""
        from lerobot.envs.<benchmark> import create_<benchmark>_envs
        return create_<benchmark>_envs(task=self.task, n_envs=n_envs, ...)
@@ -90,17 +90,11 @@ The same policy can work with different environment processors, and the same env

 ```python
 # Use SmolVLA policy with LIBERO environment
-# Use SmolVLA policy with LIBERO environment
-libero_preprocessor, libero_postprocessor = make_env_pre_post_processors(
-    env_cfg=libero_cfg,
-    policy_cfg=smolvla_cfg,
-)
+libero_preprocessor, libero_postprocessor = make_env_pre_post_processors(libero_cfg)
 smolvla_preprocessor, smolvla_postprocessor = make_pre_post_processors(smolvla_cfg)
+
 # Or use ACT policy with the same LIBERO environment
-libero_preprocessor, libero_postprocessor = make_env_pre_post_processors(
-    env_cfg=libero_cfg,
-    policy_cfg=act_cfg,
-)
+libero_preprocessor, libero_postprocessor = make_env_pre_post_processors(libero_cfg)
 act_preprocessor, act_postprocessor = make_pre_post_processors(act_cfg)
 ```

@@ -209,10 +203,7 @@ def eval_main(cfg: EvalPipelineConfig):
    )

    # Create environment processors (NEW!)
-    env_preprocessor, env_postprocessor = make_env_pre_post_processors(
-        env_cfg=cfg.env,
-        policy_cfg=cfg.policy,
-    )
+    env_preprocessor, env_postprocessor = make_env_pre_post_processors(env_cfg=cfg.env)

    # Run evaluation with both processor types
    eval_policy_all(
@@ -316,22 +307,21 @@ class MyEnvProcessorStep(ObservationProcessorStep):
        return processed
 ```

-### 2. Update Your `EnvConfig` Subclass
+### 2. Update the Factory

 ```python
-# In src/lerobot/envs/configs.py
-@EnvConfig.register_subclass("myenv")
-@dataclass
-class MyEnvConfig(EnvConfig):
-    # ... task/features/gym kwargs ...
+# In src/lerobot/envs/factory.py

-    def get_env_processors(self):
-        from lerobot.processor.pipeline import PolicyProcessorPipeline
+def make_env_pre_post_processors(env_cfg: EnvConfig):
+    if isinstance(env_cfg, LiberoEnv) or "libero" in env_cfg.type:
+        preprocessor = PolicyProcessorPipeline(steps=[LiberoProcessorStep()])
+    elif isinstance(env_cfg, MyEnvConfig) or "myenv" in env_cfg.type:
+        preprocessor = PolicyProcessorPipeline(steps=[MyEnvProcessorStep()])
+    else:
+        preprocessor = PolicyProcessorPipeline(steps=[])

-        return (
-            PolicyProcessorPipeline(steps=[MyEnvProcessorStep()]),
-            PolicyProcessorPipeline(steps=[]),
-        )
+    postprocessor = PolicyProcessorPipeline(steps=[])
+    return preprocessor, postprocessor
 ```

 ### 3. Use in Evaluation
@@ -131,4 +131,4 @@ lerobot-record \

 ## License

-This model follows NVIDIA's proprietary license, consistent with the original [GR00T repository](https://github.com/NVIDIA/Isaac-GR00T). Future versions (starting from N1.7) will follow **Apache 2.0 License**.
+This model follows the **Apache 2.0 License**, consistent with the original [GR00T repository](https://github.com/NVIDIA/Isaac-GR00T).
@@ -1,6 +1,6 @@
 # Installation

-This guide uses `conda` (via miniforge) to manage environments (recommended). If you prefer another environment manager (e.g. `uv`, `venv`), ensure you have Python >=3.12 and support PyTorch >= 2.10, then skip ahead to [Environment Setup](#step-2-environment-setup).
+This guide uses `conda` (via miniforge) to manage environments (recommended). If you prefer another environment manager (e.g. `uv`, `venv`), ensure you have Python >=3.12 and `ffmpeg` installed with the `libsvtav1` encoder, then skip ahead to [Environment Setup](#step-2-environment-setup).

 ## Step 1 (`conda` only): Install [`miniforge`](https://conda-forge.org/download/)

@@ -20,7 +20,7 @@ Create a virtual environment with Python 3.12:
 conda create -y -n lerobot python=3.12
 ```
 </hfoption>
-<hfoption id="uv (PyTorch >= 2.10 only)">
+<hfoption id="uv">
 ```bash
 uv python install 3.12
 uv venv --python 3.12
@@ -32,87 +32,48 @@ uv venv --python 3.12
 Then activate your virtual environment, you have to do this each time you open a shell to use lerobot:

 <!-- prettier-ignore-start -->
-
 <hfoptions id="activate_venv">
-<hfoption id="conda">
-```bash
+<hfoption id="conda">```bash
 conda activate lerobot
+```</hfoption>
+<hfoption id="uv">
+```bash
+# Linux/macOSsource
+source .venv/bin/activate
+# Windows PowerShell
+source .venv\Scripts\Activate.ps1
+```
+</hfoption>
+</hfoptions>
+<!-- prettier-ignore-end -->
+
+When using `conda`, install `ffmpeg` in your environment:
+
+```bash
+conda install ffmpeg -c conda-forge
+ffmpeg -version  # ffmpeg 8.X is not yet supported !
 ```

+> [!TIP]
+> This usually installs `ffmpeg 7.X` for your platform compiled with the `libsvtav1` encoder. If `libsvtav1` is not supported (check supported encoders with `ffmpeg -encoders`), you can:
+>
+> - _[On any platform]_ Explicitly install `ffmpeg 7.X` using:
+>
+> ```bash
+> conda install ffmpeg=7.1.1 -c conda-forge
+> ```
+>
+> - _[On Linux only]_ If you want to bring your own ffmpeg: Install [ffmpeg build dependencies](https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu#GettheDependencies) and [compile ffmpeg from source with libsvtav1](https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu#libsvtav1), and make sure you use the corresponding ffmpeg binary to your install with `which ffmpeg`.
+
 > [!NOTE]
-> When installing LeRobot inside WSL (Windows Subsystem for Linux), make sure to also install `evdev`:
+> When installing LeRobot inside WSL (Windows Subsystem for Linux), make sure to install `evdev` with the following command:
 >
 > ```bash
 > conda install evdev -c conda-forge
 > ```

-</hfoption>
-<hfoption id="uv (PyTorch >= 2.10 only)">
-```bash
-# Linux/macOS
-source .venv/bin/activate
-# Windows PowerShell
-.venv\Scripts\activate
-```
-
-> [!NOTE]
-> When installing LeRobot inside WSL (Windows Subsystem for Linux), make sure to also install `evdev`:
->
-> ```bash
-> sudo apt install libevdev-dev
-> uv pip install evdev
-> ```
-
-</hfoption>
-</hfoptions>
-<!-- prettier-ignore-end -->
-
-### Install `ffmpeg` (for video decoding)
-
-LeRobot uses [TorchCodec](https://github.com/meta-pytorch/torchcodec) for video decoding by default, which requires `ffmpeg`.
-
-> [!NOTE]
-> **Platform support:** TorchCodec is **not available** on macOS Intel (x86_64), Linux ARM (aarch64, arm64, armv7l), or Windows with PyTorch < 2.8. On these platforms, LeRobot automatically falls back to `pyav` — so you do not need to install `ffmpeg` and can skip to Step 3.
-
-If your platform supports TorchCodec, install `ffmpeg` using one of the methods below:
-
-<!-- prettier-ignore-start -->
-
-<hfoptions id="install_ffmpeg">
-<hfoption id="conda (any PyTorch version)">
-
-Install `ffmpeg` in your conda environment. This works with **all PyTorch versions** and is **required for PyTorch < 2.10**:
-
-```bash
-conda install ffmpeg -c conda-forge
-```
-
-> [!TIP]
-> This usually installs `ffmpeg 8.X` with the `libsvtav1` encoder. If you run into issues (e.g. `libsvtav1` missing — check with `ffmpeg -encoders` — or a version mismatch with `torchcodec`), you can explicitly install `ffmpeg 7.1.1` using:
->
-> ```bash
-> conda install ffmpeg=7.1.1 -c conda-forge
-> ```
-
-</hfoption>
-<hfoption id="uv (PyTorch >= 2.10 only)">
-
-Starting with **PyTorch >= 2.10** (TorchCodec ≥ 0.10), TorchCodec can dynamically link to a system-wide `ffmpeg` installation. This is useful when using `uv` or other non-`conda` environment managers:
-
-```bash
-# Ubuntu/Debian
-sudo apt install ffmpeg
-
-# macOS (Apple Silicon)
-brew install ffmpeg
-```
-
 > [!IMPORTANT]
-> System-wide `ffmpeg` is **only supported with PyTorch >= 2.10** (TorchCodec ≥ 0.10). For older PyTorch versions, you **must** use `conda install ffmpeg -c conda-forge` instead.
-
-</hfoption>
-</hfoptions>
-<!-- prettier-ignore-end -->
+> If you are using `uv` you will have to install `ffmpeg` system-wide (outside of the virtual environment). You rely on `uv` and `torchcodec` ability to dynamically link to the system `ffmpeg`.

 ## Step 3: Install LeRobot 🤗

@@ -331,54 +331,6 @@ lerobot-train \
  --wandb.project=multitask_dit
 ```

-## Libero Results
-
-```
-python -m lerobot.scripts.lerobot_train \
-  --dataset.repo_id=HuggingFaceVLA/libero \
-  --policy.type=multi_task_dit \
-  --policy.push_to_hub=false \
-  --output_dir="./outputs/multitask_dit_libero" \
-  --job_name="multitask-dit-libero" \
-  --wandb.enable=true \
-  --wandb.project=multitask_dit_libero \
-  --dataset.image_transforms.enable=true \
-  --dataset.image_transforms.max_num_transforms=4 \
-  --dataset.image_transforms.tfs='{"brightness":{"type":"ColorJitter","kwargs":{"brightness":[0.75,1.25]}},"contrast":{"type":"ColorJitter","kwargs":{"contrast":[0.6,1.4]}},"saturation":{"type":"ColorJitter","kwargs":{"saturation":[0.8,1.2]}},"hue":{"type":"ColorJitter","kwargs":{"hue":[-0.05,0.05]}},"sharpness":{"type":"SharpnessJitter","kwargs":{"sharpness":[0.6,1.4]}},"rotation":{"type":"RandomRotation","kwargs":{"degrees":[-5,5]}},"translation":{"type":"RandomAffine","kwargs":{"degrees":0,"translate":[0.1,0.1]}}}' \
-  --dataset.video_backend=torchcodec \
-  --policy.use_amp=true \
-  --policy.horizon=48 \
-  --policy.n_obs_steps=2 \
-  --policy.use_rope=true \
-  --policy.use_positional_encoding=false \
-  --policy.hidden_dim=768 \
-  --policy.num_layers=8 \
-  --policy.num_heads=12 \
-  --policy.dropout=0.1 \
-  --policy.timestep_embed_dim=256 \
-  --policy.objective=diffusion \
-  --policy.optimizer_lr=3e-4 \
-  --policy.optimizer_weight_decay=0 \
-  --policy.scheduler_warmup_steps=0 \
-  --policy.vision_encoder_name=openai/clip-vit-base-patch16 \
-  --policy.image_resize_shape=[256,256] \
-  --policy.image_crop_is_random=true \
-  --policy.text_encoder_name=openai/clip-vit-base-patch16 \
-  --policy.vision_encoder_lr_multiplier=0.1 \
-  --policy.device=cuda \
-  --num_workers=8 \
-  --save_freq=4000 \
-  --log_freq=100 \
-  --steps=100000 \
-  --batch_size=320
-```
-
-Results:
-
-| LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average |
-| -------------- | ------------- | ----------- | --------- | ------- |
-| 87.0           | 98.2          | 93.8        | 83.2      | 90.6    |
-
 ## References

 For more details on the technical implementation and architecture, see:
@@ -1,91 +0,0 @@
-# π₀.₅ (pi05)
-
-This repository contains the Hugging Face port of **π₀.₅**, adapted from [OpenPI](https://github.com/Physical-Intelligence/openpi) by the Physical Intelligence.
-It is designed as a **Vision-Language-Action model with open-world generalization**.
-
---
-
-## Model Overview
-
-| Feature              | π₀                                                     | π₀.₅                                      |
-| -------------------- | ------------------------------------------------------ | ----------------------------------------- |
-| Time Conditioning    | Concatenates time with actions via `action_time_mlp_*` | Uses `time_mlp_*` for AdaRMS conditioning |
-| AdaRMS               | Not used                                               | Used in action expert                     |
-| Tokenizer Length     | 48 tokens                                              | 200 tokens                                |
-| Discrete State Input | False (Uses `state_proj` layer)                        | True                                      |
-| Parameter Count      | Higher (includes state embedding)                      | Lower (no state embedding)                |
-
---
-
-## Relative Actions
-
-π₀.₅ supports training with **relative actions**, where the model learns relative offsets
-from the current robot state instead of absolute joint positions. This mirrors the
-relative-action transform in OpenPI (`DeltaActions`) and can improve performance.
-
-### How it works
-
-1. **During preprocessing**, absolute actions are converted to relative offsets:
-   `relative = action - state` (for selected joints).
-2. The relative actions are normalized using statistics computed from the relative distribution.
-3. **During postprocessing**, predicted relative actions are converted back to absolute:
-   `absolute = relative + state`.
-
-Joints listed in `relative_exclude_joints` (e.g., gripper) are kept absolute.
-
-### Configuration
-
-| Parameter                 | Type        | Default       | Description                                                      |
-| ------------------------- | ----------- | ------------- | ---------------------------------------------------------------- |
-| `use_relative_actions`    | `bool`      | `False`       | Enable relative-action training                                  |
-| `relative_exclude_joints` | `list[str]` | `["gripper"]` | Joint names to keep absolute (matched by substring)              |
-| `action_feature_names`    | `list[str]` | `None`        | Auto-populated from dataset metadata at runtime by `make_policy` |
-
-### Training example
-
-```bash
-python -m lerobot.scripts.lerobot_train \
-  --policy.type=pi05 \
-  --dataset.repo_id=your_org/your_dataset \
-  --policy.use_relative_actions=true \
-  --policy.relative_exclude_joints='["gripper"]'
-```
-
-When `use_relative_actions=true`, the training script automatically:
-
- Computes relative action statistics from the dataset (sampled chunk-level relative actions)
- Replaces the standard action stats with relative stats for normalization
- Broadcasts these stats across all ranks in distributed training
-
---
-
-## Citation
-
-If you use this work, please cite both **OpenPI** and the π₀.₅ paper:
-
-```bibtex
-@misc{openpi2024,
-  author       = {Physical Intelligence Lab},
-  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
-  year         = {2024},
-  publisher    = {GitHub},
-  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
-  license      = {Apache-2.0}
-}
-
-@misc{intelligence2025pi05visionlanguageactionmodelopenworld,
-  title        = {π₀.₅: a Vision-Language-Action Model with Open-World Generalization},
-  author       = {Physical Intelligence and Kevin Black and Noah Brown and James Darpinian and Karan Dhabalia and Danny Driess and Adnan Esmail and Michael Equi and Chelsea Finn and Niccolo Fusai and Manuel Y. Galliker and Dibya Ghosh and Lachy Groom and Karol Hausman and Brian Ichter and Szymon Jakubczak and Tim Jones and Liyiming Ke and Devin LeBlanc and Sergey Levine and Adrian Li-Bell and Mohith Mothukuri and Suraj Nair and Karl Pertsch and Allen Z. Ren and Lucy Xiaoyang Shi and Laura Smith and Jost Tobias Springenberg and Kyle Stachowicz and James Tanner and Quan Vuong and Homer Walke and Anna Walling and Haohuan Wang and Lili Yu and Ury Zhilinsky},
-  year         = {2025},
-  eprint       = {2504.16054},
-  archivePrefix= {arXiv},
-  primaryClass = {cs.LG},
-  url          = {https://arxiv.org/abs/2504.16054},
-}
-```
-
---
-
-## License
-
-This port follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi).
@@ -1,108 +0,0 @@
-# π₀ (pi0)
-
-This repository contains the Hugging Face port of **π₀**, adapted from [OpenPI](https://github.com/Physical-Intelligence/openpi) by the Physical Intelligence.
-It is designed as a **Vision-Language-Action model for general robot control**.
-
---
-
-## Model Overview
-
-| Feature              | π₀                                                     | π₀.₅                                      |
-| -------------------- | ------------------------------------------------------ | ----------------------------------------- |
-| Time Conditioning    | Concatenates time with actions via `action_time_mlp_*` | Uses `time_mlp_*` for AdaRMS conditioning |
-| AdaRMS               | Not used                                               | Used in action expert                     |
-| Tokenizer Length     | 48 tokens                                              | 200 tokens                                |
-| Discrete State Input | False (Uses `state_proj` layer)                        | True                                      |
-| Parameter Count      | Higher (includes state embedding)                      | Lower (no state embedding)                |
-
---
-
-## Relative Actions
-
-π₀ supports training with **relative actions**, where the model learns relative offsets
-from the current robot state instead of absolute joint positions. This mirrors the
-relative-action transform in OpenPI (`DeltaActions`) and can improve performance.
-
-### How it works
-
-1. **During preprocessing**, absolute actions are converted to relative offsets:
-   `relative = action - state` (for selected joints).
-2. The relative actions are normalized using statistics computed from the relative distribution.
-3. **During postprocessing**, predicted relative actions are converted back to absolute:
-   `absolute = relative + state`.
-
-Joints listed in `relative_exclude_joints` (e.g., gripper) are kept absolute.
-
-### Configuration
-
-| Parameter                 | Type        | Default       | Description                                                      |
-| ------------------------- | ----------- | ------------- | ---------------------------------------------------------------- |
-| `use_relative_actions`    | `bool`      | `False`       | Enable relative-action training                                  |
-| `relative_exclude_joints` | `list[str]` | `["gripper"]` | Joint names to keep absolute (matched by substring)              |
-| `action_feature_names`    | `list[str]` | `None`        | Auto-populated from dataset metadata at runtime by `make_policy` |
-
-### Training example
-
-```bash
-python -m lerobot.scripts.lerobot_train \
-  --policy.type=pi0 \
-  --dataset.repo_id=your_org/your_dataset \
-  --policy.use_relative_actions=true \
-  --policy.relative_exclude_joints='["gripper"]'
-```
-
-When `use_relative_actions=true`, the training script automatically:
-
- Computes relative action statistics from the dataset (sampled chunk-level relative actions)
- Replaces the standard action stats with relative stats for normalization
- Broadcasts these stats across all ranks in distributed training
-
-### Recomputing stats for an existing dataset
-
-If you want to precompute relative action stats offline, use `recompute_stats` from
-`lerobot.datasets.dataset_tools`:
-
-```python
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-from lerobot.datasets.dataset_tools import recompute_stats
-
-dataset = LeRobotDataset("your_org/your_dataset")
-dataset = recompute_stats(
-    dataset,
-    relative_action=True,
-    relative_exclude_joints=["gripper"],
-)
-```
-
---
-
-## Citation
-
-If you use this work, please cite both **OpenPI** and the π₀ paper:
-
-```bibtex
-@misc{openpi2024,
-  author       = {Physical Intelligence Lab},
-  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
-  year         = {2024},
-  publisher    = {GitHub},
-  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
-  license      = {Apache-2.0}
-}
-
-@misc{black2024pi0visionlanguageactionflowmodel,
-  title        = {π₀: A Vision-Language-Action Flow Model for General Robot Control},
-  author       = {Kevin Black and Noah Brown and Danny Driess and Adnan Esmail and Michael Equi and Chelsea Finn and Niccolo Fusai and Lachy Groom and Karol Hausman and Brian Ichter and Szymon Jakubczak and Tim Jones and Liyiming Ke and Sergey Levine and Adrian Li-Bell and Mohith Mothukuri and Suraj Nair and Karl Pertsch and Lucy Xiaoyang Shi and James Tanner and Quan Vuong and Anna Walling and Haohuan Wang and Ury Zhilinsky},
-  year         = {2024},
-  eprint       = {2410.24164},
-  archivePrefix= {arXiv},
-  primaryClass = {cs.LG},
-  url          = {https://arxiv.org/abs/2410.24164},
-}
-```
-
---
-
-## License
-
-This port follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi).
@@ -1,38 +0,0 @@
-# Real-Time Chunking (RTC)
-
-This module contains the LeRobot implementation of **Real-Time Chunking (RTC)**, an inference-time technique for flow-matching based policies.
-
-**Note**: RTC is not a policy itself, but rather an inference enhancement that works with flow-matching based policies including [π₀](../pi0/), [π₀.₅](../pi05/), and [SmolVLA](../smolvla/).
-
---
-
-## Citation
-
-If you use Real-Time Chunking in your work, please cite:
-
-```bibtex
-@misc{openpi2024,
-  author       = {Physical Intelligence Lab},
-  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
-  year         = {2024},
-  publisher    = {GitHub},
-  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
-  license      = {Apache-2.0}
-}
-
-@misc{black2025realtimeexecutionactionchunking,
-      title={Real-Time Execution of Action Chunking Flow Policies},
-      author={Kevin Black and Manuel Y. Galliker and Sergey Levine},
-      year={2025},
-      eprint={2506.07339},
-      archivePrefix={arXiv},
-      primaryClass={cs.RO},
-      url={https://arxiv.org/abs/2506.07339},
-}
-```
-
---
-
-## License
-
-This implementation follows the **Apache 2.0 License**, consistent with the LeRobot project.
@@ -1,14 +0,0 @@
-## Paper
-
-https://arxiv.org/abs/2509.25358
-
-## Citation
-
-```bibtex
-@article{chen2025sarm,
-  title={SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation},
-  author={Chen, Qianzhong and Yu, Justin and Schwager, Mac and Abbeel, Pieter and Shentu, Yide and Wu, Philipp},
-  journal={arXiv preprint arXiv:2509.25358},
-  year={2025}
-}
-```
@@ -25,7 +25,7 @@ discord = "https://discord.gg/s3KuuzsPFb"

 [project]
 name = "lerobot"
-version = "0.5.2"
+version = "0.5.1"
 description = "🤗 LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch"
 dynamic = ["readme"]
 license = { text = "Apache-2.0" }
@@ -71,9 +71,9 @@ dependencies = [
    "cmake>=3.29.0.1,<4.2.0",
    "packaging>=24.2,<26.0",

-    "torch>=2.7,<2.11.0",
-    "torchcodec>=0.3.0,<0.11.0; sys_platform != 'win32' and (sys_platform != 'linux' or (platform_machine != 'aarch64' and platform_machine != 'arm64' and platform_machine != 'armv7l')) and (sys_platform != 'darwin' or platform_machine != 'x86_64')", # NOTE: Windows support starts at version 0.7 (needs torch==2.8), ffmpeg>=8 support starts at version 0.8.1 (needs torch==2.9), system-wide ffmpeg support starts at version 0.10 (needs torch==2.10).
-    "torchvision>=0.22.0,<0.26.0",
+    "torch>=2.2.1,<2.11.0",
+    "torchcodec>=0.2.1,<0.11.0; sys_platform != 'win32' and (sys_platform != 'linux' or (platform_machine != 'aarch64' and platform_machine != 'arm64' and platform_machine != 'armv7l')) and (sys_platform != 'darwin' or platform_machine != 'x86_64')",
+    "torchvision>=0.21.0,<0.26.0",

    "einops>=0.8.0,<0.9.0",
    "opencv-python-headless>=4.9.0,<4.14.0",
@@ -220,6 +220,8 @@ lerobot-replay="lerobot.scripts.lerobot_replay:main"
 lerobot-setup-motors="lerobot.scripts.lerobot_setup_motors:main"
 lerobot-teleoperate="lerobot.scripts.lerobot_teleoperate:main"
 lerobot-eval="lerobot.scripts.lerobot_eval:main"
+lerobot-eval-parallel="lerobot.scripts.lerobot_eval_parallel:main"
+lerobot-eval-autotune="lerobot.scripts.lerobot_eval_autotune:main"
 lerobot-train="lerobot.scripts.lerobot_train:main"
 lerobot-train-tokenizer="lerobot.scripts.lerobot_train_tokenizer:main"
 lerobot-dataset-viz="lerobot.scripts.lerobot_dataset_viz:main"
@@ -67,7 +67,13 @@ class EvalConfig:
    # `batch_size` specifies the number of environments to use in a gym.vector.VectorEnv.
    batch_size: int = 50
    # `use_async_envs` specifies whether to use asynchronous environments (multiprocessing).
-    use_async_envs: bool = False
+    # Defaults to True; automatically downgraded to SyncVectorEnv when batch_size=1.
+    use_async_envs: bool = True
+    # Sharding: split n_episodes across independent processes.
+    # shard_id=0, num_shards=1 is the default (no sharding, existing behaviour).
+    # Set via lerobot_eval_parallel or manually: --eval.shard_id=K --eval.num_shards=N
+    shard_id: int = 0
+    num_shards: int = 1

    def __post_init__(self) -> None:
        if self.batch_size > self.n_episodes:
@@ -79,6 +85,12 @@ class EvalConfig:
                f"to increase the number of episodes to match the batch size (e.g. `eval.n_episodes={self.batch_size}`), "
                f"or lower the batch size (e.g. `eval.batch_size={self.n_episodes}`)."
            )
+        if self.num_shards < 1:
+            raise ValueError(f"`num_shards` must be >= 1, got {self.num_shards}")
+        if not (0 <= self.shard_id < self.num_shards):
+            raise ValueError(
+                f"`shard_id` must be in [0, num_shards), got shard_id={self.shard_id}, num_shards={self.num_shards}"
+            )


@dataclass
@@ -151,11 +151,9 @@ class LeRobotDataset(torch.utils.data.Dataset):
                ``$HF_LEROBOT_HOME/hub``.
            episodes (list[int] | None, optional): If specified, this will only load episodes specified by
                their episode_index in this list. Defaults to None.
-            image_transforms (Callable | None, optional):
-                Transform applied to visual modalities inside `__getitem__` after image decoding / tensor
-                conversion. This works for both image-backed and video-backed observations and can later be
-                updated with `set_image_transforms()` or cleared with `clear_image_transforms()`.
-                Defaults to None.
+            image_transforms (Callable | None, optional): You can pass standard v2 image transforms from
+                torchvision.transforms.v2 here which will be applied to visual modalities (whether they come
+                from videos or images). Defaults to None.
            delta_timestamps (dict[list[float]] | None, optional): _description_. Defaults to None.
            tolerance_s (float, optional): Tolerance in seconds used to ensure data timestamps are actually in
                sync with the fps value. It is used at the init of the dataset to make sure that each
@@ -194,8 +192,7 @@ class LeRobotDataset(torch.utils.data.Dataset):
        super().__init__()
        self.repo_id = repo_id
        self._requested_root = Path(root) if root else None
-        self.reader = None
-        self.set_image_transforms(image_transforms)
+        self.image_transforms = image_transforms
        self.delta_timestamps = delta_timestamps
        self.episodes = episodes
        self.tolerance_s = tolerance_s
@@ -478,18 +475,6 @@ class LeRobotDataset(torch.utils.data.Dataset):
            f"}})"
        )

-    def set_image_transforms(self, image_transforms: Callable | None) -> None:
-        """Replace the transform applied to visual observations."""
-        if image_transforms is not None and not callable(image_transforms):
-            raise TypeError("image_transforms must be callable or None.")
-        self.image_transforms = image_transforms
-        if self.reader is not None:
-            self.reader._image_transforms = image_transforms
-
-    def clear_image_transforms(self) -> None:
-        """Remove the transform applied to visual observations."""
-        self.set_image_transforms(None)
-
    # ── Hub methods (stay on facade) ──────────────────────────────────

    def push_to_hub(
@@ -89,24 +89,12 @@ class MultiLeRobotDataset(torch.utils.data.Dataset):
                )
                self.disabled_features.update(extra_keys)

+        self.image_transforms = image_transforms
        self.delta_timestamps = delta_timestamps
        # TODO(rcadene, aliberts): We should not perform this aggregation for datasets
        # with multiple robots of different ranges. Instead we should have one normalization
        # per robot.
        self.stats = aggregate_stats([dataset.meta.stats for dataset in self._datasets])
-        self.set_image_transforms(image_transforms)
-
-    def set_image_transforms(self, image_transforms: Callable | None) -> None:
-        """Replace the transform for this dataset and its children."""
-        if image_transforms is not None and not callable(image_transforms):
-            raise TypeError("image_transforms must be callable or None.")
-        self.image_transforms = image_transforms
-        for dataset in getattr(self, "_datasets", []):
-            dataset.set_image_transforms(self.image_transforms)
-
-    def clear_image_transforms(self) -> None:
-        """Remove the transform from this dataset and its children."""
-        self.set_image_transforms(None)

    @property
    def repo_id_to_index(self):
@@ -44,6 +44,13 @@ from lerobot.utils.constants import (
 )


+def _make_vec_env_cls(use_async: bool, n_envs: int):
+    """Return the right VectorEnv constructor."""
+    if use_async and n_envs > 1:
+        return gym.vector.AsyncVectorEnv
+    return gym.vector.SyncVectorEnv
+
+
@dataclass
 class EnvConfig(draccus.ChoiceRegistry, abc.ABC):
    task: str | None = None
@@ -75,13 +82,14 @@ class EnvConfig(draccus.ChoiceRegistry, abc.ABC):
    def create_envs(
        self,
        n_envs: int,
-        use_async_envs: bool = False,
+        use_async_envs: bool = True,
    ) -> dict[str, dict[int, gym.vector.VectorEnv]]:
        """Create {suite: {task_id: VectorEnv}}.

        Default: single-task env via gym.make(). Multi-task benchmarks override.
+        AsyncVectorEnv is the default for n_envs > 1; auto-downgraded to Sync for n_envs=1.
        """
-        env_cls = gym.vector.AsyncVectorEnv if use_async_envs else gym.vector.SyncVectorEnv
+        env_cls = gym.vector.AsyncVectorEnv if (use_async_envs and n_envs > 1) else gym.vector.SyncVectorEnv

        if self.gym_id not in gym_registry:
            print(f"gym id '{self.gym_id}' not found, attempting to import '{self.package_name}'...")
@@ -399,12 +407,12 @@ class LiberoEnv(EnvConfig):
            kwargs["task_ids"] = self.task_ids
        return kwargs

-    def create_envs(self, n_envs: int, use_async_envs: bool = False):
+    def create_envs(self, n_envs: int, use_async_envs: bool = True):
        from lerobot.envs.libero import create_libero_envs

        if self.task is None:
            raise ValueError("LiberoEnv requires a task to be specified")
-        env_cls = gym.vector.AsyncVectorEnv if use_async_envs else gym.vector.SyncVectorEnv
+        env_cls = _make_vec_env_cls(use_async_envs, n_envs)
        return create_libero_envs(
            task=self.task,
            n_envs=n_envs,
@@ -468,12 +476,12 @@ class MetaworldEnv(EnvConfig):
            "render_mode": self.render_mode,
        }

-    def create_envs(self, n_envs: int, use_async_envs: bool = False):
+    def create_envs(self, n_envs: int, use_async_envs: bool = True):
        from lerobot.envs.metaworld import create_metaworld_envs

        if self.task is None:
            raise ValueError("MetaWorld requires a task to be specified")
-        env_cls = gym.vector.AsyncVectorEnv if use_async_envs else gym.vector.SyncVectorEnv
+        env_cls = _make_vec_env_cls(use_async_envs, n_envs)
        return create_metaworld_envs(
            task=self.task,
            n_envs=n_envs,
@@ -58,7 +58,7 @@ def make_env_pre_post_processors(
 def make_env(
    cfg: EnvConfig | str,
    n_envs: int = 1,
-    use_async_envs: bool = False,
+    use_async_envs: bool = True,
    hub_cache_dir: str | None = None,
    trust_remote_code: bool = False,
 ) -> dict[str, dict[int, gym.vector.VectorEnv]]:
@@ -150,7 +150,17 @@ class LiberoEnv(gym.Env):

        self.init_state_id = self.episode_index  # tie each sub-env to a fixed init state

-        self._env = self._make_envs_task(task_suite, self.task_id)
+        # Extract task metadata without allocating GPU resources (safe before fork).
+        task = task_suite.get_task(task_id)
+        self.task = task.name
+        self.task_description = task.language
+        self._task_bddl_file = os.path.join(
+            get_libero_path("bddl_files"), task.problem_folder, task.bddl_file
+        )
+        self._env: OffScreenRenderEnv | None = (
+            None  # deferred — created on first reset() inside the worker subprocess
+        )
+
        default_steps = 500
        self._max_episode_steps = (
            TASK_SUITE_MAX_STEPS.get(task_suite_name, default_steps)
@@ -221,29 +231,33 @@ class LiberoEnv(gym.Env):
            low=ACTION_LOW, high=ACTION_HIGH, shape=(ACTION_DIM,), dtype=np.float32
        )

+    def _ensure_env(self) -> None:
+        """Create the underlying OffScreenRenderEnv on first use.
+
+        Called inside the worker subprocess after fork(), so each worker gets
+        its own clean EGL context rather than inheriting a stale one from the
+        parent process (which causes EGL_BAD_CONTEXT crashes with AsyncVectorEnv).
+        """
+        if self._env is not None:
+            return
+        env = OffScreenRenderEnv(
+            bddl_file_name=self._task_bddl_file,
+            camera_heights=self.observation_height,
+            camera_widths=self.observation_width,
+        )
+        env.reset()
+        self._env = env
+
    def render(self):
+        self._ensure_env()
        raw_obs = self._env.env._get_observations()
        pixels = self._format_raw_obs(raw_obs)["pixels"]
        image = next(iter(pixels.values()))
        image = image[::-1, ::-1]  # flip both H and W for visualization
        return image

-    def _make_envs_task(self, task_suite: Any, task_id: int = 0):
-        task = task_suite.get_task(task_id)
-        self.task = task.name
-        self.task_description = task.language
-        task_bddl_file = os.path.join(get_libero_path("bddl_files"), task.problem_folder, task.bddl_file)
-
-        env_args = {
-            "bddl_file_name": task_bddl_file,
-            "camera_heights": self.observation_height,
-            "camera_widths": self.observation_width,
-        }
-        env = OffScreenRenderEnv(**env_args)
-        env.reset()
-        return env
-
    def _format_raw_obs(self, raw_obs: RobotObservation) -> RobotObservation:
+        assert self._env is not None, "_format_raw_obs called before _ensure_env()"
        images = {}
        for camera_name in self.camera_name:
            image = raw_obs[camera_name]
@@ -295,6 +309,7 @@ class LiberoEnv(gym.Env):
        )

    def reset(self, seed=None, **kwargs):
+        self._ensure_env()
        super().reset(seed=seed)
        self._env.seed(seed)
        raw_obs = self._env.reset()
@@ -321,6 +336,8 @@ class LiberoEnv(gym.Env):
        return observation, info

    def step(self, action: np.ndarray) -> tuple[RobotObservation, float, bool, bool, dict[str, Any]]:
+        self._ensure_env()
+        assert self._env is not None
        if action.ndim != 1:
            raise ValueError(
                f"Expected action to be 1-D (shape (action_dim,)), "
@@ -345,7 +362,8 @@ class LiberoEnv(gym.Env):
        return observation, reward, terminated, truncated, info

    def close(self):
-        self._env.close()
+        if self._env is not None:
+            self._env.close()


 def _make_env_fns(
@@ -385,6 +403,57 @@ def _make_env_fns(
    return fns


+class _LazyAsyncVectorEnv:
+    """Wrapper that defers AsyncVectorEnv creation until first use.
+
+    Creating all tasks' AsyncVectorEnvs upfront spawns N_tasks × n_envs worker
+    processes, all of which allocate EGL/GPU resources immediately. Since tasks
+    are evaluated sequentially, only one task's workers need to be alive at a
+    time. This wrapper stores the factory functions and creates the real
+    AsyncVectorEnv on first reset(), keeping peak process count = n_envs.
+    """
+
+    def __init__(self, env_fns: list[Callable]):
+        self._env_fns = env_fns
+        self._env: gym.vector.AsyncVectorEnv | None = None
+        self.num_envs = len(env_fns)
+        # Instantiate one env to expose spaces (no GPU — _ensure_env is lazy).
+        tmp = env_fns[0]()
+        self.observation_space = tmp.observation_space
+        self.action_space = tmp.action_space
+        self.single_observation_space = tmp.observation_space
+        self.single_action_space = tmp.action_space
+        tmp.close()
+
+    def _ensure(self):
+        if self._env is None:
+            self._env = gym.vector.AsyncVectorEnv(self._env_fns, context="forkserver")
+
+    def reset(self, **kwargs):
+        self._ensure()
+        return self._env.reset(**kwargs)
+
+    def step(self, actions):
+        self._ensure()
+        return self._env.step(actions)
+
+    def call(self, name, *args, **kwargs):
+        self._ensure()
+        return self._env.call(name, *args, **kwargs)
+
+    def get_attr(self, name):
+        self._ensure()
+        return self._env.get_attr(name)
+
+    def close(self):
+        if self._env is not None:
+            self._env.close()
+            self._env = None
+
+    def __del__(self):
+        self.close()
+
+
 # ---- Main API ----------------------------------------------------------------


@@ -428,6 +497,8 @@ def create_libero_envs(
    if task_ids_filter is not None:
        print(f"Restricting to task_ids={task_ids_filter}")

+    is_async = env_cls is gym.vector.AsyncVectorEnv
+
    out: dict[str, dict[int, Any]] = defaultdict(dict)
    for suite_name in suite_names:
        suite = _get_suite(suite_name)
@@ -449,8 +520,10 @@ def create_libero_envs(
                control_mode=control_mode,
                camera_name_mapping=camera_name_mapping,
            )
-            out[suite_name][tid] = env_cls(fns)
+            if is_async:
+                out[suite_name][tid] = _LazyAsyncVectorEnv(fns)
+            else:
+                out[suite_name][tid] = env_cls(fns)
            print(f"Built vec env | suite={suite_name} | task_id={tid} | n_envs={n_envs}")

-    # return plain dicts for predictability
    return {suite: dict(task_map) for suite, task_map in out.items()}
@@ -97,8 +97,9 @@ class MetaworldEnv(gym.Env):
        self.visualization_height = visualization_height
        self.camera_name = camera_name

-        self._env = self._make_envs_task(self.task)
-        self._max_episode_steps = self._env.max_path_length
+        self._env_name = self.task  # already stripped of "metaworld-" prefix above
+        self._env = None  # deferred — created on first reset() inside the worker subprocess
+        self._max_episode_steps = 500  # MT1 environments always have max_path_length=500
        self.task_description = TASK_DESCRIPTIONS[self.task]

        self.expert_policy = TASK_POLICY_MAPPING[self.task]()
@@ -136,6 +137,24 @@ class MetaworldEnv(gym.Env):

        self.action_space = spaces.Box(low=-1, high=1, shape=(ACTION_DIM,), dtype=np.float32)

+    def _ensure_env(self) -> None:
+        """Create the underlying MetaWorld env on first use.
+
+        Called inside the worker subprocess after fork(), so each worker gets
+        its own clean rendering context rather than inheriting a stale one from
+        the parent process (which causes crashes with AsyncVectorEnv).
+        """
+        if self._env is not None:
+            return
+        mt1 = metaworld.MT1(self._env_name, seed=42)
+        env = mt1.train_classes[self._env_name](render_mode="rgb_array", camera_name=self.camera_name)
+        env.set_task(mt1.train_tasks[0])
+        if self.camera_name == "corner2":
+            env.model.cam_pos[2] = [0.75, 0.075, 0.7]
+        env.reset()
+        env._freeze_rand_vec = False  # otherwise no randomization
+        self._env = env
+
    def render(self) -> np.ndarray:
        """
        Render the current environment frame.
@@ -143,26 +162,13 @@ class MetaworldEnv(gym.Env):
        Returns:
            np.ndarray: The rendered RGB image from the environment.
        """
+        self._ensure_env()
        image = self._env.render()
        if self.camera_name == "corner2":
            # Images from this camera are flipped — correct them
            image = np.flip(image, (0, 1))
        return image

-    def _make_envs_task(self, env_name: str):
-        mt1 = metaworld.MT1(env_name, seed=42)
-        env = mt1.train_classes[env_name](render_mode="rgb_array", camera_name=self.camera_name)
-        env.set_task(mt1.train_tasks[0])
-        if self.camera_name == "corner2":
-            env.model.cam_pos[2] = [
-                0.75,
-                0.075,
-                0.7,
-            ]  # corner2 position, similar to https://arxiv.org/pdf/2206.14244
-        env.reset()
-        env._freeze_rand_vec = False  # otherwise no randomization
-        return env
-
    def _format_raw_obs(self, raw_obs: np.ndarray) -> RobotObservation:
        image = None
        if self._env is not None:
@@ -209,6 +215,7 @@ class MetaworldEnv(gym.Env):
            observation (RobotObservation): The initial formatted observation.
            info (Dict[str, Any]): Additional info about the reset state.
        """
+        self._ensure_env()
        super().reset(seed=seed)

        raw_obs, info = self._env.reset(seed=seed)
@@ -232,6 +239,7 @@ class MetaworldEnv(gym.Env):
            truncated (bool): Whether the episode was truncated due to a time limit.
            info (Dict[str, Any]): Additional environment info.
        """
+        self._ensure_env()
        if action.ndim != 1:
            raise ValueError(
                f"Expected action to be 1-D (shape (action_dim,)), "
@@ -263,7 +271,8 @@ class MetaworldEnv(gym.Env):
        return observation, reward, terminated, truncated, info

    def close(self):
-        self._env.close()
+        if self._env is not None:
+            self._env.close()


 # ---- Main API ----------------------------------------------------------------
@@ -130,56 +130,51 @@ def env_to_policy_features(env_cfg: EnvConfig) -> dict[str, PolicyFeature]:
    return policy_features


-def are_all_envs_same_type(env: gym.vector.VectorEnv) -> bool:
-    first_type = type(env.envs[0])  # Get type of first env
-    return all(type(e) is first_type for e in env.envs)  # Fast type check
+def _get_sub_env_attr(env: gym.vector.VectorEnv, attr: str, index: int = 0):
+    """Retrieve an attribute from a sub-environment, works for both Sync and Async."""
+    try:
+        return env.get_attr(attr)[index]
+    except (AttributeError, Exception):
+        return None
+
+
+def _sub_env_has_attr(env: gym.vector.VectorEnv, attr: str) -> bool:
+    try:
+        env.get_attr(attr)
+        return True
+    except (AttributeError, Exception):
+        return False


 def check_env_attributes_and_types(env: gym.vector.VectorEnv) -> None:
    with warnings.catch_warnings():
-        warnings.simplefilter("once", UserWarning)  # Apply filter only in this function
+        warnings.simplefilter("once", UserWarning)

-        if not (hasattr(env.envs[0], "task_description") and hasattr(env.envs[0], "task")):
+        if not (_sub_env_has_attr(env, "task_description") and _sub_env_has_attr(env, "task")):
            warnings.warn(
                "The environment does not have 'task_description' and 'task'. Some policies require these features.",
                UserWarning,
                stacklevel=2,
            )
-        if not are_all_envs_same_type(env):
-            warnings.warn(
-                "The environments have different types. Make sure you infer the right task from each environment. Empty task will be passed instead.",
-                UserWarning,
-                stacklevel=2,
-            )


 def add_envs_task(env: gym.vector.VectorEnv, observation: RobotObservation) -> RobotObservation:
    """Adds task feature to the observation dict with respect to the first environment attribute."""
-    if hasattr(env.envs[0], "task_description"):
-        task_result = env.call("task_description")
+    if _sub_env_has_attr(env, "task_description"):
+        task_result = list(env.call("task_description"))

-        if isinstance(task_result, tuple):
-            task_result = list(task_result)
-
-        if not isinstance(task_result, list):
-            raise TypeError(f"Expected task_description to return a list, got {type(task_result)}")
        if not all(isinstance(item, str) for item in task_result):
            raise TypeError("All items in task_description result must be strings")

        observation["task"] = task_result
-    elif hasattr(env.envs[0], "task"):
-        task_result = env.call("task")
+    elif _sub_env_has_attr(env, "task"):
+        task_result = list(env.call("task"))

-        if isinstance(task_result, tuple):
-            task_result = list(task_result)
-
-        if not isinstance(task_result, list):
-            raise TypeError(f"Expected task to return a list, got {type(task_result)}")
        if not all(isinstance(item, str) for item in task_result):
            raise TypeError("All items in task result must be strings")

        observation["task"] = task_result
-    else:  #  For envs without language instructions, e.g. aloha transfer cube and etc.
+    else:
        num_envs = observation[list(observation.keys())[0]].shape[0]
        observation["task"] = ["" for _ in range(num_envs)]
    return observation
@@ -1 +0,0 @@
-../../../../docs/source/policy_multi_task_dit_README.md
@@ -0,0 +1,37 @@
+# Multitask DiT Policy
+
+## Citation
+
+If you use this work, please cite the following works:
+
+```bibtex
+@misc{jones2025multitaskditpolicy,
+  author = {Bryson Jones},
+  title = {Dissecting and Open-Sourcing Multitask Diffusion Transformer Policy},
+  year = {2025},
+  url = {https://brysonkjones.substack.com/p/dissecting-and-open-sourcing-multitask-diffusion-transformer-policy},
+  note = {Blog post}
+}
+```
+
+```bibtex
+@misc{trilbmteam2025carefulexaminationlargebehaviormodels,
+  author       = {TRI LBM Team},
+  title        = {A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation},
+  year         = {2025},
+  eprint       = {arXiv:2507.05331},
+  archivePrefix = {arXiv},
+  primaryClass = {cs.RO},
+  url          = {https://arxiv.org/abs/2507.05331}
+}
+```
+
+```bibtex
+@misc{bostondynamics2025largebehaviormodelsatlas,
+  author       = {Boston Dynamics and TRI Research Team},
+  title        = {Large Behavior Models and Atlas Find New Footing},
+  year         = {2025},
+  url          = {https://bostondynamics.com/blog/large-behavior-models-atlas-find-new-footing/},
+  note         = {Blog post}
+}
+```
@@ -1 +0,0 @@
-../../../../docs/source/policy_pi0_README.md
@@ -0,0 +1,108 @@
+# π₀ (pi0)
+
+This repository contains the Hugging Face port of **π₀**, adapted from [OpenPI](https://github.com/Physical-Intelligence/openpi) by the Physical Intelligence.
+It is designed as a **Vision-Language-Action model for general robot control**.
+
+---
+
+## Model Overview
+
+| Feature              | π₀                                                     | π₀.₅                                      |
+| -------------------- | ------------------------------------------------------ | ----------------------------------------- |
+| Time Conditioning    | Concatenates time with actions via `action_time_mlp_*` | Uses `time_mlp_*` for AdaRMS conditioning |
+| AdaRMS               | Not used                                               | Used in action expert                     |
+| Tokenizer Length     | 48 tokens                                              | 200 tokens                                |
+| Discrete State Input | False (Uses `state_proj` layer)                        | True                                      |
+| Parameter Count      | Higher (includes state embedding)                      | Lower (no state embedding)                |
+
+---
+
+## Relative Actions
+
+π₀ supports training with **relative actions**, where the model learns relative offsets
+from the current robot state instead of absolute joint positions. This mirrors the
+relative-action transform in OpenPI (`DeltaActions`) and can improve performance.
+
+### How it works
+
+1. **During preprocessing**, absolute actions are converted to relative offsets:
+   `relative = action - state` (for selected joints).
+2. The relative actions are normalized using statistics computed from the relative distribution.
+3. **During postprocessing**, predicted relative actions are converted back to absolute:
+   `absolute = relative + state`.
+
+Joints listed in `relative_exclude_joints` (e.g., gripper) are kept absolute.
+
+### Configuration
+
+| Parameter                 | Type        | Default       | Description                                                      |
+| ------------------------- | ----------- | ------------- | ---------------------------------------------------------------- |
+| `use_relative_actions`    | `bool`      | `False`       | Enable relative-action training                                  |
+| `relative_exclude_joints` | `list[str]` | `["gripper"]` | Joint names to keep absolute (matched by substring)              |
+| `action_feature_names`    | `list[str]` | `None`        | Auto-populated from dataset metadata at runtime by `make_policy` |
+
+### Training example
+
+```bash
+python -m lerobot.scripts.lerobot_train \
+  --policy.type=pi0 \
+  --dataset.repo_id=your_org/your_dataset \
+  --policy.use_relative_actions=true \
+  --policy.relative_exclude_joints='["gripper"]'
+```
+
+When `use_relative_actions=true`, the training script automatically:
+
+- Computes relative action statistics from the dataset (sampled chunk-level relative actions)
+- Replaces the standard action stats with relative stats for normalization
+- Broadcasts these stats across all ranks in distributed training
+
+### Recomputing stats for an existing dataset
+
+If you want to precompute relative action stats offline, use `recompute_stats` from
+`lerobot.datasets.dataset_tools`:
+
+```python
+from lerobot.datasets.lerobot_dataset import LeRobotDataset
+from lerobot.datasets.dataset_tools import recompute_stats
+
+dataset = LeRobotDataset("your_org/your_dataset")
+dataset = recompute_stats(
+    dataset,
+    relative_action=True,
+    relative_exclude_joints=["gripper"],
+)
+```
+
+---
+
+## Citation
+
+If you use this work, please cite both **OpenPI** and the π₀ paper:
+
+```bibtex
+@misc{openpi2024,
+  author       = {Physical Intelligence Lab},
+  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
+  year         = {2024},
+  publisher    = {GitHub},
+  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
+  license      = {Apache-2.0}
+}
+
+@misc{black2024pi0visionlanguageactionflowmodel,
+  title        = {π₀: A Vision-Language-Action Flow Model for General Robot Control},
+  author       = {Kevin Black and Noah Brown and Danny Driess and Adnan Esmail and Michael Equi and Chelsea Finn and Niccolo Fusai and Lachy Groom and Karol Hausman and Brian Ichter and Szymon Jakubczak and Tim Jones and Liyiming Ke and Sergey Levine and Adrian Li-Bell and Mohith Mothukuri and Suraj Nair and Karl Pertsch and Lucy Xiaoyang Shi and James Tanner and Quan Vuong and Anna Walling and Haohuan Wang and Ury Zhilinsky},
+  year         = {2024},
+  eprint       = {2410.24164},
+  archivePrefix= {arXiv},
+  primaryClass = {cs.LG},
+  url          = {https://arxiv.org/abs/2410.24164},
+}
+```
+
+---
+
+## License
+
+This port follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi).
@@ -1 +0,0 @@
-../../../../docs/source/policy_pi05_README.md
@@ -0,0 +1,91 @@
+# π₀.₅ (pi05)
+
+This repository contains the Hugging Face port of **π₀.₅**, adapted from [OpenPI](https://github.com/Physical-Intelligence/openpi) by the Physical Intelligence.
+It is designed as a **Vision-Language-Action model with open-world generalization**.
+
+---
+
+## Model Overview
+
+| Feature              | π₀                                                     | π₀.₅                                      |
+| -------------------- | ------------------------------------------------------ | ----------------------------------------- |
+| Time Conditioning    | Concatenates time with actions via `action_time_mlp_*` | Uses `time_mlp_*` for AdaRMS conditioning |
+| AdaRMS               | Not used                                               | Used in action expert                     |
+| Tokenizer Length     | 48 tokens                                              | 200 tokens                                |
+| Discrete State Input | False (Uses `state_proj` layer)                        | True                                      |
+| Parameter Count      | Higher (includes state embedding)                      | Lower (no state embedding)                |
+
+---
+
+## Relative Actions
+
+π₀.₅ supports training with **relative actions**, where the model learns relative offsets
+from the current robot state instead of absolute joint positions. This mirrors the
+relative-action transform in OpenPI (`DeltaActions`) and can improve performance.
+
+### How it works
+
+1. **During preprocessing**, absolute actions are converted to relative offsets:
+   `relative = action - state` (for selected joints).
+2. The relative actions are normalized using statistics computed from the relative distribution.
+3. **During postprocessing**, predicted relative actions are converted back to absolute:
+   `absolute = relative + state`.
+
+Joints listed in `relative_exclude_joints` (e.g., gripper) are kept absolute.
+
+### Configuration
+
+| Parameter                 | Type        | Default       | Description                                                      |
+| ------------------------- | ----------- | ------------- | ---------------------------------------------------------------- |
+| `use_relative_actions`    | `bool`      | `False`       | Enable relative-action training                                  |
+| `relative_exclude_joints` | `list[str]` | `["gripper"]` | Joint names to keep absolute (matched by substring)              |
+| `action_feature_names`    | `list[str]` | `None`        | Auto-populated from dataset metadata at runtime by `make_policy` |
+
+### Training example
+
+```bash
+python -m lerobot.scripts.lerobot_train \
+  --policy.type=pi05 \
+  --dataset.repo_id=your_org/your_dataset \
+  --policy.use_relative_actions=true \
+  --policy.relative_exclude_joints='["gripper"]'
+```
+
+When `use_relative_actions=true`, the training script automatically:
+
+- Computes relative action statistics from the dataset (sampled chunk-level relative actions)
+- Replaces the standard action stats with relative stats for normalization
+- Broadcasts these stats across all ranks in distributed training
+
+---
+
+## Citation
+
+If you use this work, please cite both **OpenPI** and the π₀.₅ paper:
+
+```bibtex
+@misc{openpi2024,
+  author       = {Physical Intelligence Lab},
+  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
+  year         = {2024},
+  publisher    = {GitHub},
+  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
+  license      = {Apache-2.0}
+}
+
+@misc{intelligence2025pi05visionlanguageactionmodelopenworld,
+  title        = {π₀.₅: a Vision-Language-Action Model with Open-World Generalization},
+  author       = {Physical Intelligence and Kevin Black and Noah Brown and James Darpinian and Karan Dhabalia and Danny Driess and Adnan Esmail and Michael Equi and Chelsea Finn and Niccolo Fusai and Manuel Y. Galliker and Dibya Ghosh and Lachy Groom and Karol Hausman and Brian Ichter and Szymon Jakubczak and Tim Jones and Liyiming Ke and Devin LeBlanc and Sergey Levine and Adrian Li-Bell and Mohith Mothukuri and Suraj Nair and Karl Pertsch and Allen Z. Ren and Lucy Xiaoyang Shi and Laura Smith and Jost Tobias Springenberg and Kyle Stachowicz and James Tanner and Quan Vuong and Homer Walke and Anna Walling and Haohuan Wang and Lili Yu and Ury Zhilinsky},
+  year         = {2025},
+  eprint       = {2504.16054},
+  archivePrefix= {arXiv},
+  primaryClass = {cs.LG},
+  url          = {https://arxiv.org/abs/2504.16054},
+}
+```
+
+---
+
+## License
+
+This port follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi).
@@ -1 +0,0 @@
-../../../../docs/source/policy_rtc_README.md
@@ -0,0 +1,38 @@
+# Real-Time Chunking (RTC)
+
+This module contains the LeRobot implementation of **Real-Time Chunking (RTC)**, an inference-time technique for flow-matching based policies.
+
+**Note**: RTC is not a policy itself, but rather an inference enhancement that works with flow-matching based policies including [π₀](../pi0/), [π₀.₅](../pi05/), and [SmolVLA](../smolvla/).
+
+---
+
+## Citation
+
+If you use Real-Time Chunking in your work, please cite:
+
+```bibtex
+@misc{openpi2024,
+  author       = {Physical Intelligence Lab},
+  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
+  year         = {2024},
+  publisher    = {GitHub},
+  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
+  license      = {Apache-2.0}
+}
+
+@misc{black2025realtimeexecutionactionchunking,
+      title={Real-Time Execution of Action Chunking Flow Policies},
+      author={Kevin Black and Manuel Y. Galliker and Sergey Levine},
+      year={2025},
+      eprint={2506.07339},
+      archivePrefix={arXiv},
+      primaryClass={cs.RO},
+      url={https://arxiv.org/abs/2506.07339},
+}
+```
+
+---
+
+## License
+
+This implementation follows the **Apache 2.0 License**, consistent with the LeRobot project.
@@ -1 +0,0 @@
-../../../../docs/source/policy_sarm_README.md
@@ -0,0 +1,14 @@
+## Paper
+
+https://arxiv.org/abs/2509.25358
+
+## Citation
+
+```bibtex
+@article{chen2025sarm,
+  title={SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation},
+  author={Chen, Qianzhong and Yu, Justin and Schwager, Mac and Abbeel, Pieter and Shentu, Yide and Wu, Philipp},
+  journal={arXiv preprint arXiv:2509.25358},
+  year={2025}
+}
+```
@@ -136,8 +136,8 @@ class TokenizerProcessorStep(ObservationProcessorStep):
        # Standardize to a list of strings for the tokenizer
        if isinstance(task, str):
            return [task]
-        elif isinstance(task, list) and all(isinstance(t, str) for t in task):
-            return task
+        elif isinstance(task, (list, tuple)) and all(isinstance(t, str) for t in task):
+            return list(task)

        return None

@@ -47,8 +47,10 @@ You can learn about the CLI options for this script in the `EvalPipelineConfig`
 """

 import concurrent.futures as cf
+import copy
 import json
 import logging
+import math
 import threading
 import time
 from collections import defaultdict
@@ -56,7 +58,6 @@ from collections.abc import Callable
 from contextlib import nullcontext
 from copy import deepcopy
 from dataclasses import asdict
-from functools import partial
 from pathlib import Path
 from pprint import pformat
 from typing import Any, TypedDict
@@ -73,7 +74,6 @@ from lerobot.configs import parser
 from lerobot.configs.eval import EvalPipelineConfig
 from lerobot.envs.factory import make_env, make_env_pre_post_processors
 from lerobot.envs.utils import (
-    add_envs_task,
    check_env_attributes_and_types,
    close_envs,
    preprocess_observation,
@@ -93,6 +93,14 @@ from lerobot.utils.utils import (
 )


+def _shard_episodes(n_episodes: int, shard_id: int, num_shards: int) -> list[int]:
+    """Return the episode indices assigned to this shard (round-robin distribution).
+
+    Example: _shard_episodes(10, 1, 4) -> [1, 5, 9]
+    """
+    return list(range(shard_id, n_episodes, num_shards))
+
+
 def rollout(
    env: gym.vector.VectorEnv,
    policy: PreTrainedPolicy,
@@ -166,9 +174,15 @@ def rollout(
        if return_observations:
            all_observations.append(deepcopy(observation))

-        # Infer "task" from attributes of environments.
-        # TODO: works with SyncVectorEnv but not AsyncVectorEnv
-        observation = add_envs_task(env, observation)
+        # Infer "task" from sub-environments (prefer natural language description).
+        # env.call() works with both SyncVectorEnv and AsyncVectorEnv.
+        try:
+            observation["task"] = list(env.call("task_description"))
+        except Exception:
+            try:
+                observation["task"] = list(env.call("task"))
+            except Exception:
+                observation["task"] = [""] * env.num_envs

        # Apply environment-specific preprocessing (e.g., LiberoProcessorStep for LIBERO)
        observation = env_preprocessor(observation)
@@ -193,14 +207,8 @@ def rollout(

        # VectorEnv stores is_success in `info["final_info"][env_index]["is_success"]`. "final_info" isn't
        # available if none of the envs finished.
-        if "final_info" in info:
-            final_info = info["final_info"]
-            if not isinstance(final_info, dict):
-                raise RuntimeError(
-                    "Unsupported `final_info` format: expected dict (Gymnasium >= 1.0). "
-                    "You're likely using an older version of gymnasium (< 1.0). Please upgrade."
-                )
-            successes = final_info["is_success"].tolist()
+        if "final_info" in info and isinstance(info["final_info"], dict):
+            successes = info["final_info"]["is_success"].tolist()
        elif "is_success" in info:
            is_success = info["is_success"]
            successes = (
@@ -554,6 +562,14 @@ def eval_main(cfg: EvalPipelineConfig):
    # Create environment-specific preprocessor and postprocessor (e.g., for LIBERO environments)
    env_preprocessor, env_postprocessor = make_env_pre_post_processors(env_cfg=cfg.env, policy_cfg=cfg.policy)

+    # Sharding: each shard runs a subset of n_episodes with non-overlapping seeds.
+    shard_id = cfg.eval.shard_id
+    num_shards = cfg.eval.num_shards
+    episodes_for_shard = _shard_episodes(cfg.eval.n_episodes, shard_id, num_shards)
+    n_per_shard = len(episodes_for_shard)
+    # Shift the seed so each shard gets a different, non-overlapping seed range.
+    shard_seed = (cfg.seed or 0) + shard_id * math.ceil(cfg.eval.n_episodes / num_shards)
+
    with torch.no_grad(), torch.autocast(device_type=device.type) if cfg.policy.use_amp else nullcontext():
        info = eval_policy_all(
            envs=envs,
@@ -562,10 +578,10 @@ def eval_main(cfg: EvalPipelineConfig):
            env_postprocessor=env_postprocessor,
            preprocessor=preprocessor,
            postprocessor=postprocessor,
-            n_episodes=cfg.eval.n_episodes,
+            n_episodes=n_per_shard,
            max_episodes_rendered=10,
            videos_dir=Path(cfg.output_dir) / "videos",
-            start_seed=cfg.seed,
+            start_seed=shard_seed,
            max_parallel_tasks=cfg.env.max_parallel_tasks,
        )
        print("Overall Aggregated Metrics:")
@@ -578,8 +594,13 @@ def eval_main(cfg: EvalPipelineConfig):
    # Close all vec envs
    close_envs(envs)

-    # Save info
-    with open(Path(cfg.output_dir) / "eval_info.json", "w") as f:
+    # Save info — use shard-specific filename when running in parallel mode.
+    if num_shards > 1:
+        out_path = Path(cfg.output_dir) / f"shard_{shard_id}_of_{num_shards}.json"
+    else:
+        out_path = Path(cfg.output_dir) / "eval_info.json"
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    with open(out_path, "w") as f:
        json.dump(info, f, indent=2)

    logging.info("End of eval")
@@ -739,39 +760,58 @@ def eval_policy_all(
            group_acc[group]["video_paths"].extend(paths)
            overall["video_paths"].extend(paths)

+    def _make_thread_policy(p: PreTrainedPolicy) -> PreTrainedPolicy:
+        """Shallow copy sharing weight tensors, with independent per-thread state.
+
+        copy.copy() gives a new Python object whose _parameters dict is a shared
+        reference (same tensor storage, zero extra VRAM). reset() then rebinds
+        mutable state (action queues etc.) to fresh per-thread objects.
+
+        Note: does NOT work for ACT with temporal_ensemble_coeff — that policy's
+        reset() mutates a shared sub-object. Use max_parallel_tasks=1 for that config.
+        """
+        thread_p = copy.copy(p)
+        thread_p.reset()
+        return thread_p
+
    # Choose runner (sequential vs threaded)
-    task_runner = partial(
-        run_one,
-        policy=policy,
-        env_preprocessor=env_preprocessor,
-        env_postprocessor=env_postprocessor,
-        preprocessor=preprocessor,
-        postprocessor=postprocessor,
-        n_episodes=n_episodes,
-        max_episodes_rendered=max_episodes_rendered,
-        videos_dir=videos_dir,
-        return_episode_data=return_episode_data,
-        start_seed=start_seed,
-    )
+    _runner_kwargs = {
+        "env_preprocessor": env_preprocessor,
+        "env_postprocessor": env_postprocessor,
+        "preprocessor": preprocessor,
+        "postprocessor": postprocessor,
+        "n_episodes": n_episodes,
+        "max_episodes_rendered": max_episodes_rendered,
+        "videos_dir": videos_dir,
+        "return_episode_data": return_episode_data,
+        "start_seed": start_seed,
+    }

    if max_parallel_tasks <= 1:
-        # sequential path (single accumulator path on the main thread)
-        # NOTE: keeping a single-threaded accumulator avoids concurrent list appends or locks
        for task_group, task_id, env in tasks:
-            tg, tid, metrics = task_runner(task_group, task_id, env)
-            _accumulate_to(tg, metrics)
-            per_task_infos.append({"task_group": tg, "task_id": tid, "metrics": metrics})
+            try:
+                tg, tid, metrics = run_one(task_group, task_id, env, policy=policy, **_runner_kwargs)
+                _accumulate_to(tg, metrics)
+                per_task_infos.append({"task_group": tg, "task_id": tid, "metrics": metrics})
+            finally:
+                env.close()
    else:
-        # threaded path: submit all tasks, consume completions on main thread and accumulate there
+        # threaded path: each thread gets a shallow policy copy (shared weights, independent state)
        with cf.ThreadPoolExecutor(max_workers=max_parallel_tasks) as executor:
            fut2meta = {}
            for task_group, task_id, env in tasks:
-                fut = executor.submit(task_runner, task_group, task_id, env)
-                fut2meta[fut] = (task_group, task_id)
+                fut = executor.submit(
+                    run_one, task_group, task_id, env, policy=_make_thread_policy(policy), **_runner_kwargs
+                )
+                fut2meta[fut] = (task_group, task_id, env)
            for fut in cf.as_completed(fut2meta):
-                tg, tid, metrics = fut.result()
-                _accumulate_to(tg, metrics)
-                per_task_infos.append({"task_group": tg, "task_id": tid, "metrics": metrics})
+                tg, tid, env = fut2meta[fut]
+                try:
+                    tg, tid, metrics = fut.result()
+                    _accumulate_to(tg, metrics)
+                    per_task_infos.append({"task_group": tg, "task_id": tid, "metrics": metrics})
+                finally:
+                    env.close()

    # compute aggregated metrics helper (robust to lists/scalars)
    def _agg_from_list(xs):
@@ -0,0 +1,249 @@
+#!/usr/bin/env python
+
+# Copyright 2024 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Probe hardware and recommend optimal lerobot-eval-parallel flags.
+
+Run standalone:
+    lerobot-eval-autotune --policy.path=lerobot/smolvla_libero --env.type=libero
+
+Or called programmatically from lerobot_eval_parallel when --num-shards auto.
+
+Steps:
+    1. Probe GPU VRAM and CPU core count.
+    2. Measure model VRAM footprint (load policy, delta of cuda.memory_allocated).
+    3. Compute max shards limited by VRAM (85% of total).
+    4. Probe env step time (optional, skipped when skip_timing=True).
+    5. Probe inference time (optional, skipped when skip_timing=True).
+    6. Derive num_shards = min(vram_limit, saturation_shards).
+    7. Choose MUJOCO_GL (egl vs osmesa) based on remaining VRAM headroom.
+    8. Compute batch_size = max(4, min(floor(cpu_cores * 0.8 / num_shards), 64)).
+    9. Print paste-ready command.
+"""
+
+import math
+import os
+import sys
+import time
+from dataclasses import dataclass
+
+
+@dataclass
+class AutotuneRecommendation:
+    num_shards: int
+    batch_size: int
+    mujoco_gl: str
+    use_amp: bool
+    # Probed values
+    gpu_name: str
+    vram_gb: float
+    cpu_cores: int
+    model_gb: float
+    env_step_ms: float | None
+    infer_ms: float | None
+
+
+_DEFAULT_ENV_STEP_MS = 22.0  # LIBERO on GPU, typical value
+_DEFAULT_INFER_MS = 5.0  # SmolVLA fp16 on H100
+
+
+def _probe_gpu() -> tuple[str, float]:
+    """Return (gpu_name, vram_gb). Falls back to CPU sentinel on non-CUDA systems."""
+    try:
+        import torch
+
+        if not torch.cuda.is_available():
+            return "CPU (no CUDA)", 0.0
+        props = torch.cuda.get_device_properties(0)
+        return props.name, props.total_memory / (1024**3)
+    except Exception:
+        return "unknown", 0.0
+
+
+def _probe_model_gb(passthrough: list[str]) -> float:
+    """Load the policy (from --policy.path) and measure VRAM delta. Returns GB."""
+    # Extract policy path from passthrough args
+    policy_path = None
+    for tok in passthrough:
+        if tok.startswith("policy.path="):
+            policy_path = tok.split("=", 1)[1]
+            break
+        if tok.startswith("--policy.path="):
+            policy_path = tok.split("=", 1)[1]
+            break
+    if policy_path is None:
+        return 0.0
+
+    try:
+        import torch
+
+        from lerobot.policies.factory import make_policy
+        from lerobot.policies.pretrained import PreTrainedConfig
+
+        if not torch.cuda.is_available():
+            return 0.0
+        torch.cuda.synchronize()
+        before = torch.cuda.memory_allocated(0)
+        cfg = PreTrainedConfig.from_pretrained(policy_path)
+        cfg.pretrained_path = policy_path  # type: ignore[assignment]
+        policy = make_policy(cfg=cfg)
+        policy.eval()
+        torch.cuda.synchronize()
+        after = torch.cuda.memory_allocated(0)
+        del policy
+        torch.cuda.empty_cache()
+        return (after - before) / (1024**3)
+    except Exception as e:
+        print(f"[autotune] could not measure model VRAM: {e}", file=sys.stderr)
+        return 0.0
+
+
+def _probe_env_step_ms(passthrough: list[str], batch_size: int = 8, n_steps: int = 30) -> float | None:
+    """Run a short env warmup and return median step latency in ms. Returns None on failure."""
+    try:
+        import numpy as np
+
+        from lerobot.envs.factory import make_env
+
+        # Parse env config from passthrough using lerobot's own parser
+        env_type = None
+        for tok in passthrough:
+            if tok.startswith("env.type=") or tok.startswith("--env.type="):
+                env_type = tok.split("=", 1)[1]
+                break
+        if env_type is None:
+            return None
+
+        # Minimal env config
+        from lerobot.envs.factory import make_env_config
+
+        env_cfg = make_env_config(env_type)
+        envs = make_env(env_cfg, n_envs=batch_size, use_async_envs=(batch_size > 1))
+        # Get first vec env
+        first_suite = next(iter(envs.values()))
+        env = next(iter(first_suite.values()))
+
+        env.reset()
+        dummy_action = np.zeros((batch_size, env.single_action_space.shape[0]))
+        timings = []
+        for _ in range(n_steps):
+            t0 = time.perf_counter()
+            env.step(dummy_action)
+            timings.append((time.perf_counter() - t0) * 1000)
+        env.close()
+        return float(np.median(timings))
+    except Exception as e:
+        print(f"[autotune] env step probe failed: {e}", file=sys.stderr)
+        return None
+
+
+def probe_and_recommend(
+    passthrough: list[str],
+    skip_timing: bool = False,
+) -> AutotuneRecommendation:
+    """Probe hardware + model and return the recommended configuration."""
+    gpu_name, vram_gb = _probe_gpu()
+    cpu_cores = os.cpu_count() or 4
+
+    # Model footprint
+    model_gb = _probe_model_gb(passthrough)
+    if model_gb == 0.0:
+        # Unknown model: assume a conservative 14 GB (SmolVLA fp16) as placeholder
+        model_gb = 14.0
+        print("[autotune] model size unknown, assuming 14 GB (SmolVLA fp16)", file=sys.stderr)
+
+    # Max shards from VRAM (leave 15% headroom for activations + env frames)
+    max_shards_vram = max(1, math.floor(vram_gb * 0.85 / model_gb)) if vram_gb > 0 else 1
+
+    # Timing probes
+    env_step_ms: float | None = None
+    infer_ms: float | None = None
+    if not skip_timing:
+        env_step_ms = _probe_env_step_ms(passthrough)
+        # Inference time: assume ~infer = env_step / saturation_factor heuristic
+        # Full probe would require loading policy — skip for now to stay fast.
+        infer_ms = _DEFAULT_INFER_MS
+
+    # Number of shards to saturate GPU: ceil(env_step / infer)
+    _step = env_step_ms or _DEFAULT_ENV_STEP_MS
+    _infer = infer_ms or _DEFAULT_INFER_MS
+    saturation_shards = max(1, math.ceil(_step / _infer))
+
+    num_shards = min(max_shards_vram, saturation_shards)
+
+    # Rendering mode: EGL if all model copies + env frame buffers fit in VRAM
+    env_vram_per_shard_gb = 0.01  # ~10 MB overhead per env batch
+    total_with_egl = num_shards * (model_gb + env_vram_per_shard_gb)
+    mujoco_gl = "egl" if (vram_gb == 0 or total_with_egl < vram_gb * 0.85) else "osmesa"
+
+    # Batch size: fill CPU cores evenly across shards
+    batch_size = max(4, min(math.floor(cpu_cores * 0.8 / num_shards), 64))
+
+    # Recommend AMP when model is large (saves ~50% VRAM)
+    use_amp = model_gb > 8.0
+
+    return AutotuneRecommendation(
+        num_shards=num_shards,
+        batch_size=batch_size,
+        mujoco_gl=mujoco_gl,
+        use_amp=use_amp,
+        gpu_name=gpu_name,
+        vram_gb=vram_gb,
+        cpu_cores=cpu_cores,
+        model_gb=model_gb,
+        env_step_ms=env_step_ms,
+        infer_ms=infer_ms,
+    )
+
+
+def main(argv: list[str] | None = None) -> None:
+    passthrough = argv if argv is not None else sys.argv[1:]
+
+    rec = probe_and_recommend(passthrough)
+
+    env_step_str = (
+        f"{rec.env_step_ms:.0f}ms" if rec.env_step_ms else f"~{_DEFAULT_ENV_STEP_MS:.0f}ms (estimated)"
+    )
+    infer_str = f"{rec.infer_ms:.0f}ms" if rec.infer_ms else f"~{_DEFAULT_INFER_MS:.0f}ms (estimated)"
+
+    print()
+    print(
+        f"GPU: {rec.gpu_name}  |  VRAM: {rec.vram_gb:.1f} GB  |  CPU cores: {rec.cpu_cores}  |  Model: {rec.model_gb:.1f} GB"
+    )
+    print()
+    print(f"  env_step_ms: {env_step_str}  |  infer_ms: {infer_str}")
+    print()
+    print(f"  num_shards:  {rec.num_shards}")
+    print(f"  batch_size:  {rec.batch_size}")
+    print(f"  MUJOCO_GL:   {rec.mujoco_gl}")
+    if rec.use_amp:
+        print("  use_amp:     true  (recommended — halves VRAM, faster matmuls)")
+    print()
+
+    # Build paste-ready command
+    flags = [f"--num-shards {rec.num_shards}", f"eval.batch_size={rec.batch_size}"]
+    if rec.use_amp:
+        flags.append("policy.use_amp=true")
+    flags_str = " \\\n    ".join(flags)
+    passthrough_str = " \\\n    ".join(passthrough) if passthrough else "[your flags]"
+
+    print("  Paste-ready command:")
+    print(f"  MUJOCO_GL={rec.mujoco_gl} lerobot-eval-parallel \\")
+    print(f"    {flags_str} \\")
+    print(f"    {passthrough_str}")
+    print()
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,185 @@
+#!/usr/bin/env python
+
+# Copyright 2024 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run lerobot-eval across N independent subprocesses (shards) for maximum GPU utilization.
+
+Each shard handles a disjoint subset of episodes and writes its own JSON results file.
+Results are merged and printed when all shards complete.
+
+Usage:
+    lerobot-eval-parallel --num-shards 4 [any lerobot-eval flags]
+    lerobot-eval-parallel --num-shards auto [any lerobot-eval flags]
+    lerobot-eval-parallel --num-shards auto --render-device cpu [any lerobot-eval flags]
+
+--num-shards auto:
+    Calls lerobot-eval-autotune to probe hardware and determine the optimal number of shards.
+
+--render-device gpu|cpu|auto:
+    Controls MUJOCO_GL env var. 'gpu' -> EGL (faster, ~3ms/frame, ~200KB VRAM/env).
+    'cpu' -> osmesa (slower, ~12ms/frame, 0 VRAM). 'auto' picks based on VRAM headroom.
+    Default: auto.
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+
+def _parse_known(argv: list[str]) -> tuple[argparse.Namespace, list[str]]:
+    p = argparse.ArgumentParser(add_help=False)
+    p.add_argument("--num-shards", default="1")
+    p.add_argument("--render-device", choices=["gpu", "cpu", "auto"], default="auto")
+    p.add_argument("--output-dir", default=None)
+    return p.parse_known_args(argv)
+
+
+def _resolve_num_shards(num_shards_str: str, passthrough: list[str]) -> int:
+    if num_shards_str == "auto":
+        from lerobot.scripts.lerobot_eval_autotune import probe_and_recommend
+
+        rec = probe_and_recommend(passthrough)
+        print(
+            f"[autotune] recommended num_shards={rec.num_shards}, batch_size={rec.batch_size}, MUJOCO_GL={rec.mujoco_gl}"
+        )
+        return rec.num_shards
+    return int(num_shards_str)
+
+
+def _resolve_mujoco_gl(render_device: str, num_shards: int, passthrough: list[str]) -> str:
+    if render_device == "gpu":
+        return "egl"
+    if render_device == "cpu":
+        return "osmesa"
+    # auto: use EGL for single shard; for multiple shards check VRAM headroom
+    if num_shards == 1:
+        return "egl"
+    try:
+        from lerobot.scripts.lerobot_eval_autotune import probe_and_recommend
+
+        rec = probe_and_recommend(passthrough, skip_timing=True)
+        return rec.mujoco_gl
+    except Exception:
+        # Conservative fallback: osmesa avoids EGL VRAM contention
+        return "osmesa"
+
+
+def _extract_output_dir(passthrough: list[str]) -> str | None:
+    for tok in passthrough:
+        if tok.startswith("--output-dir="):
+            return tok.split("=", 1)[1]
+        if tok == "--output-dir":
+            idx = passthrough.index(tok)
+            if idx + 1 < len(passthrough):
+                return passthrough[idx + 1]
+    return None
+
+
+def _merge_shards(output_dir: str, num_shards: int) -> dict:
+    """Merge per-shard JSON files into a single result dict and write eval_info.json."""
+    all_per_task: list[dict] = []
+    per_group: dict[str, dict] = {}
+
+    for k in range(num_shards):
+        shard_path = Path(output_dir) / f"shard_{k}_of_{num_shards}.json"
+        if not shard_path.exists():
+            print(f"[warning] shard file not found: {shard_path}", file=sys.stderr)
+            continue
+        with open(shard_path) as f:
+            shard = json.load(f)
+        all_per_task.extend(shard.get("per_task", []))
+        for group, metrics in shard.get("per_group", {}).items():
+            if group not in per_group:
+                per_group[group] = {"sum_rewards": [], "max_rewards": [], "successes": []}
+            for key in ("sum_rewards", "max_rewards", "successes"):
+                # metrics may store aggregates; reconstruct lists if possible
+                per_group[group][key].extend(metrics.get(key, []))
+
+    # Re-aggregate
+    import numpy as np
+
+    def _nanmean(xs: list) -> float:
+        return float(np.nanmean(xs)) if xs else float("nan")
+
+    groups_out = {}
+    all_sr, all_mr, all_succ = [], [], []
+    for group, acc in per_group.items():
+        groups_out[group] = {
+            "avg_sum_reward": _nanmean(acc["sum_rewards"]),
+            "avg_max_reward": _nanmean(acc["max_rewards"]),
+            "pc_success": _nanmean(acc["successes"]) * 100 if acc["successes"] else float("nan"),
+            "n_episodes": len(acc["sum_rewards"]),
+        }
+        all_sr.extend(acc["sum_rewards"])
+        all_mr.extend(acc["max_rewards"])
+        all_succ.extend(acc["successes"])
+
+    overall = {
+        "avg_sum_reward": _nanmean(all_sr),
+        "avg_max_reward": _nanmean(all_mr),
+        "pc_success": _nanmean(all_succ) * 100 if all_succ else float("nan"),
+        "n_episodes": len(all_sr),
+    }
+
+    merged = {"per_task": all_per_task, "per_group": groups_out, "overall": overall}
+    out_path = Path(output_dir) / "eval_info.json"
+    with open(out_path, "w") as f:
+        json.dump(merged, f, indent=2)
+    return merged
+
+
+def main(argv: list[str] | None = None) -> None:
+    args, passthrough = _parse_known(argv if argv is not None else sys.argv[1:])
+
+    num_shards = _resolve_num_shards(args.num_shards, passthrough)
+    mujoco_gl = _resolve_mujoco_gl(args.render_device, num_shards, passthrough)
+
+    output_dir = args.output_dir or _extract_output_dir(passthrough)
+
+    print(f"[lerobot-eval-parallel] launching {num_shards} shard(s), MUJOCO_GL={mujoco_gl}")
+
+    child_env = {**os.environ, "MUJOCO_GL": mujoco_gl, "OMP_NUM_THREADS": "1"}
+
+    procs = []
+    for k in range(num_shards):
+        cmd = [
+            sys.executable,
+            "-m",
+            "lerobot.scripts.lerobot_eval",
+            f"eval.shard_id={k}",
+            f"eval.num_shards={num_shards}",
+            *passthrough,
+        ]
+        if output_dir:
+            # Each shard shares the same output_dir; shard files are named shard_K_of_N.json
+            cmd.append(f"output_dir={output_dir}")
+        procs.append(subprocess.Popen(cmd, env=child_env))
+
+    return_codes = [p.wait() for p in procs]
+    if any(rc != 0 for rc in return_codes):
+        failed = [k for k, rc in enumerate(return_codes) if rc != 0]
+        print(f"[lerobot-eval-parallel] shards {failed} failed with non-zero exit codes.", file=sys.stderr)
+        sys.exit(1)
+
+    if output_dir and num_shards > 1:
+        merged = _merge_shards(output_dir, num_shards)
+        print("\n=== Merged Results ===")
+        print(json.dumps(merged["overall"], indent=2))
+
+
+if __name__ == "__main__":
+    main()
@@ -421,7 +421,6 @@ def record_loop(
                act_processed_policy: RobotAction = make_robot_action(action_values, dataset.features)
                # Applies a pipeline to the action, default is IdentityProcessor
                robot_action_to_send = robot_action_processor((act_processed_policy, obs))
-                action_values = robot_action_to_send

        elif policy is None and isinstance(teleop, Teleoperator):
            act = teleop.get_action()
@@ -24,7 +24,6 @@ import torch
 from huggingface_hub import HfApi
 from PIL import Image
 from safetensors.torch import load_file
-from torchvision.transforms import v2

 import lerobot
 from lerobot.configs.default import DatasetConfig
@@ -35,7 +34,6 @@ from lerobot.datasets.image_writer import image_array_to_pil_image
 from lerobot.datasets.io_utils import hf_transform_to_torch
 from lerobot.datasets.lerobot_dataset import LeRobotDataset
 from lerobot.datasets.multi_dataset import MultiLeRobotDataset
-from lerobot.datasets.transforms import ImageTransforms, ImageTransformsConfig
 from lerobot.datasets.utils import (
    DEFAULT_CHUNK_SIZE,
    DEFAULT_DATA_FILE_SIZE_IN_MB,
@@ -357,62 +355,6 @@ def test_add_frame_image_pil(image_dataset):
    assert dataset[0]["image"].shape == torch.Size(DUMMY_CHW)


-def test_set_image_transforms_applies_transparently(image_dataset):
-    dataset = image_dataset
-    dataset.add_frame({"image": np.random.rand(*DUMMY_CHW), "task": "Dummy task"})
-    dataset.save_episode()
-    dataset.finalize()
-
-    dataset.set_image_transforms(v2.Resize((224, 224)))
-    assert dataset[0]["image"].shape == torch.Size((3, 224, 224))
-
-    dataset.set_image_transforms(v2.Resize((128, 128)))
-    assert dataset[0]["image"].shape == torch.Size((3, 128, 128))
-
-    dataset.clear_image_transforms()
-    assert dataset[0]["image"].shape == torch.Size(DUMMY_CHW)
-
-
-def test_set_image_transforms_supports_lerobot_image_transforms(image_dataset):
-    dataset = image_dataset
-    dataset.add_frame({"image": np.random.rand(*DUMMY_CHW), "task": "Dummy task"})
-    dataset.save_episode()
-    dataset.finalize()
-
-    image_transforms = ImageTransforms(ImageTransformsConfig(enable=False))
-    dataset.set_image_transforms(image_transforms)
-
-    assert dataset.image_transforms is image_transforms
-    assert dataset[0]["image"].shape == torch.Size(DUMMY_CHW)
-
-
-def test_set_image_transforms_supports_loaded_dataset(tmp_path, lerobot_dataset_factory):
-    dataset = lerobot_dataset_factory(root=tmp_path / "test", use_videos=False)
-    dataset.set_image_transforms(v2.Compose([v2.Resize((224, 224)), v2.Resize((112, 112))]))
-
-    camera_key = dataset.meta.camera_keys[0]
-    assert dataset[0][camera_key].shape == torch.Size((3, 112, 112))
-
-
-def test_multilerobot_dataset_set_image_transforms_propagates(tmp_path, lerobot_dataset_factory):
-    root = tmp_path / "multi"
-    repo_ids = ["lerobot/test_multi_a", "lerobot/test_multi_b"]
-
-    for repo_id in repo_ids:
-        lerobot_dataset_factory(root=root / repo_id, repo_id=repo_id, use_videos=False)
-
-    dataset = MultiLeRobotDataset(repo_ids, root=root, download_videos=False)
-    dataset.set_image_transforms(v2.Resize((96, 96)))
-
-    camera_key = dataset.camera_keys[0]
-    assert dataset[0][camera_key].shape == torch.Size((3, 96, 96))
-    assert all(child.image_transforms is dataset.image_transforms for child in dataset._datasets)
-
-    dataset.clear_image_transforms()
-    assert dataset.image_transforms is None
-    assert all(child.image_transforms is None for child in dataset._datasets)
-
-
 def test_image_array_to_pil_image_wrong_range_float_0_255():
    image = np.random.rand(*DUMMY_HWC) * 255
    with pytest.raises(ValueError):
@@ -90,7 +90,7 @@ def test_base_create_envs():
        envs = _Env().create_envs(n_envs=2)
        assert "_dispatch_base_test" in envs
        env = envs["_dispatch_base_test"][0]
-        assert isinstance(env, gym.vector.SyncVectorEnv)
+        assert isinstance(env, gym.vector.VectorEnv)
        assert env.num_envs == 2
        env.close()
    finally:
@@ -189,6 +189,30 @@ def test_list_of_strings_tokenization(mock_auto_tokenizer):
    assert attention_mask.shape == (2, 8)


+@require_package("transformers")
+@patch("lerobot.processor.tokenizer_processor.AutoTokenizer")
+def test_tuple_of_strings_tokenization(mock_auto_tokenizer):
+    """Test tokenization of a tuple of strings (returned by VectorEnv.call())."""
+    mock_tokenizer = MockTokenizer(vocab_size=100)
+    mock_auto_tokenizer.from_pretrained.return_value = mock_tokenizer
+
+    processor = TokenizerProcessorStep(tokenizer_name="test-tokenizer", max_length=8)
+
+    transition = create_transition(
+        observation={"state": torch.tensor([1.0, 2.0])},
+        action=torch.tensor([0.1, 0.2]),
+        complementary_data={"task": ("pick up cube", "place on table")},
+    )
+
+    result = processor(transition)
+
+    observation = result[TransitionKey.OBSERVATION]
+    tokens = observation[f"{OBS_LANGUAGE}.tokens"]
+    attention_mask = observation[f"{OBS_LANGUAGE}.attention_mask"]
+    assert tokens.shape == (2, 8)
+    assert attention_mask.shape == (2, 8)
+
+
@require_package("transformers")
@patch("lerobot.processor.tokenizer_processor.AutoTokenizer")
 def test_custom_keys(mock_auto_tokenizer):
Author	SHA1	Message	Date
Pepijn	66276f1efd	feat(eval): thread-safe policy copies for max_parallel_tasks > 1 eval_policy_all already supports running multiple task groups concurrently via ThreadPoolExecutor, but policy.reset() was not thread-safe: all threads shared the same policy object and its mutable state (action queues, temporal buffers). Fix: each thread receives a shallow copy of the policy. copy.copy() creates a new Python object whose _parameters dict is a shared reference — same tensor storage, zero extra VRAM — while reset() rebinds per-episode state to fresh objects per thread. Caveat: ACT with temporal_ensemble_coeff is not safe with this approach (its reset() mutates a shared sub-object). Keep max_parallel_tasks=1 for that config. For MetaWorld (50 tasks, no temporal ensembling), max_parallel_tasks=4 raises GPU utilization from ~20% to ~60-80% with no additional VRAM cost. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-07 13:43:42 +02:00
Pepijn	5972a85ec7	feat(eval): episode sharding, parallel launcher, and autotune Add lerobot-eval-parallel and lerobot-eval-autotune entry points for multi-process evaluation. A single H100 running 4 shards of SmolVLA achieves ~100% GPU utilisation vs ~0.5% with the serial baseline. - EvalConfig: add shard_id / num_shards fields; validate ranges - lerobot_eval.py: _shard_episodes() splits n_episodes round-robin; eval_main uses per-shard n_episodes + seed offset; writes shard_K_of_N.json when num_shards > 1 - lerobot_eval_parallel.py: spawns K subprocesses with disjoint shard IDs, sets MUJOCO_GL and OMP_NUM_THREADS, merges results on completion - lerobot_eval_autotune.py: probes GPU VRAM, CPU cores, optional model footprint and env step time; derives optimal num_shards / batch_size / MUJOCO_GL; prints a paste-ready command - pyproject.toml: register lerobot-eval-parallel and lerobot-eval-autotune Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-07 13:43:03 +02:00
Pepijn Kooijmans	800b0a5f26	docs: update adding_benchmarks for async env changes - Replace add_envs_task reference with env.call("task_description") - Update use_async_envs default to True - Add note about lazy GPU init for AsyncVectorEnv compatibility Made-with: Cursor	2026-04-07 13:38:37 +02:00
Pepijn Kooijmans	6aeb7c54f9	fix(eval): use task_description instead of task for language conditioning env.call("task") returns the LIBERO task name with underscores (e.g. "pick_up_the_black_bowl_...") instead of the natural language description ("pick up the black bowl ..."). The VLM tokenizes these completely differently, causing 0.0 reward across all episodes. Made-with: Cursor	2026-04-07 13:12:42 +02:00
Pepijn Kooijmans	1f7e7b4a90	fix: close envs between tasks to prevent worker process accumulation eval_policy_all never closed environments after each task completed, causing AsyncVectorEnv worker processes to accumulate (N_tasks × n_envs). This led to OOM, BrokenPipeError and EOFError on multi-task benchmarks. Also fixes: - AsyncVectorEnv compat in envs/utils.py (use get_attr/call instead of .envs) - Tuple task handling in tokenizer_processor and lerobot_eval - _LazyAsyncVectorEnv for deferred worker spawning in LIBERO Made-with: Cursor	2026-04-07 12:30:22 +02:00
Pepijn	681cc59ed2	feat(envs): lazy env init + AsyncVectorEnv as default for n_envs > 1 LiberoEnv and MetaworldEnv previously allocated GPU resources (EGL context, OpenGL framebuffer) in __init__, before AsyncVectorEnv's fork(). Worker processes inherited stale GPU handles, causing EGL_BAD_CONTEXT crashes on first render. Fix: defer OffScreenRenderEnv / MT1 construction to _ensure_env(), called on first reset() or step() inside the worker subprocess. Each worker creates its own clean context after fork(). Also fixes lerobot_eval.py:170 (add_envs_task TODO): replace with env.call("task") which works with both SyncVectorEnv and AsyncVectorEnv. AsyncVectorEnv is now the default for n_envs > 1; auto-downgraded to SyncVectorEnv when n_envs=1 (no benefit, less overhead). Expected speedup: ~15-20x for LIBERO Spatial with batch_size=50. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-07 11:31:32 +02:00
Pepijn Kooijmans	d9edc12e00	refactor: revert policy changes, keep env-only camera mapping fixes - Revert GR00T N1.5 default_factory/default changes (transformers compat) - Revert SmolVLA use_peft legacy field - Apply ruff formatting fixes - camera_name_mapping stays entirely in env/eval layer (no policy changes) Made-with: Cursor	2026-04-07 11:25:49 +02:00
Pepijn Kooijmans	fd2bad9b42	fix: handle gymnasium < 1.0 without AutoresetMode Made-with: Cursor	2026-04-07 11:20:38 +02:00
Pepijn Kooijmans	7e729e33c9	fix: use direct AutoresetMode import for gymnasium compat Made-with: Cursor	2026-04-07 11:19:17 +02:00
Pepijn Kooijmans	e383207a15	fix: enable SmolVLA eval on LIBERO with custom camera mappings - Thread camera_name_mapping from LiberoEnv config through to gym envs - Sync features_map with camera_name_mapping in LiberoEnv.__post_init__ - Fix render() to use first available camera instead of hardcoded "image" - Handle non-dict final_info in rollout by falling back to info["is_success"] - Add use_peft legacy field to SmolVLAConfig for checkpoint compat - Add defaults to GR00TN15Config init=False fields for transformers 5.3 Made-with: Cursor	2026-04-07 11:18:29 +02:00
Pepijn	8ed658c6aa	fix(tests): fix 3 failing dispatch tests - test_registry_all_types: skip non-EnvConfig stubs (e.g. TestPluginConfig) - test_processors_delegation: use None instead of abstract PreTrainedConfig - test_custom_get_env_processors_override: use DataProcessorPipeline for isinstance check (PolicyProcessorPipeline is a subscripted generic) Made-with: Cursor	2026-04-03 17:19:27 +02:00
Pepijn	0045f88355	merge: resolve conflicts from main into refactor/benchmark-dispatch Keep refactored dispatch pattern (no factory.py edits for new benchmarks). Incorporate main's "Verifying your integration" section and class naming fix. Made-with: Cursor	2026-04-03 14:49:36 +02:00
Pepijn	89ce91f69f	Merge branch 'docs/adding-benchmarks-guide' into refactor/benchmark-dispatch	2026-04-03 13:56:49 +02:00
Pepijn	90e614f6b9	fix task count	2026-04-03 13:48:37 +02:00
Pepijn	ff4f860e5d	fix link	2026-04-03 13:47:17 +02:00
Pepijn	6f2823bfc4	merge: resolve conflicts with docs/adding-benchmarks-guide Incorporate cleaner writing from the docs branch while reflecting the refactored dispatch pattern (no factory.py edits needed for new benchmarks). Made-with: Cursor	2026-04-03 13:45:12 +02:00
Pepijn	77415559b8	docs(benchmarks): clean up adding-benchmarks guide for clarity Rewrite for simpler language, better structure, and easier navigation. Move quick-reference table to the top, fold eval explanation into architecture section, condense the doc template to a bulleted outline. Made-with: Cursor	2026-04-03 13:36:16 +02:00
Pepijn	24d9b74d81	refactor(envs): move dispatch logic from factory into EnvConfig subclasses Replace hardcoded if/elif chains in factory.py with create_envs() and get_env_processors() methods on EnvConfig. New benchmarks now only need to register a config subclass — no factory.py edits required. Net -23 lines: factory.py shrinks from ~200 to ~70 lines of logic. Made-with: Cursor	2026-04-03 13:23:44 +02:00
Pepijn	508358749a	docs(benchmarks): add benchmark integration guide and standardize benchmark docs Add a comprehensive guide for adding new benchmarks to LeRobot, and refactor the existing LIBERO and Meta-World docs to follow the new standardized template. Made-with: Cursor	2026-04-02 20:43:31 +02:00
				`@@ -1 +0,0 @@`
				`../../../../docs/source/policy_multi_task_dit_README.md`
				`@@ -1 +0,0 @@`
				`../../../../docs/source/policy_pi0_README.md`
				`@@ -1 +0,0 @@`
				`../../../../docs/source/policy_pi05_README.md`
				`@@ -1 +0,0 @@`
				`../../../../docs/source/policy_rtc_README.md`
				`@@ -1 +0,0 @@`
				`../../../../docs/source/policy_sarm_README.md`