add_envs_task is replaced by env.call("task_description") in this PR.
Remove it from the pipeline walkthrough and renumber the steps (8→7).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_get_sub_env_attr was defined but never called anywhere in the codebase.
_sub_env_has_attr (its sibling) is kept — it is actively used in utils.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
isinstance(env, AsyncVectorEnv) silently skipped _LazyAsyncVectorEnv,
causing video rendering to produce no frames on the default async path.
Switch to hasattr(env, "call") so any async-compatible env (including
_LazyAsyncVectorEnv) hits the call("render") branch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
num2words (required by SmolVLM processor) is declared in lerobot[smolvla],
not lerobot[libero/metaworld]. Install both extras together.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The config was pointing to /tmp/libero_init which doesn't exist.
Use importlib.util.find_spec to locate the hf-libero package directory
and write paths to the actual bundled bddl_files/init_files/assets.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The multiline RUN python -c "..." was being parsed as Dockerfile
instructions. Use printf to write ~/.libero/config.yaml directly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
libero/__init__.py calls input() when ~/.libero/config.yaml is missing.
We write the config at image build time (without importing libero) so
the prompt never fires at runtime. Also trigger CI on pyproject.toml changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
libero/__init__.py calls input() to ask about a custom dataset path,
which raises EOFError when stdin is closed inside Docker. Setting
LIBERO_DATA_FOLDER skips the prompt entirely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each benchmark gets its own Docker image (lerobot[libero] / lerobot[metaworld]
only) so incompatible dep trees cannot collide. A 1-episode smoke eval runs
per benchmark on GPU runners.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AsyncVectorEnv now uses shared_memory=True for zero-copy observation transfer
- LiberoEnvConfig.gym_kwargs passes observation_height/width to the env
- eval_policy_all prefetches next task's workers while current task runs
Made-with: Cursor
- New docs/source/evaluation.mdx covering lerobot-eval usage, batch_size
auto-tuning, AsyncVectorEnv performance, tuning tips, output format,
multi-task evaluation, and programmatic usage.
- Add evaluation page to _toctree.yml under Benchmarks section.
- Update adding_benchmarks.mdx to reference batch_size auto default and
link to the evaluation guide.
Made-with: Cursor
- batch_size=0 (default) auto-tunes based on CPU cores, capped by
n_episodes and 64. Removes the need for users to guess the right
value. The old batch_size > n_episodes error is replaced by silently
clamping to n_episodes.
- _LazyAsyncVectorEnv accepts pre-computed spaces so only one temp env
is created per suite (not per task). For libero_spatial (10 tasks)
this avoids 9 redundant LiberoEnv instantiations during env setup.
Made-with: Cursor
env.call("task") returns the LIBERO task name with underscores
(e.g. "pick_up_the_black_bowl_...") instead of the natural language
description ("pick up the black bowl ..."). The VLM tokenizes these
completely differently, causing 0.0 reward across all episodes.
Made-with: Cursor
eval_policy_all never closed environments after each task completed,
causing AsyncVectorEnv worker processes to accumulate (N_tasks × n_envs).
This led to OOM, BrokenPipeError and EOFError on multi-task benchmarks.
Also fixes:
- AsyncVectorEnv compat in envs/utils.py (use get_attr/call instead of .envs)
- Tuple task handling in tokenizer_processor and lerobot_eval
- _LazyAsyncVectorEnv for deferred worker spawning in LIBERO
Made-with: Cursor
LiberoEnv and MetaworldEnv previously allocated GPU resources (EGL context,
OpenGL framebuffer) in __init__, before AsyncVectorEnv's fork(). Worker
processes inherited stale GPU handles, causing EGL_BAD_CONTEXT crashes on
first render.
Fix: defer OffScreenRenderEnv / MT1 construction to _ensure_env(), called on
first reset() or step() inside the worker subprocess. Each worker creates its
own clean context after fork().
Also fixes lerobot_eval.py:170 (add_envs_task TODO): replace with
env.call("task") which works with both SyncVectorEnv and AsyncVectorEnv.
AsyncVectorEnv is now the default for n_envs > 1; auto-downgraded to
SyncVectorEnv when n_envs=1 (no benefit, less overhead).
Expected speedup: ~15-20x for LIBERO Spatial with batch_size=50.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* add: a flexible transformation registry
* fix: image transforms can be set both at init and after
* add: tests
* fix: take in review
* feat(datasets): add image transform setters
* fix: pre-commit
* fix: CI
---------
Signed-off-by: Francesco Capuano <74058581+fracapuano@users.noreply.github.com>
- Thread camera_name_mapping from LiberoEnv config through to gym envs
- Sync features_map with camera_name_mapping in LiberoEnv.__post_init__
- Fix render() to use first available camera instead of hardcoded "image"
- Handle non-dict final_info in rollout by falling back to info["is_success"]
- Add use_peft legacy field to SmolVLAConfig for checkpoint compat
- Add defaults to GR00TN15Config init=False fields for transformers 5.3
Made-with: Cursor
* feat(ci): add uv.lock
* feat(ci): use uv.lock in CI PR testing
* chore(ci): rename nightly to docker publish and test
* feat(ci): automated update of uv.lock + remove unbound check + docker images now use uv.lock
* fix(ci): add --force-with-lease + set -e for silent erros
* feat(ffmpeg): updating ffmpeg verion to 8.X
* Revert "feat(ffmpeg): updating ffmpeg verion to 8.X"
This reverts commit bb0f03185c.
* chore(pyproject): updating pyproject to fit the minimally required version of torchcodec
* chore(docs): updating doc with specific instructions for ffmpeg/torchcodec installation
* fix(typo): reverting ceiling bound on pytorch to 2.11.0
* chore(format): removing empty line
* chore(typo): fixing typo
* chore(docs): adding warning in case of torchcodec/ffmpeg version mismatch
* chore(docs): applying comments
* chore(docs): adding uv commands for evdev on WSL
* fix(typo): fixing typo
* fix(typo): fixing typos again
* chore(ruff): format
* fix(evdev install): splitting evdev install instructions between conda and uv
* chore(ruff): format
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
- test_registry_all_types: skip non-EnvConfig stubs (e.g. TestPluginConfig)
- test_processors_delegation: use None instead of abstract PreTrainedConfig
- test_custom_get_env_processors_override: use DataProcessorPipeline for isinstance check (PolicyProcessorPipeline is a subscripted generic)
Made-with: Cursor
Pepijn