- envs(vlabench): drop the unused `episode_index` / `n_envs`
constructor args — they were stored but never referenced.
- envs(vlabench): replace module-level `print()` calls with the
standard `logging.getLogger(__name__)` logger (matches the rest
of the repo's script convention). Covers the two PhysicsError
retry/step paths and the `create_vlabench_envs` progress lines.
- envs(vlabench): drop `_make_env_fns` and build factories
MetaWorld-style via a lambda list comprehension — simpler, no
extra helper.
- docker(vlabench): pin upstream clones to commit SHAs
(`OpenMOSS/VLABench@cf588fe6`, `motion-planning/rrt-algorithms@e51d95ee`)
so benchmark images are reproducible.
- ci(vlabench): run `scripts/ci/extract_task_descriptions.py` after
the eval so `metrics.json` carries per-task natural-language
labels, matching LIBERO / MetaWorld jobs. Added a VLABench
extractor that produces cleaned-name labels keyed by `<task>_0`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two independent fixes:
1. Port of #3416: add `metadata` / `unwrapped` to `_LazyAsyncVectorEnv`
and cache the metadata alongside obs/action spaces in the LIBERO,
MetaWorld, and VLABench factories so async eval keeps
`env.unwrapped.metadata['render_fps']` working.
2. VLABench `select_mahjong` kept failing the smoke eval with
`mjWARN_BADQACC` after the 5-retry ceiling. Bump
`_ENSURE_ENV_MAX_ATTEMPTS` to 20, reseed NumPy's global RNG
between attempts (VLABench's upstream layout sampler is
deterministic given current RNG state — without a reseed, retries
replay the same diverging configuration), and raise a clearer
`RuntimeError` on exhaustion. Also swap `select_mahjong` out of
the CI task list in favor of `select_billiards`, which has a
stable layout sampler.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- processor(vlabench): drop the no-op `VLABenchProcessorStep` and
let `VLABenchEnv` fall back to the identity `get_env_processors`
default. Removes the class, its export, and its override.
- envs(vlabench): hoist `import cv2` to module scope; propagate the
gym `seed` into the inner dm_control env (re-seeding `task.random`
and the env's `_random_state`) so layout sampling is reproducible.
- docs(vlabench): mirror the CI eval command across all four
snippets (`--eval.use_async_envs=false`, `--policy.device=cuda`,
and the `rename_map` for image/second_image/wrist_image →
camera1/2/3). Intro now calls out that LeRobot exposes 43 of
VLABench's 100 upstream task categories via `--env.task`.
- docker(vlabench): drop the `|| echo WARN` fallback on the gdown
asset-download path so a genuine download failure surfaces
instead of being masked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(record): pass rename_map to make_policy in lerobot-record
Fixes#3181. The rename_map from dataset config was used for preprocessor
construction but not passed to make_policy(), causing feature mismatch
errors when camera key names differ between dataset and model config.
make_policy() already accepts a rename_map parameter and uses it to skip
visual feature consistency validation when remapping is active, but
lerobot_record.py was not passing it through.
* style: fix ruff format for ternary expression
---------
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
VLABench's LM4ManipDMEnv.reset() runs 20 warm-up `step()` calls while
toggling gravity / fluid coefficients to settle the scene. For some
random layout samples MuJoCo's integrator diverges and raises
`mjWARN_BADQACC` -> `dm_control.rl.control.PhysicsError` before we
ever get to return from reset. In a multi-task smoke eval that kills
the whole run at the next task's env construction.
- `_ensure_env`: catch PhysicsError and retry `load_env()` up to five
times with a fresh random layout. Almost always converges; if it
really can't, re-raise the last exception.
- `step`: wrap `self._env.step(ctrl)` in a PhysicsError handler. On
divergence, terminate the episode gracefully (is_success=False,
terminated=True, info['physics_error']=True) and drop the stale
inner env so the next reset rebuilds it. Keeps the rest of the
multi-task eval moving instead of propagating a hard crash.
Made-with: Cursor
VLABench is not on PyPI — its only distribution is the
OpenMOSS/VLABench GitHub repo (package name `VLABench`, no PyPI
release) — so the `vlabench>=0.1.0` spec in the `[vlabench]` extra
was flatly unsatisfiable. Every `uv sync` (even with unrelated
extras like `--extra test`) validates the full lockfile graph and
tripped on this, breaking the fast-test CI step.
- Remove `vlabench = [...]` from pyproject.toml (left an explanatory
comment in its place) and drop the `lerobot[vlabench]` entry from
the `all` extra.
- Update docs/source/vlabench.mdx to document the manual
`git clone` + editable install flow for VLABench, rrt-algorithms
and the MuJoCo / dm_control pins — same pattern as robocasa.
- Docker image already does this via explicit git clones, so no CI
image change is needed.
Made-with: Cursor
Rework docs/source/vlabench.mdx to follow the standard benchmark doc
structure: intro + links + available tasks (with cleaner counts and
comma-list shortcuts) + installation + eval + recommended episodes +
policy I/O + training + reproducing results.
- Point everything at lerobot/smolvla_vlabench (the released
checkpoint) rather than a generic "your-policy-id" placeholder.
- Document the three valid `--env.task` forms (single, comma list,
suite shortcut).
- Drop the "Evaluation tracks" table — it reflected the upstream
paper rather than what the LeRobot env exposes, and was confusing.
- Add an explicit "Reproducing published results" section pointing
at the CI smoke eval.
Made-with: Cursor
Broader coverage on the VLABench benchmark CI job: bump the smoke eval
from 1 task to 10 (one episode each), all drawn from PRIMITIVE_TASKS.
Tasks now run: select_fruit, select_toy, select_book, select_painting,
select_drink, select_ingredient, select_mahjong, select_poker,
add_condiment, insert_flower.
Bumped the parse_eval_metrics.py `--task` label to the full comma
list so the metrics artifact reflects what was actually run.
`parse_eval_metrics.py` already reads `overall` for multi-task runs,
so no parser change is needed.
Made-with: Cursor
`env.physics` in dm_control is a weakref proxy; caching it in
`_ensure_env` and reusing across steps/resets yields a
`ReferenceError: weakly-referenced object no longer exists` once the
underlying sim is rebuilt (e.g. after the first reset of a composer
environment). Drop the cache and refetch `self._env.physics` at the
call sites (step and IK) — it's a cheap attribute access.
Made-with: Cursor
The previous HF Hub snapshot_download step could complete silently
without actually populating the asset tree — several files in
lerobot/vlabench-assets are Xet-stored, and the stock huggingface_hub
install in the base image can skip them. The first runtime signal was
an IndexError deep inside VLABench's task builder (random.choice on an
empty XML list in config_manager.py), far from the real cause.
- Install huggingface_hub[hf_xet]>=0.26 before the snapshot download
so Xet-stored assets are fetched reliably.
- Scope the download with allow_patterns=['obj/**', 'scenes/**'] (the
only subtrees the env needs), reducing image size and surface area.
- Add a build-time validator: after the download, walk four
task-critical subtrees (plates, basket, fruit, tray) and fail the
build loudly with the list of empty dirs if any is missing XMLs.
Prints XML counts when successful so CI logs confirm completeness.
Made-with: Cursor
Pass VLABENCH_ASSETS_REPO=lerobot/vlabench-assets so the image pulls
mesh + scene assets from HF Hub (reliable) instead of hitting Google
Drive quotas.
Made-with: Cursor
Google Drive returns HTTP 429 ("too many users") on VLABench's shared
academic asset file, blocking CI builds unpredictably.
- Add VLABENCH_ASSETS_REPO build arg: when set, snapshot_download the
given HF Hub dataset repo straight into VLABench/assets/. This is the
reliable path for CI once a mirror is uploaded.
- Keep the gdown script as fallback but make it best-effort (|| echo WARN)
so quota errors don't block the image build.
If neither source succeeds, the constant.py patch (empty-dir guard)
keeps module import working; only task-build at runtime will surface
missing-mesh errors.
Made-with: Cursor
VLABench's dataset logs 7D actions [x, y, z, rx, ry, rz, gripper] in
end-effector space, but its dm_control task writes `data.ctrl[:] = action`
directly, which for Franka expects 9 joint entries (7 arm + 2 gripper
fingers). Previously we zero-padded the action, which physically moved
nothing meaningful.
Add `_build_ctrl_from_action` that mirrors what VLABench's data collector
uses (SingleArm.get_qpos_from_ee_pos, backed by dm_control's
qpos_from_site_pose IK solver):
- decompose EEF action into (pos, euler, gripper)
- euler XYZ -> quat (w, x, y, z)
- IK solve for 7 arm joint qposes targeting that EEF pose
- map gripper scalar [0..1] linearly to Franka finger qpos [0.04..0.0]
Joint-space actions (shape == ctrl_dim) still pass through unchanged.
Made-with: Cursor
VLABench's dm_control task does `data.ctrl[:] = action` without any
adaptation. Franka has 9 actuators (7 arm + 2 gripper fingers) but
our policy emits a 7D action, causing a broadcast error. Pad with
zeros / repeat the gripper command for trailing actuators.
Made-with: Cursor
Previous resize path only kicked in when shape[:2] mismatched, so
arrays with wrong channel count or unexpected layouts (e.g. HWC
arriving as HWC but treated as CHW and transposed into junk shape)
could still slip through. New _to_hwc3 helper:
- strips leading singleton batch dims
- auto-detects CHW vs HWC via channel size
- normalizes 1/4-channel to 3 channels
- resizes to (h, w) and casts to uint8 unconditionally
Made-with: Cursor
- The nightly base image has stale lerobot code (pre-dates VLABenchEnv
addition), so COPY . . doesn't propagate into .venv. Run
`uv pip install --no-deps -e .` after COPY, same fix as robocasa365.
- Coerce ee_state to exactly shape (7,) so SyncVectorEnv.concatenate
doesn't raise "Output array is the wrong shape" when VLABench's
actual ee_state length differs from our declared observation_space.
Made-with: Cursor
VLABench's load_env doesn't always honor render_resolution, so the
returned rgb frames may not match our declared (h, w, 3) observation
space. gymnasium's SyncVectorEnv.concatenate then raises
"Output array is the wrong shape". Resize to the declared (h, w).
Made-with: Cursor
Task configs reference object meshes (obj/meshes/fruit/, containers/*)
via name2class_xml; without them, random.choice([]) crashes during
task build. Install gdown and run VLABench's download_assets.py script
to fetch the asset + scene archives from Google Drive.
Made-with: Cursor
VLABench.utils.gpt_utils imports openai (for LLM instruction generation).
Triggered transitively via dm_task.py. Not used by eval but required
at import time.
Made-with: Cursor
rrt_algorithms/rrt/ has no __init__.py, so find_packages() excludes
it from the built wheel (same issue as VLABench/utils/). Clone the
repo and install with -e so imports resolve from the source tree.
Made-with: Cursor
VLABench.utils.skill_lib imports rrt_algorithms (GitHub-only package,
not on PyPI). Triggered transitively via dm_task.py on any task import.
Made-with: Cursor
VLABench's configs/constant.py calls os.listdir() on ~100 asset
directories (obj/meshes/*) at module import time. These dirs come
from a Google Drive download we skip in CI. Patch get_object_list
to return [] for missing dirs instead of crashing with FileNotFoundError.
Made-with: Cursor
VLABench uses decorator-based registration (@register.add_robot,
@register.add_task). Importing only VLABench.envs doesn't trigger
the robot/task module imports, so the registry is empty at load_env
time. Import VLABench.robots and VLABench.tasks explicitly.
Made-with: Cursor
download_assets lives in scripts/download_assets.py (not
VLABench.utils) and downloads from Google Drive via gdown, which
is unreliable in CI. Our smoke test uses its own eval pipeline and
does not need these assets.
Made-with: Cursor
VLABench/utils/ has no __init__.py, so find_packages() omits it
from built wheels. Use editable install (-e) so Python imports
directly from the source tree via implicit namespace packages.
Made-with: Cursor
The VLABench repo's nested submodules (openpi -> aloha, libero) use
git@github.com SSH URLs which fail in Docker builds without SSH.
Clone with --depth 1 (no submodule recursion) and install from the
local checkout instead.
Made-with: Cursor
The base image (lerobot-gpu) uses uv, not pip. The previous `pip install`
silently failed ("pip: not found") and `|| true` hid the error, resulting
in an image with no VLABench installed.
Made-with: Cursor
The docker/login-action fails hard when DOCKERHUB_LEROBOT_USERNAME is
empty (e.g. fork PRs). Since all jobs use push: false (local build only),
the login is just for pull rate limit avoidance — make it conditional.
Made-with: Cursor
Pepijn