- envs(vlabench): drop the unused `episode_index` / `n_envs`
constructor args — they were stored but never referenced.
- envs(vlabench): replace module-level `print()` calls with the
standard `logging.getLogger(__name__)` logger (matches the rest
of the repo's script convention). Covers the two PhysicsError
retry/step paths and the `create_vlabench_envs` progress lines.
- envs(vlabench): drop `_make_env_fns` and build factories
MetaWorld-style via a lambda list comprehension — simpler, no
extra helper.
- docker(vlabench): pin upstream clones to commit SHAs
(`OpenMOSS/VLABench@cf588fe6`, `motion-planning/rrt-algorithms@e51d95ee`)
so benchmark images are reproducible.
- ci(vlabench): run `scripts/ci/extract_task_descriptions.py` after
the eval so `metrics.json` carries per-task natural-language
labels, matching LIBERO / MetaWorld jobs. Added a VLABench
extractor that produces cleaned-name labels keyed by `<task>_0`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- processor(vlabench): drop the no-op `VLABenchProcessorStep` and
let `VLABenchEnv` fall back to the identity `get_env_processors`
default. Removes the class, its export, and its override.
- envs(vlabench): hoist `import cv2` to module scope; propagate the
gym `seed` into the inner dm_control env (re-seeding `task.random`
and the env's `_random_state`) so layout sampling is reproducible.
- docs(vlabench): mirror the CI eval command across all four
snippets (`--eval.use_async_envs=false`, `--policy.device=cuda`,
and the `rename_map` for image/second_image/wrist_image →
camera1/2/3). Intro now calls out that LeRobot exposes 43 of
VLABench's 100 upstream task categories via `--env.task`.
- docker(vlabench): drop the `|| echo WARN` fallback on the gdown
asset-download path so a genuine download failure surfaces
instead of being masked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous HF Hub snapshot_download step could complete silently
without actually populating the asset tree — several files in
lerobot/vlabench-assets are Xet-stored, and the stock huggingface_hub
install in the base image can skip them. The first runtime signal was
an IndexError deep inside VLABench's task builder (random.choice on an
empty XML list in config_manager.py), far from the real cause.
- Install huggingface_hub[hf_xet]>=0.26 before the snapshot download
so Xet-stored assets are fetched reliably.
- Scope the download with allow_patterns=['obj/**', 'scenes/**'] (the
only subtrees the env needs), reducing image size and surface area.
- Add a build-time validator: after the download, walk four
task-critical subtrees (plates, basket, fruit, tray) and fail the
build loudly with the list of empty dirs if any is missing XMLs.
Prints XML counts when successful so CI logs confirm completeness.
Made-with: Cursor
Google Drive returns HTTP 429 ("too many users") on VLABench's shared
academic asset file, blocking CI builds unpredictably.
- Add VLABENCH_ASSETS_REPO build arg: when set, snapshot_download the
given HF Hub dataset repo straight into VLABench/assets/. This is the
reliable path for CI once a mirror is uploaded.
- Keep the gdown script as fallback but make it best-effort (|| echo WARN)
so quota errors don't block the image build.
If neither source succeeds, the constant.py patch (empty-dir guard)
keeps module import working; only task-build at runtime will surface
missing-mesh errors.
Made-with: Cursor
- The nightly base image has stale lerobot code (pre-dates VLABenchEnv
addition), so COPY . . doesn't propagate into .venv. Run
`uv pip install --no-deps -e .` after COPY, same fix as robocasa365.
- Coerce ee_state to exactly shape (7,) so SyncVectorEnv.concatenate
doesn't raise "Output array is the wrong shape" when VLABench's
actual ee_state length differs from our declared observation_space.
Made-with: Cursor
Task configs reference object meshes (obj/meshes/fruit/, containers/*)
via name2class_xml; without them, random.choice([]) crashes during
task build. Install gdown and run VLABench's download_assets.py script
to fetch the asset + scene archives from Google Drive.
Made-with: Cursor
VLABench.utils.gpt_utils imports openai (for LLM instruction generation).
Triggered transitively via dm_task.py. Not used by eval but required
at import time.
Made-with: Cursor
rrt_algorithms/rrt/ has no __init__.py, so find_packages() excludes
it from the built wheel (same issue as VLABench/utils/). Clone the
repo and install with -e so imports resolve from the source tree.
Made-with: Cursor
VLABench.utils.skill_lib imports rrt_algorithms (GitHub-only package,
not on PyPI). Triggered transitively via dm_task.py on any task import.
Made-with: Cursor
VLABench's configs/constant.py calls os.listdir() on ~100 asset
directories (obj/meshes/*) at module import time. These dirs come
from a Google Drive download we skip in CI. Patch get_object_list
to return [] for missing dirs instead of crashing with FileNotFoundError.
Made-with: Cursor
download_assets lives in scripts/download_assets.py (not
VLABench.utils) and downloads from Google Drive via gdown, which
is unreliable in CI. Our smoke test uses its own eval pipeline and
does not need these assets.
Made-with: Cursor
VLABench/utils/ has no __init__.py, so find_packages() omits it
from built wheels. Use editable install (-e) so Python imports
directly from the source tree via implicit namespace packages.
Made-with: Cursor
The VLABench repo's nested submodules (openpi -> aloha, libero) use
git@github.com SSH URLs which fail in Docker builds without SSH.
Clone with --depth 1 (no submodule recursion) and install from the
local checkout instead.
Made-with: Cursor
The base image (lerobot-gpu) uses uv, not pip. The previous `pip install`
silently failed ("pip: not found") and `|| true` hid the error, resulting
in an image with no VLABench installed.
Made-with: Cursor
Add VLABench (language-conditioned manipulation with long-horizon
reasoning) as a new simulation benchmark, following the established
LIBERO/MetaWorld patterns.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pepijn