- processor(vlabench): drop the no-op `VLABenchProcessorStep` and
let `VLABenchEnv` fall back to the identity `get_env_processors`
default. Removes the class, its export, and its override.
- envs(vlabench): hoist `import cv2` to module scope; propagate the
gym `seed` into the inner dm_control env (re-seeding `task.random`
and the env's `_random_state`) so layout sampling is reproducible.
- docs(vlabench): mirror the CI eval command across all four
snippets (`--eval.use_async_envs=false`, `--policy.device=cuda`,
and the `rename_map` for image/second_image/wrist_image →
camera1/2/3). Intro now calls out that LeRobot exposes 43 of
VLABench's 100 upstream task categories via `--env.task`.
- docker(vlabench): drop the `|| echo WARN` fallback on the gdown
asset-download path so a genuine download failure surfaces
instead of being masked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VLABench is not on PyPI — its only distribution is the
OpenMOSS/VLABench GitHub repo (package name `VLABench`, no PyPI
release) — so the `vlabench>=0.1.0` spec in the `[vlabench]` extra
was flatly unsatisfiable. Every `uv sync` (even with unrelated
extras like `--extra test`) validates the full lockfile graph and
tripped on this, breaking the fast-test CI step.
- Remove `vlabench = [...]` from pyproject.toml (left an explanatory
comment in its place) and drop the `lerobot[vlabench]` entry from
the `all` extra.
- Update docs/source/vlabench.mdx to document the manual
`git clone` + editable install flow for VLABench, rrt-algorithms
and the MuJoCo / dm_control pins — same pattern as robocasa.
- Docker image already does this via explicit git clones, so no CI
image change is needed.
Made-with: Cursor
Rework docs/source/vlabench.mdx to follow the standard benchmark doc
structure: intro + links + available tasks (with cleaner counts and
comma-list shortcuts) + installation + eval + recommended episodes +
policy I/O + training + reproducing results.
- Point everything at lerobot/smolvla_vlabench (the released
checkpoint) rather than a generic "your-policy-id" placeholder.
- Document the three valid `--env.task` forms (single, comma list,
suite shortcut).
- Drop the "Evaluation tracks" table — it reflected the upstream
paper rather than what the LeRobot env exposes, and was confusing.
- Add an explicit "Reproducing published results" section pointing
at the CI smoke eval.
Made-with: Cursor
Add VLABench (language-conditioned manipulation with long-horizon
reasoning) as a new simulation benchmark, following the established
LIBERO/MetaWorld patterns.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pepijn