VLABench is not on PyPI — its only distribution is the
OpenMOSS/VLABench GitHub repo (package name `VLABench`, no PyPI
release) — so the `vlabench>=0.1.0` spec in the `[vlabench]` extra
was flatly unsatisfiable. Every `uv sync` (even with unrelated
extras like `--extra test`) validates the full lockfile graph and
tripped on this, breaking the fast-test CI step.
- Remove `vlabench = [...]` from pyproject.toml (left an explanatory
comment in its place) and drop the `lerobot[vlabench]` entry from
the `all` extra.
- Update docs/source/vlabench.mdx to document the manual
`git clone` + editable install flow for VLABench, rrt-algorithms
and the MuJoCo / dm_control pins — same pattern as robocasa.
- Docker image already does this via explicit git clones, so no CI
image change is needed.
Made-with: Cursor
Rework docs/source/vlabench.mdx to follow the standard benchmark doc
structure: intro + links + available tasks (with cleaner counts and
comma-list shortcuts) + installation + eval + recommended episodes +
policy I/O + training + reproducing results.
- Point everything at lerobot/smolvla_vlabench (the released
checkpoint) rather than a generic "your-policy-id" placeholder.
- Document the three valid `--env.task` forms (single, comma list,
suite shortcut).
- Drop the "Evaluation tracks" table — it reflected the upstream
paper rather than what the LeRobot env exposes, and was confusing.
- Add an explicit "Reproducing published results" section pointing
at the CI smoke eval.
Made-with: Cursor
Add VLABench (language-conditioned manipulation with long-horizon
reasoning) as a new simulation benchmark, following the established
LIBERO/MetaWorld patterns.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pepijn