diff --git a/docs/source/vlabench.mdx b/docs/source/vlabench.mdx index 20107276f..c70b15b71 100644 --- a/docs/source/vlabench.mdx +++ b/docs/source/vlabench.mdx @@ -1,59 +1,49 @@ # VLABench -VLABench is a large-scale benchmark for **language-conditioned robotic manipulation with long-horizon reasoning**. It provides 100 task categories across 2000+ objects, evaluating six dimensions of robot intelligence: mesh & texture understanding, spatial reasoning, world knowledge transfer, semantic instruction comprehension, physical law understanding, and long-horizon reasoning. VLABench is built on MuJoCo/dm_control and uses a Franka Panda 7-DOF arm. +[VLABench](https://github.com/OpenMOSS/VLABench) is a large-scale benchmark for **language-conditioned robotic manipulation with long-horizon reasoning**. It covers 100 task categories across 2,000+ objects and evaluates six dimensions of robot intelligence: mesh & texture understanding, spatial reasoning, world-knowledge transfer, semantic instruction comprehension, physical-law understanding, and long-horizon planning. Built on MuJoCo / dm_control with a Franka Panda 7-DOF arm. - Paper: [VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning](https://arxiv.org/abs/2412.18194) - GitHub: [OpenMOSS/VLABench](https://github.com/OpenMOSS/VLABench) - Project website: [vlabench.github.io](https://vlabench.github.io) +- Pretrained policy: [`lerobot/smolvla_vlabench`](https://huggingface.co/lerobot/smolvla_vlabench) ## Available tasks -VLABench includes **two task suites** covering **100 task categories**: +VLABench ships two task suites covering **43 task categories** in LeRobot's `--env.task` surface: | Suite | CLI name | Tasks | Description | | --------- | ----------- | ----- | ---------------------------------------------------------------- | -| Primitive | `primitive` | 21 | Single/few skill combinations (select, insert, physics QA) | +| Primitive | `primitive` | 21 | Single / few-skill combinations (select, insert, physics QA) | | Composite | `composite` | 22 | Multi-step reasoning and long-horizon planning (cook, rearrange) | -### Primitive tasks +**Primitive tasks:** `select_fruit`, `select_toy`, `select_chemistry_tube`, `add_condiment`, `select_book`, `select_painting`, `select_drink`, `insert_flower`, `select_billiards`, `select_ingredient`, `select_mahjong`, `select_poker`, and physical-reasoning tasks (`density_qa`, `friction_qa`, `magnetism_qa`, `reflection_qa`, `simple_cuestick_usage`, `simple_seesaw_usage`, `sound_speed_qa`, `thermal_expansion_qa`, `weight_qa`). -Includes `select_fruit`, `select_toy`, `select_chemistry_tube`, `add_condiment`, `select_book`, `select_painting`, `select_drink`, `insert_flower`, `select_billiards`, `select_ingredient`, `select_mahjong`, `select_poker`, and physical reasoning tasks (`density_qa`, `friction_qa`, `magnetism_qa`, `reflection_qa`, `simple_cuestick_usage`, `simple_seesaw_usage`, `sound_speed_qa`, `thermal_expansion_qa`, `weight_qa`). +**Composite tasks:** `cluster_billiards`, `cluster_book`, `cluster_drink`, `cluster_toy`, `cook_dishes`, `cool_drink`, `find_unseen_object`, `get_coffee`, `hammer_nail`, `heat_food`, `make_juice`, `play_mahjong`, `play_math_game`, `play_poker`, `play_snooker`, `rearrange_book`, `rearrange_chemistry_tube`, `set_dining_table`, `set_study_table`, `store_food`, `take_chemistry_experiment`, `use_seesaw_complex`. -### Composite tasks +`--env.task` accepts three forms: -Includes `cluster_billiards`, `cluster_book`, `cluster_drink`, `cluster_toy`, `cook_dishes`, `cool_drink`, `find_unseen_object`, `get_coffee`, `hammer_nail`, `heat_food`, `make_juice`, `play_mahjong`, `play_math_game`, `play_poker`, `play_snooker`, `rearrange_book`, `rearrange_chemistry_tube`, `set_dining_table`, `set_study_table`, `store_food`, `take_chemistry_experiment`, `use_seesaw_complex`. - -### Evaluation tracks - -VLABench defines five standard evaluation tracks: - -| Track | Focus | -| ----- | ----------------------------- | -| 1 | In-distribution task learning | -| 2 | Cross-category generalization | -| 3 | Commonsense reasoning | -| 4 | Semantic instruction | -| 6 | Unseen texture robustness | +- a single task name (`select_fruit`) +- a comma-separated list (`select_fruit,heat_food`) +- a suite shortcut (`primitive`, `composite`, or `primitive,composite`) ## Installation -After following the LeRobot installation instructions: +After following the standard LeRobot installation instructions: ```bash pip install -e ".[vlabench]" ``` -VLABench also requires downloading simulation assets: +VLABench also ships simulation assets separately. Clone the repo and download them: ```bash -# Clone the VLABench repo and download assets git clone https://github.com/OpenMOSS/VLABench.git cd VLABench python scripts/download_assets.py ``` -VLABench requires Linux (`sys_platform == 'linux'`) and Python 3.10+. Set the MuJoCo rendering backend before training or evaluation: +VLABench requires Linux (`sys_platform == 'linux'`) and Python 3.10+. Set the MuJoCo rendering backend before running: ```bash export MUJOCO_GL=egl # for headless servers (HPC, cloud) @@ -63,26 +53,37 @@ export MUJOCO_GL=egl # for headless servers (HPC, cloud) ## Evaluation -### Default evaluation (recommended) - -Evaluate on a single task (10 episodes): +### Single-task evaluation (recommended for quick iteration) ```bash lerobot-eval \ - --policy.path="your-policy-id" \ + --policy.path=lerobot/smolvla_vlabench \ --env.type=vlabench \ --env.task=select_fruit \ --eval.batch_size=1 \ --eval.n_episodes=10 ``` -### Suite-wide evaluation +### Multi-task evaluation -Evaluate across all primitive tasks: +Pass a comma-separated list of tasks: ```bash lerobot-eval \ - --policy.path="your-policy-id" \ + --policy.path=lerobot/smolvla_vlabench \ + --env.type=vlabench \ + --env.task=select_fruit,select_toy,add_condiment,heat_food \ + --eval.batch_size=1 \ + --eval.n_episodes=10 +``` + +### Suite-wide evaluation + +Run an entire suite (all 21 primitives or all 22 composites): + +```bash +lerobot-eval \ + --policy.path=lerobot/smolvla_vlabench \ --env.type=vlabench \ --env.task=primitive \ --eval.batch_size=1 \ @@ -90,13 +91,11 @@ lerobot-eval \ --env.max_parallel_tasks=1 ``` -### Multi-suite evaluation - -Evaluate across both primitive and composite suites: +Or both suites: ```bash lerobot-eval \ - --policy.path="your-policy-id" \ + --policy.path=lerobot/smolvla_vlabench \ --env.type=vlabench \ --env.task=primitive,composite \ --eval.batch_size=1 \ @@ -104,59 +103,54 @@ lerobot-eval \ --env.max_parallel_tasks=1 ``` -### Individual task evaluation +### Recommended evaluation episodes -Evaluate on specific tasks by name: - -```bash -lerobot-eval \ - --policy.path="your-policy-id" \ - --env.type=vlabench \ - --env.task=select_fruit,heat_food \ - --eval.batch_size=1 \ - --eval.n_episodes=10 -``` +**10 episodes per task** for reproducible benchmarking (210 total for the full primitive suite, 220 for composite). Matches the protocol in the VLABench paper. ## Policy inputs and outputs **Observations:** -- `observation.state` — 7-dim end-effector state (position xyz, euler xyz, gripper) -- `observation.images.image` — front camera view, 480x480 HWC uint8 -- `observation.images.second_image` — second camera view, 480x480 HWC uint8 -- `observation.images.wrist_image` — wrist camera view, 480x480 HWC uint8 +- `observation.state` — 7-dim end-effector state (position xyz + Euler xyz + gripper) +- `observation.images.image` — front camera, 480×480 HWC uint8 +- `observation.images.second_image` — second camera, 480×480 HWC uint8 +- `observation.images.wrist_image` — wrist camera, 480×480 HWC uint8 **Actions:** -- Continuous control in `Box(-1, 1, shape=(7,))` — 3D position + 3D euler orientation + 1D gripper - -### Recommended evaluation episodes - -For reproducible benchmarking, use **10 episodes per task**. For the full primitive suite this gives 210 episodes; for the full composite suite, 220 episodes. +- Continuous control in `Box(-1, 1, shape=(7,))` — 3D position + 3D Euler orientation + 1D gripper. ## Training ### Datasets -Pre-collected VLABench datasets in LeRobot format are available on the Hugging Face Hub: +Pre-collected VLABench datasets in LeRobot format on the Hub: -- Primitive tasks: [VLABench/vlabench_primitive_ft_lerobot_video](https://huggingface.co/datasets/VLABench/vlabench_primitive_ft_lerobot_video) (5,000 episodes, 128 tasks, 480x480 images) -- Composite tasks: [VLABench/vlabench_composite_ft_lerobot_video](https://huggingface.co/datasets/VLABench/vlabench_composite_ft_lerobot_video) (5,977 episodes, 167 tasks, 224x224 images) +- [`VLABench/vlabench_primitive_ft_lerobot_video`](https://huggingface.co/datasets/VLABench/vlabench_primitive_ft_lerobot_video) — 5,000 episodes, 128 tasks, 480×480 images. +- [`VLABench/vlabench_composite_ft_lerobot_video`](https://huggingface.co/datasets/VLABench/vlabench_composite_ft_lerobot_video) — 5,977 episodes, 167 tasks, 224×224 images. ### Example training command +Fine-tune a SmolVLA base on the primitive suite: + ```bash lerobot-train \ --policy.type=smolvla \ - --policy.repo_id=${HF_USER}/vlabench-test \ + --policy.repo_id=${HF_USER}/smolvla_vlabench_primitive \ --policy.load_vlm_weights=true \ + --policy.push_to_hub=true \ --dataset.repo_id=VLABench/vlabench_primitive_ft_lerobot_video \ --env.type=vlabench \ --env.task=select_fruit \ - --output_dir=./outputs/ \ + --output_dir=./outputs/smolvla_vlabench_primitive \ --steps=100000 \ --batch_size=4 \ + --eval_freq=5000 \ --eval.batch_size=1 \ --eval.n_episodes=1 \ - --eval_freq=1000 + --save_freq=10000 ``` + +## Reproducing published results + +The released checkpoint [`lerobot/smolvla_vlabench`](https://huggingface.co/lerobot/smolvla_vlabench) was trained on the primitive-suite dataset above and is evaluated with the [Single-task](#single-task-evaluation-recommended-for-quick-iteration) / [Suite-wide](#suite-wide-evaluation) commands. CI runs a 10-primitive-task smoke eval (one episode each) on every PR touching the benchmark.