From 87334c1be4a711b189b73c59dc25e6cde4e0b51d Mon Sep 17 00:00:00 2001 From: Pepijn Date: Fri, 17 Apr 2026 13:45:30 +0100 Subject: [PATCH] docs(robocasa): align page with adding_benchmarks template Rework docs/source/robocasa.mdx to follow the standard benchmark doc structure: intro + links + available tasks (with family breakdown and first-class benchmark-group shortcuts) + installation + eval + recommended episodes + policy I/O + training + reproducing results. - Fix the paper link (was pointing at a non-existent arxiv ID). - Surface lerobot/smolvla_robocasa and pepijn223/robocasa_CloseFridge in the top-of-page links so they're findable without reading the training section. - Add an explicit "Object registries" subsection explaining the `--env.obj_registries=[objaverse,lightwheel]` override path. - Add an explicit "Reproducing published results" section pointing at the CI smoke eval. Made-with: Cursor --- docs/source/robocasa.mdx | 132 ++++++++++++++++++++++----------------- 1 file changed, 76 insertions(+), 56 deletions(-) diff --git a/docs/source/robocasa.mdx b/docs/source/robocasa.mdx index db39e82f3..a24a6038e 100644 --- a/docs/source/robocasa.mdx +++ b/docs/source/robocasa.mdx @@ -1,33 +1,35 @@ # RoboCasa365 -RoboCasa365 is a large-scale simulation framework for training and benchmarking **generalist robots** in everyday kitchen tasks. It provides 365 diverse manipulation tasks across 2,500 kitchen environments, with over 3,200 object assets and 600+ hours of human demonstration data. The benchmark tests whether robots can handle the diversity and complexity of real-world household environments. +[RoboCasa365](https://robocasa.ai) is a large-scale simulation framework for training and benchmarking **generalist robots** in everyday kitchen tasks. It ships 365 diverse manipulation tasks across 2,500 kitchen environments, 3,200+ object assets and 600+ hours of human demonstration data, on a PandaOmron 12-DOF mobile manipulator (Franka arm on a holonomic base). -- Paper: [RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots](https://arxiv.org/abs/2603.04356) +- Paper: [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://arxiv.org/abs/2406.02523) - GitHub: [robocasa/robocasa](https://github.com/robocasa/robocasa) - Project website: [robocasa.ai](https://robocasa.ai) +- Pretrained policy: [`lerobot/smolvla_robocasa`](https://huggingface.co/lerobot/smolvla_robocasa) +- Single-task dataset (CloseFridge): [`pepijn223/robocasa_CloseFridge`](https://huggingface.co/datasets/pepijn223/robocasa_CloseFridge) ## Available tasks -RoboCasa365 includes **365 tasks** organized into atomic (single-skill) and composite (multi-step) categories: +RoboCasa365 organizes its 365 tasks into two families and three upstream benchmark groups that LeRobot exposes as first-class `--env.task` shortcuts: -| Category | Tasks | Description | +| Family | Tasks | Description | | --------- | ----- | ------------------------------------------------------------------------------- | | Atomic | ~65 | Single-skill tasks: pick-and-place, door/drawer manipulation, appliance control | | Composite | ~300 | Multi-step tasks across 60+ categories: cooking, cleaning, organizing, etc. | -**Atomic task examples:** `CloseFridge`, `OpenBlenderLid`, `PickPlaceCoffee`, `ManipulateStoveKnob`, `NavigateKitchen`, `TurnOnToaster` +**Atomic task examples:** `CloseFridge`, `OpenDrawer`, `OpenCabinet`, `TurnOnMicrowave`, `TurnOffStove`, `NavigateKitchen`, `PickPlaceCounterToStove`. -**Composite task categories:** baking, boiling, brewing, chopping food, clearing table, defrosting food, loading dishwasher, making tea, microwaving food, washing dishes, and many more. +**Composite task categories:** baking, boiling, brewing, chopping, clearing table, defrosting food, loading dishwasher, making tea, microwaving food, washing dishes, and more. `--env.task` accepts three forms: -- a single task name (e.g. `CloseFridge`) -- a comma-separated list of task names (e.g. `CloseFridge,OpenBlenderLid`) +- a single task name (`CloseFridge`) +- a comma-separated list (`CloseFridge,OpenBlenderLid,PickPlaceCoffee`) - a benchmark-group shortcut — `atomic_seen`, `composite_seen`, `composite_unseen`, `pretrain50`, `pretrain100`, `pretrain200`, `pretrain300` — which auto-expands to the upstream task list and auto-sets the dataset `split` (`target` or `pretrain`). ## Installation -RoboCasa and its dependency `robosuite` are not published on PyPI, and RoboCasa's own `setup.py` hardcodes `lerobot==0.3.3`, which conflicts with this repo's `lerobot`. Install them manually as editable clones (using `--no-deps` on `robocasa` to skip its shadowed `lerobot` pin): +RoboCasa and its dependency `robosuite` are not published on PyPI, and RoboCasa's own `setup.py` hardcodes `lerobot==0.3.3`, which conflicts with this repo's `lerobot`. LeRobot therefore does **not** expose a `robocasa` extra — install the two packages manually as editable clones (using `--no-deps` on `robocasa` to skip its shadowed `lerobot` pin): ```bash # After following the standard LeRobot installation instructions. @@ -37,26 +39,24 @@ git clone https://github.com/ARISE-Initiative/robosuite.git ~/robosuite pip install -e ~/robocasa --no-deps pip install -e ~/robosuite -# Robocasa's runtime deps (the ones its setup.py would have pulled, -# minus the bad lerobot pin). +# Robocasa's runtime deps (the ones its setup.py would have pulled, minus +# the bad lerobot pin). pip install numpy numba scipy mujoco pygame Pillow opencv-python \ pyyaml pynput tqdm termcolor imageio h5py lxml hidapi \ tianshou gymnasium python -m robocasa.scripts.setup_macros -# Lightweight assets only (lightwheel registry, ~2GB + the generative -# textures needed by the fixture XMLs). Enough for the default env out -# of the box. +# Lightweight assets (lightwheel object meshes + textures). Enough for +# the default env out of the box. python -m robocasa.scripts.download_kitchen_assets \ --type tex tex_generative fixtures_lw objs_lw # Optional: full objaverse/aigen registries (~30GB) for richer object -# variety. If you download these, pass them via `--env.obj_registries` -# at eval/train time (see below). +# variety. Enable at eval time via --env.obj_registries (see below). # python -m robocasa.scripts.download_kitchen_assets --type objs_objaverse ``` -RoboCasa365 requires MuJoCo for simulation. Set the rendering backend before training or evaluation: +RoboCasa requires MuJoCo. Set the rendering backend before training or evaluation: ```bash export MUJOCO_GL=egl # for headless servers (HPC, cloud) @@ -64,31 +64,78 @@ export MUJOCO_GL=egl # for headless servers (HPC, cloud) -By default the env samples objects only from the `lightwheel` registry (the one shipped by `--type objs_lw`), which avoids a `Probabilities contain NaN` crash when the objaverse/aigen packs are absent. If you've downloaded the full asset set, enable it with: +### Object registries + +By default the env samples objects only from the `lightwheel` registry (what `--type objs_lw` ships), which avoids a `Probabilities contain NaN` crash when the objaverse / aigen packs aren't on disk. If you've downloaded the full asset set, enable the full registry at runtime: ```bash --env.obj_registries='[objaverse,lightwheel]' ``` +## Evaluation + +### Single-task evaluation (recommended for quick iteration) + +```bash +lerobot-eval \ + --policy.path=lerobot/smolvla_robocasa \ + --env.type=robocasa \ + --env.task=CloseFridge \ + --eval.batch_size=1 \ + --eval.n_episodes=20 +``` + +### Multi-task evaluation + +Pass a comma-separated list of tasks: + +```bash +lerobot-eval \ + --policy.path=lerobot/smolvla_robocasa \ + --env.type=robocasa \ + --env.task=CloseFridge,OpenCabinet,OpenDrawer,TurnOnMicrowave,TurnOffStove \ + --eval.batch_size=1 \ + --eval.n_episodes=20 +``` + +### Benchmark-group evaluation + +Run an entire upstream group (e.g. all 18 `atomic_seen` tasks with `split=target`): + +```bash +lerobot-eval \ + --policy.path=lerobot/smolvla_robocasa \ + --env.type=robocasa \ + --env.task=atomic_seen \ + --eval.batch_size=1 \ + --eval.n_episodes=20 +``` + +### Recommended evaluation episodes + +**20 episodes per task** for reproducible benchmarking. Matches the protocol used in published results. + ## Policy inputs and outputs **Observations** (raw RoboCasa camera names are preserved verbatim): -- `observation.state` -- 16-dim proprioceptive state (base position, base quaternion, relative end-effector position, relative end-effector quaternion, gripper qpos) -- `observation.images.robot0_agentview_left` -- left agent view, 256x256 HWC uint8 -- `observation.images.robot0_eye_in_hand` -- wrist camera view, 256x256 HWC uint8 -- `observation.images.robot0_agentview_right` -- right agent view, 256x256 HWC uint8 +- `observation.state` — 16-dim proprioceptive state (base position, base quaternion, relative end-effector position, relative end-effector quaternion, gripper qpos) +- `observation.images.robot0_agentview_left` — left agent view, 256×256 HWC uint8 +- `observation.images.robot0_eye_in_hand` — wrist camera view, 256×256 HWC uint8 +- `observation.images.robot0_agentview_right` — right agent view, 256×256 HWC uint8 **Actions:** -- Continuous control in `Box(-1, 1, shape=(12,))` -- base motion (4D) + control mode (1D) + end-effector position (3D) + end-effector rotation (3D) + gripper (1D) +- Continuous control in `Box(-1, 1, shape=(12,))` — base motion (4D) + control mode (1D) + end-effector position (3D) + end-effector rotation (3D) + gripper (1D). -## Training on a single task +## Training -A ready-to-use single-task dataset is available on the Hub: +### Single-task example + +A ready-to-use single-task dataset is on the Hub: [`pepijn223/robocasa_CloseFridge`](https://huggingface.co/datasets/pepijn223/robocasa_CloseFridge). -Train a SmolVLA policy on `CloseFridge`: +Fine-tune a SmolVLA base on `CloseFridge`: ```bash lerobot-train \ @@ -108,7 +155,7 @@ lerobot-train \ --save_freq=10000 ``` -Evaluate the trained checkpoint on the same task: +Evaluate the resulting checkpoint: ```bash lerobot-eval \ @@ -119,33 +166,6 @@ lerobot-eval \ --eval.n_episodes=20 ``` -## Multi-task evaluation +## Reproducing published results -Evaluate across several tasks at once: - -```bash -lerobot-eval \ - --policy.path=your-policy-id \ - --env.type=robocasa \ - --env.task=CloseFridge,OpenBlenderLid,PickPlaceCoffee \ - --eval.batch_size=1 \ - --eval.n_episodes=20 -``` - -Or run a full benchmark group: - -```bash -lerobot-eval \ - --policy.path=your-policy-id \ - --env.type=robocasa \ - --env.task=atomic_seen \ - --eval.batch_size=1 \ - --eval.n_episodes=20 -``` - -- `--eval.batch_size` controls how many environments run in parallel. -- `--eval.n_episodes` sets how many episodes to run per task. - -### Recommended evaluation episodes - -For reproducible benchmarking, use **20 episodes per task**. This matches the protocol used in published results. +The released checkpoint [`lerobot/smolvla_robocasa`](https://huggingface.co/lerobot/smolvla_robocasa) is evaluated with the commands in the [Evaluation](#evaluation) section. CI runs a 10-atomic-task smoke eval (one episode each) on every PR touching the benchmark, picking fixture-centric tasks that don't require the objaverse asset pack.