admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-26 11:16:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(robocasa): align page with adding_benchmarks template

Browse Source 
Rework docs/source/robocasa.mdx to follow the standard benchmark doc
structure: intro + links + available tasks (with family breakdown and
first-class benchmark-group shortcuts) + installation + eval +
recommended episodes + policy I/O + training + reproducing results.

- Fix the paper link (was pointing at a non-existent arxiv ID).
- Surface lerobot/smolvla_robocasa and pepijn223/robocasa_CloseFridge
  in the top-of-page links so they're findable without reading the
  training section.
- Add an explicit "Object registries" subsection explaining the
  `--env.obj_registries=[objaverse,lightwheel]` override path.
- Add an explicit "Reproducing published results" section pointing
  at the CI smoke eval.

Made-with: Cursor
This commit is contained in:
Pepijn
2026-04-17 13:45:30 +01:00
parent 5a065c8f87
commit 87334c1be4
1 changed files with 76 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/robocasa.mdx
+76 -56
View File
@@ -1,33 +1,35 @@
# RoboCasa365 # RoboCasa365
RoboCasa365 is a large-scale simulation framework for training and benchmarking **generalist robots** in everyday kitchen tasks. It provides 365 diverse manipulation tasks across 2,500 kitchen environments, with over 3,200 object assets and 600+ hours of human demonstration data. The benchmark tests whether robots can handle the diversity and complexity of real-world household environments. [RoboCasa365](https://robocasa.ai) is a large-scale simulation framework for training and benchmarking **generalist robots** in everyday kitchen tasks. It ships 365 diverse manipulation tasks across 2,500 kitchen environments, 3,200+ object assets and 600+ hours of human demonstration data, on a PandaOmron 12-DOF mobile manipulator (Franka arm on a holonomic base).
- Paper: [RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots](https://arxiv.org/abs/2603.04356) - Paper: [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://arxiv.org/abs/2406.02523)
- GitHub: [robocasa/robocasa](https://github.com/robocasa/robocasa) - GitHub: [robocasa/robocasa](https://github.com/robocasa/robocasa)
- Project website: [robocasa.ai](https://robocasa.ai) - Project website: [robocasa.ai](https://robocasa.ai)
- Pretrained policy: [`lerobot/smolvla_robocasa`](https://huggingface.co/lerobot/smolvla_robocasa)
- Single-task dataset (CloseFridge): [`pepijn223/robocasa_CloseFridge`](https://huggingface.co/datasets/pepijn223/robocasa_CloseFridge)
## Available tasks ## Available tasks
RoboCasa365 includes **365 tasks** organized into atomic (single-skill) and composite (multi-step) categories: RoboCasa365 organizes its 365 tasks into two families and three upstream benchmark groups that LeRobot exposes as first-class `--env.task` shortcuts:
| Category | Tasks | Description | | Family | Tasks | Description |
| --------- | ----- | ------------------------------------------------------------------------------- | | --------- | ----- | ------------------------------------------------------------------------------- |
| Atomic | ~65 | Single-skill tasks: pick-and-place, door/drawer manipulation, appliance control | | Atomic | ~65 | Single-skill tasks: pick-and-place, door/drawer manipulation, appliance control |
| Composite | ~300 | Multi-step tasks across 60+ categories: cooking, cleaning, organizing, etc. | | Composite | ~300 | Multi-step tasks across 60+ categories: cooking, cleaning, organizing, etc. |
**Atomic task examples:** `CloseFridge`, `OpenBlenderLid`, `PickPlaceCoffee`, `ManipulateStoveKnob`, `NavigateKitchen`, `TurnOnToaster` **Atomic task examples:** `CloseFridge`, `OpenDrawer`, `OpenCabinet`, `TurnOnMicrowave`, `TurnOffStove`, `NavigateKitchen`, `PickPlaceCounterToStove`.
**Composite task categories:** baking, boiling, brewing, chopping food, clearing table, defrosting food, loading dishwasher, making tea, microwaving food, washing dishes, and many more. **Composite task categories:** baking, boiling, brewing, chopping, clearing table, defrosting food, loading dishwasher, making tea, microwaving food, washing dishes, and more.
`--env.task` accepts three forms: `--env.task` accepts three forms:
- a single task name (e.g. `CloseFridge`) - a single task name (`CloseFridge`)
- a comma-separated list of task names (e.g. `CloseFridge,OpenBlenderLid`) - a comma-separated list (`CloseFridge,OpenBlenderLid,PickPlaceCoffee`)
- a benchmark-group shortcut — `atomic_seen`, `composite_seen`, `composite_unseen`, `pretrain50`, `pretrain100`, `pretrain200`, `pretrain300` — which auto-expands to the upstream task list and auto-sets the dataset `split` (`target` or `pretrain`). - a benchmark-group shortcut — `atomic_seen`, `composite_seen`, `composite_unseen`, `pretrain50`, `pretrain100`, `pretrain200`, `pretrain300` — which auto-expands to the upstream task list and auto-sets the dataset `split` (`target` or `pretrain`).
## Installation ## Installation
RoboCasa and its dependency `robosuite` are not published on PyPI, and RoboCasa's own `setup.py` hardcodes `lerobot==0.3.3`, which conflicts with this repo's `lerobot`. Install them manually as editable clones (using `--no-deps` on `robocasa` to skip its shadowed `lerobot` pin): RoboCasa and its dependency `robosuite` are not published on PyPI, and RoboCasa's own `setup.py` hardcodes `lerobot==0.3.3`, which conflicts with this repo's `lerobot`. LeRobot therefore does **not** expose a `robocasa` extra — install the two packages manually as editable clones (using `--no-deps` on `robocasa` to skip its shadowed `lerobot` pin):
```bash ```bash
# After following the standard LeRobot installation instructions. # After following the standard LeRobot installation instructions.
@@ -37,26 +39,24 @@ git clone https://github.com/ARISE-Initiative/robosuite.git ~/robosuite
pip install -e ~/robocasa --no-deps pip install -e ~/robocasa --no-deps
pip install -e ~/robosuite pip install -e ~/robosuite
# Robocasa's runtime deps (the ones its setup.py would have pulled, # Robocasa's runtime deps (the ones its setup.py would have pulled, minus
# minus the bad lerobot pin). # the bad lerobot pin).
pip install numpy numba scipy mujoco pygame Pillow opencv-python \ pip install numpy numba scipy mujoco pygame Pillow opencv-python \
pyyaml pynput tqdm termcolor imageio h5py lxml hidapi \ pyyaml pynput tqdm termcolor imageio h5py lxml hidapi \
tianshou gymnasium tianshou gymnasium
python -m robocasa.scripts.setup_macros python -m robocasa.scripts.setup_macros
# Lightweight assets only (lightwheel registry, ~2GB + the generative # Lightweight assets (lightwheel object meshes + textures). Enough for
# textures needed by the fixture XMLs). Enough for the default env out # the default env out of the box.
# of the box.
python -m robocasa.scripts.download_kitchen_assets \ python -m robocasa.scripts.download_kitchen_assets \
--type tex tex_generative fixtures_lw objs_lw --type tex tex_generative fixtures_lw objs_lw
# Optional: full objaverse/aigen registries (~30GB) for richer object # Optional: full objaverse/aigen registries (~30GB) for richer object
# variety. If you download these, pass them via `--env.obj_registries` # variety. Enable at eval time via --env.obj_registries (see below).
# at eval/train time (see below).
# python -m robocasa.scripts.download_kitchen_assets --type objs_objaverse # python -m robocasa.scripts.download_kitchen_assets --type objs_objaverse
``` ```
<Tip> <Tip>
RoboCasa365 requires MuJoCo for simulation. Set the rendering backend before training or evaluation: RoboCasa requires MuJoCo. Set the rendering backend before training or evaluation:
```bash ```bash
export MUJOCO_GL=egl # for headless servers (HPC, cloud) export MUJOCO_GL=egl # for headless servers (HPC, cloud)
@@ -64,31 +64,78 @@ export MUJOCO_GL=egl # for headless servers (HPC, cloud)
</Tip> </Tip>
By default the env samples objects only from the `lightwheel` registry (the one shipped by `--type objs_lw`), which avoids a `Probabilities contain NaN` crash when the objaverse/aigen packs are absent. If you've downloaded the full asset set, enable it with: ### Object registries
By default the env samples objects only from the `lightwheel` registry (what `--type objs_lw` ships), which avoids a `Probabilities contain NaN` crash when the objaverse / aigen packs aren't on disk. If you've downloaded the full asset set, enable the full registry at runtime:
```bash ```bash
--env.obj_registries='[objaverse,lightwheel]' --env.obj_registries='[objaverse,lightwheel]'
``` ```
## Evaluation
### Single-task evaluation (recommended for quick iteration)
```bash
lerobot-eval \
--policy.path=lerobot/smolvla_robocasa \
--env.type=robocasa \
--env.task=CloseFridge \
--eval.batch_size=1 \
--eval.n_episodes=20
```
### Multi-task evaluation
Pass a comma-separated list of tasks:
```bash
lerobot-eval \
--policy.path=lerobot/smolvla_robocasa \
--env.type=robocasa \
--env.task=CloseFridge,OpenCabinet,OpenDrawer,TurnOnMicrowave,TurnOffStove \
--eval.batch_size=1 \
--eval.n_episodes=20
```
### Benchmark-group evaluation
Run an entire upstream group (e.g. all 18 `atomic_seen` tasks with `split=target`):
```bash
lerobot-eval \
--policy.path=lerobot/smolvla_robocasa \
--env.type=robocasa \
--env.task=atomic_seen \
--eval.batch_size=1 \
--eval.n_episodes=20
```
### Recommended evaluation episodes
**20 episodes per task** for reproducible benchmarking. Matches the protocol used in published results.
## Policy inputs and outputs ## Policy inputs and outputs
**Observations** (raw RoboCasa camera names are preserved verbatim): **Observations** (raw RoboCasa camera names are preserved verbatim):
- `observation.state` -- 16-dim proprioceptive state (base position, base quaternion, relative end-effector position, relative end-effector quaternion, gripper qpos) - `observation.state` — 16-dim proprioceptive state (base position, base quaternion, relative end-effector position, relative end-effector quaternion, gripper qpos)
- `observation.images.robot0_agentview_left` -- left agent view, 256x256 HWC uint8 - `observation.images.robot0_agentview_left` — left agent view, 256×256 HWC uint8
- `observation.images.robot0_eye_in_hand` -- wrist camera view, 256x256 HWC uint8 - `observation.images.robot0_eye_in_hand` — wrist camera view, 256×256 HWC uint8
- `observation.images.robot0_agentview_right` -- right agent view, 256x256 HWC uint8 - `observation.images.robot0_agentview_right` — right agent view, 256×256 HWC uint8
**Actions:** **Actions:**
- Continuous control in `Box(-1, 1, shape=(12,))` -- base motion (4D) + control mode (1D) + end-effector position (3D) + end-effector rotation (3D) + gripper (1D) - Continuous control in `Box(-1, 1, shape=(12,))` — base motion (4D) + control mode (1D) + end-effector position (3D) + end-effector rotation (3D) + gripper (1D).
## Training on a single task ## Training
A ready-to-use single-task dataset is available on the Hub: ### Single-task example
A ready-to-use single-task dataset is on the Hub:
[`pepijn223/robocasa_CloseFridge`](https://huggingface.co/datasets/pepijn223/robocasa_CloseFridge). [`pepijn223/robocasa_CloseFridge`](https://huggingface.co/datasets/pepijn223/robocasa_CloseFridge).
Train a SmolVLA policy on `CloseFridge`: Fine-tune a SmolVLA base on `CloseFridge`:
```bash ```bash
lerobot-train \ lerobot-train \
@@ -108,7 +155,7 @@ lerobot-train \
--save_freq=10000 --save_freq=10000
``` ```
Evaluate the trained checkpoint on the same task: Evaluate the resulting checkpoint:
```bash ```bash
lerobot-eval \ lerobot-eval \
@@ -119,33 +166,6 @@ lerobot-eval \
--eval.n_episodes=20 --eval.n_episodes=20
``` ```
## Multi-task evaluation ## Reproducing published results
Evaluate across several tasks at once: The released checkpoint [`lerobot/smolvla_robocasa`](https://huggingface.co/lerobot/smolvla_robocasa) is evaluated with the commands in the [Evaluation](#evaluation) section. CI runs a 10-atomic-task smoke eval (one episode each) on every PR touching the benchmark, picking fixture-centric tasks that don't require the objaverse asset pack.
```bash
lerobot-eval \
--policy.path=your-policy-id \
--env.type=robocasa \
--env.task=CloseFridge,OpenBlenderLid,PickPlaceCoffee \
--eval.batch_size=1 \
--eval.n_episodes=20
```
Or run a full benchmark group:
```bash
lerobot-eval \
--policy.path=your-policy-id \
--env.type=robocasa \
--env.task=atomic_seen \
--eval.batch_size=1 \
--eval.n_episodes=20
```
- `--eval.batch_size` controls how many environments run in parallel.
- `--eval.n_episodes` sets how many episodes to run per task.
### Recommended evaluation episodes
For reproducible benchmarking, use **20 episodes per task**. This matches the protocol used in published results.