mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-26 11:16:00 +00:00
feat(envs): add RoboCasa365 benchmark integration
Add RoboCasa365 (arXiv:2603.04356) as a new simulation benchmark with 365 everyday kitchen manipulation tasks across 2,500 diverse environments. New files: - src/lerobot/envs/robocasa.py: gym.Env wrapper with deferred env creation, flat 12D action / 16D state vectors, 3-camera support - docs/source/robocasa.mdx: user-facing documentation - docker/Dockerfile.benchmark.robocasa: CI benchmark image Modified files: - src/lerobot/envs/configs.py: RoboCasaEnv config (--env.type=robocasa) - pyproject.toml: robocasa optional dependency group - docs/source/_toctree.yml: sidebar entry - .github/workflows/benchmark_tests.yml: integration test job Refs: https://arxiv.org/abs/2603.04356, https://robocasa.ai Related: huggingface/lerobot#321 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -79,6 +79,8 @@
|
||||
title: LIBERO
|
||||
- local: metaworld
|
||||
title: Meta-World
|
||||
- local: robocasa
|
||||
title: RoboCasa365
|
||||
- local: envhub_isaaclab_arena
|
||||
title: NVIDIA IsaacLab Arena Environments
|
||||
title: "Benchmarks"
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
# RoboCasa365
|
||||
|
||||
RoboCasa365 is a large-scale simulation framework for training and benchmarking **generalist robots** in everyday kitchen tasks. It provides 365 diverse manipulation tasks across 2,500 kitchen environments, with over 3,200 object assets and 600+ hours of human demonstration data. The benchmark tests whether robots can handle the diversity and complexity of real-world household environments.
|
||||
|
||||
- Paper: [RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots](https://arxiv.org/abs/2603.04356)
|
||||
- GitHub: [robocasa/robocasa](https://github.com/robocasa/robocasa)
|
||||
- Project website: [robocasa.ai](https://robocasa.ai)
|
||||
|
||||
## Available tasks
|
||||
|
||||
RoboCasa365 includes **365 tasks** organized into atomic (single-skill) and composite (multi-step) categories:
|
||||
|
||||
| Category | Tasks | Description |
|
||||
| --------- | ----- | ------------------------------------------------------------------------------- |
|
||||
| Atomic | ~65 | Single-skill tasks: pick-and-place, door/drawer manipulation, appliance control |
|
||||
| Composite | ~300 | Multi-step tasks across 60+ categories: cooking, cleaning, organizing, etc. |
|
||||
|
||||
**Atomic task examples:** `CloseFridge`, `OpenBlenderLid`, `PickPlaceCoffee`, `ManipulateStoveKnob`, `NavigateKitchen`, `TurnOnToaster`
|
||||
|
||||
**Composite task categories:** baking, boiling, brewing, chopping food, clearing table, defrosting food, loading dishwasher, making tea, microwaving food, washing dishes, and many more.
|
||||
|
||||
Pass individual task class names directly as `--env.task` (e.g., `CloseFridge`, `PickPlaceCoffee`). Multiple tasks can be comma-separated for multi-task evaluation.
|
||||
|
||||
## Installation
|
||||
|
||||
After following the LeRobot installation instructions:
|
||||
|
||||
```bash
|
||||
pip install -e ".[robocasa]"
|
||||
python -m robocasa.scripts.setup_macros
|
||||
python -m robocasa.scripts.download_kitchen_assets
|
||||
```
|
||||
|
||||
<Tip>
|
||||
RoboCasa365 requires MuJoCo for simulation. Set the rendering backend before training or evaluation:
|
||||
|
||||
```bash
|
||||
export MUJOCO_GL=egl # for headless servers (HPC, cloud)
|
||||
```
|
||||
|
||||
</Tip>
|
||||
|
||||
## Evaluation
|
||||
|
||||
### Single-task evaluation
|
||||
|
||||
Evaluate a policy on a single RoboCasa task:
|
||||
|
||||
```bash
|
||||
lerobot-eval \
|
||||
--policy.path="your-policy-id" \
|
||||
--env.type=robocasa \
|
||||
--env.task=CloseFridge \
|
||||
--eval.batch_size=1 \
|
||||
--eval.n_episodes=20
|
||||
```
|
||||
|
||||
### Multi-task evaluation
|
||||
|
||||
Evaluate across multiple tasks at once by passing a comma-separated list:
|
||||
|
||||
```bash
|
||||
lerobot-eval \
|
||||
--policy.path="your-policy-id" \
|
||||
--env.type=robocasa \
|
||||
--env.task=CloseFridge,OpenBlenderLid,PickPlaceCoffee \
|
||||
--eval.batch_size=1 \
|
||||
--eval.n_episodes=20
|
||||
```
|
||||
|
||||
- `--env.task` accepts individual task names (comma-separated).
|
||||
- `--eval.batch_size` controls how many environments run in parallel.
|
||||
- `--eval.n_episodes` sets how many episodes to run per task.
|
||||
|
||||
### Policy inputs and outputs
|
||||
|
||||
**Observations:**
|
||||
|
||||
- `observation.state` -- 16-dim proprioceptive state (base position, base quaternion, relative end-effector position, relative end-effector quaternion, gripper qpos)
|
||||
- `observation.images.image` -- left agent view (`robot0_agentview_left`), 256x256 HWC uint8
|
||||
- `observation.images.image2` -- wrist camera view (`robot0_eye_in_hand`), 256x256 HWC uint8
|
||||
- `observation.images.image3` -- right agent view (`robot0_agentview_right`), 256x256 HWC uint8
|
||||
|
||||
**Actions:**
|
||||
|
||||
- Continuous control in `Box(-1, 1, shape=(12,))` -- base motion (4D) + control mode (1D) + end-effector position (3D) + end-effector rotation (3D) + gripper (1D)
|
||||
|
||||
### Recommended evaluation episodes
|
||||
|
||||
For reproducible benchmarking, use **20 episodes per task**. This matches the protocol used in published results.
|
||||
|
||||
## Training
|
||||
|
||||
### Example training command
|
||||
|
||||
```bash
|
||||
lerobot-train \
|
||||
--policy.type=smolvla \
|
||||
--policy.repo_id=${HF_USER}/robocasa-test \
|
||||
--policy.load_vlm_weights=true \
|
||||
--dataset.repo_id=your-robocasa-dataset \
|
||||
--env.type=robocasa \
|
||||
--env.task=CloseFridge \
|
||||
--output_dir=./outputs/ \
|
||||
--steps=100000 \
|
||||
--batch_size=4 \
|
||||
--eval.batch_size=1 \
|
||||
--eval.n_episodes=1 \
|
||||
--eval_freq=1000
|
||||
```
|
||||
Reference in New Issue
Block a user