mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-26 11:16:00 +00:00
Pepijn
5a065c8f87
robocasa's upstream setup.py hardcodes `lerobot==0.3.3` in install_requires. Exposing it as the `lerobot[robocasa]` extra made uv's dep resolver cycle: `lerobot[robocasa]` -> robocasa -> lerobot (a different version) -> unsolvable. This broke every `uv sync` — even invocations with an unrelated extra like `--extra test` — because uv validates the whole lockfile graph. - Remove the `robocasa` extra from pyproject.toml. Installation instructions in docs/source/robocasa.mdx now walk users through the manual `git clone` + `pip install --no-deps` flow, which matches what the Docker image already does and sidesteps the cyclic dep entirely. - Dockerfile: `uv pip install -e ~/robocasa --no-deps` so the shadowed lerobot==0.3.3 never lands in the image; install robocasa's actual runtime deps (numpy, numba, scipy, mujoco, tianshou, etc.) explicitly. Made-with: Cursor
152 lines
6.1 KiB
Plaintext
152 lines
6.1 KiB
Plaintext
# RoboCasa365
|
|
|
|
RoboCasa365 is a large-scale simulation framework for training and benchmarking **generalist robots** in everyday kitchen tasks. It provides 365 diverse manipulation tasks across 2,500 kitchen environments, with over 3,200 object assets and 600+ hours of human demonstration data. The benchmark tests whether robots can handle the diversity and complexity of real-world household environments.
|
|
|
|
- Paper: [RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots](https://arxiv.org/abs/2603.04356)
|
|
- GitHub: [robocasa/robocasa](https://github.com/robocasa/robocasa)
|
|
- Project website: [robocasa.ai](https://robocasa.ai)
|
|
|
|
## Available tasks
|
|
|
|
RoboCasa365 includes **365 tasks** organized into atomic (single-skill) and composite (multi-step) categories:
|
|
|
|
| Category | Tasks | Description |
|
|
| --------- | ----- | ------------------------------------------------------------------------------- |
|
|
| Atomic | ~65 | Single-skill tasks: pick-and-place, door/drawer manipulation, appliance control |
|
|
| Composite | ~300 | Multi-step tasks across 60+ categories: cooking, cleaning, organizing, etc. |
|
|
|
|
**Atomic task examples:** `CloseFridge`, `OpenBlenderLid`, `PickPlaceCoffee`, `ManipulateStoveKnob`, `NavigateKitchen`, `TurnOnToaster`
|
|
|
|
**Composite task categories:** baking, boiling, brewing, chopping food, clearing table, defrosting food, loading dishwasher, making tea, microwaving food, washing dishes, and many more.
|
|
|
|
`--env.task` accepts three forms:
|
|
|
|
- a single task name (e.g. `CloseFridge`)
|
|
- a comma-separated list of task names (e.g. `CloseFridge,OpenBlenderLid`)
|
|
- a benchmark-group shortcut — `atomic_seen`, `composite_seen`, `composite_unseen`, `pretrain50`, `pretrain100`, `pretrain200`, `pretrain300` — which auto-expands to the upstream task list and auto-sets the dataset `split` (`target` or `pretrain`).
|
|
|
|
## Installation
|
|
|
|
RoboCasa and its dependency `robosuite` are not published on PyPI, and RoboCasa's own `setup.py` hardcodes `lerobot==0.3.3`, which conflicts with this repo's `lerobot`. Install them manually as editable clones (using `--no-deps` on `robocasa` to skip its shadowed `lerobot` pin):
|
|
|
|
```bash
|
|
# After following the standard LeRobot installation instructions.
|
|
|
|
git clone https://github.com/robocasa/robocasa.git ~/robocasa
|
|
git clone https://github.com/ARISE-Initiative/robosuite.git ~/robosuite
|
|
pip install -e ~/robocasa --no-deps
|
|
pip install -e ~/robosuite
|
|
|
|
# Robocasa's runtime deps (the ones its setup.py would have pulled,
|
|
# minus the bad lerobot pin).
|
|
pip install numpy numba scipy mujoco pygame Pillow opencv-python \
|
|
pyyaml pynput tqdm termcolor imageio h5py lxml hidapi \
|
|
tianshou gymnasium
|
|
|
|
python -m robocasa.scripts.setup_macros
|
|
# Lightweight assets only (lightwheel registry, ~2GB + the generative
|
|
# textures needed by the fixture XMLs). Enough for the default env out
|
|
# of the box.
|
|
python -m robocasa.scripts.download_kitchen_assets \
|
|
--type tex tex_generative fixtures_lw objs_lw
|
|
# Optional: full objaverse/aigen registries (~30GB) for richer object
|
|
# variety. If you download these, pass them via `--env.obj_registries`
|
|
# at eval/train time (see below).
|
|
# python -m robocasa.scripts.download_kitchen_assets --type objs_objaverse
|
|
```
|
|
|
|
<Tip>
|
|
RoboCasa365 requires MuJoCo for simulation. Set the rendering backend before training or evaluation:
|
|
|
|
```bash
|
|
export MUJOCO_GL=egl # for headless servers (HPC, cloud)
|
|
```
|
|
|
|
</Tip>
|
|
|
|
By default the env samples objects only from the `lightwheel` registry (the one shipped by `--type objs_lw`), which avoids a `Probabilities contain NaN` crash when the objaverse/aigen packs are absent. If you've downloaded the full asset set, enable it with:
|
|
|
|
```bash
|
|
--env.obj_registries='[objaverse,lightwheel]'
|
|
```
|
|
|
|
## Policy inputs and outputs
|
|
|
|
**Observations** (raw RoboCasa camera names are preserved verbatim):
|
|
|
|
- `observation.state` -- 16-dim proprioceptive state (base position, base quaternion, relative end-effector position, relative end-effector quaternion, gripper qpos)
|
|
- `observation.images.robot0_agentview_left` -- left agent view, 256x256 HWC uint8
|
|
- `observation.images.robot0_eye_in_hand` -- wrist camera view, 256x256 HWC uint8
|
|
- `observation.images.robot0_agentview_right` -- right agent view, 256x256 HWC uint8
|
|
|
|
**Actions:**
|
|
|
|
- Continuous control in `Box(-1, 1, shape=(12,))` -- base motion (4D) + control mode (1D) + end-effector position (3D) + end-effector rotation (3D) + gripper (1D)
|
|
|
|
## Training on a single task
|
|
|
|
A ready-to-use single-task dataset is available on the Hub:
|
|
[`pepijn223/robocasa_CloseFridge`](https://huggingface.co/datasets/pepijn223/robocasa_CloseFridge).
|
|
|
|
Train a SmolVLA policy on `CloseFridge`:
|
|
|
|
```bash
|
|
lerobot-train \
|
|
--policy.type=smolvla \
|
|
--policy.repo_id=${HF_USER}/smolvla_robocasa_CloseFridge \
|
|
--policy.load_vlm_weights=true \
|
|
--policy.push_to_hub=true \
|
|
--dataset.repo_id=pepijn223/robocasa_CloseFridge \
|
|
--env.type=robocasa \
|
|
--env.task=CloseFridge \
|
|
--output_dir=./outputs/smolvla_robocasa_CloseFridge \
|
|
--steps=100000 \
|
|
--batch_size=4 \
|
|
--eval_freq=5000 \
|
|
--eval.batch_size=1 \
|
|
--eval.n_episodes=5 \
|
|
--save_freq=10000
|
|
```
|
|
|
|
Evaluate the trained checkpoint on the same task:
|
|
|
|
```bash
|
|
lerobot-eval \
|
|
--policy.path=${HF_USER}/smolvla_robocasa_CloseFridge \
|
|
--env.type=robocasa \
|
|
--env.task=CloseFridge \
|
|
--eval.batch_size=1 \
|
|
--eval.n_episodes=20
|
|
```
|
|
|
|
## Multi-task evaluation
|
|
|
|
Evaluate across several tasks at once:
|
|
|
|
```bash
|
|
lerobot-eval \
|
|
--policy.path=your-policy-id \
|
|
--env.type=robocasa \
|
|
--env.task=CloseFridge,OpenBlenderLid,PickPlaceCoffee \
|
|
--eval.batch_size=1 \
|
|
--eval.n_episodes=20
|
|
```
|
|
|
|
Or run a full benchmark group:
|
|
|
|
```bash
|
|
lerobot-eval \
|
|
--policy.path=your-policy-id \
|
|
--env.type=robocasa \
|
|
--env.task=atomic_seen \
|
|
--eval.batch_size=1 \
|
|
--eval.n_episodes=20
|
|
```
|
|
|
|
- `--eval.batch_size` controls how many environments run in parallel.
|
|
- `--eval.n_episodes` sets how many episodes to run per task.
|
|
|
|
### Recommended evaluation episodes
|
|
|
|
For reproducible benchmarking, use **20 episodes per task**. This matches the protocol used in published results.
|