mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-31 13:39:40 +00:00
Xingdong Zuo
6f2e71ec31
- Dataset section: compare lerobot/libero (1.9 GB, MP4) with HuggingFaceVLA/libero (69.9 GB, PNG-in-parquet) — same demonstrations and schema, equivalent loading throughput, 37x smaller download. - Training example: use lerobot/libero with video_backend=torchcodec. - Tips: pin --dataset.revision when reporting results; deterministic paired evaluation (seed, init_states, single batch per task); average over >=3 eval seeds. Co-authored-by: Xingdong Zuo <18168681+zuoxingdong@users.noreply.github.com>
202 lines
9.3 KiB
Plaintext
202 lines
9.3 KiB
Plaintext
# LIBERO
|
||
|
||
LIBERO is a benchmark designed to study **lifelong robot learning** — the idea that robots need to keep learning and adapting with their users over time, not just be pretrained once. It provides a set of standardized manipulation tasks that focus on **knowledge transfer**: how well a robot can apply what it has already learned to new situations. By evaluating on LIBERO, different algorithms can be compared fairly and researchers can build on each other's work.
|
||
|
||
- Paper: [Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://arxiv.org/abs/2306.03310)
|
||
- GitHub: [Lifelong-Robot-Learning/LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO)
|
||
- Project website: [libero-project.github.io](https://libero-project.github.io)
|
||
|
||
|
||
|
||
## Available tasks
|
||
|
||
LIBERO includes **five task suites** covering **130 tasks**, ranging from simple object manipulations to complex multi-step scenarios:
|
||
|
||
| Suite | CLI name | Tasks | Description |
|
||
| -------------- | ---------------- | ----- | -------------------------------------------------- |
|
||
| LIBERO-Spatial | `libero_spatial` | 10 | Tasks requiring reasoning about spatial relations |
|
||
| LIBERO-Object | `libero_object` | 10 | Tasks centered on manipulating different objects |
|
||
| LIBERO-Goal | `libero_goal` | 10 | Goal-conditioned tasks with changing targets |
|
||
| LIBERO-90 | `libero_90` | 90 | Short-horizon tasks from the LIBERO-100 collection |
|
||
| LIBERO-Long | `libero_10` | 10 | Long-horizon tasks from the LIBERO-100 collection |
|
||
|
||
## Installation
|
||
|
||
After following the LeRobot installation instructions:
|
||
|
||
```bash
|
||
pip install -e ".[libero]"
|
||
```
|
||
|
||
<Tip>
|
||
LIBERO requires Linux (`sys_platform == 'linux'`). LeRobot uses MuJoCo for simulation — set the rendering backend before training or evaluation:
|
||
|
||
```bash
|
||
export MUJOCO_GL=egl # for headless servers (HPC, cloud)
|
||
```
|
||
|
||
</Tip>
|
||
|
||
## Evaluation
|
||
|
||
### Default evaluation (recommended)
|
||
|
||
Evaluate across the four standard suites (10 episodes per task):
|
||
|
||
```bash
|
||
lerobot-eval \
|
||
--policy.path="your-policy-id" \
|
||
--env.type=libero \
|
||
--env.task=libero_spatial,libero_object,libero_goal,libero_10 \
|
||
--eval.batch_size=1 \
|
||
--eval.n_episodes=10 \
|
||
--env.max_parallel_tasks=1
|
||
```
|
||
|
||
### Single-suite evaluation
|
||
|
||
Evaluate on one LIBERO suite:
|
||
|
||
```bash
|
||
lerobot-eval \
|
||
--policy.path="your-policy-id" \
|
||
--env.type=libero \
|
||
--env.task=libero_object \
|
||
--eval.batch_size=2 \
|
||
--eval.n_episodes=3
|
||
```
|
||
|
||
- `--env.task` picks the suite (`libero_object`, `libero_spatial`, etc.).
|
||
- `--env.task_ids` restricts to specific task indices (`[0]`, `[1,2,3]`, etc.). Omit to run all tasks in the suite.
|
||
- `--eval.batch_size` controls how many environments run in parallel.
|
||
- `--eval.n_episodes` sets how many episodes to run per task.
|
||
|
||
### Multi-suite evaluation
|
||
|
||
Benchmark a policy across multiple suites at once by passing a comma-separated list:
|
||
|
||
```bash
|
||
lerobot-eval \
|
||
--policy.path="your-policy-id" \
|
||
--env.type=libero \
|
||
--env.task=libero_object,libero_spatial \
|
||
--eval.batch_size=1 \
|
||
--eval.n_episodes=2
|
||
```
|
||
|
||
### Control mode
|
||
|
||
LIBERO supports two control modes — `relative` (default) and `absolute`. Different VLA checkpoints are trained with different action parameterizations, so make sure the mode matches your policy:
|
||
|
||
```bash
|
||
--env.control_mode=relative # or "absolute"
|
||
```
|
||
|
||
### Policy inputs and outputs
|
||
|
||
**Observations:**
|
||
|
||
- `observation.state` — 8-dim proprioceptive features (eef position, axis-angle orientation, gripper qpos)
|
||
- `observation.images.image` — main camera view (`agentview_image`), HWC uint8
|
||
- `observation.images.image2` — wrist camera view (`robot0_eye_in_hand_image`), HWC uint8
|
||
|
||
<Tip warning={true}>
|
||
LeRobot enforces the `.images.*` prefix for visual features. Ensure your
|
||
policy config `input_features` use the same naming keys, and that your dataset
|
||
metadata keys follow this convention. If your data contains different keys,
|
||
you must rename the observations to match what the policy expects, since
|
||
naming keys are encoded inside the normalization statistics layer.
|
||
</Tip>
|
||
|
||
**Actions:**
|
||
|
||
- Continuous control in `Box(-1, 1, shape=(7,))` — 6D end-effector delta + 1D gripper
|
||
|
||
### Recommended evaluation episodes
|
||
|
||
For reproducible benchmarking, use **10 episodes per task** across all four standard suites (Spatial, Object, Goal, Long). This gives 400 total episodes and matches the protocol used for published results. Success rates may vary by a few percent across evaluation seeds, so we recommend averaging over 3 seeds.
|
||
|
||
<Tip>
|
||
To compare two policies on the same episodes, use the same `--seed`, keep
|
||
`--env.init_states=true`, and run each task in a single batch
|
||
(`--eval.batch_size` equal to episodes per task).
|
||
</Tip>
|
||
|
||
## Training
|
||
|
||
### Dataset
|
||
|
||
Two preprocessed LIBERO datasets are fully compatible with LeRobot. They contain the same demonstrations with the same schema and differ in how camera frames are stored:
|
||
|
||
| | [lerobot/libero](https://huggingface.co/datasets/lerobot/libero) | [HuggingFaceVLA/libero](https://huggingface.co/datasets/HuggingFaceVLA/libero) |
|
||
| ------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------ |
|
||
| episodes / frames / tasks | 1,693 / 273,465 / 40 | 1,693 / 273,465 / 40 |
|
||
| cameras | 2× 256×256×3 | 2× 256×256×3 |
|
||
| state / action dims | 8 / 7 | 8 / 7 |
|
||
| dataset format | v3.0 | v3.0 |
|
||
| camera encoding | MP4 video | PNG in parquet |
|
||
| download size | **1.9 GB** | 69.9 GB |
|
||
| extra dependency | video backend (`torchcodec` or `pyav`) | none |
|
||
|
||
**We recommend [lerobot/libero](https://huggingface.co/datasets/lerobot/libero)**: **37× smaller download** with **equivalent loading speed** (~330 samples/s per worker). Video re-encoding is slightly lossy; use the image-based variant if you cannot install a video decoding backend.
|
||
|
||
For reference, the original dataset published by Physical Intelligence:
|
||
|
||
- [physical-intelligence/libero](https://huggingface.co/datasets/physical-intelligence/libero)
|
||
|
||
<Tip>
|
||
Pin `--dataset.revision=<commit-sha>` when reporting results — Hub datasets can be re-uploaded, and success rates are only comparable against the same data revision.
|
||
</Tip>
|
||
|
||
### Example training command
|
||
|
||
Train SmolVLA on the recommended dataset:
|
||
|
||
```bash
|
||
lerobot-train \
|
||
--policy.type=smolvla \
|
||
--policy.load_vlm_weights=true \
|
||
--policy.push_to_hub=false \
|
||
--dataset.repo_id=lerobot/libero \
|
||
--dataset.video_backend=torchcodec \
|
||
--output_dir=./outputs/libero_smolvla \
|
||
--steps=100000 \
|
||
--batch_size=64
|
||
```
|
||
|
||
To share the result on the Hub, replace `--policy.push_to_hub=false` with `--policy.repo_id=${HF_USER}/libero-smolvla`. Evaluate saved checkpoints with `lerobot-eval` as shown in the [Evaluation](#evaluation) section.
|
||
|
||
## Reproducing published results
|
||
|
||
We reproduce the results of Pi0.5 on the LIBERO benchmark. We take the Physical Intelligence LIBERO base model (`pi05_libero`) and finetune for an additional 6k steps in bfloat16, with batch size of 256 on 8 H100 GPUs using the [HuggingFace LIBERO dataset](https://huggingface.co/datasets/HuggingFaceVLA/libero).
|
||
|
||
The finetuned model: [lerobot/pi05_libero_finetuned](https://huggingface.co/lerobot/pi05_libero_finetuned)
|
||
|
||
### Evaluation command
|
||
|
||
```bash
|
||
lerobot-eval \
|
||
--output_dir=./eval_logs/ \
|
||
--env.type=libero \
|
||
--env.task=libero_spatial,libero_object,libero_goal,libero_10 \
|
||
--eval.batch_size=1 \
|
||
--eval.n_episodes=10 \
|
||
--policy.path=pi05_libero_finetuned \
|
||
--policy.n_action_steps=10 \
|
||
--env.max_parallel_tasks=1
|
||
```
|
||
|
||
We set `n_action_steps=10`, matching the original OpenPI implementation.
|
||
|
||
### Results
|
||
|
||
| Model | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average |
|
||
| ------------------- | -------------- | ------------- | ----------- | --------- | -------- |
|
||
| **Pi0.5 (LeRobot)** | 97.0 | 99.0 | 98.0 | 96.0 | **97.5** |
|
||
|
||
These results are consistent with the [original results](https://github.com/Physical-Intelligence/openpi/tree/main/examples/libero#results) reported by Physical Intelligence:
|
||
|
||
| Model | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average |
|
||
| ------------------ | -------------- | ------------- | ----------- | --------- | --------- |
|
||
| **Pi0.5 (OpenPI)** | 98.8 | 98.2 | 98.0 | 92.4 | **96.85** |
|