mirror of
https://github.com/huggingface/lerobot.git
synced 2026-08-01 05:59:44 +00:00
Steven Palma
2d8f5f314e
* perf(libero): skip the discarded scene rebuild on reset LIBERO's OffScreenRenderEnv defaults to hard_reset=True, so every reset() frees the MjSim, re-serialises the scene with model.get_xml(), recompiles it with MjSim.from_xml_string(), constructs a fresh offscreen GL context and re-wires every observable. When init states are in use, LiberoEnv.reset() immediately calls set_init_state(), which overwrites the whole sim state -- so all of that work is discarded. This passes hard_reset=not init_states instead. Without init states the randomisation reset() performs is the only thing placing the objects, so the hard reset is kept. Measured on an RTX 3060 Ti (EGL, robosuite 1.4.0, mujoco 3.2.7, 256x256 x2 cameras), through LiberoEnv.reset(), fresh env per arm: suite hard soft saved libero_spatial 1697 ms 233 ms 1464 ms libero_object 1394 ms 172 ms 1222 ms libero_goal 1177 ms 164 ms 1013 ms libero_10 1528 ms 207 ms 1321 ms Equivalence ----------- Immediately after set_init_state, qpos, qvel, ctrl and act are bit-identical between the two paths on every suite tested. After the 10 settle steps that reset() runs, 9 of 41 qpos entries differ: robot0_joint1..7 (<= 2.4e-5 rad) and gripper0_finger_joint1/2 (<= 2.1e-4 rad). No object joint differs on any suite. The drift is driven by the gripper component of the settle action ([0,0,0,0,0,0,-1]); replacing it with zeros keeps the two paths bit-identical for 12 further steps, and with num_steps_wait=0 there is no divergence at all. Wrist-camera pixels can differ by up to ~87/255, because a sub-millimetre finger displacement crosses rasterisation boundaries at 256x256. The pixel metric badly overstates the physical difference here; 2.1e-4 rad is 0.012 degrees. So this is not bit-identical end to end, and reviewers should decide whether 0.012 degrees of gripper drift is acceptable for the benchmark. It does not change object placement, which is what the fixed init states exist to control. * refactor(env): config param libero + docs --------- Co-authored-by: Dimitar Dimitrov <dvdimitrov13@gmail.com>
216 lines
9.8 KiB
Plaintext
216 lines
9.8 KiB
Plaintext
# LIBERO
|
||
|
||
LIBERO is a benchmark designed to study **lifelong robot learning** — the idea that robots need to keep learning and adapting with their users over time, not just be pretrained once. It provides a set of standardized manipulation tasks that focus on **knowledge transfer**: how well a robot can apply what it has already learned to new situations. By evaluating on LIBERO, different algorithms can be compared fairly and researchers can build on each other's work.
|
||
|
||
- Paper: [Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://arxiv.org/abs/2306.03310)
|
||
- GitHub: [Lifelong-Robot-Learning/LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO)
|
||
- Project website: [libero-project.github.io](https://libero-project.github.io)
|
||
|
||
|
||
|
||
## Available tasks
|
||
|
||
LIBERO includes **five task suites** covering **130 tasks**, ranging from simple object manipulations to complex multi-step scenarios:
|
||
|
||
| Suite | CLI name | Tasks | Description |
|
||
| -------------- | ---------------- | ----- | -------------------------------------------------- |
|
||
| LIBERO-Spatial | `libero_spatial` | 10 | Tasks requiring reasoning about spatial relations |
|
||
| LIBERO-Object | `libero_object` | 10 | Tasks centered on manipulating different objects |
|
||
| LIBERO-Goal | `libero_goal` | 10 | Goal-conditioned tasks with changing targets |
|
||
| LIBERO-90 | `libero_90` | 90 | Short-horizon tasks from the LIBERO-100 collection |
|
||
| LIBERO-Long | `libero_10` | 10 | Long-horizon tasks from the LIBERO-100 collection |
|
||
|
||
## Installation
|
||
|
||
After following the LeRobot installation instructions:
|
||
|
||
```bash
|
||
pip install -e ".[libero]"
|
||
```
|
||
|
||
<Tip>
|
||
LIBERO requires Linux (`sys_platform == 'linux'`). LeRobot uses MuJoCo for simulation — set the rendering backend before training or evaluation:
|
||
|
||
```bash
|
||
export MUJOCO_GL=egl # for headless servers (HPC, cloud)
|
||
```
|
||
|
||
</Tip>
|
||
|
||
## Evaluation
|
||
|
||
### Default evaluation (recommended)
|
||
|
||
Evaluate across the four standard suites (10 episodes per task):
|
||
|
||
```bash
|
||
lerobot-eval \
|
||
--policy.path="your-policy-id" \
|
||
--env.type=libero \
|
||
--env.task=libero_spatial,libero_object,libero_goal,libero_10 \
|
||
--eval.batch_size=1 \
|
||
--eval.n_episodes=10 \
|
||
--env.max_parallel_tasks=1
|
||
```
|
||
|
||
### Single-suite evaluation
|
||
|
||
Evaluate on one LIBERO suite:
|
||
|
||
```bash
|
||
lerobot-eval \
|
||
--policy.path="your-policy-id" \
|
||
--env.type=libero \
|
||
--env.task=libero_object \
|
||
--eval.batch_size=2 \
|
||
--eval.n_episodes=3
|
||
```
|
||
|
||
- `--env.task` picks the suite (`libero_object`, `libero_spatial`, etc.).
|
||
- `--env.task_ids` restricts to specific task indices (`[0]`, `[1,2,3]`, etc.). Omit to run all tasks in the suite.
|
||
- `--eval.batch_size` controls how many environments run in parallel.
|
||
- `--eval.n_episodes` sets how many episodes to run per task.
|
||
|
||
### Multi-suite evaluation
|
||
|
||
Benchmark a policy across multiple suites at once by passing a comma-separated list:
|
||
|
||
```bash
|
||
lerobot-eval \
|
||
--policy.path="your-policy-id" \
|
||
--env.type=libero \
|
||
--env.task=libero_object,libero_spatial \
|
||
--eval.batch_size=1 \
|
||
--eval.n_episodes=2
|
||
```
|
||
|
||
### Control mode
|
||
|
||
LIBERO supports two control modes — `relative` (default) and `absolute`. Different VLA checkpoints are trained with different action parameterizations, so make sure the mode matches your policy:
|
||
|
||
```bash
|
||
--env.control_mode=relative # or "absolute"
|
||
```
|
||
|
||
### Reset performance
|
||
|
||
By default, LeRobot preserves LIBERO's hard-reset behavior. With fixed initial
|
||
states enabled, you can opt into soft resets to skip rebuilding the simulator
|
||
model and renderer on every episode:
|
||
|
||
```bash
|
||
--env.init_states=true --env.hard_reset=false
|
||
```
|
||
|
||
Soft resets are faster but are not bit-identical to hard resets after the
|
||
environment's settling steps, so camera observations and policy results may
|
||
differ slightly. Use hard resets when reproducing benchmark results.
|
||
|
||
### Policy inputs and outputs
|
||
|
||
**Observations:**
|
||
|
||
- `observation.state` — 8-dim proprioceptive features (eef position, axis-angle orientation, gripper qpos)
|
||
- `observation.images.image` — main camera view (`agentview_image`), HWC uint8
|
||
- `observation.images.image2` — wrist camera view (`robot0_eye_in_hand_image`), HWC uint8
|
||
|
||
<Tip warning={true}>
|
||
LeRobot enforces the `.images.*` prefix for visual features. Ensure your
|
||
policy config `input_features` use the same naming keys, and that your dataset
|
||
metadata keys follow this convention. If your data contains different keys,
|
||
you must rename the observations to match what the policy expects, since
|
||
naming keys are encoded inside the normalization statistics layer.
|
||
</Tip>
|
||
|
||
**Actions:**
|
||
|
||
- Continuous control in `Box(-1, 1, shape=(7,))` — 6D end-effector delta + 1D gripper
|
||
|
||
### Recommended evaluation episodes
|
||
|
||
For reproducible benchmarking, use **10 episodes per task** across all four standard suites (Spatial, Object, Goal, Long). This gives 400 total episodes and matches the protocol used for published results. Success rates may vary by a few percent across evaluation seeds, so we recommend averaging over 3 seeds.
|
||
|
||
<Tip>
|
||
To compare two policies on the same episodes, use the same `--seed`, keep
|
||
`--env.init_states=true`, and run each task in a single batch
|
||
(`--eval.batch_size` equal to episodes per task).
|
||
</Tip>
|
||
|
||
## Training
|
||
|
||
### Dataset
|
||
|
||
Two preprocessed LIBERO datasets are fully compatible with LeRobot. They contain the same demonstrations with the same schema and differ in how camera frames are stored:
|
||
|
||
| | [lerobot/libero](https://huggingface.co/datasets/lerobot/libero) | [HuggingFaceVLA/libero](https://huggingface.co/datasets/HuggingFaceVLA/libero) |
|
||
| ------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------ |
|
||
| episodes / frames / tasks | 1,693 / 273,465 / 40 | 1,693 / 273,465 / 40 |
|
||
| cameras | 2× 256×256×3 | 2× 256×256×3 |
|
||
| state / action dims | 8 / 7 | 8 / 7 |
|
||
| dataset format | v3.0 | v3.0 |
|
||
| camera encoding | MP4 video | PNG in parquet |
|
||
| download size | **1.9 GB** | 69.9 GB |
|
||
| extra dependency | video backend (`torchcodec` or `pyav`) | none |
|
||
|
||
**We recommend [lerobot/libero](https://huggingface.co/datasets/lerobot/libero)**: **37× smaller download** with **equivalent loading speed** (~330 samples/s per worker). Video re-encoding is slightly lossy; use the image-based variant if you cannot install a video decoding backend.
|
||
|
||
For reference, the original dataset published by Physical Intelligence:
|
||
|
||
- [physical-intelligence/libero](https://huggingface.co/datasets/physical-intelligence/libero)
|
||
|
||
<Tip>
|
||
Pin `--dataset.revision=<commit-sha>` when reporting results — Hub datasets can be re-uploaded, and success rates are only comparable against the same data revision.
|
||
</Tip>
|
||
|
||
### Example training command
|
||
|
||
Train SmolVLA on the recommended dataset:
|
||
|
||
```bash
|
||
lerobot-train \
|
||
--policy.type=smolvla \
|
||
--policy.load_vlm_weights=true \
|
||
--policy.push_to_hub=false \
|
||
--dataset.repo_id=lerobot/libero \
|
||
--dataset.video_backend=torchcodec \
|
||
--output_dir=./outputs/libero_smolvla \
|
||
--steps=100000 \
|
||
--batch_size=64
|
||
```
|
||
|
||
To share the result on the Hub, replace `--policy.push_to_hub=false` with `--policy.repo_id=${HF_USER}/libero-smolvla`. Evaluate saved checkpoints with `lerobot-eval` as shown in the [Evaluation](#evaluation) section.
|
||
|
||
## Reproducing published results
|
||
|
||
We reproduce the results of Pi0.5 on the LIBERO benchmark. We take the Physical Intelligence LIBERO base model (`pi05_libero`) and finetune for an additional 6k steps in bfloat16, with batch size of 256 on 8 H100 GPUs using the [HuggingFace LIBERO dataset](https://huggingface.co/datasets/HuggingFaceVLA/libero).
|
||
|
||
The finetuned model: [lerobot/pi05_libero_finetuned](https://huggingface.co/lerobot/pi05_libero_finetuned)
|
||
|
||
### Evaluation command
|
||
|
||
```bash
|
||
lerobot-eval \
|
||
--output_dir=./eval_logs/ \
|
||
--env.type=libero \
|
||
--env.task=libero_spatial,libero_object,libero_goal,libero_10 \
|
||
--eval.batch_size=1 \
|
||
--eval.n_episodes=10 \
|
||
--policy.path=pi05_libero_finetuned \
|
||
--policy.n_action_steps=10 \
|
||
--env.max_parallel_tasks=1
|
||
```
|
||
|
||
We set `n_action_steps=10`, matching the original OpenPI implementation.
|
||
|
||
### Results
|
||
|
||
| Model | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average |
|
||
| ------------------- | -------------- | ------------- | ----------- | --------- | -------- |
|
||
| **Pi0.5 (LeRobot)** | 97.0 | 99.0 | 98.0 | 96.0 | **97.5** |
|
||
|
||
These results are consistent with the [original results](https://github.com/Physical-Intelligence/openpi/tree/main/examples/libero#results) reported by Physical Intelligence:
|
||
|
||
| Model | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average |
|
||
| ------------------ | -------------- | ------------- | ----------- | --------- | --------- |
|
||
| **Pi0.5 (OpenPI)** | 98.8 | 98.2 | 98.0 | 92.4 | **96.85** |
|