Remove open pi patch and use custom transformer branch for now

2026-05-20 11:09:59 +00:00 · 2025-09-23 17:03:23 +02:00
parent 969e8eeae1
commit d71f0d4a05
16 changed files with 60 additions and 6112 deletions
@@ -127,4 +127,44 @@ LeRobot uses MuJoCo for simulation. You need to set the rendering backend before

 ## Reproducing π₀ and π₀.₅ results

-We can also reproduce the results of π₀ and π₀.₅ on the Libero benchmark by using the finetuned libero models.
+We reproduce the results of π₀ and π₀.₅ on the LIBERO benchmark using the LeRobot implementation. We take the Physical Intelligence LIBERO base models (`pi0_libero` and `pi05_libero`) and finetune them for an additional 6k steps in bfloat16, with batch size of 256 on 8 H100 GPUs using the [HuggingFace LIBERO dataset](https://huggingface.co/datasets/HuggingFaceVLA/libero).
+
+The finetuned models can be found here:
+
+- **π₀ LIBERO**: [pepijn223/pi0_libero_fp32](https://huggingface.co/pepijn223/pi0_libero_fp32)
+- **π₀.₅ LIBERO**: [pepijn223/pi05_libero_fp32](https://huggingface.co/pepijn223/pi05_libero_fp32)
+
+We then evaluate the finetuned models using the LeRobot LIBERO implementation, by running the following command:
+
+```bash
+python src/lerobot/scripts/eval.py \
+  --output_dir=/logs/ \
+  --env.type=libero \
+  --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
+  --eval.batch_size=1 \
+  --eval.n_episodes=5 \
+  --policy.path=pepijn223/pi0_libero_fp32 \
+  --env.multitask_eval=true \
+  --output_dir=./eval_logs/ \
+  --policy.compile_model=false \
+  --policy.gradient_checkpointing=false \
+  --env.max_parallel_tasks=1
+```
+
+**Note:** We set `n_action_steps=10` which normally has a default value of 50.
+
+### Results
+
+We obtain the following results on the LIBERO benchmark:
+
+| Model    | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average |
+| -------- | -------------- | ------------- | ----------- | --------- | ------- |
+| **π₀**   | x              | x             | x           | x         | **x**   |
+| **π₀.₅** | 98.0           | 99.0          | 97.0        | 93.0      | **x**   |
+
+These results are consistent with the original results reported by Physical Intelligence:
+
+| Model    | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average   |
+| -------- | -------------- | ------------- | ----------- | --------- | --------- |
+| **π₀**   | 96.8           | 98.8          | 95.8        | 85.2      | **94.15** |
+| **π₀.₅** | 98.8           | 98.2          | 98.0        | 92.4      | **96.85** |