mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-31 21:49:45 +00:00
Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 2aba372b4e | |||
| c841a0c258 | |||
| a3ddba2454 |
@@ -19,8 +19,8 @@ on:
|
|||||||
workflow_dispatch:
|
workflow_dispatch:
|
||||||
|
|
||||||
# Runs at 02:00
|
# Runs at 02:00
|
||||||
# schedule:
|
schedule:
|
||||||
# - cron: "0 2 * * *"
|
- cron: "0 2 * * *"
|
||||||
|
|
||||||
env:
|
env:
|
||||||
CLOSE_ISSUE_MESSAGE: >
|
CLOSE_ISSUE_MESSAGE: >
|
||||||
@@ -31,7 +31,7 @@ env:
|
|||||||
Feel free to reopen if is still relevant, or to ping a collaborator if you have any questions.
|
Feel free to reopen if is still relevant, or to ping a collaborator if you have any questions.
|
||||||
WARN_ISSUE_MESSAGE: >
|
WARN_ISSUE_MESSAGE: >
|
||||||
This issue has been automatically marked as stale because it has not had
|
This issue has been automatically marked as stale because it has not had
|
||||||
recent activity (1 year). It will be closed if no further activity occurs.
|
recent activity (1 year). It will be closed if no further activity occurs within 30 days.
|
||||||
Any change, comment or update to this issue will reset this count.
|
Any change, comment or update to this issue will reset this count.
|
||||||
Thank you for your contributions.
|
Thank you for your contributions.
|
||||||
WARN_PR_MESSAGE: >
|
WARN_PR_MESSAGE: >
|
||||||
@@ -61,8 +61,8 @@ jobs:
|
|||||||
exempt-pr-labels: never-stale
|
exempt-pr-labels: never-stale
|
||||||
days-before-issue-stale: 365
|
days-before-issue-stale: 365
|
||||||
days-before-issue-close: 30
|
days-before-issue-close: 30
|
||||||
days-before-pr-stale: 365
|
days-before-pr-stale: -1
|
||||||
days-before-pr-close: 30
|
days-before-pr-close: -1
|
||||||
delete-branch: true
|
delete-branch: true
|
||||||
close-issue-message: ${{ env.CLOSE_ISSUE_MESSAGE }}
|
close-issue-message: ${{ env.CLOSE_ISSUE_MESSAGE }}
|
||||||
close-pr-message: ${{ env.CLOSE_PR_MESSAGE }}
|
close-pr-message: ${{ env.CLOSE_PR_MESSAGE }}
|
||||||
|
|||||||
+218
-11
@@ -23,18 +23,18 @@ The broader EVO1 project may include additional training scripts and dataset too
|
|||||||
2. Install EVO1 dependencies:
|
2. Install EVO1 dependencies:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install -e ".[evo1]"
|
pip install -e ".[training,evo1]"
|
||||||
```
|
```
|
||||||
|
|
||||||
For LIBERO evaluation, install the LIBERO extra as well:
|
For LIBERO training and evaluation, install the LIBERO extra as well:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install -e ".[evo1,libero]"
|
pip install -e ".[training,evo1,libero]"
|
||||||
```
|
```
|
||||||
|
|
||||||
3. Install a `flash-attn` wheel only if it is compatible with your Python, PyTorch, CUDA, and GPU stack. EVO1 falls back to standard attention when `flash_attn` is not available.
|
3. Install a `flash-attn` wheel only if it is compatible with your Python, PyTorch, CUDA, and GPU stack. EVO1 falls back to standard attention when `flash_attn` is not available.
|
||||||
|
|
||||||
EVO1 uses the native Hugging Face `transformers` InternVL implementation, so `policy.vlm_model_name` must point to a natively converted checkpoint such as `OpenGVLab/InternVL3-1B-hf` (note the `-hf` suffix). The first run may download the configured VLM checkpoint unless `policy.vlm_model_name` points to a local model directory.
|
EVO1 uses the native Hugging Face `transformers` InternVL implementation, so `policy.vlm_model_name` must point to a natively converted checkpoint such as `OpenGVLab/InternVL3-1B-hf` (note the `-hf` suffix). The first run downloads the configured VLM checkpoint and later runs reuse it from the Hugging Face cache.
|
||||||
|
|
||||||
## Data Requirements
|
## Data Requirements
|
||||||
|
|
||||||
@@ -92,7 +92,7 @@ lerobot-train \
|
|||||||
|
|
||||||
### Stage 2
|
### Stage 2
|
||||||
|
|
||||||
Stage 2 finetunes the VLM branches and action head. A common workflow starts from a Stage 1 checkpoint:
|
Stage 2 loads the Stage 1 policy, but starts a fresh optimizer and scheduler:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
lerobot-train \
|
lerobot-train \
|
||||||
@@ -152,16 +152,154 @@ lerobot-rollout \
|
|||||||
|
|
||||||
### LIBERO Evaluation
|
### LIBERO Evaluation
|
||||||
|
|
||||||
> [!NOTE]
|
#### Reference result
|
||||||
> Benchmark results for a `lerobot`-hosted LIBERO checkpoint trained with this implementation
|
|
||||||
> will be added once training completes.
|
|
||||||
|
|
||||||
The official EVO1 LIBERO rollout protocol uses the raw LIBERO camera feature names
|
> [!NOTE]
|
||||||
|
> The released Stage-2 checkpoint passed clean-download and rollout verification:
|
||||||
|
> [`zuoxingdong/evo1_libero`](https://huggingface.co/zuoxingdong/evo1_libero), revision
|
||||||
|
> [`515921f4a2c1d3f3ad523721eafa26fdf2af315b`](https://huggingface.co/zuoxingdong/evo1_libero/commit/515921f4a2c1d3f3ad523721eafa26fdf2af315b).
|
||||||
|
> The clean-download evaluation used LeRobot revision
|
||||||
|
> [`e40b58a8dfa9e7b86918c374791599d070518d11`](https://github.com/huggingface/lerobot/commit/e40b58a8dfa9e7b86918c374791599d070518d11).
|
||||||
|
|
||||||
|
The single-run Stage-2 checkpoint at step 70,000 produced:
|
||||||
|
|
||||||
|
| Suite | Successful episodes | Episodes | Success rate |
|
||||||
|
| -------------- | ------------------: | --------: | -----------: |
|
||||||
|
| LIBERO Spatial | 485 | 500 | 97.0% |
|
||||||
|
| LIBERO Object | 496 | 500 | 99.2% |
|
||||||
|
| LIBERO Goal | 483 | 500 | 96.6% |
|
||||||
|
| LIBERO-10 | 469 | 500 | 93.8% |
|
||||||
|
| **Overall** | **1,933** | **2,000** | **96.65%** |
|
||||||
|
|
||||||
|
These results use one trained checkpoint and evaluation seed `1000`; they are not a multi-seed
|
||||||
|
mean or confidence estimate.
|
||||||
|
|
||||||
|
#### Reference training recipe
|
||||||
|
|
||||||
|
The released checkpoint records the complete resolved Stage-2 configuration in
|
||||||
|
[`train_config.json`](https://huggingface.co/zuoxingdong/evo1_libero/blob/515921f4a2c1d3f3ad523721eafa26fdf2af315b/train_config.json).
|
||||||
|
The measured run used two H100 GPUs with two DDP processes and batch 64 per process, giving global batch 128. Both stages used the same topology. The base VLM came from revision
|
||||||
|
`014c0583a0d4bedf29fbe2dbff4f865eb998e171` of `OpenGVLab/InternVL3-1B-hf`.
|
||||||
|
The released artifact does not record the exact LeRobot training commit or its original dependency lock,
|
||||||
|
so the commands below reproduce the recorded configuration and topology from a current checkout rather
|
||||||
|
than reconstructing the software environment bit for bit.
|
||||||
|
|
||||||
|
From a LeRobot source checkout, install the locked dependencies and download that exact VLM revision:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv sync --locked --extra training --extra evo1 --extra libero
|
||||||
|
VLM_DIR=$(uv run hf download OpenGVLab/InternVL3-1B-hf \
|
||||||
|
--revision=014c0583a0d4bedf29fbe2dbff4f865eb998e171)
|
||||||
|
```
|
||||||
|
|
||||||
|
Stage 1 freezes the VLM and trains the action head for 5,000 steps:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run accelerate launch --num_processes=2 -m lerobot.scripts.lerobot_train \
|
||||||
|
--dataset.repo_id=lerobot/libero \
|
||||||
|
--dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
|
||||||
|
--dataset.video_backend=torchcodec \
|
||||||
|
--dataset.return_uint8=true \
|
||||||
|
--dataset.image_transforms.enable=true \
|
||||||
|
--dataset.use_imagenet_stats=true \
|
||||||
|
--dataset.eval_split=0.0 \
|
||||||
|
--policy.type=evo1 \
|
||||||
|
--policy.training_stage=stage1 \
|
||||||
|
--policy.apply_training_stage_defaults=true \
|
||||||
|
--policy.vlm_model_name="${VLM_DIR}" \
|
||||||
|
--policy.vlm_num_layers=14 \
|
||||||
|
--policy.vlm_dtype=bfloat16 \
|
||||||
|
--policy.device=cuda \
|
||||||
|
--policy.use_amp=true \
|
||||||
|
--policy.use_flash_attn=true \
|
||||||
|
--policy.enable_gradient_checkpointing=true \
|
||||||
|
--policy.gradient_checkpointing_use_reentrant=false \
|
||||||
|
--policy.image_resolution='[448,448]' \
|
||||||
|
--policy.chunk_size=50 \
|
||||||
|
--policy.n_action_steps=50 \
|
||||||
|
--policy.max_state_dim=24 \
|
||||||
|
--policy.max_action_dim=24 \
|
||||||
|
--policy.dropout=0.2 \
|
||||||
|
--policy.optimizer_lr=1e-5 \
|
||||||
|
--policy.optimizer_weight_decay=1e-3 \
|
||||||
|
--policy.optimizer_grad_clip_norm=1.0 \
|
||||||
|
--policy.scheduler_warmup_steps=1000 \
|
||||||
|
--policy.push_to_hub=false \
|
||||||
|
--use_policy_training_preset=true \
|
||||||
|
--batch_size=64 \
|
||||||
|
--steps=5000 \
|
||||||
|
--save_checkpoint=true \
|
||||||
|
--save_checkpoint_to_hub=false \
|
||||||
|
--save_freq=2500 \
|
||||||
|
--log_freq=10 \
|
||||||
|
--env_eval_freq=0 \
|
||||||
|
--num_workers=4 \
|
||||||
|
--prefetch_factor=2 \
|
||||||
|
--persistent_workers=true \
|
||||||
|
--seed=1000 \
|
||||||
|
--wandb.enable=false \
|
||||||
|
--output_dir=./outputs/evo1-libero-stage1-g128-5k
|
||||||
|
```
|
||||||
|
|
||||||
|
Stage 2 loads the Stage-1 policy but starts a fresh optimizer and scheduler. It trains for 80,000 steps;
|
||||||
|
the reported checkpoint is the save at step 70,000:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run accelerate launch --num_processes=2 -m lerobot.scripts.lerobot_train \
|
||||||
|
--dataset.repo_id=lerobot/libero \
|
||||||
|
--dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
|
||||||
|
--dataset.video_backend=torchcodec \
|
||||||
|
--dataset.return_uint8=true \
|
||||||
|
--dataset.image_transforms.enable=true \
|
||||||
|
--dataset.use_imagenet_stats=true \
|
||||||
|
--dataset.eval_split=0.0 \
|
||||||
|
--policy.path=./outputs/evo1-libero-stage1-g128-5k/checkpoints/005000/pretrained_model \
|
||||||
|
--policy.training_stage=stage2 \
|
||||||
|
--policy.apply_training_stage_defaults=true \
|
||||||
|
--policy.vlm_model_name="${VLM_DIR}" \
|
||||||
|
--policy.vlm_num_layers=14 \
|
||||||
|
--policy.vlm_dtype=float32 \
|
||||||
|
--policy.device=cuda \
|
||||||
|
--policy.use_amp=true \
|
||||||
|
--policy.use_flash_attn=true \
|
||||||
|
--policy.enable_gradient_checkpointing=true \
|
||||||
|
--policy.gradient_checkpointing_use_reentrant=false \
|
||||||
|
--policy.image_resolution='[448,448]' \
|
||||||
|
--policy.chunk_size=50 \
|
||||||
|
--policy.n_action_steps=50 \
|
||||||
|
--policy.max_state_dim=24 \
|
||||||
|
--policy.max_action_dim=24 \
|
||||||
|
--policy.dropout=0.2 \
|
||||||
|
--policy.optimizer_lr=1e-5 \
|
||||||
|
--policy.optimizer_weight_decay=1e-3 \
|
||||||
|
--policy.optimizer_grad_clip_norm=1.0 \
|
||||||
|
--policy.scheduler_warmup_steps=1000 \
|
||||||
|
--policy.push_to_hub=false \
|
||||||
|
--use_policy_training_preset=true \
|
||||||
|
--batch_size=64 \
|
||||||
|
--steps=80000 \
|
||||||
|
--resume=false \
|
||||||
|
--save_checkpoint=true \
|
||||||
|
--save_checkpoint_to_hub=false \
|
||||||
|
--save_freq=10000 \
|
||||||
|
--log_freq=10 \
|
||||||
|
--env_eval_freq=0 \
|
||||||
|
--num_workers=4 \
|
||||||
|
--prefetch_factor=2 \
|
||||||
|
--persistent_workers=true \
|
||||||
|
--seed=1000 \
|
||||||
|
--wandb.enable=false \
|
||||||
|
--output_dir=./outputs/evo1-libero-stage2-g128-80k
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Author-format evaluation profile
|
||||||
|
|
||||||
|
The author-format EVO1 LIBERO profile uses the raw LIBERO camera feature names
|
||||||
(`observation.images.agentview_image` and `observation.images.robot0_eye_in_hand_image`), replans every
|
(`observation.images.agentview_image` and `observation.images.robot0_eye_in_hand_image`), replans every
|
||||||
14 actions, and binarizes the gripper command before stepping the simulator. The EVO1 policy postprocessor
|
14 actions, and binarizes the gripper command before stepping the simulator. The EVO1 policy postprocessor
|
||||||
can crop the padded 24D action back to the 7D LIBERO action space and apply that gripper binarization. To
|
can crop the padded 24D action back to the 7D LIBERO action space and apply that gripper binarization. To
|
||||||
evaluate a LIBERO checkpoint under the same one-episode-per-task setting, keep the raw camera names instead
|
evaluate an author-format checkpoint under the same one-episode-per-task setting, keep the raw camera names
|
||||||
of the default `image`/`image2` mapping and set the LIBERO action postprocessing flags:
|
instead of the default `image`/`image2` mapping and set the LIBERO action postprocessing flags:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
lerobot-eval \
|
lerobot-eval \
|
||||||
@@ -181,6 +319,75 @@ lerobot-eval \
|
|||||||
--eval.n_episodes=1
|
--eval.n_episodes=1
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### Native `lerobot/libero` v3 profile
|
||||||
|
|
||||||
|
Revision `a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4` stores camera features as `image` and
|
||||||
|
`image2`. This example evaluates all ten LIBERO Object tasks, launching each task in a fresh process:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export MUJOCO_GL=egl
|
||||||
|
export PYOPENGL_PLATFORM=egl
|
||||||
|
|
||||||
|
suite=libero_object
|
||||||
|
horizon=280
|
||||||
|
for task_id in {0..9}; do
|
||||||
|
lerobot-eval \
|
||||||
|
--policy.path=zuoxingdong/evo1_libero \
|
||||||
|
--policy.pretrained_revision=515921f4a2c1d3f3ad523721eafa26fdf2af315b \
|
||||||
|
--policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
|
||||||
|
--policy.device=cuda \
|
||||||
|
--policy.use_amp=true \
|
||||||
|
--policy.vlm_dtype=bfloat16 \
|
||||||
|
--policy.use_flash_attn=false \
|
||||||
|
--policy.enable_gradient_checkpointing=false \
|
||||||
|
--policy.vlm_num_layers=14 \
|
||||||
|
--policy.image_resolution='[448,448]' \
|
||||||
|
--policy.max_text_length=1024 \
|
||||||
|
--policy.chunk_size=50 \
|
||||||
|
--policy.n_action_steps=14 \
|
||||||
|
--policy.max_state_dim=24 \
|
||||||
|
--policy.max_action_dim=24 \
|
||||||
|
--policy.num_inference_timesteps=32 \
|
||||||
|
--policy.postprocess_action_dim=7 \
|
||||||
|
--policy.binarize_gripper=true \
|
||||||
|
--policy.gripper_threshold=0.0 \
|
||||||
|
--policy.gripper_below_threshold_value=-1.0 \
|
||||||
|
--policy.gripper_above_threshold_value=1.0 \
|
||||||
|
--env.type=libero \
|
||||||
|
--env.task="${suite}" \
|
||||||
|
--env.task_ids="[${task_id}]" \
|
||||||
|
--env.camera_name=agentview_image,robot0_eye_in_hand_image \
|
||||||
|
--env.camera_name_mapping="{agentview_image: image, robot0_eye_in_hand_image: image2}" \
|
||||||
|
--env.control_mode=relative \
|
||||||
|
--env.obs_type=pixels_agent_pos \
|
||||||
|
--env.observation_width=448 \
|
||||||
|
--env.observation_height=448 \
|
||||||
|
--env.init_states=true \
|
||||||
|
--env.episode_length="${horizon}" \
|
||||||
|
--env.render_mode=rgb_array \
|
||||||
|
--env.max_parallel_tasks=1 \
|
||||||
|
--eval.n_episodes=50 \
|
||||||
|
--eval.batch_size=1 \
|
||||||
|
--eval.use_async_envs=false \
|
||||||
|
--eval.recording=false \
|
||||||
|
--seed=1000 \
|
||||||
|
--output_dir="./outputs/evo1-libero-stage2-70k-eval/${suite}/task-${task_id}" \
|
||||||
|
--job_name="evo1-libero-stage2-70k-${suite}-task-${task_id}"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
Run all ten task IDs for each suite with these horizons:
|
||||||
|
|
||||||
|
| `env.task` | `env.episode_length` |
|
||||||
|
| ---------------- | -------------------: |
|
||||||
|
| `libero_spatial` | `280` |
|
||||||
|
| `libero_object` | `280` |
|
||||||
|
| `libero_goal` | `300` |
|
||||||
|
| `libero_10` | `520` |
|
||||||
|
|
||||||
|
Set `suite` and `horizon` for each row. This gives 500 episodes per suite and 2,000 episodes overall, while
|
||||||
|
the loop's fresh process per task matches the measured RNG-reset topology.
|
||||||
|
|
||||||
## References
|
## References
|
||||||
|
|
||||||
- [EVO1 repository](https://github.com/MINT-SJTU/Evo-1)
|
- [EVO1 repository](https://github.com/MINT-SJTU/Evo-1)
|
||||||
|
|||||||
@@ -102,6 +102,7 @@ class ImageServer:
|
|||||||
fps=self.fps,
|
fps=self.fps,
|
||||||
width=shape[1],
|
width=shape[1],
|
||||||
height=shape[0],
|
height=shape[0],
|
||||||
|
fourcc=cfg.get("fourcc", "MJPG"),
|
||||||
color_mode=ColorMode.RGB,
|
color_mode=ColorMode.RGB,
|
||||||
)
|
)
|
||||||
camera = OpenCVCamera(cam_config)
|
camera = OpenCVCamera(cam_config)
|
||||||
|
|||||||
Reference in New Issue
Block a user