Compare commits

..

3 Commits

Author SHA1 Message Date
Steven Palma 2aba372b4e feat(ci): re-enable stale issues countdown (#4279) 2026-07-31 19:17:23 +02:00
Steven Palma c841a0c258 docs(evo1): add LIBERO reproduction recipe (#4278)
* docs(evo1): add LIBERO reproduction recipe

* docs(evo1): link verified LIBERO checkpoint

* docs(policies): address comments libero results

---------

Co-authored-by: Xingdong Zuo <18168681+zuoxingdong@users.noreply.github.com>
2026-07-31 17:23:10 +02:00
Steven Palma a3ddba2454 feat(cameras): prefer MJPG when fourcc unspecified in ZMQ (#4277)
* feat(cameras): prefer MJPG when fourcc unspecified; allow fourcc in ZMQ image server

With no fourcc set, OpenCV's V4L2 auto-negotiation selects uncompressed
YUYV when the camera offers it: ~16x the USB bandwidth of MJPG for
identical frames, which silently caps the frame rate on shared USB 2.0
buses (e.g. a Raspberry Pi with multiple cameras).

OpenCVCamera now prefers MJPG when config.fourcc is None: it requests
MJPG and keeps it only if the camera reports support (read-back),
falling back to the camera's own negotiation otherwise. An explicit
config.fourcc always takes precedence, so opting out is fourcc="YUYV".
The preference mirrors the existing pre/post size-and-fps ordering to
preserve the Windows DSHOW FOURCC-override handling.

Also adds a fourcc passthrough to the ZMQ image server, which previously
could not request a pixel format at all.

Behavior change: cameras that relied on implicit uncompressed capture
now receive camera-side JPEG unless fourcc is set explicitly.

* chore(cameras): address feedback

---------

Co-authored-by: Xingdong Zuo <18168681+zuoxingdong@users.noreply.github.com>
2026-07-31 17:13:32 +02:00
4 changed files with 224 additions and 37 deletions
+5 -5
View File
@@ -19,8 +19,8 @@ on:
workflow_dispatch:
# Runs at 02:00
# schedule:
# - cron: "0 2 * * *"
schedule:
- cron: "0 2 * * *"
env:
CLOSE_ISSUE_MESSAGE: >
@@ -31,7 +31,7 @@ env:
Feel free to reopen if is still relevant, or to ping a collaborator if you have any questions.
WARN_ISSUE_MESSAGE: >
This issue has been automatically marked as stale because it has not had
recent activity (1 year). It will be closed if no further activity occurs.
recent activity (1 year). It will be closed if no further activity occurs within 30 days.
Any change, comment or update to this issue will reset this count.
Thank you for your contributions.
WARN_PR_MESSAGE: >
@@ -61,8 +61,8 @@ jobs:
exempt-pr-labels: never-stale
days-before-issue-stale: 365
days-before-issue-close: 30
days-before-pr-stale: 365
days-before-pr-close: 30
days-before-pr-stale: -1
days-before-pr-close: -1
delete-branch: true
close-issue-message: ${{ env.CLOSE_ISSUE_MESSAGE }}
close-pr-message: ${{ env.CLOSE_PR_MESSAGE }}
+218 -11
View File
@@ -23,18 +23,18 @@ The broader EVO1 project may include additional training scripts and dataset too
2. Install EVO1 dependencies:
```bash
pip install -e ".[evo1]"
pip install -e ".[training,evo1]"
```
For LIBERO evaluation, install the LIBERO extra as well:
For LIBERO training and evaluation, install the LIBERO extra as well:
```bash
pip install -e ".[evo1,libero]"
pip install -e ".[training,evo1,libero]"
```
3. Install a `flash-attn` wheel only if it is compatible with your Python, PyTorch, CUDA, and GPU stack. EVO1 falls back to standard attention when `flash_attn` is not available.
EVO1 uses the native Hugging Face `transformers` InternVL implementation, so `policy.vlm_model_name` must point to a natively converted checkpoint such as `OpenGVLab/InternVL3-1B-hf` (note the `-hf` suffix). The first run may download the configured VLM checkpoint unless `policy.vlm_model_name` points to a local model directory.
EVO1 uses the native Hugging Face `transformers` InternVL implementation, so `policy.vlm_model_name` must point to a natively converted checkpoint such as `OpenGVLab/InternVL3-1B-hf` (note the `-hf` suffix). The first run downloads the configured VLM checkpoint and later runs reuse it from the Hugging Face cache.
## Data Requirements
@@ -92,7 +92,7 @@ lerobot-train \
### Stage 2
Stage 2 finetunes the VLM branches and action head. A common workflow starts from a Stage 1 checkpoint:
Stage 2 loads the Stage 1 policy, but starts a fresh optimizer and scheduler:
```bash
lerobot-train \
@@ -152,16 +152,154 @@ lerobot-rollout \
### LIBERO Evaluation
> [!NOTE]
> Benchmark results for a `lerobot`-hosted LIBERO checkpoint trained with this implementation
> will be added once training completes.
#### Reference result
The official EVO1 LIBERO rollout protocol uses the raw LIBERO camera feature names
> [!NOTE]
> The released Stage-2 checkpoint passed clean-download and rollout verification:
> [`zuoxingdong/evo1_libero`](https://huggingface.co/zuoxingdong/evo1_libero), revision
> [`515921f4a2c1d3f3ad523721eafa26fdf2af315b`](https://huggingface.co/zuoxingdong/evo1_libero/commit/515921f4a2c1d3f3ad523721eafa26fdf2af315b).
> The clean-download evaluation used LeRobot revision
> [`e40b58a8dfa9e7b86918c374791599d070518d11`](https://github.com/huggingface/lerobot/commit/e40b58a8dfa9e7b86918c374791599d070518d11).
The single-run Stage-2 checkpoint at step 70,000 produced:
| Suite | Successful episodes | Episodes | Success rate |
| -------------- | ------------------: | --------: | -----------: |
| LIBERO Spatial | 485 | 500 | 97.0% |
| LIBERO Object | 496 | 500 | 99.2% |
| LIBERO Goal | 483 | 500 | 96.6% |
| LIBERO-10 | 469 | 500 | 93.8% |
| **Overall** | **1,933** | **2,000** | **96.65%** |
These results use one trained checkpoint and evaluation seed `1000`; they are not a multi-seed
mean or confidence estimate.
#### Reference training recipe
The released checkpoint records the complete resolved Stage-2 configuration in
[`train_config.json`](https://huggingface.co/zuoxingdong/evo1_libero/blob/515921f4a2c1d3f3ad523721eafa26fdf2af315b/train_config.json).
The measured run used two H100 GPUs with two DDP processes and batch 64 per process, giving global batch 128. Both stages used the same topology. The base VLM came from revision
`014c0583a0d4bedf29fbe2dbff4f865eb998e171` of `OpenGVLab/InternVL3-1B-hf`.
The released artifact does not record the exact LeRobot training commit or its original dependency lock,
so the commands below reproduce the recorded configuration and topology from a current checkout rather
than reconstructing the software environment bit for bit.
From a LeRobot source checkout, install the locked dependencies and download that exact VLM revision:
```bash
uv sync --locked --extra training --extra evo1 --extra libero
VLM_DIR=$(uv run hf download OpenGVLab/InternVL3-1B-hf \
--revision=014c0583a0d4bedf29fbe2dbff4f865eb998e171)
```
Stage 1 freezes the VLM and trains the action head for 5,000 steps:
```bash
uv run accelerate launch --num_processes=2 -m lerobot.scripts.lerobot_train \
--dataset.repo_id=lerobot/libero \
--dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
--dataset.video_backend=torchcodec \
--dataset.return_uint8=true \
--dataset.image_transforms.enable=true \
--dataset.use_imagenet_stats=true \
--dataset.eval_split=0.0 \
--policy.type=evo1 \
--policy.training_stage=stage1 \
--policy.apply_training_stage_defaults=true \
--policy.vlm_model_name="${VLM_DIR}" \
--policy.vlm_num_layers=14 \
--policy.vlm_dtype=bfloat16 \
--policy.device=cuda \
--policy.use_amp=true \
--policy.use_flash_attn=true \
--policy.enable_gradient_checkpointing=true \
--policy.gradient_checkpointing_use_reentrant=false \
--policy.image_resolution='[448,448]' \
--policy.chunk_size=50 \
--policy.n_action_steps=50 \
--policy.max_state_dim=24 \
--policy.max_action_dim=24 \
--policy.dropout=0.2 \
--policy.optimizer_lr=1e-5 \
--policy.optimizer_weight_decay=1e-3 \
--policy.optimizer_grad_clip_norm=1.0 \
--policy.scheduler_warmup_steps=1000 \
--policy.push_to_hub=false \
--use_policy_training_preset=true \
--batch_size=64 \
--steps=5000 \
--save_checkpoint=true \
--save_checkpoint_to_hub=false \
--save_freq=2500 \
--log_freq=10 \
--env_eval_freq=0 \
--num_workers=4 \
--prefetch_factor=2 \
--persistent_workers=true \
--seed=1000 \
--wandb.enable=false \
--output_dir=./outputs/evo1-libero-stage1-g128-5k
```
Stage 2 loads the Stage-1 policy but starts a fresh optimizer and scheduler. It trains for 80,000 steps;
the reported checkpoint is the save at step 70,000:
```bash
uv run accelerate launch --num_processes=2 -m lerobot.scripts.lerobot_train \
--dataset.repo_id=lerobot/libero \
--dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
--dataset.video_backend=torchcodec \
--dataset.return_uint8=true \
--dataset.image_transforms.enable=true \
--dataset.use_imagenet_stats=true \
--dataset.eval_split=0.0 \
--policy.path=./outputs/evo1-libero-stage1-g128-5k/checkpoints/005000/pretrained_model \
--policy.training_stage=stage2 \
--policy.apply_training_stage_defaults=true \
--policy.vlm_model_name="${VLM_DIR}" \
--policy.vlm_num_layers=14 \
--policy.vlm_dtype=float32 \
--policy.device=cuda \
--policy.use_amp=true \
--policy.use_flash_attn=true \
--policy.enable_gradient_checkpointing=true \
--policy.gradient_checkpointing_use_reentrant=false \
--policy.image_resolution='[448,448]' \
--policy.chunk_size=50 \
--policy.n_action_steps=50 \
--policy.max_state_dim=24 \
--policy.max_action_dim=24 \
--policy.dropout=0.2 \
--policy.optimizer_lr=1e-5 \
--policy.optimizer_weight_decay=1e-3 \
--policy.optimizer_grad_clip_norm=1.0 \
--policy.scheduler_warmup_steps=1000 \
--policy.push_to_hub=false \
--use_policy_training_preset=true \
--batch_size=64 \
--steps=80000 \
--resume=false \
--save_checkpoint=true \
--save_checkpoint_to_hub=false \
--save_freq=10000 \
--log_freq=10 \
--env_eval_freq=0 \
--num_workers=4 \
--prefetch_factor=2 \
--persistent_workers=true \
--seed=1000 \
--wandb.enable=false \
--output_dir=./outputs/evo1-libero-stage2-g128-80k
```
#### Author-format evaluation profile
The author-format EVO1 LIBERO profile uses the raw LIBERO camera feature names
(`observation.images.agentview_image` and `observation.images.robot0_eye_in_hand_image`), replans every
14 actions, and binarizes the gripper command before stepping the simulator. The EVO1 policy postprocessor
can crop the padded 24D action back to the 7D LIBERO action space and apply that gripper binarization. To
evaluate a LIBERO checkpoint under the same one-episode-per-task setting, keep the raw camera names instead
of the default `image`/`image2` mapping and set the LIBERO action postprocessing flags:
evaluate an author-format checkpoint under the same one-episode-per-task setting, keep the raw camera names
instead of the default `image`/`image2` mapping and set the LIBERO action postprocessing flags:
```bash
lerobot-eval \
@@ -181,6 +319,75 @@ lerobot-eval \
--eval.n_episodes=1
```
#### Native `lerobot/libero` v3 profile
Revision `a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4` stores camera features as `image` and
`image2`. This example evaluates all ten LIBERO Object tasks, launching each task in a fresh process:
```bash
export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl
suite=libero_object
horizon=280
for task_id in {0..9}; do
lerobot-eval \
--policy.path=zuoxingdong/evo1_libero \
--policy.pretrained_revision=515921f4a2c1d3f3ad523721eafa26fdf2af315b \
--policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
--policy.device=cuda \
--policy.use_amp=true \
--policy.vlm_dtype=bfloat16 \
--policy.use_flash_attn=false \
--policy.enable_gradient_checkpointing=false \
--policy.vlm_num_layers=14 \
--policy.image_resolution='[448,448]' \
--policy.max_text_length=1024 \
--policy.chunk_size=50 \
--policy.n_action_steps=14 \
--policy.max_state_dim=24 \
--policy.max_action_dim=24 \
--policy.num_inference_timesteps=32 \
--policy.postprocess_action_dim=7 \
--policy.binarize_gripper=true \
--policy.gripper_threshold=0.0 \
--policy.gripper_below_threshold_value=-1.0 \
--policy.gripper_above_threshold_value=1.0 \
--env.type=libero \
--env.task="${suite}" \
--env.task_ids="[${task_id}]" \
--env.camera_name=agentview_image,robot0_eye_in_hand_image \
--env.camera_name_mapping="{agentview_image: image, robot0_eye_in_hand_image: image2}" \
--env.control_mode=relative \
--env.obs_type=pixels_agent_pos \
--env.observation_width=448 \
--env.observation_height=448 \
--env.init_states=true \
--env.episode_length="${horizon}" \
--env.render_mode=rgb_array \
--env.max_parallel_tasks=1 \
--eval.n_episodes=50 \
--eval.batch_size=1 \
--eval.use_async_envs=false \
--eval.recording=false \
--seed=1000 \
--output_dir="./outputs/evo1-libero-stage2-70k-eval/${suite}/task-${task_id}" \
--job_name="evo1-libero-stage2-70k-${suite}-task-${task_id}"
done
```
Run all ten task IDs for each suite with these horizons:
| `env.task` | `env.episode_length` |
| ---------------- | -------------------: |
| `libero_spatial` | `280` |
| `libero_object` | `280` |
| `libero_goal` | `300` |
| `libero_10` | `520` |
Set `suite` and `horizon` for each row. This gives 500 episodes per suite and 2,000 episodes overall, while
the loop's fresh process per task matches the measured RNG-reset topology.
## References
- [EVO1 repository](https://github.com/MINT-SJTU/Evo-1)
+1
View File
@@ -102,6 +102,7 @@ class ImageServer:
fps=self.fps,
width=shape[1],
height=shape[0],
fourcc=cfg.get("fourcc", "MJPG"),
color_mode=ColorMode.RGB,
)
camera = OpenCVCamera(cam_config)
@@ -604,12 +604,6 @@ class PI0FastPytorch(nn.Module): # see openpi `PI0Pytorch`
) -> torch.Tensor:
"""
Optimized autoregressive decoding for FAST tokens using KV Caching.
Greedy decoding stops once every sequence emits the end-of-action marker. The
returned tensor keeps its fixed shape, with positions not generated after the
batch-wide stop left zero-filled. Stochastic decoding always runs to
``max_decoding_steps`` so early stopping does not change the RNG state used by
subsequent calls.
"""
if max_decoding_steps is None:
max_decoding_steps = self.config.max_action_tokens
@@ -618,12 +612,6 @@ class PI0FastPytorch(nn.Module): # see openpi `PI0Pytorch`
device = tokens.device
lm_head = self.paligemma_with_expert.paligemma.lm_head
# detokenize_actions() cuts at the first "|", so greedy decoding can stop once
# every sequence has emitted it. Keep stochastic decoding unchanged because
# skipping multinomial calls would shift the RNG state for subsequent calls.
end_of_action_token_id = self._paligemma_tokenizer.convert_tokens_to_ids("|")
finished = torch.zeros(bsize, dtype=torch.bool, device=device) if temperature == 0 else None
# --- 1. PREFILL PHASE ---
# Process Images + Text Prompt + BOS token once to populate the KV cache.
@@ -675,10 +663,6 @@ class PI0FastPytorch(nn.Module): # see openpi `PI0Pytorch`
# Initialize storage for generated tokens
generated_action_tokens = torch.zeros((bsize, max_decoding_steps), dtype=torch.long, device=device)
generated_action_tokens[:, 0] = next_token.squeeze(-1)
if finished is not None:
finished |= next_token.squeeze(-1) == end_of_action_token_id
if bool(finished.all()):
return generated_action_tokens
# Track valid tokens mask (0 for pad, 1 for valid)
# We need this to tell the new token what it can attend to (images + text + past actions)
@@ -729,11 +713,6 @@ class PI0FastPytorch(nn.Module): # see openpi `PI0Pytorch`
generated_action_tokens[:, t] = next_token.squeeze(-1)
if finished is not None:
finished |= next_token.squeeze(-1) == end_of_action_token_id
if bool(finished.all()):
break
return generated_action_tokens