lerobot/docs/source/evo1.mdx

# EVO1

EVO1 is a Vision-Language-Action policy for robot control built around an InternVL3 backbone and a continuous flow-matching action head. This LeRobot integration exposes EVO1 as a standard policy type so it can be trained and evaluated with the usual LeRobot dataset, checkpoint, and processor APIs.

## Model Overview

The policy embeds one or more camera images and the language task prompt with InternVL3, pads robot state/action vectors to fixed maximum dimensions, and predicts future action chunks with a flow-matching action head. During inference, the policy samples an action chunk and returns `n_action_steps` actions from that chunk before sampling again.

### What the LeRobot Integration Covers

- Standard `policy.type=evo1` configuration through LeRobot
- InternVL3 image/text embedding with optional FlashAttention fallback
- Stage-based finetuning controls for action-head-only and VLM finetuning runs
- Continuous flow-matching action prediction
- Checkpoint save/load through LeRobot policy APIs
- Training with `lerobot-train` and evaluation with standard policy inference APIs

The broader EVO1 project may include additional training scripts and dataset tooling. This page focuses on the LeRobot robot-control policy path.

## Installation Requirements

1. Install LeRobot by following the [Installation Guide](./installation).
2. Install EVO1 dependencies:

   ```bash
   pip install -e ".[training,evo1]"
   ```

   For LIBERO training and evaluation, install the LIBERO extra as well:

   ```bash
   pip install -e ".[training,evo1,libero]"
   ```

3. Install a `flash-attn` wheel only if it is compatible with your Python, PyTorch, CUDA, and GPU stack. EVO1 falls back to standard attention when `flash_attn` is not available.

EVO1 uses the native Hugging Face `transformers` InternVL implementation, so `policy.vlm_model_name` must point to a natively converted checkpoint such as `OpenGVLab/InternVL3-1B-hf` (note the `-hf` suffix). The first run downloads the configured VLM checkpoint and later runs reuse it from the Hugging Face cache.

## Data Requirements

EVO1 expects a LeRobot dataset with:

- One to `policy.max_views` visual observations, for example `observation.images.image`
- `observation.state`
- `action`
- A language task instruction in the dataset `task` field, or another field configured with `policy.task_field`

State and action vectors are padded to `policy.max_state_dim` and `policy.max_action_dim`. Predictions are cropped back to the dataset action dimension before being returned.

## Usage

To use EVO1 in a LeRobot configuration, specify:

```python
policy.type=evo1
```

By default, a new EVO1 policy initializes its VLM from:

```python
policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf
```

Once a LeRobot-format EVO1 checkpoint is available, load it with:

```python
policy.path=your-org/your-evo1-checkpoint
```

## Training

### Stage 1

Stage 1 freezes the VLM and trains the action head:

```bash
lerobot-train \
  --dataset.repo_id=your_org/your_dataset \
  --policy.type=evo1 \
  --policy.training_stage=stage1 \
  --policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
  --policy.device=cuda \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.max_state_dim=24 \
  --policy.max_action_dim=24 \
  --policy.optimizer_lr=1e-5 \
  --batch_size=4 \
  --steps=5000 \
  --output_dir=./outputs/evo1_stage1
```

### Stage 2

Stage 2 loads the Stage 1 policy, but starts a fresh optimizer and scheduler:

```bash
lerobot-train \
  --dataset.repo_id=your_org/your_dataset \
  --policy.path=./outputs/evo1_stage1/checkpoints/005000/pretrained_model \
  --policy.training_stage=stage2 \
  --policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
  --policy.device=cuda \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.max_state_dim=24 \
  --policy.max_action_dim=24 \
  --policy.optimizer_lr=1e-5 \
  --batch_size=4 \
  --steps=80000 \
  --output_dir=./outputs/evo1_stage2
```

By default, `policy.training_stage` reapplies the finetuning defaults for that stage. This is important when
starting Stage 2 from a Stage 1 checkpoint, because the Stage 1 checkpoint config stores the VLM finetuning
flags as disabled. These stage defaults take precedence over saved or manually supplied `policy.finetune_*`
flags unless `policy.apply_training_stage_defaults=false`, so set that flag only when manually controlling
every finetuning flag.

### Key Training Parameters

| Parameter                                     | Default                     | Description                                                       |
| --------------------------------------------- | --------------------------- | ----------------------------------------------------------------- |
| `policy.vlm_model_name`                       | `OpenGVLab/InternVL3-1B-hf` | Natively converted InternVL3 checkpoint or local model directory  |
| `policy.training_stage`                       | `stage1`                    | `stage1` trains the action head; `stage2` finetunes VLM branches  |
| `policy.apply_training_stage_defaults`        | `true`                      | Reapplies stage finetuning defaults after loading a checkpoint    |
| `policy.vlm_num_layers`                       | `14`                        | Number of InternVL3 language layers kept for the policy           |
| `policy.vlm_dtype`                            | `bfloat16`                  | Requested VLM dtype                                               |
| `policy.use_flash_attn`                       | `true`                      | Requests FlashAttention when installed; otherwise falls back      |
| `policy.enable_gradient_checkpointing`        | `true`                      | Enables checkpointing on supported InternVL3 modules              |
| `policy.gradient_checkpointing_use_reentrant` | `false`                     | Reentrant setting passed to gradient checkpointing when supported |
| `policy.chunk_size`                           | `50`                        | Number of future actions predicted per chunk                      |
| `policy.n_action_steps`                       | `50`                        | Number of actions consumed from a sampled chunk                   |
| `policy.max_state_dim`                        | `24`                        | State padding dimension                                           |
| `policy.max_action_dim`                       | `24`                        | Action padding dimension                                          |
| `policy.postprocess_action_dim`               | `null`                      | Optional action dimension returned after EVO1 postprocessing      |
| `policy.binarize_gripper`                     | `false`                     | Binarizes the postprocessed gripper channel for LIBERO-style eval |
| `policy.task_field`                           | `task`                      | Batch field used as the language prompt                           |

## Inference

Try it out with a trained EVO1 checkpoint:

```bash
lerobot-rollout \
  --policy.path=your-org/your-evo1-checkpoint \
  --inference.type=rtc \ # optional
  ...
```

## Results

### LIBERO Evaluation

#### Reference result

> [!NOTE]
> The released Stage-2 checkpoint passed clean-download and rollout verification:
> [`zuoxingdong/evo1_libero`](https://huggingface.co/zuoxingdong/evo1_libero), revision
> [`515921f4a2c1d3f3ad523721eafa26fdf2af315b`](https://huggingface.co/zuoxingdong/evo1_libero/commit/515921f4a2c1d3f3ad523721eafa26fdf2af315b).
> The clean-download evaluation used LeRobot revision
> [`e40b58a8dfa9e7b86918c374791599d070518d11`](https://github.com/huggingface/lerobot/commit/e40b58a8dfa9e7b86918c374791599d070518d11).

The single-run Stage-2 checkpoint at step 70,000 produced:

| Suite          | Successful episodes |  Episodes | Success rate |
| -------------- | ------------------: | --------: | -----------: |
| LIBERO Spatial |                 485 |       500 |        97.0% |
| LIBERO Object  |                 496 |       500 |        99.2% |
| LIBERO Goal    |                 483 |       500 |        96.6% |
| LIBERO-10      |                 469 |       500 |        93.8% |
| **Overall**    |           **1,933** | **2,000** |   **96.65%** |

These results use one trained checkpoint and evaluation seed `1000`; they are not a multi-seed
mean or confidence estimate.

#### Reference training recipe

The released checkpoint records the complete resolved Stage-2 configuration in
[`train_config.json`](https://huggingface.co/zuoxingdong/evo1_libero/blob/515921f4a2c1d3f3ad523721eafa26fdf2af315b/train_config.json).
The measured run used two H100 GPUs with two DDP processes and batch 64 per process, giving global batch 128. Both stages used the same topology. The base VLM came from revision
`014c0583a0d4bedf29fbe2dbff4f865eb998e171` of `OpenGVLab/InternVL3-1B-hf`.
The released artifact does not record the exact LeRobot training commit or its original dependency lock,
so the commands below reproduce the recorded configuration and topology from a current checkout rather
than reconstructing the software environment bit for bit.

From a LeRobot source checkout, install the locked dependencies and download that exact VLM revision:

```bash
uv sync --locked --extra training --extra evo1 --extra libero
VLM_DIR=$(uv run hf download OpenGVLab/InternVL3-1B-hf \
  --revision=014c0583a0d4bedf29fbe2dbff4f865eb998e171)
```

Stage 1 freezes the VLM and trains the action head for 5,000 steps:

```bash
uv run accelerate launch --num_processes=2 -m lerobot.scripts.lerobot_train \
  --dataset.repo_id=lerobot/libero \
  --dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
  --dataset.video_backend=torchcodec \
  --dataset.return_uint8=true \
  --dataset.image_transforms.enable=true \
  --dataset.use_imagenet_stats=true \
  --dataset.eval_split=0.0 \
  --policy.type=evo1 \
  --policy.training_stage=stage1 \
  --policy.apply_training_stage_defaults=true \
  --policy.vlm_model_name="${VLM_DIR}" \
  --policy.vlm_num_layers=14 \
  --policy.vlm_dtype=bfloat16 \
  --policy.device=cuda \
  --policy.use_amp=true \
  --policy.use_flash_attn=true \
  --policy.enable_gradient_checkpointing=true \
  --policy.gradient_checkpointing_use_reentrant=false \
  --policy.image_resolution='[448,448]' \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.max_state_dim=24 \
  --policy.max_action_dim=24 \
  --policy.dropout=0.2 \
  --policy.optimizer_lr=1e-5 \
  --policy.optimizer_weight_decay=1e-3 \
  --policy.optimizer_grad_clip_norm=1.0 \
  --policy.scheduler_warmup_steps=1000 \
  --policy.push_to_hub=false \
  --use_policy_training_preset=true \
  --batch_size=64 \
  --steps=5000 \
  --save_checkpoint=true \
  --save_checkpoint_to_hub=false \
  --save_freq=2500 \
  --log_freq=10 \
  --env_eval_freq=0 \
  --num_workers=4 \
  --prefetch_factor=2 \
  --persistent_workers=true \
  --seed=1000 \
  --wandb.enable=false \
  --output_dir=./outputs/evo1-libero-stage1-g128-5k
```

Stage 2 loads the Stage-1 policy but starts a fresh optimizer and scheduler. It trains for 80,000 steps;
the reported checkpoint is the save at step 70,000:

```bash
uv run accelerate launch --num_processes=2 -m lerobot.scripts.lerobot_train \
  --dataset.repo_id=lerobot/libero \
  --dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
  --dataset.video_backend=torchcodec \
  --dataset.return_uint8=true \
  --dataset.image_transforms.enable=true \
  --dataset.use_imagenet_stats=true \
  --dataset.eval_split=0.0 \
  --policy.path=./outputs/evo1-libero-stage1-g128-5k/checkpoints/005000/pretrained_model \
  --policy.training_stage=stage2 \
  --policy.apply_training_stage_defaults=true \
  --policy.vlm_model_name="${VLM_DIR}" \
  --policy.vlm_num_layers=14 \
  --policy.vlm_dtype=float32 \
  --policy.device=cuda \
  --policy.use_amp=true \
  --policy.use_flash_attn=true \
  --policy.enable_gradient_checkpointing=true \
  --policy.gradient_checkpointing_use_reentrant=false \
  --policy.image_resolution='[448,448]' \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.max_state_dim=24 \
  --policy.max_action_dim=24 \
  --policy.dropout=0.2 \
  --policy.optimizer_lr=1e-5 \
  --policy.optimizer_weight_decay=1e-3 \
  --policy.optimizer_grad_clip_norm=1.0 \
  --policy.scheduler_warmup_steps=1000 \
  --policy.push_to_hub=false \
  --use_policy_training_preset=true \
  --batch_size=64 \
  --steps=80000 \
  --resume=false \
  --save_checkpoint=true \
  --save_checkpoint_to_hub=false \
  --save_freq=10000 \
  --log_freq=10 \
  --env_eval_freq=0 \
  --num_workers=4 \
  --prefetch_factor=2 \
  --persistent_workers=true \
  --seed=1000 \
  --wandb.enable=false \
  --output_dir=./outputs/evo1-libero-stage2-g128-80k
```

#### Author-format evaluation profile

The author-format EVO1 LIBERO profile uses the raw LIBERO camera feature names
(`observation.images.agentview_image` and `observation.images.robot0_eye_in_hand_image`), replans every
14 actions, and binarizes the gripper command before stepping the simulator. The EVO1 policy postprocessor
can crop the padded 24D action back to the 7D LIBERO action space and apply that gripper binarization. To
evaluate an author-format checkpoint under the same one-episode-per-task setting, keep the raw camera names
instead of the default `image`/`image2` mapping and set the LIBERO action postprocessing flags:

```bash
lerobot-eval \
  --policy.path=your-org/your-evo1-libero-checkpoint \
  --policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
  --policy.device=cuda \
  --policy.use_flash_attn=true \
  --policy.n_action_steps=14 \
  --policy.postprocess_action_dim=7 \
  --policy.binarize_gripper=true \
  --env.type=libero \
  --env.task=libero_object \
  --env.camera_name_mapping="{agentview_image: agentview_image, robot0_eye_in_hand_image: robot0_eye_in_hand_image}" \
  --env.observation_height=448 \
  --env.observation_width=448 \
  --eval.batch_size=1 \
  --eval.n_episodes=1
```

#### Native `lerobot/libero` v3 profile

Revision `a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4` stores camera features as `image` and
`image2`. This example evaluates all ten LIBERO Object tasks, launching each task in a fresh process:

```bash
export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl

suite=libero_object
horizon=280
for task_id in {0..9}; do
  lerobot-eval \
    --policy.path=zuoxingdong/evo1_libero \
    --policy.pretrained_revision=515921f4a2c1d3f3ad523721eafa26fdf2af315b \
    --policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
    --policy.device=cuda \
    --policy.use_amp=true \
    --policy.vlm_dtype=bfloat16 \
    --policy.use_flash_attn=false \
    --policy.enable_gradient_checkpointing=false \
    --policy.vlm_num_layers=14 \
    --policy.image_resolution='[448,448]' \
    --policy.max_text_length=1024 \
    --policy.chunk_size=50 \
    --policy.n_action_steps=14 \
    --policy.max_state_dim=24 \
    --policy.max_action_dim=24 \
    --policy.num_inference_timesteps=32 \
    --policy.postprocess_action_dim=7 \
    --policy.binarize_gripper=true \
    --policy.gripper_threshold=0.0 \
    --policy.gripper_below_threshold_value=-1.0 \
    --policy.gripper_above_threshold_value=1.0 \
    --env.type=libero \
    --env.task="${suite}" \
    --env.task_ids="[${task_id}]" \
    --env.camera_name=agentview_image,robot0_eye_in_hand_image \
    --env.camera_name_mapping="{agentview_image: image, robot0_eye_in_hand_image: image2}" \
    --env.control_mode=relative \
    --env.obs_type=pixels_agent_pos \
    --env.observation_width=448 \
    --env.observation_height=448 \
    --env.init_states=true \
    --env.episode_length="${horizon}" \
    --env.render_mode=rgb_array \
    --env.max_parallel_tasks=1 \
    --eval.n_episodes=50 \
    --eval.batch_size=1 \
    --eval.use_async_envs=false \
    --eval.recording=false \
    --seed=1000 \
    --output_dir="./outputs/evo1-libero-stage2-70k-eval/${suite}/task-${task_id}" \
    --job_name="evo1-libero-stage2-70k-${suite}-task-${task_id}"
done
```

Run all ten task IDs for each suite with these horizons:

| `env.task`       | `env.episode_length` |
| ---------------- | -------------------: |
| `libero_spatial` |                `280` |
| `libero_object`  |                `280` |
| `libero_goal`    |                `300` |
| `libero_10`      |                `520` |

Set `suite` and `horizon` for each row. This gives 500 episodes per suite and 2,000 episodes overall, while
the loop's fresh process per task matches the measured RNG-reset topology.

## References

- [EVO1 repository](https://github.com/MINT-SJTU/Evo-1)
- [InternVL3-1B-hf](https://huggingface.co/OpenGVLab/InternVL3-1B-hf)

## License

This LeRobot integration follows the Apache 2.0 License used by LeRobot. Check the upstream EVO1 and InternVL3 model pages for the licenses of released checkpoints and data.