lerobot/docs/source/g05.mdx

# OpenGalaxea G0.5

G0.5 is a Qwen3.5-2B vision-language-action model that can generate embodied
reasoning and actions in one stream. LeRobot exposes the action phase as System 1
and the optional native chain-of-thought phase as System 2. They are not separate
models: the runtime obtains both from one inference call and the action stays
conditioned on the same post-reasoning KV state.

Transformers includes the native multimodal Qwen3.5 backbone, vision tower, and
processor. G0.5 is not a stock `Qwen3_5ForConditionalGeneration` checkpoint,
however: it adds the proprioception/action path, action expert, flow-matching
head, ActionCodec, and unified CoT/action decode. LeRobot implements those G0.5
components natively and loads converted checkpoints without the OpenGalaxea
Python package, Hydra, or OmegaConf.

> [!WARNING]
> G0.5 code and checkpoints use the
> [G0.5 Community License](https://huggingface.co/OpenGalaxea/G05/blob/main/licenses/LICENSE-G0.5),
> including non-commercial restrictions. LeRobot does not redistribute checkpoint
> weights, download gated files, or imply that Apache-2.0 applies to those materials.
> Accept the license yourself and use a private or local checkpoint.

## Supported checkpoint contracts

| Profile          | Released action path                       | Raw → policy layout                          | Cameras                               | Predicted | Executed | Normalization                   |
| ---------------- | ------------------------------------------ | -------------------------------------------- | ------------------------------------- | --------: | -------: | ------------------------------- |
| `g05-base`       | Explicitly selected AR ActionCodec or flow | R1 Lite/Pro joints → 27D whole-body layout   | head + both wrists, 6-step history    |        32 |       16 | z-score-tail + q01/q99 grippers |
| `g05-libero`     | Continuous flow                            | right EEF 6 + gripper 1 → 20D grouped layout | exterior, right wrist                 |        32 |       10 | stepwise q01/q99                |
| `g05-robotwin20` | Continuous flow                            | two arms 6+gripper → 20D grouped layout      | high, left wrist, right wrist         |        32 |        8 | stepwise q01/q99                |
| `g05-so101`      | Flow or AR ActionCodec + native CoT        | right arm joints 6 → 20D grouped layout      | exterior, optional left + right wrist |        32 |       16 | stepwise q01/q99                |

Each packaged checkpoint stores the resolved model and processor configuration,
ActionCodec metadata, statistics, exact prompt template, and license. Loading
rejects a different head, horizon, processor mode, or
normalization contract.

The converted checkpoints are private under the LeRobot organization:

| Repository               | Contract                     |
| ------------------------ | ---------------------------- |
| `lerobot/g05_base`       | 27D base, R1 Lite processor  |
| `lerobot/g05_libero`     | 20D LIBERO                   |
| `lerobot/g05_robotwin20` | 20D RoboTwin                 |
| `lerobot/g05_so101`      | 20D SO-101 (`so100` profile) |

`lerobot/g05_base` supplies those 27D model weights, both action heads, the
ActionCodec tokenizer, and the released six-step R1 Lite processor/statistics
contract. OpenGalaxea's base release does not contain an `atomic_4` processor or
Atomic-4 dataset statistics. Loading the base weights for Atomic-4 therefore
requires an `atomic_4` `G05Config` plus statistics computed from the target
Atomic-4 dataset; reusing the R1 Lite statistics would be incorrect.

## Install

Install LeRobot with the G0.5 Transformers dependency:

```bash
uv sync --extra g05 --extra test
```

The LeRobot organization hosts the prepared base, LIBERO, RoboTwin, and SO-101
checkpoints privately. Authenticate with `hf auth login` before loading them.
SO-100 and SO-101 share the released `so100` embodiment token and six-joint
right-arm contract. A missing left-wrist camera is zero-padded exactly as in the
author deployment client.

## Interactive System 1 and System 2 runtime

System 1 executes the selected ActionCodec or flow chunk directly:

```bash
lerobot-rollout \
  --policy.path=lerobot/g05_so101 \
  --language --direct_subtask \
  --task="pick up the cup" \
  --mode=action
```

System 2 is available only when the packaged checkpoint metadata has
`predict_cot=true`. The adapter forwards the operator task byte-for-byte and
returns CoT telemetry and the matching action chunk atomically. It never samples
`task_aug` text, launches a second planner, or feeds generated CoT back as a
replacement task.

```bash
lerobot-rollout \
  --policy.path=lerobot/g05_so101 \
  --language \
  --task="clear the table" \
  --mode=action
```

For a physical SO-101, start paused, confirm the camera indices and arm port,
then type `/action <task>`:

```bash
lerobot-rollout \
  --language \
  --policy.path=lerobot/g05_so101 \
  --policy.device=cuda \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower \
  --robot.cameras='{exterior: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist_right: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}' \
  --task="pick up the cube" \
  --mode=paused \
  --ctrl_hz=30 \
  --chunk_hz=1
```

The checkpoint is non-commercial and may be private; authenticate with
`hf auth login` before loading it. Do not add `--direct_subtask` when inspecting
the checkpoint's native System 2 CoT telemetry.

## Fine-tune with `lerobot-train`

G0.5 implements LeRobot's training surface natively: `forward` computes
assistant-token cross entropy and flow-matching loss, the policy exposes
VLM/vision/action optimizer groups, and the checkpoint can be saved, resumed,
and loaded by the normal LeRobot scripts.
For example, fine-tune the private SO-101 checkpoint on a LeRobot dataset:

```bash
export HF_USER=your_hf_username

lerobot-train \
  --dataset.repo_id=${HF_USER}/my_so101_dataset_annotated \
  --policy.path=lerobot/g05_so101 \
  --policy.device=cuda \
  --policy.recipe_path=recipes/g05_bbox_subtask.yaml \
  --policy.cot_bbox_camera=observation.images.exterior \
  --policy.repo_id=${HF_USER}/g05_so101_finetuned \
  --policy.private=true \
  --output_dir=outputs/train/g05_so101 \
  --job_name=g05_so101 \
  --batch_size=16 \
  --steps=10000 \
  --save_freq=1000
```

The bundled `g05_bbox_subtask.yaml` recipe resolves the active
`language_persistent` `subtask` and camera-scoped grounded `vqa` event at each
sample timestamp. It emits each optional BBox or Subtask target when that
annotation is present; unavailable annotations are skipped through the recipe's
existing `if_present` guard.

Grounded VQA boxes are converted from pixel-space `xyxy` JSON using the source
camera dimensions captured before image resizing, then serialized as G0.5
`<locXXXX>` tokens. Joint samples preserve the released checkpoint's
`BBox → Subtask → Action` order. The user/task conditioning tokens remain
masked; the native backend applies its language/action objective to the
assistant sequence.

Generate the required `subtask` and grounded `vqa` language columns with
`lerobot-annotate` as described in the
[annotation pipeline](./annotation_pipeline). The bundled recipe targets
`observation.images.exterior`; copy the YAML and change its camera-filtered
bindings when training an embodiment with a different grounded camera.

The SO-101 recipe uses AdamW at `8e-5` with 1,000 warmup steps. The packaged
LIBERO and RoboTwin configurations use their released `1e-5` recipe, with
1,000 and 500 warmup steps respectively. All profiles preserve G0.5's six
decay/no-decay parameter groups and the configured VLM and vision learning-rate
multipliers. Override these only when deliberately changing the author recipe:

```bash
--policy.optimizer_lr=2e-5 \
--policy.optimizer_backbone_lr_multiplier=0.5 \
--policy.optimizer_vision_lr_multiplier=0.1
```

The dataset must expose the state, action, camera, and task features matching the
selected checkpoint contract in the table above. For SO-101, use camera names
`exterior` and `wrist_right`; the optional `wrist_left` input is zero-filled.
Training System 2 language targets additionally requires the checkpoint's
annotated CoT fields; a normal LeRobot recording supplies action supervision but
does not synthesize CoT labels.

Resume a saved run with the standard LeRobot checkpoint:

```bash
lerobot-train \
  --config_path=outputs/train/g05_so101/checkpoints/last/pretrained_model/train_config.json \
  --resume=true
```

## Validation status

CPU unit tests cover factory loading, config incompatibilities, prompt pass-through,
LIBERO and `atomic_4` mappings, padding masks, inverse action projection, a finite
forward/backward/update, optimizer-group wiring, and save/reload parity:

```bash
uv run pytest tests/policies/g05 tests/runtime/test_g05_adapter.py -q
uv run ruff check src/lerobot/policies/g05 tests/policies/g05
```

The converted private `g05_base` checkpoint strict-loaded all 945 native model
tensors, and its ActionCodec sidecar strict-loaded all 208 tensors, with no
missing, unexpected, or shape-mismatched keys. Prompt token IDs and masks match
the released runtime exactly. ActionCodec code IDs match exactly; decoded values
differ only by normal floating-point noise. A batch-size-one System 1 flow smoke
produced a finite `[1, 32, 27]` action chunk on an RTX 5090.

The same native base checkpoint completed a real joint language/action forward
and backward on the RTX 5090. It produced finite `ce_loss=13.8141` and
`fm_loss=1.32647`, finite gradients for all 945 model tensors, all six optimizer
groups, and 19.32 GiB peak allocated CUDA memory. A System 2 smoke generated
`Subtask: grasp and lift the red cup with the right gripper` and a finite
same-pass `[1, 32, 27]` action in 1.07 seconds at 11.0 GiB peak.

A 50-episode LIBERO/RoboTwin success-rate comparison additionally requires the
matching simulator, task assets, reset seeds, and author evaluator; no task-level
benchmark number is claimed until that separate gate runs.

Once the pinned LIBERO simulator and assets are installed, run the matching
LeRobot rollout with the author camera names and relative control:

```bash
lerobot-eval \
  --policy.path=lerobot/g05_libero \
  --policy.device=cuda \
  --env.type=libero \
  --env.task=libero_goal \
  --env.control_mode=relative \
  --env.observation_height=512 \
  --env.observation_width=512 \
  '--env.camera_name_mapping={"agentview_image": "image", "robot0_eye_in_hand_image": "wrist_image"}' \
  --eval.batch_size=1 \
  --eval.n_episodes=50 \
  --seed=0 \
  --output_dir=outputs/g05-libero-lerobot-eval
```

The author-oracle command for the same 50-trial gate is:

```bash
LIBERO_CONFIG_PATH=$(pwd)/experiments/libero \
bash scripts/run/eval_libero.sh checkpoints/g05-libero/model.pt \
  --suites "libero_goal" \
  --num_trials 50 \
  --num_parallel 1 \
  --output_dir outputs/g05-libero-author-eval
```