feat(policies): add joint frame transform and hardware deployment docs for MolmoAct2

Add MolmoAct2StateFrameTransformStep and MolmoAct2ActionFrameTransformStep
processor steps for cross-calibration compatibility on SO-100/101. Add
joint_signs and joint_offsets config fields. Add hardware deployment section
to molmoact2.mdx with camera naming convention, joint frame correction, and
safety guidance.
This commit is contained in:
Khalil Meftah
2026-06-26 14:16:18 +02:00
parent bef7ac1cca
commit a9322d1033
3 changed files with 170 additions and 0 deletions
+62
View File
@@ -386,6 +386,68 @@ These results demonstrate MolmoAct2's strong performance across diverse robotic
manipulation tasks. To reproduce them, follow the instructions in the LIBERO
evaluation section.
## Hardware Deployment (lerobot-rollout)
LeRobot-format checkpoints are available on the Hub for direct use with
`lerobot-rollout`. Each checkpoint uses specific camera names that must
match your robot's camera configuration.
### Camera naming convention
Each checkpoint expects specific `observation.images.*` keys.
If your robot cameras have different names, use `--rename_map` to map them:
| Checkpoint | Camera keys | Description |
| ----------------------------- | ---------------------- | ------------------------ |
| MolmoAct2-LIBERO-LeRobot | `image`, `wrist_image` | LIBERO sim cameras |
| MolmoAct2-BimanualYAM-LeRobot | `top`, `left`, `right` | YAM 3-camera setup |
| MolmoAct2-DROID-LeRobot | `cam0`, `cam1` | External + wrist |
| MolmoAct2-SO100_101-LeRobot | `cam0`, `cam1` | Primary + secondary view |
Example with an SO-100 robot using top and side cameras:
```bash
lerobot-rollout \
--policy.path=lerobot/MolmoAct2-SO100_101-LeRobot \
--rename_map='{"top": "cam0", "side": "cam1"}' \
--robot.type=so100_follower \
--robot.port=/dev/ttyACM0 \
--robot.cameras='{
top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30},
side: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}
}' \
--task="pick up the red cube" --duration=30
```
To use a wrist camera instead, just change the rename mapping:
```bash
--rename_map='{"top": "cam0", "wrist": "cam1"}'
```
### Joint frame transform (SO-100/101 zero-shot)
<Tip warning={true}>
The MolmoAct2-SO100_101 checkpoint was trained on data that uses a different
joint calibration convention than LeRobot >= 0.5.0. Without a frame
correction, the arm may move in the wrong direction.
This affects both **zero-shot deployment** and **fine-tuning** from the
original checkpoint. The pretrained weights expect the old convention, so
all joint data (observations and actions) must be transformed to match.
The converted LeRobot checkpoint (`lerobot/MolmoAct2-SO100_101-LeRobot`)
already includes this correction in its processor pipeline. If you convert
or fine-tune the checkpoint yourself, set the following in the policy config:
- `joint_signs`: `[1, -1, 1, 1, 1, 1]` (flips shoulder_lift direction)
- `joint_offsets`: `[0, 90, 90, 0, 0, 0]` (shifts shoulder_lift and elbow_flex by 90°)
See the [backward compatibility guide](./backwardcomp) for details on the
calibration change.
</Tip>
## Differences From the Original Implementation
This LeRobot port is intended to match MolmoAct2 behavior while using LeRobot's