mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-09 11:01:53 +00:00
feat(policies): add joint frame transform and hardware deployment docs for MolmoAct2
Add MolmoAct2StateFrameTransformStep and MolmoAct2ActionFrameTransformStep processor steps for cross-calibration compatibility on SO-100/101. Add joint_signs and joint_offsets config fields. Add hardware deployment section to molmoact2.mdx with camera naming convention, joint frame correction, and safety guidance.
This commit is contained in:
@@ -386,6 +386,68 @@ These results demonstrate MolmoAct2's strong performance across diverse robotic
|
||||
manipulation tasks. To reproduce them, follow the instructions in the LIBERO
|
||||
evaluation section.
|
||||
|
||||
## Hardware Deployment (lerobot-rollout)
|
||||
|
||||
LeRobot-format checkpoints are available on the Hub for direct use with
|
||||
`lerobot-rollout`. Each checkpoint uses specific camera names that must
|
||||
match your robot's camera configuration.
|
||||
|
||||
### Camera naming convention
|
||||
|
||||
Each checkpoint expects specific `observation.images.*` keys.
|
||||
If your robot cameras have different names, use `--rename_map` to map them:
|
||||
|
||||
| Checkpoint | Camera keys | Description |
|
||||
| ----------------------------- | ---------------------- | ------------------------ |
|
||||
| MolmoAct2-LIBERO-LeRobot | `image`, `wrist_image` | LIBERO sim cameras |
|
||||
| MolmoAct2-BimanualYAM-LeRobot | `top`, `left`, `right` | YAM 3-camera setup |
|
||||
| MolmoAct2-DROID-LeRobot | `cam0`, `cam1` | External + wrist |
|
||||
| MolmoAct2-SO100_101-LeRobot | `cam0`, `cam1` | Primary + secondary view |
|
||||
|
||||
Example with an SO-100 robot using top and side cameras:
|
||||
|
||||
```bash
|
||||
lerobot-rollout \
|
||||
--policy.path=lerobot/MolmoAct2-SO100_101-LeRobot \
|
||||
--rename_map='{"top": "cam0", "side": "cam1"}' \
|
||||
--robot.type=so100_follower \
|
||||
--robot.port=/dev/ttyACM0 \
|
||||
--robot.cameras='{
|
||||
top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30},
|
||||
side: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}
|
||||
}' \
|
||||
--task="pick up the red cube" --duration=30
|
||||
```
|
||||
|
||||
To use a wrist camera instead, just change the rename mapping:
|
||||
|
||||
```bash
|
||||
--rename_map='{"top": "cam0", "wrist": "cam1"}'
|
||||
```
|
||||
|
||||
### Joint frame transform (SO-100/101 zero-shot)
|
||||
|
||||
<Tip warning={true}>
|
||||
The MolmoAct2-SO100_101 checkpoint was trained on data that uses a different
|
||||
joint calibration convention than LeRobot >= 0.5.0. Without a frame
|
||||
correction, the arm may move in the wrong direction.
|
||||
|
||||
This affects both **zero-shot deployment** and **fine-tuning** from the
|
||||
original checkpoint. The pretrained weights expect the old convention, so
|
||||
all joint data (observations and actions) must be transformed to match.
|
||||
|
||||
The converted LeRobot checkpoint (`lerobot/MolmoAct2-SO100_101-LeRobot`)
|
||||
already includes this correction in its processor pipeline. If you convert
|
||||
or fine-tune the checkpoint yourself, set the following in the policy config:
|
||||
|
||||
- `joint_signs`: `[1, -1, 1, 1, 1, 1]` (flips shoulder_lift direction)
|
||||
- `joint_offsets`: `[0, 90, 90, 0, 0, 0]` (shifts shoulder_lift and elbow_flex by 90°)
|
||||
|
||||
See the [backward compatibility guide](./backwardcomp) for details on the
|
||||
calibration change.
|
||||
|
||||
</Tip>
|
||||
|
||||
## Differences From the Original Implementation
|
||||
|
||||
This LeRobot port is intended to match MolmoAct2 behavior while using LeRobot's
|
||||
|
||||
Reference in New Issue
Block a user