admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-29 20:49:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

remove ort_providers

Browse Source
This commit is contained in:
Martino Russi
2026-07-29 17:01:17 +02:00
parent 9f0663e9e3
commit cdf5141688
5 changed files with 113 additions and 164 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/unitree_g1.mdx
+57 -1
View File
@@ -8,6 +8,15 @@
The Unitree G1 humanoid is now supported in LeRobot! You can teleoperate, train locomanipulation policies, test in sim, and more. Both 29 and 23 DoF variants are supported. The Unitree G1 humanoid is now supported in LeRobot! You can teleoperate, train locomanipulation policies, test in sim, and more. Both 29 and 23 DoF variants are supported.
<Tip>
**New: SONIC whole-body control.** The `SonicWholeBodyController` runs NVIDIA's
[GEAR-SONIC](https://huggingface.co/nvidia/GEAR-SONIC) decoder on the G1, turning a
64-D latent motion token into full-body joint targets at 50 Hz. This lets you drive
the robot from a VLA policy trained on SONIC motion tokens (token in → whole-body
motion out) with `lerobot-rollout`, in sim or on the physical robot. See
[Whole-body control with SONIC](#whole-body-control-with-sonic) below.
</Tip>
--- ---
## Part 1: Getting Started ## Part 1: Getting Started
@@ -59,7 +68,7 @@ lerobot-teleoperate \
--robot.controller=GrootLocomotionController --robot.controller=GrootLocomotionController
``` ```
This will launch a [MuJoCo sim instance](https://huggingface.co/lerobot/unitree-g1-mujoco/tree/main) for the G1. You can connect a gamepad to your machine before launching in order to control the robot's locomotion in sim. We support both [HolosomaLocomotionController](https://github.com/amazon-far/holosoma) and [GrootLocomotionController](https://github.com/NVlabs/GR00T-WholeBodyControl) via `--robot.controller`. This will launch a [MuJoCo sim instance](https://huggingface.co/lerobot/unitree-g1-mujoco/tree/main) for the G1. You can connect a gamepad to your machine before launching in order to control the robot's locomotion in sim. We support [HolosomaLocomotionController](https://github.com/amazon-far/holosoma), [GrootLocomotionController](https://github.com/NVlabs/GR00T-WholeBodyControl), and [SonicWholeBodyController](https://huggingface.co/nvidia/GEAR-SONIC) via `--robot.controller`.
- Press `9` to release the robot - Press `9` to release the robot
- Press `7` / `8` to increase / decrease waist height - Press `7` / `8` to increase / decrease waist height
@@ -290,6 +299,53 @@ lerobot-rollout \
--- ---
## Whole-body control with SONIC
The `SonicWholeBodyController` runs NVIDIA's [GEAR-SONIC](https://huggingface.co/nvidia/GEAR-SONIC)
decoder on the G1. Each 50 Hz tick it consumes a **64-D latent motion token** and emits
full-body joint targets — the encoder is bypassed, so a policy feeds tokens in and the
decoder turns them into motion. Before the first token arrives the controller holds a
neutral (idle) pose.
This makes the G1 drivable by a VLA policy trained to output SONIC motion tokens (token
as both `observation.state` and `action`, e.g. [`nepyope/sonic_walk`](https://huggingface.co/nepyope/sonic_walk))
using the standard `lerobot-rollout`. The controller always runs **onboard** the robot;
the laptop is a thin client that streams tokens and receives camera frames over ZMQ.
**On the robot** — start the server in handshake mode so it instantiates and runs the
controller onboard against local DDS at full rate:
```bash
cd ~/lerobot
python src/lerobot/robots/unitree_g1/run_g1_server.py --handshake --camera
```
**From your laptop** — run the token policy; `--robot.sonic_token_action=true` switches the
robot to the 64-D latent-token action/observation interface:
```bash
lerobot-rollout \
--policy.path=nepyope/sonic_walk \
--policy.device=cuda \
--robot.type=unitree_g1 \
--robot.is_simulation=false \
--robot.robot_ip=<ROBOT_IP> \
--robot.controller=SonicWholeBodyController \
--robot.sonic_token_action=true \
--robot.cameras='{"ego_view": {"type": "zmq", "server_address": "<ROBOT_IP>", "port": 5555, "camera_name": "head_camera", "width": 640, "height": 480, "fps": 30}}' \
--task="walk back and forth" \
--duration=1000 \
--fps=30
```
<Tip>
SONIC is a token-only decoder in LeRobot: the only input path is the 64-D latent
vector. To train your own token policy, expose the 64-D token as the action (a config
choice, e.g. `pi05` with a 64-D action dim) — no policy code changes are needed.
</Tip>
---
## Additional Resources ## Additional Resources
- [Unitree SDK Documentation](https://github.com/unitreerobotics/unitree_sdk2_python) - [Unitree SDK Documentation](https://github.com/unitreerobotics/unitree_sdk2_python)
src/lerobot/robots/unitree_g1/README.md
-103
View File
@@ -1,103 +0,0 @@
# Unitree G1 — SONIC decoder whole-body control
This package runs NVIDIA's **SONIC** decoder on the Unitree G1, in MuJoCo simulation or
on real hardware, driven by a **64-D latent motion token**. It is a pure-Python/ONNX
reimplementation of the decode half of the SONIC deploy stack (no `gear_sonic`/torch
dependency): the decoder maps a 64-D latent token + proprioception history into 50 Hz
joint-position targets for the robot's PD controller. The encoder is bypassed — a policy
(e.g. `nepyope/sonic_walk`) emits the token directly.
## Controllers
Selected with `--robot.controller=<ClassName>`:
| Controller | Purpose |
| ------------------------------ | --------------------------------------------------- |
| `SonicWholeBodyController` | SONIC decoder driven by a 64-D latent motion token |
| `GrootLocomotionController` | GR00T locomotion policy |
| `HolosomaLocomotionController` | Holosoma locomotion policy |
The rest of this document covers the SONIC token path.
Each tick the `SonicWholeBodyController` takes a 64-D latent token
(`motion_token.0.pos … motion_token.63.pos`) and decodes it directly (encoder bypassed).
Before the first token arrives it holds a captured **neutral token** (a stable standing
pose), then holds the last token received between ticks (the ~30 Hz token stream vs. the
~50 Hz control loop). On startup the controller **interpolates** from the robot's measured
pose into the policy's commanded target over ~3 s (no snap).
## Requirements
- `onnxruntime` (CPU) **or** `onnxruntime-gpu` (recommended). Verify with:
```bash
python -c "import onnxruntime as ort; print(ort.get_available_providers())"
```
- `mujoco` for simulation (`is_simulation=True`).
- The SONIC encoder/decoder ONNX models download automatically from the
`nvidia/GEAR-SONIC` Hub repo.
## Architecture: controller always runs onboard
The controller runs **on the robot**, never on the laptop. The laptop is a thin client:
it negotiates the controller with `run_g1_server` (handshake), then PUSHes the 64-D token
and reads back the `observation.state` echo + camera frames over ZMQ.
## Running a rollout (real robot)
On the robot — host the SONIC decoder + camera onboard:
```bash
python -m lerobot.robots.unitree_g1.run_g1_server --handshake \
--cameras "ego_view:/dev/v4l/by-path/platform-3610000.usb-usb-0:2.1:1.3-video-index0:640x480"
```
On the laptop — `lerobot-rollout` drives the thin client:
```bash
lerobot-rollout \
--policy.path=nepyope/sonic_walk \
--robot.type=unitree_g1 \
--robot.is_simulation=false --robot.onboard=false \
--robot.robot_ip=<ROBOT_IP> \
--robot.controller=SonicWholeBodyController --robot.sonic_token_action=true \
--robot.cameras='{ego_view: {type: zmq, server_address: <ROBOT_IP>, port: 5555, camera_name: ego_view, width: 640, height: 480, fps: 30}}' \
--task="walk back and forth" --device=cuda
```
## Training a token policy (no pi05 code patch)
The SONIC token interface needs **no modeling changes** to pi05. A 64-D token action is
handled entirely by config: pi05 builds its action projections straight from config
(`action_in_proj = nn.Linear(max_action_dim, …)`, `action_out_proj = nn.Linear(…,
max_action_dim)`), pads the action to `max_action_dim`, then slices back to the dataset's
action dim. Set both dims to 64 and the pad/slice is a no-op, so the full 64-D token is
supervised.
Requirements:
1. The dataset carries a 64-D `action` and 64-D `observation.state` (the motion tokens).
2. Pass the dims to `lerobot-train`:
```bash
lerobot-train \
--dataset.repo_id=nepyope/walk_back_and_forth \
--policy.type=pi05 \
--policy.max_action_dim=64 \
--policy.max_state_dim=64 \
--policy.chunk_size=50 --policy.n_action_steps=50
```
`nepyope/sonic_walk` was trained exactly this way (`config.json`: `max_action_dim=64`,
`max_state_dim=64`, `output_features.action.shape=[64]`). Same stock code path for train
and inference — the checkpoint's 64-wide `Linear`s load with unmodified pi05.
## Observation / action interface (token mode)
With `--robot.sonic_token_action=true` the robot advertises:
- action: 64-D `motion_token.{i}.pos` (the decoder consumes it directly),
- `observation.state`: 64-D `motion_token_state.{i}.pos` (the last commanded token,
echoed so a token-output VLA closes the loop on its own previous token),
plus the ego camera image. The controller always runs onboard (or in sim); it is never
built on the laptop client.
src/lerobot/robots/unitree_g1/controllers/holosoma_locomotion.py
+14 -13
View File
@@ -16,11 +16,9 @@
from __future__ import annotations from __future__ import annotations
import json
import logging import logging
import numpy as np import numpy as np
import onnx
import onnxruntime as ort import onnxruntime as ort
from huggingface_hub import hf_hub_download from huggingface_hub import hf_hub_download
@@ -28,6 +26,7 @@ from ..g1_utils import (
REMOTE_AXES, REMOTE_AXES,
G1_29_JointArmIndex, G1_29_JointArmIndex,
G1_29_JointIndex, G1_29_JointIndex,
compute_pd_gains,
get_gravity_orientation, get_gravity_orientation,
) )
@@ -59,12 +58,23 @@ POLICY_FILES = {
"ppo": "ppo_g1_29dof.onnx", "ppo": "ppo_g1_29dof.onnx",
} }
# Per-joint motor model in Holosoma's joint order, plus the joints that get a x2
# stiffness/damping factor. These reproduce the kp/kd that used to be read from the
# policy's ONNX metadata exactly (both fastsac and ppo), so gains are now derived from
# the shared motor model (see g1_utils.compute_pd_gains) instead.
HOLOSOMA_MOTOR_MODELS = (
["7520_14", "7520_22", "7520_14", "7520_22", "5020", "5020"] * 2
+ ["7520_14", "5020", "5020"]
+ ["5020", "5020", "5020", "5020", "5020", "4010", "4010"] * 2
)
HOLOSOMA_DOUBLE = {4, 5, 10, 11, 13, 14}
def load_policy( def load_policy(
repo_id: str = DEFAULT_HOLOSOMA_REPO_ID, repo_id: str = DEFAULT_HOLOSOMA_REPO_ID,
policy_type: str = "fastsac", policy_type: str = "fastsac",
) -> tuple[ort.InferenceSession, np.ndarray, np.ndarray]: ) -> tuple[ort.InferenceSession, np.ndarray, np.ndarray]:
"""Load Holosoma locomotion policy and extract KP/KD from metadata. """Load the Holosoma locomotion policy and its motor-model-derived PD gains.
Args: Args:
repo_id: Hugging Face Hub repo ID repo_id: Hugging Face Hub repo ID
@@ -83,16 +93,7 @@ def load_policy(
policy = ort.InferenceSession(policy_path) policy = ort.InferenceSession(policy_path)
logger.info(f"Policy loaded: {policy.get_inputs()[0].shape} → {policy.get_outputs()[0].shape}") logger.info(f"Policy loaded: {policy.get_inputs()[0].shape} → {policy.get_outputs()[0].shape}")
# Extract KP/KD from ONNX metadata kp, kd = compute_pd_gains(HOLOSOMA_MOTOR_MODELS, HOLOSOMA_DOUBLE)
model = onnx.load(policy_path, load_external_data=False)
metadata = {prop.key: prop.value for prop in model.metadata_props}
if "kp" not in metadata or "kd" not in metadata:
raise ValueError("ONNX model must contain 'kp' and 'kd' in metadata")
kp = np.array(json.loads(metadata["kp"]), dtype=np.float32)
kd = np.array(json.loads(metadata["kd"]), dtype=np.float32)
logger.info(f"Loaded KP/KD from ONNX ({len(kp)} joints)")
return policy, kp, kd return policy, kp, kd
src/lerobot/robots/unitree_g1/controllers/sonic_whole_body.py
+16 -39
View File
@@ -38,12 +38,14 @@ from huggingface_hub import hf_hub_download
from ..g1_utils import ( from ..g1_utils import (
ISAACLAB_TO_MUJOCO, ISAACLAB_TO_MUJOCO,
MOTOR_ARMATURE,
MUJOCO_TO_ISAACLAB, MUJOCO_TO_ISAACLAB,
NATURAL_FREQ,
G1_29_JointIndex, G1_29_JointIndex,
compute_pd_gains,
get_gravity_orientation, get_gravity_orientation,
lowstate_to_obs, lowstate_to_obs,
make_ort_session_options, make_ort_session_options,
ort_providers,
) )
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -65,16 +67,14 @@ DEFAULT_ANGLES = np.array(
dtype=np.float32, dtype=np.float32,
) )
# Per-motor-type parameters used to derive action scaling and PD gains. Keys are Unitree # Per-motor torque limits (N·m), used only for SONIC's residual-action scaling. The
# motor model names; ARMATURE = rotor inertia, EFFORT = torque limit (N·m). # armature / bandwidth constants and the PD-gain formula are shared (see g1_utils).
NATURAL_FREQ = 10.0 * 2.0 * np.pi # target closed-loop stiffness bandwidth (rad/s)
ARMATURE = {"5020": 0.003609725, "7520_14": 0.010177520, "7520_22": 0.025101925, "4010": 0.00425}
EFFORT = {"5020": 25.0, "7520_14": 88.0, "7520_22": 139.0, "4010": 5.0} EFFORT = {"5020": 25.0, "7520_14": 88.0, "7520_22": 139.0, "4010": 5.0}
def _action_scale(k): def _action_scale(k):
"""Per-motor residual-action scale (maps policy output to joint-angle delta).""" """Per-motor residual-action scale (maps policy output to joint-angle delta)."""
return 0.25 * EFFORT[k] / (ARMATURE[k] * NATURAL_FREQ**2) return 0.25 * EFFORT[k] / (MOTOR_ARMATURE[k] * NATURAL_FREQ**2)
# Per-joint motor model (IsaacLab order): legs, waist, then arms. Single source of truth # Per-joint motor model (IsaacLab order): legs, waist, then arms. Single source of truth
@@ -101,23 +101,13 @@ def _to_mujoco(a):
DEFAULT_ANGLES_MUJOCO = _to_mujoco(DEFAULT_ANGLES) DEFAULT_ANGLES_MUJOCO = _to_mujoco(DEFAULT_ANGLES)
# Ankle + waist joint indices (IsaacLab order) that get a x2 stiffness/damping factor.
_SONIC_DOUBLE = {4, 5, 10, 11, 13, 14}
def compute_kp_kd(): def compute_kp_kd():
"""Derive per-joint PD gains (kp, kd) from motor armature and target bandwidth. """SONIC per-joint PD gains (kp, kd), (29,) float32 in IsaacLab joint order."""
return compute_pd_gains(MOTOR_MODELS, _SONIC_DOUBLE)
Ankle and waist joints get a x2 factor for extra stiffness. Returns two (29,) float32
arrays in IsaacLab joint order.
"""
def s(k):
return ARMATURE[k] * NATURAL_FREQ**2
def d(k):
return 2.0 * 2.0 * ARMATURE[k] * NATURAL_FREQ
_double = {4, 5, 10, 11, 13, 14} # ankle + waist indices with factor 2
kp = np.array([2 * s(k) if i in _double else s(k) for i, k in enumerate(MOTOR_MODELS)], dtype=np.float32)
kd = np.array([2 * d(k) if i in _double else d(k) for i, k in enumerate(MOTOR_MODELS)], dtype=np.float32)
return kp, kd
# Action-feature prefix for the latent-token interface (see _extract_token_from_action). # Action-feature prefix for the latent-token interface (see _extract_token_from_action).
@@ -303,24 +293,11 @@ class SonicRuntime:
latent token supplied directly by the policy. latent token supplied directly by the policy.
""" """
def __init__(self, force_cpu: bool = False): def __init__(self):
decoder_path = hf_hub_download(repo_id="nvidia/GEAR-SONIC", filename="model_decoder.onnx") decoder_path = hf_hub_download(repo_id="nvidia/GEAR-SONIC", filename="model_decoder.onnx")
providers = ort_providers(force_cpu=force_cpu)
so = make_ort_session_options() so = make_ort_session_options()
decoder_sess = ort.InferenceSession(decoder_path, sess_options=so, providers=providers) decoder_sess = ort.InferenceSession(decoder_path, sess_options=so)
# Report the provider actually bound, not the one requested: ORT silently falls back
# to CPU if CUDA can't load (e.g. libcudnn not on LD_LIBRARY_PATH), and a CPU decoder
# drifts the closed-loop heading. Warn loudly so it can't hide.
self.use_gpu = decoder_sess.get_providers()[0] == "CUDAExecutionProvider"
if not force_cpu and not self.use_gpu:
print(
"[SONIC] WARNING: decoder bound to CPUExecutionProvider (CUDA unavailable). "
"Closed-loop replay/control will drift. Ensure libcudnn is on LD_LIBRARY_PATH "
"(site-packages/nvidia/*/lib).",
flush=True,
)
self.kp, self.kd = compute_kp_kd() self.kp, self.kd = compute_kp_kd()
self.controller = SonicDecoder(decoder_sess) self.controller = SonicDecoder(decoder_sess)
@@ -342,9 +319,9 @@ class SonicWholeBodyController:
control_dt = CONTROL_DT control_dt = CONTROL_DT
full_body = True full_body = True
def __init__(self, force_cpu: bool = False): def __init__(self):
logger.info("Loading SONIC whole-body controller...") logger.info("Loading SONIC whole-body controller...")
self._runtime = SonicRuntime(force_cpu=force_cpu) self._runtime = SonicRuntime()
self.kp = self._runtime.kp self.kp = self._runtime.kp
self.kd = self._runtime.kd self.kd = self._runtime.kd
self.controller = self._runtime.controller self.controller = self._runtime.controller
src/lerobot/robots/unitree_g1/g1_utils.py
+26 -8
View File
@@ -119,15 +119,33 @@ def get_gravity_orientation(quaternion: list[float] | np.ndarray) -> np.ndarray:
return gravity_orientation return gravity_orientation
def ort_providers(force_cpu: bool = False) -> list[str]: # Unitree motor-model parameters shared by the controllers that derive their PD gains
"""ONNX Runtime providers, preferring CUDA when available (shared by the ONNX # from motor physics rather than hand-tuning (SONIC decoder, Holosoma). NATURAL_FREQ is
controllers: SONIC decoder, GR00T). Falls back to CPU.""" # the target closed-loop stiffness bandwidth (rad/s); MOTOR_ARMATURE is per-model rotor
import onnxruntime as ort # inertia (keys are Unitree motor model names). From these: kp = armature * w**2 and
# kd = 4 * armature * w, with an optional x2 factor on stiff joints (ankles/waist).
NATURAL_FREQ = 10.0 * 2.0 * np.pi
MOTOR_ARMATURE = {"5020": 0.003609725, "7520_14": 0.010177520, "7520_22": 0.025101925, "4010": 0.00425}
avail = ort.get_available_providers()
if not force_cpu and "CUDAExecutionProvider" in avail: def compute_pd_gains(motor_models, double_indices=()) -> tuple[np.ndarray, np.ndarray]:
return ["CUDAExecutionProvider", "CPUExecutionProvider"] """Derive per-joint PD gains (kp, kd) from motor armature and target bandwidth.
return ["CPUExecutionProvider"]
``motor_models`` is a per-joint sequence of Unitree motor model names (in the
controller's own joint order); joints whose index is in ``double_indices`` get a
x2 stiffness/damping factor. Returns two (N,) float32 arrays in that same order.
"""
double = set(double_indices)
def s(k):
return MOTOR_ARMATURE[k] * NATURAL_FREQ**2
def d(k):
return 4.0 * MOTOR_ARMATURE[k] * NATURAL_FREQ
kp = np.array([2 * s(k) if i in double else s(k) for i, k in enumerate(motor_models)], dtype=np.float32)
kd = np.array([2 * d(k) if i in double else d(k) for i, k in enumerate(motor_models)], dtype=np.float32)
return kp, kd
def make_ort_session_options(intra_op_num_threads: int | None = None, inter_op_num_threads: int | None = None): def make_ort_session_options(intra_op_num_threads: int | None = None, inter_op_num_threads: int | None = None):