The all-zero token is off the encoder's learned FSQ manifold and decodes to a
slightly goofy stance. Replace it with a NEUTRAL_TOKEN captured from the encoder's
own idle output in sim (stored as integer FSQ codes, rescaled by the encoder's
1/16 quantization step to an exact on-grid token). token_mode now seeds this
neutral, and the onboard sender starts observation.state from it so the first
inference sees the token the decoder is actually holding.
Move the token-hold idle logic into SonicWholeBodyController via a
token_mode flag (set by UnitreeG1 when sonic_token_action is enabled):
before any real token arrives the decoder is fed the all-zero neutral
token (stable neutral stance), and afterwards the last received token is
held between control ticks (the ~50 Hz control loop outruns the ~30 Hz
token stream). Living in the controller, this applies uniformly to
run_g1_onboard, lerobot-rollout and the sim replays, so the explicit
neutral seeding in run_g1_onboard is removed.
Run the whole-body controller (SONIC decoder / GR00T) onboard the G1 against
local DDS at full rate, with the laptop shipping only high-level actions over
ZMQ instead of 50Hz lowcmd via the socket bridge.
- config: add onboard, dds_interface, release_motion_control, physical_remote
- unitree_g1: onboard connect() branch (local DDS + MotionSwitcher release +
physical wireless remote), _release_motion_control, _wireless_remote_input,
controller-loop wireless priority; SDK channels when sim OR onboard
- run_g1_server: port Gripper/build_gripper/parse_camera_specs; add --cameras
spec supporting by-path device names (survive USB re-enumeration) + FOURCC
- run_g1_onboard: onboard entry point (ZMQ actions -> send_action), with a
--sonic-token-action flag for the 64-D latent-token interface
- infer_sonic_g1_onboard: laptop-side sender that runs nepyope/sonic_walk
(pi0.5) and PUSHes 64-D tokens to the onboard controller
Add a token-output VLA path (sonic_token_action) so a policy trained on 64-D SONIC
motion tokens (e.g. nepyope/sonic_walk) drives the decoder directly via lerobot-rollout:
the robot advertises a 64-D motion_token.{i}.pos action and echoes the last commanded
token as a 64-D observation.state (motion_token_state.{i}.pos), encoder bypassed.
Also:
- gr00t_locomotion: allow an external upper-body IK to override the 3 waist joints, and
cap ORT to 1 intra/inter thread so the 50Hz loop doesn't stutter under contention.
- sonic_pipeline: make_ort_session_options takes optional thread caps; report the
provider actually bound.
- unitree_g1: build the sim env with publish_images=False/cameras=[] to avoid the
offscreen EGL context crash (we drive image policies from recorded/live frames), and
guard the startup sim-step race (zero-norm pelvis quat) so the sim thread survives.
* (depth image processing): excluding depth frames from the RGB to BGR image processing
* test(update): updating tests to include RGB/BGR conversion checks
* fix(deps): bump diffusers cap to <0.39.0 (security)
Diffusers 0.35.x is affected by GHSA-98h9-4798-4q5v (HIGH, CVSS 8.8):
'trust_remote_code bypass via custom_pipeline and local custom components'.
Fixed in diffusers 0.38.0.
Current cap 'diffusers<0.36.0' blocks downstream consumers (e.g.
strands-labs/robots) from picking up the security fix.
The lerobot diffusers surface area is narrow and stable across 0.36-0.38:
- diffusers.schedulers.scheduling_ddim.DDIMScheduler
- diffusers.schedulers.scheduling_ddpm.DDPMScheduler
- diffusers.optimization.get_scheduler
- diffusers.ConfigMixin / ModelMixin / register_to_config
- diffusers.models.attention.{Attention,FeedForward}
- diffusers.models.embeddings.*
None of these were removed, renamed, or had breaking changes in 0.36, 0.37,
or 0.38 release notes. Bumping the cap to <0.39.0 unblocks the security
fix while keeping a major-version safety bound.
* chore(dependecies): bump diffusers
* chore(deps): update uv.lock
---------
Co-authored-by: Cagatay Cali <cagataycali@users.noreply.github.com>
* Make SO follower P coefficient configurable
* chore(test): minimize tests
* feat(robots): expose PID coeff in SO arms
---------
Co-authored-by: taivu1998 <46636857+taivu1998@users.noreply.github.com>
* chore(agents): adding additional infos to AGENTS.md
* adding `lerobot-train` requirement inside PR checklist
* prefer using code already implemented from transformers / diffusers instead of re-implementing in tree
---------
Signed-off-by: Maxime Ellerbach <maxime.ellerbach@huggingface.co>
- reset(): pause the background controller and, for full-body controllers,
publish the default pose directly (new _controller_paused flag) so reset and
the controller loop aren't both writing low commands.
- SONIC pipeline: add reset() to StandingEncoderDecoder and PlannerController
(clear token/proprio history/heading, rewind motion buffer); SonicRuntime.reset()
now calls controller.reset().
- sonic_whole_body: require the full dense 34-D command (no silent zero-fill of a
partial action) and integrate yaw-rate (idx 33) into heading.
- controllers/__init__: import the controller classes referenced in __all__.
- unitree_g1: lazy replay-frame decode + small cache instead of decoding all
frames up front; safer disconnect (longer controller-thread join + fail-safe
that skips the graceful ramp if the thread won't stop).
- lint: ruff-format config_unitree_g1 hand_closed_pose; prettier README table.
* fix(envs): set LiberoEnvConfig.fps default to 20 to match robosuite
LiberoEnvConfig.fps was set to 30, but the underlying robosuite
OffScreenRenderEnv always runs at its default control_freq of 20 Hz
since fps is never passed through. This mismatch silently decouples
the dataset/eval loop rate from the actual simulation step rate.
Set the default to 20 to match the real sim rate and avoid the
footgun.
Fixes#3368
* fix(libero): apply configured control frequency
---------
Co-authored-by: xinmotlanthua <275663218+xinmotlanthua@users.noreply.github.com>
* fix(damiao): make is_calibrated a plain property, not cached
`is_calibrated` was a `@cached_property`, so it froze at its first-read
value and never reflected later changes to `self.calibration` (set by
connect/calibrate/load). This caused the OpenArm teleop to re-run
calibration even when a calibration file existed, and to skip
`set_zero_position()` after a fresh calibration.
Switch to `@property` (matching the MotorsBus base contract and the
Feetech/SO-100 buses) and drop the now-unused `functools.cached_property`
import.
Co-authored-by: Cursor <cursoragent@cursor.com>
* don't set_zero_position() on connect
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(diffusion): add gradient checkpointing for memory optimization
Add gradient_checkpointing config option to DiffusionPolicy. When
enabled, wraps the UNet encoder, mid, and decoder residual blocks
with torch.utils.checkpoint.checkpoint to trade compute for memory.
Allows training with larger batch sizes or higher-resolution inputs
on memory-constrained GPUs. Disabled by default.
Usage: --policy.gradient_checkpointing=true
Part of the 0.6.0 roadmap item 3.3 (gradient checkpointing for all
policies).
* test(diffusion): verify gradient checkpointing parity
---------
Co-authored-by: Jash Shah <jashshah.999@gmail.com>
* gamepad does often work on macos
* review comments
* fix(gamepad): expose hidapi fallback in config
---------
Co-authored-by: Maxim Bonnaerens <maxim@bonnaerens.be>
* feat(annotate): run lerobot-annotate on HF Jobs via --job.target
Annotation needed a hand-edited launcher script (examples/annotations/run_hf_job.py)
to reach a GPU: users copied it, rewrote the embedded CMD string for their dataset,
and ran it with `python`. Fold that into the CLI instead, mirroring `lerobot-train`:
`lerobot-annotate --job.target=h200` submits the exact command you'd run locally.
- AnnotationJobConfig extends JobConfig with the annotation runtime's defaults
(vllm/vllm-openai image, 2h cap) plus --job.lerobot_ref, so an unmerged branch
can be exercised remotely without editing a script.
- lerobot.jobs.annotate builds the pod command by replaying the user's own CLI
flags (minus --job.*/--root, with --repo_id re-emitted from the config) after a
setup prelude that installs lerobot on top of the vLLM image. Job monitoring,
log tailing and Ctrl-C-detaches reuse the training submitter's plumbing.
- Remote runs require --repo_id; a local-only dataset is pushed privately first.
The generated pod command is byte-for-byte the script's old CMD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(annotate): reject client-side config files on remote runs
draccus exposes `--config_path` plus a `--<field>` config-file arg for every
nested dataclass (`--vlm`, `--plan`, `--job`, ...). All name files on the
client's disk, so forwarding them to the pod silently dropped whatever settings
they carried. Reject them up front instead.
Bare `--job` also slipped past the `--job.` prefix filter, so a `--job=cfg.yaml`
holding `target: h200` would have reached the pod and had the job submit a job
of its own, recursively. It is dropped from the forwarded args as well.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(jobs): share the submit-and-follow loop between both submitters
`submit_annotate_to_hf` reused the leaf helpers (`_poll_until_done`, `_tail_logs`,
`_pod_forwarded_args`) but duplicated the orchestration around them: ~40 of the 50
lines that spawn the poll/log threads, install the Ctrl-C-detaches handler and
raise on a non-COMPLETED stage were identical in both files.
Extract that into `follow_job(job_id, *, detach, success_marker=None) -> bool`,
returning True when the job finished and False when we stopped watching without a
verdict (detach or Ctrl-C). Training keeps its model-pushed marker by passing it in;
annotation has no equivalent line (the CLI keeps working after the upload log to
write the card and tag) so its completion stays stage-based.
Kept in hf.py rather than a new module so every existing monkeypatch target in
test_hf.py still resolves.
Behaviour change: a training run whose job reaches COMPLETED without the marker
matching now prints its completion line instead of returning silently. The marker
was already documented as an optimisation with a stage-based fallback; the fallback
just never reported success.
Tests: adds annotate coverage for the non-detach path (completion and failure) —
previously only ever exercised with detach=true — plus a detach short-circuit test.
Both new annotate tests verified to fail under a mutation that stubs out follow_job.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Strip everything except the OpenHLM/pi0.5 -> SONIC encoder/decoder rollout
path so this branch does exactly that and nothing more:
- Remove the SONIC motion planner (planner ONNX + subprocess worker, PlannerMotion,
replanning, MovementState/LocomotionMode, joystick) from sonic_pipeline; keep the
encoder/decoder and the caller-fed reference buffer (PlannerController) intact.
- Slim SonicRuntime to load only the encoder/decoder; SonicWholeBodyController now
runs solely the 34-D whole-body command path (drop SMPL/VR3/keyboard teleop).
- Delete the pico_headset teleoperator (SONIC's SMPL/VR3 teleop source).
- Move WB action constants into g1_utils; repoint imports.
GR00T/Holosoma locomotion controllers are left untouched.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a dense 34-D whole-body command path so lerobot-rollout can drive the
G1 directly with an OpenHLM / pi0.5 policy through the SONIC encoder/decoder:
- SonicWholeBodyController: wb.{i}.pos action interface, mode-0 reference with
a rolling 50-frame trajectory (finite-diff velocities) and first-tick anchor
init; correct MuJoCo->IsaacLab joint reordering.
- unitree_g1: expose 34-D wb_state.{i}.pos proprio; empty/replay camera feeds
for image-conditioned policies; Dex3 hand publishing from the grip scalars.
- g1_utils: obs_to_wb34_state + WB action constants.
Co-authored-by: Cursor <cursoragent@cursor.com>
MihaiAnca13