Compare commits

..

83 Commits

Author SHA1 Message Date
Martino Russi 6662431923 feat(unitree_g1): derive PD gains from motor physics
Add compute_pd_gains / compute_action_scale to g1_utils (kp = armature*w^2,
kd = 4*armature*w, with a 2x factor on stiff ankle/waist joints), plus the G1
motor-model layout and shared NATURAL_FREQ/armature/effort constants. The config
now derives its default kp/kd from these instead of hand-tuned values.

Both controllers stop reading kp/kd from ONNX metadata: SONIC uses the config
gains (identical to its baked values), and Holosoma derives them with a hip-pitch
override (7520_14 instead of 7520_22) to match its checkpoint.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-31 13:48:27 +02:00
Martino Russi 09fb2b4580 update docs 2026-07-31 12:55:20 +02:00
Martino Russi 90699fdece inline token helpers in wbc, move properties to init 2026-07-31 11:50:39 +02:00
Martino Russi d1eed119a6 refactor 2026-07-31 11:43:17 +02:00
Martino Russi 87b0dc470c dedup reset 2026-07-31 11:25:08 +02:00
Martino Russi ec6da5c92e reduce verbosity 2026-07-31 11:09:59 +02:00
Martino Russi d29fecfa2c refactor sonic_wbc
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-31 10:48:37 +02:00
Martino Russi 1799eceb22 make every controller consume lowstate for obs 2026-07-31 10:31:11 +02:00
Martino Russi 89ac5b4408 remove comments 2026-07-31 10:00:22 +02:00
Martino Russi a8ea5e6ea0 (refactor) deduplicate _clear_history(), remove unused debug 2026-07-31 09:53:36 +02:00
Martino Russi 6ea5dd95e9 merge sonic runtime into sonicwbc 2026-07-30 16:22:42 +02:00
Martino Russi f8359c19d6 remove verbose 2026-07-30 16:13:58 +02:00
Martino Russi f8521958d8 (refactor) move sonic logic to controller 2026-07-30 16:00:35 +02:00
Martino Russi 35341dc603 restore REMOTE_KEYS 2026-07-30 15:53:33 +02:00
Martino Russi 686ac2cc21 move controller logic out of unitree_g1, reduce verbosity 2026-07-30 15:45:17 +02:00
Martino Russi 48e2530455 (refactor) match main sim logic, move extract_token_from_action to controller 2026-07-30 15:38:29 +02:00
Martino Russi b6a283193c (refactor) restore remote_keys 2026-07-30 15:22:38 +02:00
Martino Russi 44edbd704f refactor(unitree_g1): revert reset() to simple version
Drop the full-body pause/direct-publish reset special-casing (and the
_controller_paused plumbing) so SONIC uses the same reset path as the other
controllers. The removed logic is preserved on the better-reset branch.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-30 15:05:51 +02:00
Martino Russi 4cd9fc73f8 (refactor) token mode is default 2026-07-30 14:50:44 +02:00
Martino Russi 57d33c295e delete lowstate_to_obs 2026-07-30 14:48:09 +02:00
Martino Russi 6e23429105 inline mujoco indexing 2026-07-30 14:37:38 +02:00
Martino Russi fffa42cc5e refactor(unitree_g1): load SONIC constants from onnx 2026-07-30 13:44:24 +02:00
Martino Russi f33089b027 style(unitree_g1): ruff-format blank line in g1_utils
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-30 12:07:02 +02:00
Martino Russi 9f32f57b59 chore(unitree_g1): silence bandit B105 on SONIC token prefixes
The motion_token / motion_token_state constants are feature-key prefixes, not
secrets; mark them #nosec so the pre-commit bandit hook passes.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-30 12:03:37 +02:00
Martino Russi 0a9b9d9a93 refactor(unitree_g1): move lowstate_to_obs out of g1_utils into unitree_g1
Keep g1_utils close to main: the lowstate -> obs mapping lives in unitree_g1
(where get_observation builds it, left unchanged from main) and the SONIC
controller imports it from there.
2026-07-30 11:56:57 +02:00
Martino Russi 16b915ede5 refactor(unitree_g1): drop make_ort_session_options from SONIC branch
The ORT SessionOptions helper (thread-pool cap) is being split into its own
"ORT fix" PR. SONIC only needs quiet logs, so inline a minimal SessionOptions in
the decoder instead of depending on the shared helper.
2026-07-30 11:49:48 +02:00
Martino Russi 95d9029039 refactor(unitree_g1): reduce branch to SONIC-only diff vs main
Split the onboard-controller server/handshake/thin-client work out to
feat/g1_onboard_controller and revert it here:

- unitree_g1.py: drop client + onboard roles, wireless-remote parsing and
  motion-service release; restore main's sim/socket-bridge transport. Keep only
  the SONIC integration (implicit token action/state, full-body reset/pause,
  controller kp/kd + shutdown).
- config: drop onboard/dds_interface/release_motion_control/physical_remote.
- run_g1_server.py + unitree_g1.mdx: reverted to main.
- gr00t/holosoma: keep the move into controllers/ but revert their content to
  main (only the package-relative import changes).
2026-07-30 11:40:01 +02:00
Martino Russi bbfc4ff443 feat(unitree_g1): make SONIC token interface implicit
Drop the ``sonic_token_action`` config flag; the 64-D latent-token
action/observation interface now switches on automatically whenever the
SONIC whole-body controller is selected (``controller == "SonicWholeBodyController"``).
Keyed via a ``_sonic_token`` property so client, onboard and sim roles agree.
2026-07-30 11:30:07 +02:00
Martino Russi 0c57cd03f2 Merge branch 'main' into feat/sonic_encoder_decoder 2026-07-30 11:15:18 +02:00
Martino Russi 962ed236af config cleanup 2026-07-29 18:45:49 +02:00
Martino Russi 3ae036ee40 restore gravity_compensation 2026-07-29 18:12:19 +02:00
Martino Russi cdf5141688 remove ort_providers 2026-07-29 17:01:17 +02:00
Martino Russi 9f0663e9e3 clean init and utils 2026-07-29 16:43:36 +02:00
Martino Russi 503f3e57ae refactor(unitree_g1): minimize diff vs main (drop onnx guards, e-stop)
Revert pyproject.toml and import_utils.py to main: the onnxruntime-gpu
detection fallback and the _onnxruntime_available/_onnx_available flags
are unnecessary. Controllers now import onnx/onnxruntime unconditionally
(matching main) which also works with onnxruntime-gpu since the import
name is stable. Drop the require_package() calls we had added.

Remove the stdin e-stop listener from serve_onboard_controller and the
now-unused os/sys imports; Ctrl-C still triggers graceful shutdown.
2026-07-29 16:11:55 +02:00
Martino Russi 6d24f20eb4 drop stray artifacts 2026-07-29 15:47:46 +02:00
Martino Russi af163fd032 refactor(unitree_g1): minimize diff w.r.t main 2026-07-29 15:46:17 +02:00
Martino Russi 77259f436e feat(unitree_g1): use captured neutral SONIC token instead of zeros
The all-zero token is off the encoder's learned FSQ manifold and decodes to a
slightly goofy stance. Replace it with a NEUTRAL_TOKEN captured from the encoder's
own idle output in sim (stored as integer FSQ codes, rescaled by the encoder's
1/16 quantization step to an exact on-grid token). token_mode now seeds this
neutral, and the onboard sender starts observation.state from it so the first
inference sees the token the decoder is actually holding.
2026-07-27 11:05:42 +02:00
Martino Russi 85f5c3606d feat(unitree_g1): hold neutral SONIC token until first command
Move the token-hold idle logic into SonicWholeBodyController via a
token_mode flag (set by UnitreeG1 when sonic_token_action is enabled):
before any real token arrives the decoder is fed the all-zero neutral
token (stable neutral stance), and afterwards the last received token is
held between control ticks (the ~50 Hz control loop outruns the ~30 Hz
token stream). Living in the controller, this applies uniformly to
run_g1_onboard, lerobot-rollout and the sim replays, so the explicit
neutral seeding in run_g1_onboard is removed.
2026-07-27 10:34:42 +02:00
Martino Russi b587e81587 feat(unitree_g1): onboard controller deployment for SONIC walk
Run the whole-body controller (SONIC decoder / GR00T) onboard the G1 against
local DDS at full rate, with the laptop shipping only high-level actions over
ZMQ instead of 50Hz lowcmd via the socket bridge.

- config: add onboard, dds_interface, release_motion_control, physical_remote
- unitree_g1: onboard connect() branch (local DDS + MotionSwitcher release +
  physical wireless remote), _release_motion_control, _wireless_remote_input,
  controller-loop wireless priority; SDK channels when sim OR onboard
- run_g1_server: port Gripper/build_gripper/parse_camera_specs; add --cameras
  spec supporting by-path device names (survive USB re-enumeration) + FOURCC
- run_g1_onboard: onboard entry point (ZMQ actions -> send_action), with a
  --sonic-token-action flag for the 64-D latent-token interface
- infer_sonic_g1_onboard: laptop-side sender that runs nepyope/sonic_walk
  (pi0.5) and PUSHes 64-D tokens to the onboard controller
2026-07-26 21:36:02 +02:00
Martino Russi 4658dada9b feat(unitree_g1): 64-D SONIC token interface for lerobot-rollout + GR00T waist override
Add a token-output VLA path (sonic_token_action) so a policy trained on 64-D SONIC
motion tokens (e.g. nepyope/sonic_walk) drives the decoder directly via lerobot-rollout:
the robot advertises a 64-D motion_token.{i}.pos action and echoes the last commanded
token as a 64-D observation.state (motion_token_state.{i}.pos), encoder bypassed.

Also:
- gr00t_locomotion: allow an external upper-body IK to override the 3 waist joints, and
  cap ORT to 1 intra/inter thread so the 50Hz loop doesn't stutter under contention.
- sonic_pipeline: make_ort_session_options takes optional thread caps; report the
  provider actually bound.
- unitree_g1: build the sim env with publish_images=False/cameras=[] to avoid the
  offscreen EGL context crash (we drive image policies from recorded/live frames), and
  guard the startup sim-step race (zero-norm pelvis quat) so the sim thread survives.
2026-07-26 20:49:32 +02:00
Martino Russi 57ea6f4106 feat(unitree_g1): episode reset, lazy replay decode, safe shutdown
- reset(): pause the background controller and, for full-body controllers,
  publish the default pose directly (new _controller_paused flag) so reset and
  the controller loop aren't both writing low commands.
- SONIC pipeline: add reset() to StandingEncoderDecoder and PlannerController
  (clear token/proprio history/heading, rewind motion buffer); SonicRuntime.reset()
  now calls controller.reset().
- sonic_whole_body: require the full dense 34-D command (no silent zero-fill of a
  partial action) and integrate yaw-rate (idx 33) into heading.
- controllers/__init__: import the controller classes referenced in __all__.
- unitree_g1: lazy replay-frame decode + small cache instead of decoding all
  frames up front; safer disconnect (longer controller-thread join + fail-safe
  that skips the graceful ramp if the thread won't stop).
- lint: ruff-format config_unitree_g1 hand_closed_pose; prettier README table.
2026-07-24 12:02:49 +02:00
Martino Russi 4209639f33 refactor(unitree_g1): isolate SONIC encoder/decoder whole-body path
Strip everything except the OpenHLM/pi0.5 -> SONIC encoder/decoder rollout
path so this branch does exactly that and nothing more:

- Remove the SONIC motion planner (planner ONNX + subprocess worker, PlannerMotion,
  replanning, MovementState/LocomotionMode, joystick) from sonic_pipeline; keep the
  encoder/decoder and the caller-fed reference buffer (PlannerController) intact.
- Slim SonicRuntime to load only the encoder/decoder; SonicWholeBodyController now
  runs solely the 34-D whole-body command path (drop SMPL/VR3/keyboard teleop).
- Delete the pico_headset teleoperator (SONIC's SMPL/VR3 teleop source).
- Move WB action constants into g1_utils; repoint imports.

GR00T/Holosoma locomotion controllers are left untouched.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-20 20:15:51 +02:00
Martino Russi fc7a0bc2fd feat(unitree_g1): drive SONIC whole-body from a 34-D OpenHLM/pi0.5 VLA
Add a dense 34-D whole-body command path so lerobot-rollout can drive the
G1 directly with an OpenHLM / pi0.5 policy through the SONIC encoder/decoder:

- SonicWholeBodyController: wb.{i}.pos action interface, mode-0 reference with
  a rolling 50-frame trajectory (finite-diff velocities) and first-tick anchor
  init; correct MuJoCo->IsaacLab joint reordering.
- unitree_g1: expose 34-D wb_state.{i}.pos proprio; empty/replay camera feeds
  for image-conditioned policies; Dex3 hand publishing from the grip scalars.
- g1_utils: obs_to_wb34_state + WB action constants.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-20 20:01:48 +02:00
Martino Russi 5f6513551c Merge branch 'main' into feat/unitree_g1_sonic_rebased 2026-07-18 13:23:23 +02:00
Martino Russi 70e157e00f fix ruff 2026-07-18 13:22:53 +02:00
Martino Russi 1837be51bf add 3 point calibration + waist coupling, remote controller and smoothed motion 2026-07-17 17:56:30 +02:00
Martino Russi bedd56eed9 Remove g1_sonic_slider, examples/onnx, and SONIC debugging docs 2026-07-16 14:40:32 +02:00
Martino Russi c165e4df68 Merge branch 'main' into feat/unitree_g1_sonic_rebased 2026-07-16 14:33:10 +02:00
Martino Russi 5e24da483a (add) sonic 3-point teleop, safe startup/shutdown, tested on real g1 2026-07-16 13:38:49 +02:00
Martino Russi 9c54665a76 test 3-point teleop
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-15 18:20:26 +02:00
Martino Russi f6a845c30c Merge branch 'main' into feat/unitree_g1_sonic_rebased 2026-07-15 17:13:50 +02:00
Martino Russi 45e8336854 replace quat operations with scipy 2026-07-15 17:07:09 +02:00
Martino Russi 5046e2df32 fix ruff 2026-07-15 16:42:46 +02:00
Martino Russi 1c88e26c6d clean up sonic-side 2026-07-15 16:40:56 +02:00
Martino Russi 69a3edfa33 fix lint 2026-07-15 16:00:42 +02:00
Martino Russi 2492ce2c29 switch to logging 2026-07-15 15:30:54 +02:00
Martino Russi c8e75da55f Merge remote-tracking branch 'origin/main' into feat/unitree_g1_sonic_rebased 2026-07-15 14:59:53 +02:00
Martino Russi 2eae31ea2b fix(unitree_g1): disable SMPL root-motion anchor to prevent sim instability
Feeding the per-frame SMPL root quaternion into the mode-2 anchor produced
root-acceleration spikes (NaN QACC at DOF 0) mid-episode during replay. Keep the
anchor self-driven until the reference root trajectory is smoothed/rate-matched
(30 Hz dataset -> 50 Hz control).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-15 14:59:22 +02:00
Martino Russi c997abe739 (fix) keep num of ORTthreads under core count 2026-07-14 18:11:03 +02:00
Martino Russi c73579055e refactor(unitree_g1): drop duplicate keyboard code, clarify smpl sentinel
- Remove unused RawKeyboard/drain_keyboard/process_keyboard from sonic_pipeline
  (dead code duplicating lerobot.utils.keyboard_input); the G1 integration uses
  the joystick path. Drop now-unused sys/select/termios/tty imports.
- Add a comment explaining the smpl.0 presence check is a sentinel for a full
  SMPL window (review question).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 15:09:03 +02:00
Martino Russi 4be438161b style: apply ruff format to sonic_pipeline and smpl_fk
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 14:46:05 +02:00
Martino Russi 806d28a883 docs(unitree_g1): add docstrings and comments to sonic_pipeline
Address review feedback that sonic_pipeline.py was dense and hard to read.
Adds a module-level architecture overview plus class and key-function
docstrings (planner subprocess, encoder/decoder, movement state, input
helpers). No behavior change.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 14:40:39 +02:00
Martino Russi 573b65ff6b (fix) hardcode smpl_skeleton, remove .npz 2026-07-14 13:07:30 +02:00
Martino Russi bc55713e7c fix relative imports 2026-07-13 18:50:00 +02:00
Martino Russi 4f53c42583 Apply ruff-format
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-10 16:03:28 +02:00
Martino Russi bfced3d149 Silence ruff N817 on scipy Rotation import 2026-07-10 15:59:55 +02:00
Martino Russi 4969813d4e Silence ruff N817 on scipy Rotation import 2026-07-10 15:49:09 +02:00
Martino Russi 1c87ca31a3 remove examples inlcuding npz motion files 2026-07-10 15:47:21 +02:00
Martino Russi 4bcde762cc add heading to SMPL, stream dataset 2026-07-10 15:44:48 +02:00
Martino Russi 943ae78cfe feat(unitree_g1): standalone PICO SMPL publisher + dedup/replay fixes
Add a self-contained rt/smpl publisher in the pico_headset teleoperator
(pico_publisher.py + numpy SMPL FK in smpl_fk.py + vendored skeleton table)
so headset whole-body teleop no longer depends on gear_sonic/torch; only
xrobotoolkit_sdk is needed at the headset.

Also: share lowstate_to_obs/get_gravity_orientation via g1_utils (dedup
sonic_pipeline and UnitreeG1.get_observation), and fix dataset-replay joint
ordering (Unitree -> IsaacLab) for sonic.py --replay-dataset.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 19:13:22 +02:00
Martino Russi 3363688f1e Merge branch 'main' into feat/unitree_g1_sonic_rebased 2026-07-09 18:02:53 +02:00
Martino Russi 0876629e72 Merge branch 'main' into feat/unitree_g1_sonic_rebased 2026-07-06 18:21:16 +02:00
Martino Russi 305614b8c6 add pico teleoperator, add sonic VR support
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-06 18:16:12 +02:00
Martino Russi 02d3202c4f add SMPL wiring into sonic controller 2026-07-06 18:13:46 +02:00
Martino Russi 3b6de2fdf8 fix(unitree_g1): fix typo flagged by spellchecker in motion_loader docstring 2026-06-26 13:46:33 +02:00
Martino Russi 744f3667c0 fix(unitree_g1): silence bandit findings in SONIC example/pipeline 2026-06-26 13:40:53 +02:00
Martino Russi fdde436776 Merge branch 'main' into feat/unitree_g1_sonic_rebased 2026-06-26 13:35:28 +02:00
Martino Russi 5c683c65c6 Merge branch 'main' into feat/unitree_g1_sonic_rebased 2026-06-25 14:38:48 +02:00
Martino Russi dfbc25c58f fix(unitree_g1): satisfy ruff lint/format and address review comments 2026-06-25 14:37:44 +02:00
Martino Russi 804c76bcc2 Merge branch 'main' into feat/unitree_g1_sonic_rebased 2026-06-25 13:41:04 +02:00
Martino Russi e6afa69be9 add motion loader 2026-06-17 12:31:08 +02:00
Martino Russi 31d1439e29 add custom motion loader 2026-06-17 12:29:36 +02:00
Martino Russi 1c118c6359 feat(unitree_g1): add SONIC whole-body controller
Move GrootLocomotionController and HolosomaLocomotionController into a new
controllers/ subpackage and add the SONIC whole-body controller
(sonic_pipeline.py, sonic_whole_body.py) plus the examples/unitree_g1/sonic.py
standalone script. UnitreeG1 now honors a controller's kp/kd, calls
controller.shutdown() on disconnect, and skips arm publishing for full_body
controllers.
2026-06-16 17:12:20 +02:00
321 changed files with 3198 additions and 15177 deletions
+50 -50
View File
@@ -53,7 +53,7 @@ permissions:
contents: read
env:
UV_VERSION: "0.11.30"
UV_VERSION: "0.8.0"
PYTHON_VERSION: "3.12"
# Cancel in-flight runs for the same branch/PR.
@@ -72,19 +72,19 @@ jobs:
HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
lfs: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
if: ${{ env.DOCKERHUB_USERNAME != '' }}
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
@@ -95,7 +95,7 @@ jobs:
# from source-copy, so code-only changes skip the slow uv-sync layer
# when the runner has a warm Docker daemon cache.
- name: Build Libero benchmark image
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: docker/Dockerfile.benchmark.libero
@@ -151,7 +151,7 @@ jobs:
- name: Upload Libero rollout video
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: libero-rollout-video
path: /tmp/libero-artifacts/videos/
@@ -159,7 +159,7 @@ jobs:
- name: Upload Libero eval metrics
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: libero-metrics
path: /tmp/libero-artifacts/metrics.json
@@ -214,7 +214,7 @@ jobs:
- name: Upload Libero train-smoke eval video
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: libero-train-smoke-video
path: /tmp/libero-train-smoke-artifacts/eval/
@@ -230,19 +230,19 @@ jobs:
HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
lfs: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
if: ${{ env.DOCKERHUB_USERNAME != '' }}
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
@@ -250,7 +250,7 @@ jobs:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
- name: Build MetaWorld benchmark image
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: docker/Dockerfile.benchmark.metaworld
@@ -303,7 +303,7 @@ jobs:
- name: Upload MetaWorld rollout video
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: metaworld-rollout-video
path: /tmp/metaworld-artifacts/videos/
@@ -311,7 +311,7 @@ jobs:
- name: Upload MetaWorld eval metrics
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: metaworld-metrics
path: /tmp/metaworld-artifacts/metrics.json
@@ -332,19 +332,19 @@ jobs:
ROBOTWIN_TASKS: beat_block_hammer,click_bell,handover_block,stack_blocks_two,click_alarmclock,open_microwave,adjust_bottle,lift_pot,stamp_seal,turn_switch
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
lfs: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
if: ${{ env.DOCKERHUB_USERNAME != '' }}
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
@@ -355,7 +355,7 @@ jobs:
# simulation assets (~4 GB). Layer cache lives in the runner's local
# Docker daemon — reused across re-runs on the same machine.
- name: Build RoboTwin 2.0 benchmark image
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: docker/Dockerfile.benchmark.robotwin
@@ -413,7 +413,7 @@ jobs:
- name: Upload RoboTwin rollout video
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v4
with:
name: robotwin-rollout-video
path: /tmp/robotwin-artifacts/videos/
@@ -421,7 +421,7 @@ jobs:
- name: Upload RoboTwin eval metrics
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v4
with:
name: robotwin-metrics
path: /tmp/robotwin-artifacts/metrics.json
@@ -439,19 +439,19 @@ jobs:
HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
lfs: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
if: ${{ env.DOCKERHUB_USERNAME != '' }}
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
@@ -459,7 +459,7 @@ jobs:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
- name: Build RoboCasa365 benchmark image
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: docker/Dockerfile.benchmark.robocasa
@@ -514,7 +514,7 @@ jobs:
- name: Upload RoboCasa365 rollout video
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: robocasa-rollout-video
path: /tmp/robocasa-artifacts/videos/
@@ -522,7 +522,7 @@ jobs:
- name: Upload RoboCasa365 eval metrics
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: robocasa-metrics
path: /tmp/robocasa-artifacts/metrics.json
@@ -540,19 +540,19 @@ jobs:
HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
lfs: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
if: ${{ env.DOCKERHUB_USERNAME != '' }}
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
@@ -560,7 +560,7 @@ jobs:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
- name: Build RoboCerebra benchmark image
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: docker/Dockerfile.benchmark.robocerebra
@@ -621,7 +621,7 @@ jobs:
- name: Upload RoboCerebra rollout video
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: robocerebra-rollout-video
path: /tmp/robocerebra-artifacts/videos/
@@ -629,7 +629,7 @@ jobs:
- name: Upload RoboCerebra eval metrics
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: robocerebra-metrics
path: /tmp/robocerebra-artifacts/metrics.json
@@ -648,19 +648,19 @@ jobs:
ROBOMME_TASKS: PickXtimes,BinFill,StopCube,MoveCube,InsertPeg,SwingXtimes,VideoUnmask,ButtonUnmask,PickHighlight,PatternLock
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
lfs: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
if: ${{ env.DOCKERHUB_USERNAME != '' }}
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
@@ -668,7 +668,7 @@ jobs:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
- name: Build RoboMME benchmark image
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: docker/Dockerfile.benchmark.robomme
@@ -726,7 +726,7 @@ jobs:
- name: Upload RoboMME rollout video
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: robomme-rollout-video
path: /tmp/robomme-artifacts/videos/
@@ -734,7 +734,7 @@ jobs:
- name: Upload RoboMME eval metrics
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: robomme-metrics
path: /tmp/robomme-artifacts/metrics.json
@@ -754,19 +754,19 @@ jobs:
LIBERO_PLUS_TASK_IDS: "[0,100,260,500,1000,1500,2000,2400]"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
lfs: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
if: ${{ env.DOCKERHUB_USERNAME != '' }}
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
@@ -774,7 +774,7 @@ jobs:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
- name: Build LIBERO-plus benchmark image
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: docker/Dockerfile.benchmark.libero_plus
@@ -834,7 +834,7 @@ jobs:
- name: Upload LIBERO-plus rollout video
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: libero-plus-rollout-video
path: /tmp/libero-plus-artifacts/videos/
@@ -842,7 +842,7 @@ jobs:
- name: Upload LIBERO-plus eval metrics
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: libero-plus-metrics
path: /tmp/libero-plus-artifacts/metrics.json
@@ -858,19 +858,19 @@ jobs:
HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
lfs: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
if: ${{ env.DOCKERHUB_USERNAME != '' }}
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
@@ -878,7 +878,7 @@ jobs:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
- name: Build VLABench benchmark image
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: docker/Dockerfile.benchmark.vlabench
@@ -936,7 +936,7 @@ jobs:
- name: Upload VLABench rollout video
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: vlabench-rollout-video
path: /tmp/vlabench-artifacts/videos/
@@ -944,7 +944,7 @@ jobs:
- name: Upload VLABench eval metrics
if: always()
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: vlabench-metrics
path: /tmp/vlabench-artifacts/metrics.json
+1 -1
View File
@@ -47,7 +47,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
+9 -9
View File
@@ -27,7 +27,7 @@ on:
# Sets up the environment variables
env:
UV_VERSION: "0.11.30"
UV_VERSION: "0.8.0"
PYTHON_VERSION: "3.12"
DOCKER_IMAGE_NAME_CPU: huggingface/lerobot-cpu:latest
DOCKER_IMAGE_NAME_GPU: huggingface/lerobot-gpu:latest
@@ -52,21 +52,21 @@ jobs:
sudo apt-get update
sudo apt-get install git-lfs
git lfs install
- uses: actions/checkout@v7
- uses: actions/checkout@v6
with:
lfs: true
persist-credentials: false
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
- name: Build and push Docker image CPU
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: ./docker/Dockerfile.user
@@ -87,21 +87,21 @@ jobs:
sudo apt-get update
sudo apt-get install git-lfs
git lfs install
- uses: actions/checkout@v7
- uses: actions/checkout@v6
with:
lfs: true
persist-credentials: false
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
- name: Build and push Docker image GPU
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: ./docker/Dockerfile.internal
@@ -33,7 +33,7 @@ jobs:
github.event.workflow_run.event == 'pull_request' &&
github.event.workflow_run.conclusion == 'success' &&
github.repository == 'huggingface/lerobot'
uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@6108e850ae1cf2f71bb0815a600bcd50c39abfa7 # main
uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@2430c1ec91d04667414e2fa31ecfc36c153ea391 # main
with:
package_name: lerobot
secrets:
+7 -23
View File
@@ -24,24 +24,19 @@ on:
required: false
type: string
# Triggers on pushes to main that touch the docs or the sources the API reference is generated from.
# `src/**` is included because the API reference is built from docstrings via `[[autodoc]]`: without it,
# published API pages would go stale as soon as a docstring changed.
# Triggers the workflow on push events to main for the docs folder
push:
branches:
- main
paths:
- "docs/**"
- "src/**"
# Same for pull requests, so a docstring change gets a preview build and a broken `[[autodoc]]` path
# fails the PR rather than main.
# Triggers the workflow on pull request events targeting main for the docs folder
pull_request:
branches:
- main
paths:
- "docs/**"
- "src/**"
release:
types: [published]
@@ -60,25 +55,16 @@ jobs:
github.repository == 'huggingface/lerobot'
permissions:
contents: read
uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@6108e850ae1cf2f71bb0815a600bcd50c39abfa7 # main
uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@e60a538eea9817ab312196d0d233604b01697265 # main
with:
commit_sha: ${{ github.sha }}
package: lerobot
# doc-builder ships a mock-deps registry entry for lerobot, so the reusable workflow takes its
# "light install" path: `pip install ./lerobot --no-deps` plus a handful of real dependencies.
# That is not enough to import lerobot — draccus runs `register_subclass` at import time and
# `processor/converters.py` calls `functools.singledispatch.register(torch.Tensor)`, neither of
# which works against a mock. Install the package for real before the build.
pre_command: uv pip install "./lerobot[dataset]"
# `--version main` is load-bearing: without `--not_python_module`, doc-builder falls back to
# `lerobot.__version__` and only maps that to the default branch when it contains "dev". Our main
# branch carries a release version (0.6.2), so omitting this would publish the main docs to
# /lerobot/v0.6.2/ instead of /lerobot/main/ and disable notebook building.
additional_args: >-
--not_python_module
${{
(github.event_name == 'release' && format('--version {0}', github.event.release.tag_name)) ||
(inputs.version != '' && format('--version {0}', inputs.version)) ||
'--version main'
''
}}
secrets:
token: ${{ secrets.HUGGINGFACE_PUSH }}
@@ -92,11 +78,9 @@ jobs:
permissions:
contents: read
pull-requests: write
uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@6108e850ae1cf2f71bb0815a600bcd50c39abfa7 # main
uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@e60a538eea9817ab312196d0d233604b01697265 # main
with:
commit_sha: ${{ github.event.pull_request.head.sha }}
pr_number: ${{ github.event.number }}
package: lerobot
# See the comment on build_main_docs. The PR workflow passes its own `--version pr_<n>`, so no
# additional_args are needed here.
pre_command: uv pip install "./lerobot[dataset]"
additional_args: --not_python_module
+3 -3
View File
@@ -48,7 +48,7 @@ permissions:
# Sets up the environment variables
env:
UV_VERSION: "0.11.30"
UV_VERSION: "0.8.0"
PYTHON_VERSION: "3.12"
# Ensures that only the latest commit for a PR or branch is built, canceling older runs.
@@ -69,7 +69,7 @@ jobs:
HF_LEROBOT_HOME: /mnt/cache/.cache/huggingface/lerobot
HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
lfs: true
@@ -87,7 +87,7 @@ jobs:
libusb-1.0-0-dev speech-dispatcher libgeos-dev portaudio19-dev
- name: Setup uv and Python
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
with:
enable-cache: true
version: ${{ env.UV_VERSION }}
+7 -7
View File
@@ -37,7 +37,7 @@ permissions:
# Sets up the environment variables
env:
UV_VERSION: "0.11.30"
UV_VERSION: "0.8.0"
PYTHON_VERSION: "3.12"
DOCKER_IMAGE_NAME: huggingface/lerobot-gpu
@@ -63,7 +63,7 @@ jobs:
HF_LEROBOT_HOME: /mnt/cache/.cache/huggingface/lerobot
HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
lfs: true
persist-credentials: false
@@ -80,7 +80,7 @@ jobs:
speech-dispatcher libgeos-dev portaudio19-dev
- name: Setup uv and Python
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
with:
enable-cache: true
version: ${{ env.UV_VERSION }}
@@ -137,21 +137,21 @@ jobs:
sudo apt-get update
sudo apt-get install git-lfs
git lfs install
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
lfs: true
persist-credentials: false
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
with:
cache-binary: false
- name: Login to Docker Hub
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
- name: Build and push Docker image
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
with:
context: .
file: ./docker/Dockerfile.internal
+1 -1
View File
@@ -29,7 +29,7 @@ jobs:
runs-on: ubuntu-latest
if: github.repository == 'huggingface/lerobot'
steps:
- uses: actions/github-script@v9
- uses: actions/github-script@v8
with:
script: |
// Setup Input Text
+14 -14
View File
@@ -27,7 +27,7 @@ on:
# Sets up the environment variables
env:
UV_VERSION: "0.11.30"
UV_VERSION: "0.8.0"
PYTHON_VERSION: "3.12"
DOCKER_IMAGE_NAME: huggingface/lerobot-gpu:latest-deps
@@ -48,12 +48,12 @@ jobs:
outputs:
changed: ${{ steps.diff.outputs.changed }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Setup uv and Python
uses: astral-sh/setup-uv@v8.3.2 # zizmor: ignore[unpinned-uses]
uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
with:
version: ${{ env.UV_VERSION }}
python-version: ${{ env.PYTHON_VERSION }}
@@ -74,7 +74,7 @@ jobs:
- name: Upload updated lockfile
if: steps.diff.outputs.changed == 'true'
uses: actions/upload-artifact@v7 # zizmor: ignore[unpinned-uses]
uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: uv-lock
path: uv.lock
@@ -93,13 +93,13 @@ jobs:
HF_LEROBOT_HOME: /mnt/cache/.cache/huggingface/lerobot
HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
with:
lfs: true
persist-credentials: false
- name: Download updated lockfile
uses: actions/download-artifact@v8 # zizmor: ignore[unpinned-uses]
uses: actions/download-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: uv-lock
@@ -115,7 +115,7 @@ jobs:
speech-dispatcher libgeos-dev portaudio19-dev
- name: Setup uv and Python
uses: astral-sh/setup-uv@v8.3.2 # zizmor: ignore[unpinned-uses]
uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
with:
enable-cache: true
version: ${{ env.UV_VERSION }}
@@ -153,27 +153,27 @@ jobs:
sudo apt-get update
sudo apt-get install git-lfs
git lfs install
- uses: actions/checkout@v7
- uses: actions/checkout@v6
with:
lfs: true
persist-credentials: false
- name: Download updated lockfile
uses: actions/download-artifact@v8 # zizmor: ignore[unpinned-uses]
uses: actions/download-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: uv-lock
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 # zizmor: ignore[unpinned-uses]
uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
with:
cache-binary: false
- name: Login to Docker Hub
uses: docker/login-action@v4.4.0 # zizmor: ignore[unpinned-uses]
uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
with:
username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
- name: Build and push Docker image
uses: docker/build-push-action@v7 # zizmor: ignore[unpinned-uses]
uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
with:
context: .
file: ./docker/Dockerfile.internal
@@ -247,12 +247,12 @@ jobs:
env:
GH_TOKEN: ${{ secrets.UPDATE_LOCK_TOKEN }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Download updated lockfile
uses: actions/download-artifact@v8 # zizmor: ignore[unpinned-uses]
uses: actions/download-artifact@v4 # zizmor: ignore[unpinned-uses]
with:
name: uv-lock
+1 -1
View File
@@ -33,7 +33,7 @@ jobs:
runs-on: ubuntu-latest
if: github.repository == 'huggingface/lerobot' && !github.event.pull_request.draft
steps:
- uses: actions/labeler@v7
- uses: actions/labeler@v6
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
sync-labels: true # Removes labels if files are removed from the PR
+2 -40
View File
@@ -43,12 +43,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v6
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: '3.12'
@@ -56,41 +56,3 @@ jobs:
uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1
with:
extra_args: --all-files --show-diff-on-failure --color=always
# This job runs the examples in our docstrings and validates the doctest allowlist.
# See docs/source/writing_docstrings.mdx for the standard these enforce.
doc-checks:
name: Run Documentation Checks (Doctests)
runs-on: ubuntu-latest
env:
# Examples that need a physical robot, a serial port or a Hub download are skipped by content.
# Everything else has to actually run. See src/lerobot/utils/doctest_utils.py.
SKIP_HARDWARE_DOCTEST: "1"
SKIP_CUDA_DOCTEST: "1"
steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup uv and Python
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
with:
enable-cache: true
version: "0.11.30"
python-version: "3.12"
- name: Install dependencies
run: uv sync --locked --extra test --extra dataset
- name: Check the doctest list is sorted and its paths exist
run: make check-doctest-list
- name: Check documented arguments match their signatures
run: make check-docstrings
- name: Check docstring coverage has not regressed
run: uv run --with interrogate interrogate --config=pyproject.toml
- name: Run doctests
run: make doctest
+7 -7
View File
@@ -21,7 +21,7 @@ on:
# Sets up the environment variables
env:
UV_VERSION: "0.11.30"
UV_VERSION: "0.8.0"
PYTHON_VERSION: "3.12"
jobs:
@@ -38,12 +38,12 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v6
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: '3.12'
@@ -104,7 +104,7 @@ jobs:
- name: Publish to TestPyPI for pre-releases
# True for tags like 'v0.2.0-rc1'
if: startsWith(github.ref, 'refs/tags/v') && contains(github.ref, '-')
uses: pypa/gh-action-pypi-publish@ba38be9e461d3875417946c167d0b5f3d385a247 # v1.14.1
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0
with:
repository-url: https://test.pypi.org/legacy/
verbose: true
@@ -112,7 +112,7 @@ jobs:
- name: Publish to PyPI
if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref, '-')
uses: pypa/gh-action-pypi-publish@ba38be9e461d3875417946c167d0b5f3d385a247 # v1.14.1
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0
with:
verbose: true
print-hash: true
@@ -127,7 +127,7 @@ jobs:
env:
MUJOCO_GL: egl
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
lfs: true
persist-credentials: false
@@ -137,7 +137,7 @@ jobs:
git curl libglib2.0-0 libegl1-mesa-dev ffmpeg libusb-1.0-0-dev \
speech-dispatcher libgeos-dev portaudio19-dev
- name: Setup uv and Python
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
with:
enable-cache: true # zizmor: ignore[cache-poisoning]
version: ${{ env.UV_VERSION }}
+2 -2
View File
@@ -43,12 +43,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0
persist-credentials: false
- name: Secret Scanning
uses: trufflesecurity/trufflehog@27b0417c16317ca9a472a9a8092acce143b49c55 # v3.95.9
uses: trufflesecurity/trufflehog@eafb8c5f6a06175141c27f17bcc17941853d0047 # v3.90.0
with:
extra_args: --only-verified
+5 -5
View File
@@ -19,8 +19,8 @@ on:
workflow_dispatch:
# Runs at 02:00
schedule:
- cron: "0 2 * * *"
# schedule:
# - cron: "0 2 * * *"
env:
CLOSE_ISSUE_MESSAGE: >
@@ -31,7 +31,7 @@ env:
Feel free to reopen if is still relevant, or to ping a collaborator if you have any questions.
WARN_ISSUE_MESSAGE: >
This issue has been automatically marked as stale because it has not had
recent activity (1 year). It will be closed if no further activity occurs within 30 days.
recent activity (1 year). It will be closed if no further activity occurs.
Any change, comment or update to this issue will reset this count.
Thank you for your contributions.
WARN_PR_MESSAGE: >
@@ -61,8 +61,8 @@ jobs:
exempt-pr-labels: never-stale
days-before-issue-stale: 365
days-before-issue-close: 30
days-before-pr-stale: -1
days-before-pr-close: -1
days-before-pr-stale: 365
days-before-pr-close: 30
delete-branch: true
close-issue-message: ${{ env.CLOSE_ISSUE_MESSAGE }}
close-pr-message: ${{ env.CLOSE_PR_MESSAGE }}
+2 -11
View File
@@ -67,11 +67,7 @@ repos:
args: [--prose-wrap=preserve]
# Jinja2 model-card templates use a .md extension but contain {% ... %} /
# {{ ... }} tags that prettier's Markdown formatter mangles (e.g. table loops).
#
# docs/source/api/ holds the generated API reference. Its `[[autodoc]]` blocks restrict output
# to an indented `- member` list, which prettier reads as a lazy paragraph continuation and
# joins onto one line — silently turning a member list into part of the directive.
exclude: ^(src/lerobot/templates/.*\.md|docs/source/api/.*\.mdx)$
exclude: ^src/lerobot/templates/.*\.md$
##### Security #####
- repo: https://github.com/gitleaks/gitleaks
@@ -108,13 +104,8 @@ repos:
# args: ["--docstring-style", "google", "-v", "2"]
# exclude: ^tests/.*$
# interrogate runs in CI (quality.yml, doc-checks job) rather than here. Its 1.7.0 release still imports
# the deprecated `py` package, which resolves against whatever `py` happens to be importable in
# pre-commit's isolated env — on a machine with miniconda on the path that is a stray `py.py` and the
# hook dies before it reads any config. The gate is the same either way; the CI step is just reliable.
# - repo: https://github.com/econchick/interrogate
# rev: 1.7.0
# hooks:
# - id: interrogate
# args: ["--config=pyproject.toml"]
# pass_filenames: false
# args: ["-vv", "--config=pyproject.toml"]
-4
View File
@@ -50,10 +50,6 @@ To run checks manually on all files:
pre-commit run --all-files
```
### Docstrings
The API reference is generated from the docstrings in `src/lerobot/`. If you add or change anything public, follow the [docstring standard](https://huggingface.co/docs/lerobot/writing_docstrings) — the format is parsed by the renderer and checked in CI.
### Running Tests
We use `pytest`. First, ensure you have test artifacts by installing **git-lfs**:
-26
View File
@@ -184,29 +184,3 @@ test-smolvla-ete-eval:
# backend, so it does not require a real model checkpoint or GPU.
annotation-e2e:
uv run python -m tests.annotations.run_e2e_smoke
# Docstring & doctest checks. See docs/source/writing_docstrings.mdx for the standard these enforce.
# Run the examples in the docstrings listed in utils/documentation_tests.txt. Hardware and GPU examples are
# skipped by content (see src/lerobot/utils/doctest_utils.py); CI sets both flags.
doctest:
@files=$$(grep -v '^\s*#' utils/documentation_tests.txt | grep -v '^\s*$$'); \
if [ -z "$$files" ]; then \
echo "utils/documentation_tests.txt lists no files; nothing to run."; \
else \
SKIP_HARDWARE_DOCTEST=1 uv run pytest --doctest-modules --no-header -q $$files; \
fi
check-doctest-list:
uv run python utils/check_doctest_list.py
fix-doctest-list:
uv run python utils/check_doctest_list.py --fix_and_overwrite
check-docstrings:
uv run python utils/check_docstrings.py
uv run python utils/check_config_docstrings.py
fix-docstrings:
uv run python utils/check_docstrings.py --fix_and_overwrite
uv run python utils/check_doctest_list.py --fix_and_overwrite
-17
View File
@@ -128,23 +128,6 @@ lerobot-eval \
Learn how to implement your own simulation environment or benchmark and distribute it from the HF Hub by following the [EnvHub Documentation](https://huggingface.co/docs/lerobot/envhub).
### Third-Party Hardware
Beyond the natively supported hardware, the community maintains a growing ecosystem of plugins for other robots, teleoperators, cameras, and sensors - UFACTORY xArm, Universal Robots UR5e, Franka, AgileX Piper, Trossen WidowX, ARX5, I2RT YAM, GELLO, SpaceMouse, Meta Quest, ROS 2 bridges, tactile and depth cameras, and more.
Plugins are auto-discovered by package name: LeRobot imports any installed package prefixed with `lerobot_robot_`, `lerobot_teleoperator_`, or `lerobot_camera_`. Install one and use the `type` it registers straight from the CLI:
```bash
pip install lerobot_robot_<name> lerobot_teleoperator_<name>
lerobot-record \
--robot.type=<robot_name> \
--teleop.type=<teleoperator_name> \
--dataset.repo_id=${HF_USER}/my-dataset
```
Browse the full list in the [Third-Party Robots & Teleoperators](https://huggingface.co/docs/lerobot/main/third_party_robots) and [Third-Party Cameras & Sensors](https://huggingface.co/docs/lerobot/main/third_party_sensors) documentation.
## Resources
- **[Documentation](https://huggingface.co/docs/lerobot/index):** The complete guide to tutorials & API.
-60
View File
@@ -1,60 +0,0 @@
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Root conftest: makes doctest collection use LeRobot's parser.
This only affects `--doctest-modules` runs (see `make doctest`). The test suite itself is configured by
`tests/conftest.py`.
"""
import doctest
import _pytest.doctest
from lerobot.utils.doctest_utils import LeRobotDoctestModule, LeRobotDocTestParser
# Lets an example opt out of output comparison with `# doctest: +IGNORE_RESULT`, for calls whose output is
# a progress bar or otherwise not reproducible.
IGNORE_RESULT = doctest.register_optionflag("IGNORE_RESULT")
OutputChecker = doctest.OutputChecker
class CustomOutputChecker(OutputChecker):
"""An output checker that honours the `IGNORE_RESULT` flag."""
def check_output(self, want, got, optionflags):
"""Return `True` when `IGNORE_RESULT` is set, otherwise defer to stdlib.
Args:
want (`str`):
The expected output.
got (`str`):
The actual output.
optionflags (`int`):
Bitmask of active doctest option flags.
Returns:
`bool`: Whether the output is considered a match.
"""
if IGNORE_RESULT & optionflags:
return True
return OutputChecker.check_output(self, want, got, optionflags)
# Reassigning these module attributes is how doctest behaviour is customised; mypy sees it as assigning to
# a type, which is exactly what is intended here.
doctest.OutputChecker = CustomOutputChecker # type: ignore[misc]
_pytest.doctest.DoctestModule = LeRobotDoctestModule
doctest.DocTestParser = LeRobotDocTestParser # type: ignore[misc]
-26
View File
@@ -165,8 +165,6 @@
title: OpenArm
- local: rebot_b601
title: reBot B601-DM
- local: third_party_robots
title: Third-Party Robots & Teleoperators
title: "Robots"
- sections:
- local: phone_teleop
@@ -177,8 +175,6 @@
- sections:
- local: cameras
title: Cameras
- local: third_party_sensors
title: Third-Party Cameras & Sensors
title: "Sensors"
- sections:
- local: notebooks
@@ -191,28 +187,6 @@
- sections:
- local: contributing
title: Contribute to LeRobot
- local: writing_docstrings
title: Writing docstrings
- local: backwardcomp
title: Backward compatibility
title: "About"
- sections:
- local: api/robots
title: Robots
- local: api/teleoperators
title: Teleoperators
- local: api/cameras
title: Cameras
- local: api/motors
title: Motors
- local: api/datasets
title: Datasets
- local: api/policies
title: Policies
- local: api/processor
title: Processors
- local: api/envs
title: Environments
- local: api/configs
title: Configuration
title: "API Reference"
-73
View File
@@ -1,73 +0,0 @@
# Cameras
Cameras supply the image observations a policy sees. Every backend — OpenCV, Intel RealSense, Reachy 2 —
implements the [`Camera`] interface, so swapping hardware does not change the code that reads frames.
See the [Cameras guide](../cameras) for choosing and configuring a camera, and
[Third-Party Cameras & Sensors](../third_party_sensors) for devices outside the core set.
## Camera
[[autodoc]] lerobot.cameras.Camera
- connect
- disconnect
- read
- async_read
- find_cameras
## CameraConfig
[[autodoc]] lerobot.cameras.CameraConfig
## make_cameras_from_configs
[[autodoc]] lerobot.cameras.make_cameras_from_configs
## ColorMode
[[autodoc]] lerobot.cameras.ColorMode
## Cv2Rotation
[[autodoc]] lerobot.cameras.Cv2Rotation
## Cv2Backends
[[autodoc]] lerobot.cameras.Cv2Backends
## OpenCVCamera
The default backend for USB webcams and video files, built on OpenCV.
[[autodoc]] lerobot.cameras.opencv.OpenCVCamera
- all
[[autodoc]] lerobot.cameras.opencv.OpenCVCameraConfig
## RealSenseCamera
Intel RealSense cameras, with optional depth sensing.
[[autodoc]] lerobot.cameras.realsense.RealSenseCamera
- all
[[autodoc]] lerobot.cameras.realsense.RealSenseCameraConfig
## Reachy2Camera
Cameras exposed by a Reachy 2 robot's own camera manager.
[[autodoc]] lerobot.cameras.reachy2_camera.Reachy2Camera
- all
[[autodoc]] lerobot.cameras.reachy2_camera.Reachy2CameraConfig
## ZMQCamera
Reads frames published over a ZeroMQ socket by `ImageServer`, for cameras attached to a different machine
than the one running the policy.
[[autodoc]] lerobot.cameras.zmq.ZMQCamera
- all
[[autodoc]] lerobot.cameras.zmq.ZMQCameraConfig
-27
View File
@@ -1,27 +0,0 @@
# Configuration
LeRobot configuration is plain dataclasses parsed by [draccus](https://github.com/dlwh/draccus), so every
field is settable from the CLI. [`TrainPipelineConfig`] is the top-level object for `lerobot-train`.
Polymorphic configs (policies, robots, environments) use `draccus.ChoiceRegistry`: a subclass registers
itself with `@register_subclass("name")` and is then selectable by that name on the command line.
## TrainPipelineConfig
[[autodoc]] lerobot.configs.train.TrainPipelineConfig
## PreTrainedConfig
[[autodoc]] lerobot.configs.PreTrainedConfig
## DatasetConfig
[[autodoc]] lerobot.configs.DatasetConfig
## EvalConfig
[[autodoc]] lerobot.configs.EvalConfig
## WandBConfig
[[autodoc]] lerobot.configs.WandBConfig
-23
View File
@@ -1,23 +0,0 @@
# Datasets
[`LeRobotDataset`] is the format every LeRobot script reads and writes. It is episode-aware, decodes video
observations on the fly, and round-trips to the Hugging Face Hub.
See [Using LeRobotDataset](../lerobot-dataset-v3) for the format and the common operations,
[Porting Large Datasets](../porting_datasets_v3) for migration, and [Tools](../tools) for the CLI.
## LeRobotDataset
[[autodoc]] lerobot.datasets.LeRobotDataset
## LeRobotDatasetMetadata
[[autodoc]] lerobot.datasets.LeRobotDatasetMetadata
## MultiLeRobotDataset
[[autodoc]] lerobot.datasets.MultiLeRobotDataset
## StreamingLeRobotDataset
[[autodoc]] lerobot.datasets.StreamingLeRobotDataset
-19
View File
@@ -1,19 +0,0 @@
# Environments
Simulation environments are configured through [`EnvConfig`] and built by [`make_env`]. Each subclass
declares its `gym_kwargs` and how to construct the vectorised environments.
See [Environments from the Hub](../envhub) for using published environments and
[Adding a New Benchmark](../adding_benchmarks) for contributing one.
## EnvConfig
[[autodoc]] lerobot.envs.EnvConfig
## make_env
[[autodoc]] lerobot.envs.make_env
## make_env_config
[[autodoc]] lerobot.envs.make_env_config
-23
View File
@@ -1,23 +0,0 @@
# Motors
`MotorsBus` is the low-level interface to a chain of servos on a serial bus. Robots use it to read positions
and write goal positions; you rarely touch it directly unless you are adding hardware.
See [Bring Your Own Hardware](../integrate_hardware) for adding a new bus, and
[Updating Feetech Firmware](../feetech) and [Damiao Motors and CAN Bus](../damiao) for device-specific notes.
## MotorsBus
[[autodoc]] lerobot.motors.motors_bus.MotorsBus
## Motor
[[autodoc]] lerobot.motors.Motor
## MotorCalibration
[[autodoc]] lerobot.motors.MotorCalibration
## MotorNormMode
[[autodoc]] lerobot.motors.MotorNormMode
-20
View File
@@ -1,20 +0,0 @@
# Policies
Every policy inherits [`PreTrainedPolicy`], which combines a `torch.nn.Module` with the Hub mixin, so any
policy can be pushed to and loaded from the Hugging Face Hub with the same two calls.
Each policy has its own guide with training recipes and results — [ACT](../act), [SmolVLA](../smolvla),
[π₀](../pi0), [π₀.₅](../pi05) and the rest are listed under Policies. To add one, see
[Adding a Policy](../bring_your_own_policies).
## PreTrainedPolicy
[[autodoc]] lerobot.policies.pretrained.PreTrainedPolicy
## PreTrainedConfig
[[autodoc]] lerobot.configs.PreTrainedConfig
## make_policy
[[autodoc]] lerobot.policies.factory.make_policy
-20
View File
@@ -1,20 +0,0 @@
# Processors
Processors are the data transformation layer between a robot, a dataset and a policy. A pipeline is a chain
of [`ProcessorStep`]s; each step declares how it transforms both the data and the feature contract.
See [Introduction to Robot Processors](../introduction_processors) for the concepts,
[Implement your own processor](../implement_your_own_processor) to write a step, and
[Debug your processor pipeline](../debug_processor_pipeline) when a pipeline misbehaves.
## ProcessorStep
[[autodoc]] lerobot.processor.pipeline.ProcessorStep
## DataProcessorPipeline
[[autodoc]] lerobot.processor.pipeline.DataProcessorPipeline
## PolicyProcessorPipeline
[[autodoc]] lerobot.processor.pipeline.PolicyProcessorPipeline
-147
View File
@@ -1,147 +0,0 @@
# Robots
Every robot in LeRobot implements the [`Robot`] interface: connect, read an observation, send an action,
disconnect. Writing a policy or a recording script against that interface means it works with any supported
arm without change.
This page is the generated reference. For wiring, calibration and first-run instructions, start with the
hardware guides — [SO-101](../so101), [LeKiwi](../lekiwi), [Hope Jr](../hope_jr), [Reachy 2](../reachy2),
[OpenArm](../openarm) — or [Imitation Learning for Robots](../il_robots) for the end-to-end workflow. To add
a robot of your own, see [Bring Your Own Hardware](../integrate_hardware).
## Robot
The abstract base class. Subclasses implement every method below; the contract described here is what a
policy or recording loop can rely on.
[[autodoc]] lerobot.robots.Robot
- connect
- disconnect
- configure
- calibrate
- get_observation
- send_action
- observation_features
- action_features
- is_connected
- is_calibrated
## RobotConfig
[[autodoc]] lerobot.robots.RobotConfig
## make_robot_from_config
[[autodoc]] lerobot.robots.make_robot_from_config
## SO-100 and SO-101 followers
`SO100Follower` and `SO101Follower` are aliases of the same `SOFollower` class; the two arms differ in their
configuration, not their control code. `SO100FollowerConfig` and `SO101FollowerConfig` are likewise aliases
of `SOFollowerRobotConfig`.
[[autodoc]] lerobot.robots.so_follower.SOFollower
- all
[[autodoc]] lerobot.robots.so_follower.SOFollowerRobotConfig
## BiSOFollower
Two SO followers driven as one bimanual robot.
[[autodoc]] lerobot.robots.bi_so_follower.BiSOFollower
- all
[[autodoc]] lerobot.robots.bi_so_follower.BiSOFollowerConfig
## KochFollower
[[autodoc]] lerobot.robots.koch_follower.KochFollower
- all
[[autodoc]] lerobot.robots.koch_follower.KochFollowerConfig
## LeKiwi
`LeKiwi` runs on the robot itself. `LeKiwiClient` is the host-side proxy that talks to it over the network
and presents the same [`Robot`] interface.
[[autodoc]] lerobot.robots.lekiwi.LeKiwi
- all
[[autodoc]] lerobot.robots.lekiwi.LeKiwiConfig
[[autodoc]] lerobot.robots.lekiwi.LeKiwiClient
- all
[[autodoc]] lerobot.robots.lekiwi.LeKiwiClientConfig
## OpenArmFollower
[[autodoc]] lerobot.robots.openarm_follower.OpenArmFollower
- all
[[autodoc]] lerobot.robots.openarm_follower.OpenArmFollowerConfig
## BiOpenArmFollower
[[autodoc]] lerobot.robots.bi_openarm_follower.BiOpenArmFollower
- all
[[autodoc]] lerobot.robots.bi_openarm_follower.BiOpenArmFollowerConfig
## OmxFollower
[[autodoc]] lerobot.robots.omx_follower.OmxFollower
- all
[[autodoc]] lerobot.robots.omx_follower.OmxFollowerConfig
## Reachy2Robot
[[autodoc]] lerobot.robots.reachy2.Reachy2Robot
- all
[[autodoc]] lerobot.robots.reachy2.Reachy2RobotConfig
## UnitreeG1
[[autodoc]] lerobot.robots.unitree_g1.UnitreeG1
- all
[[autodoc]] lerobot.robots.unitree_g1.UnitreeG1Config
## Hope Jr
The Hope Jr humanoid is exposed as two independent robots, an arm and a hand.
[[autodoc]] lerobot.robots.hope_jr.HopeJrArm
- all
[[autodoc]] lerobot.robots.hope_jr.HopeJrArmConfig
[[autodoc]] lerobot.robots.hope_jr.HopeJrHand
- all
[[autodoc]] lerobot.robots.hope_jr.HopeJrHandConfig
## RebotB601Follower
[[autodoc]] lerobot.robots.rebot_b601_follower.RebotB601Follower
- all
[[autodoc]] lerobot.robots.rebot_b601_follower.RebotB601FollowerRobotConfig
## BiRebotB601Follower
[[autodoc]] lerobot.robots.bi_rebot_b601_follower.BiRebotB601Follower
- all
[[autodoc]] lerobot.robots.bi_rebot_b601_follower.BiRebotB601FollowerConfig
## EarthRoverMiniPlus
[[autodoc]] lerobot.robots.earthrover_mini_plus.EarthRoverMiniPlus
- all
[[autodoc]] lerobot.robots.earthrover_mini_plus.EarthRoverMiniPlusConfig
-30
View File
@@ -1,30 +0,0 @@
# Teleoperators
A teleoperator produces actions for a robot to follow — a leader arm, a gamepad, a keyboard, a phone. All of
them implement the [`Teleoperator`] interface, so a recording script written against it works with any input
device.
See [Phone teleoperation](../phone_teleop) and [Isaac Teleop](../isaac_teleop) for setup guides, and
[Imitation Learning for Robots](../il_robots) for the recording workflow.
## Teleoperator
[[autodoc]] lerobot.teleoperators.Teleoperator
- connect
- disconnect
- configure
- calibrate
- get_action
- send_feedback
- action_features
- feedback_features
- is_connected
- is_calibrated
## TeleoperatorConfig
[[autodoc]] lerobot.teleoperators.TeleoperatorConfig
## make_teleoperator_from_config
[[autodoc]] lerobot.teleoperators.make_teleoperator_from_config
+1 -1
View File
@@ -65,7 +65,7 @@ In summary, you need to specify instructions for:
Importantly,
- `actions_per_chunk` and `chunk_size_threshold` are key parameters to tune for your setup.
- `aggregate_fn_name` is the function to aggregate actions on overlapping portions. You can either add a new one to a registry of functions, or add your own in `robot_client.py` (see [here](https://github.com/huggingface/lerobot/blob/main/src/lerobot/async_inference/robot_client.py#L224))
- `aggregate_fn_name` is the function to aggregate actions on overlapping portions. You can either add a new one to a registry of functions, or add your own in `robot_client.py` (see [here](NOTE:addlinktoLOC))
- `debug_visualize_queue_size` is a useful tool to tune the `CLIENT` parameters.
## Done! You should see your robot moving around by now 😉
+2 -12
View File
@@ -161,16 +161,6 @@ The methods called by the train/eval loops:
Batches are flat dictionaries keyed by the constants in [`lerobot.utils.constants`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/utils/constants.py): `OBS_STATE` (`observation.state.<motor>`), `OBS_IMAGES` (`observation.images.<camera>`), `OBS_LANGUAGE`, `ACTION`, etc. Reuse the constants — don't invent new prefixes.
If your model is large enough to warrant [sharded multi-GPU training](./multi_gpu_training#sharded-training-fsdp), also declare its FSDP wrap units — the repeated block classes sharding operates on:
```python
class MyPolicy(PreTrainedPolicy):
...
_fsdp_wrap_modules = ["MyTransformerBlock"]
```
With this one declaration, `--parallelism.dp_shard=N` works out of the box for your policy (users can still override it with `--accelerator.fsdp.wrap_modules`). Without any wrap source, sharded runs fail at startup by design.
### Processor functions
LeRobot uses `PolicyProcessorPipeline`s to normalize inputs and de-normalize outputs around your policy. For a concrete reference, see [`processor_act.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/act/processor_act.py) or [`processor_diffusion.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/diffusion/processor_diffusion.py).
@@ -310,7 +300,7 @@ The file names are load-bearing: the factory does lazy imports by name, and the
Two places need to know about your policy. All by name.
1. **`policies/__init__.py`** — re-export `MyPolicyConfig` and add it to `__all__`. This import is what registers your policy: `@PreTrainedConfig.register_subclass("my_policy")` runs, and from then on the factory resolves everything by convention. **Don't** re-export the modeling class; it loads lazily through the factory (so `import lerobot` stays fast).
2. **`templates/lerobot_modelcard_template.md` and the root `README.md`** — the template is what the end-of-training publisher renders into the model card of every checkpoint trained with your policy: add a one-line description of your policy in the `model_name` branches, map it in `policy_docs` so cards link to your MDX guide, and optionally add an architecture image to `diagrams`. Then add your policy to the models table in the root `README.md`, under the right category, linking to your doc page.
2. **`templates/lerobot_modelcard_template.md` and the root `README.md`** — the template is what `push_model_to_hub` renders into the model card of every checkpoint trained with your policy: add a one-line description of your policy in the `model_name` branches, map it in `policy_docs` so cards link to your MDX guide, and optionally add an architecture image to `diagrams`. Then add your policy to the models table in the root `README.md`, under the right category, linking to your doc page.
Mirror an existing policy that's structurally similar to yours; the diff is small.
@@ -354,7 +344,7 @@ A new policy is much easier to review — and far more useful — when it ships
**Pick at least one in-tree benchmark.** LeRobot ships sim benchmarks with per-benchmark Docker images (LIBERO, LIBERO-plus, Meta-World, RoboTwin 2.0, RoboCasa365, RoboCerebra, RoboMME, VLABench and more). Pick the one that matches your policy's modality — VLAs usually go to LIBERO or VLABench; image-only BC to LIBERO or Meta-World. The full list lives under [Benchmarks](./libero) in the docs sidebar.
**Push the checkpoint & processors** to the Hub under `lerobot/<policy>_<benchmark>` (or your namespace if you don't have write access; a maintainer can mirror it). The easiest way is training with `--policy.repo_id=<namespace>/<repo>` and `--policy.push_to_hub=true`: `lerobot-train` publishes the model, both processors, and a model card at the end of the run. To publish an existing checkpoint after the fact, upload its `pretrained_model/` directory (e.g. `huggingface-cli upload`), or use `lerobot-convert-dcp --push_to_hub=...` for sharded-format checkpoints.
**Push the checkpoint & processors** to the Hub under `lerobot/<policy>_<benchmark>` (or your namespace if you don't have write access; a maintainer can mirror it). Use `PreTrainedPolicy.push_model_to_hub` so the repo gets `config.json`, `model.safetensors`, and a model card.
**Report results in your policy's MDX**, with the exact `lerobot-eval` command and hardware so anyone can re-run:
+2 -2
View File
@@ -164,8 +164,8 @@ includes the range reported by the sensor. Requesting an unsupported control als
Omitted controls leave the sensor's existing automatic or manual setting unchanged. These options
require `use_rgb=True`.
Manual color controls require a dedicated RGB module. Cameras without one, such as the RealSense
D405, do not support them and raise an error at connection time.
On the RealSense D405, the color stream is provided by the Stereo Module, so changing manual
exposure or gain also affects the depth stream.
</hfoption>
</hfoptions>
+10 -217
View File
@@ -23,18 +23,18 @@ The broader EVO1 project may include additional training scripts and dataset too
2. Install EVO1 dependencies:
```bash
pip install -e ".[training,evo1]"
pip install -e ".[evo1]"
```
For LIBERO training and evaluation, install the LIBERO extra as well:
For LIBERO evaluation, install the LIBERO extra as well:
```bash
pip install -e ".[training,evo1,libero]"
pip install -e ".[evo1,libero]"
```
3. Install a `flash-attn` wheel only if it is compatible with your Python, PyTorch, CUDA, and GPU stack. EVO1 falls back to standard attention when `flash_attn` is not available.
EVO1 uses the native Hugging Face `transformers` InternVL implementation, so `policy.vlm_model_name` must point to a natively converted checkpoint such as `OpenGVLab/InternVL3-1B-hf` (note the `-hf` suffix). The first run downloads the configured VLM checkpoint and later runs reuse it from the Hugging Face cache.
EVO1 uses the native Hugging Face `transformers` InternVL implementation, so `policy.vlm_model_name` must point to a natively converted checkpoint such as `OpenGVLab/InternVL3-1B-hf` (note the `-hf` suffix). The first run may download the configured VLM checkpoint unless `policy.vlm_model_name` points to a local model directory.
## Data Requirements
@@ -92,7 +92,7 @@ lerobot-train \
### Stage 2
Stage 2 loads the Stage 1 policy, but starts a fresh optimizer and scheduler:
Stage 2 finetunes the VLM branches and action head. A common workflow starts from a Stage 1 checkpoint:
```bash
lerobot-train \
@@ -152,154 +152,16 @@ lerobot-rollout \
### LIBERO Evaluation
#### Reference result
> [!NOTE]
> The released Stage-2 checkpoint passed clean-download and rollout verification:
> [`zuoxingdong/evo1_libero`](https://huggingface.co/zuoxingdong/evo1_libero), revision
> [`515921f4a2c1d3f3ad523721eafa26fdf2af315b`](https://huggingface.co/zuoxingdong/evo1_libero/commit/515921f4a2c1d3f3ad523721eafa26fdf2af315b).
> The clean-download evaluation used LeRobot revision
> [`e40b58a8dfa9e7b86918c374791599d070518d11`](https://github.com/huggingface/lerobot/commit/e40b58a8dfa9e7b86918c374791599d070518d11).
> Benchmark results for a `lerobot`-hosted LIBERO checkpoint trained with this implementation
> will be added once training completes.
The single-run Stage-2 checkpoint at step 70,000 produced:
| Suite | Successful episodes | Episodes | Success rate |
| -------------- | ------------------: | --------: | -----------: |
| LIBERO Spatial | 485 | 500 | 97.0% |
| LIBERO Object | 496 | 500 | 99.2% |
| LIBERO Goal | 483 | 500 | 96.6% |
| LIBERO-10 | 469 | 500 | 93.8% |
| **Overall** | **1,933** | **2,000** | **96.65%** |
These results use one trained checkpoint and evaluation seed `1000`; they are not a multi-seed
mean or confidence estimate.
#### Reference training recipe
The released checkpoint records the complete resolved Stage-2 configuration in
[`train_config.json`](https://huggingface.co/zuoxingdong/evo1_libero/blob/515921f4a2c1d3f3ad523721eafa26fdf2af315b/train_config.json).
The measured run used two H100 GPUs with two DDP processes and batch 64 per process, giving global batch 128. Both stages used the same topology. The base VLM came from revision
`014c0583a0d4bedf29fbe2dbff4f865eb998e171` of `OpenGVLab/InternVL3-1B-hf`.
The released artifact does not record the exact LeRobot training commit or its original dependency lock,
so the commands below reproduce the recorded configuration and topology from a current checkout rather
than reconstructing the software environment bit for bit.
From a LeRobot source checkout, install the locked dependencies and download that exact VLM revision:
```bash
uv sync --locked --extra training --extra evo1 --extra libero
VLM_DIR=$(uv run hf download OpenGVLab/InternVL3-1B-hf \
--revision=014c0583a0d4bedf29fbe2dbff4f865eb998e171)
```
Stage 1 freezes the VLM and trains the action head for 5,000 steps:
```bash
uv run accelerate launch --num_processes=2 -m lerobot.scripts.lerobot_train \
--dataset.repo_id=lerobot/libero \
--dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
--dataset.video_backend=torchcodec \
--dataset.return_uint8=true \
--dataset.image_transforms.enable=true \
--dataset.use_imagenet_stats=true \
--dataset.eval_split=0.0 \
--policy.type=evo1 \
--policy.training_stage=stage1 \
--policy.apply_training_stage_defaults=true \
--policy.vlm_model_name="${VLM_DIR}" \
--policy.vlm_num_layers=14 \
--policy.vlm_dtype=bfloat16 \
--policy.device=cuda \
--policy.use_amp=true \
--policy.use_flash_attn=true \
--policy.enable_gradient_checkpointing=true \
--policy.gradient_checkpointing_use_reentrant=false \
--policy.image_resolution='[448,448]' \
--policy.chunk_size=50 \
--policy.n_action_steps=50 \
--policy.max_state_dim=24 \
--policy.max_action_dim=24 \
--policy.dropout=0.2 \
--policy.optimizer_lr=1e-5 \
--policy.optimizer_weight_decay=1e-3 \
--policy.optimizer_grad_clip_norm=1.0 \
--policy.scheduler_warmup_steps=1000 \
--policy.push_to_hub=false \
--use_policy_training_preset=true \
--batch_size=64 \
--steps=5000 \
--save_checkpoint=true \
--save_checkpoint_to_hub=false \
--save_freq=2500 \
--log_freq=10 \
--env_eval_freq=0 \
--num_workers=4 \
--prefetch_factor=2 \
--persistent_workers=true \
--seed=1000 \
--wandb.enable=false \
--output_dir=./outputs/evo1-libero-stage1-g128-5k
```
Stage 2 loads the Stage-1 policy but starts a fresh optimizer and scheduler. It trains for 80,000 steps;
the reported checkpoint is the save at step 70,000:
```bash
uv run accelerate launch --num_processes=2 -m lerobot.scripts.lerobot_train \
--dataset.repo_id=lerobot/libero \
--dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
--dataset.video_backend=torchcodec \
--dataset.return_uint8=true \
--dataset.image_transforms.enable=true \
--dataset.use_imagenet_stats=true \
--dataset.eval_split=0.0 \
--policy.path=./outputs/evo1-libero-stage1-g128-5k/checkpoints/005000/pretrained_model \
--policy.training_stage=stage2 \
--policy.apply_training_stage_defaults=true \
--policy.vlm_model_name="${VLM_DIR}" \
--policy.vlm_num_layers=14 \
--policy.vlm_dtype=float32 \
--policy.device=cuda \
--policy.use_amp=true \
--policy.use_flash_attn=true \
--policy.enable_gradient_checkpointing=true \
--policy.gradient_checkpointing_use_reentrant=false \
--policy.image_resolution='[448,448]' \
--policy.chunk_size=50 \
--policy.n_action_steps=50 \
--policy.max_state_dim=24 \
--policy.max_action_dim=24 \
--policy.dropout=0.2 \
--policy.optimizer_lr=1e-5 \
--policy.optimizer_weight_decay=1e-3 \
--policy.optimizer_grad_clip_norm=1.0 \
--policy.scheduler_warmup_steps=1000 \
--policy.push_to_hub=false \
--use_policy_training_preset=true \
--batch_size=64 \
--steps=80000 \
--resume=false \
--save_checkpoint=true \
--save_checkpoint_to_hub=false \
--save_freq=10000 \
--log_freq=10 \
--env_eval_freq=0 \
--num_workers=4 \
--prefetch_factor=2 \
--persistent_workers=true \
--seed=1000 \
--wandb.enable=false \
--output_dir=./outputs/evo1-libero-stage2-g128-80k
```
#### Author-format evaluation profile
The author-format EVO1 LIBERO profile uses the raw LIBERO camera feature names
The official EVO1 LIBERO rollout protocol uses the raw LIBERO camera feature names
(`observation.images.agentview_image` and `observation.images.robot0_eye_in_hand_image`), replans every
14 actions, and binarizes the gripper command before stepping the simulator. The EVO1 policy postprocessor
can crop the padded 24D action back to the 7D LIBERO action space and apply that gripper binarization. To
evaluate an author-format checkpoint under the same one-episode-per-task setting, keep the raw camera names
instead of the default `image`/`image2` mapping and set the LIBERO action postprocessing flags:
evaluate a LIBERO checkpoint under the same one-episode-per-task setting, keep the raw camera names instead
of the default `image`/`image2` mapping and set the LIBERO action postprocessing flags:
```bash
lerobot-eval \
@@ -319,75 +181,6 @@ lerobot-eval \
--eval.n_episodes=1
```
#### Native `lerobot/libero` v3 profile
Revision `a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4` stores camera features as `image` and
`image2`. This example evaluates all ten LIBERO Object tasks, launching each task in a fresh process:
```bash
export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl
suite=libero_object
horizon=280
for task_id in {0..9}; do
lerobot-eval \
--policy.path=zuoxingdong/evo1_libero \
--policy.pretrained_revision=515921f4a2c1d3f3ad523721eafa26fdf2af315b \
--policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
--policy.device=cuda \
--policy.use_amp=true \
--policy.vlm_dtype=bfloat16 \
--policy.use_flash_attn=false \
--policy.enable_gradient_checkpointing=false \
--policy.vlm_num_layers=14 \
--policy.image_resolution='[448,448]' \
--policy.max_text_length=1024 \
--policy.chunk_size=50 \
--policy.n_action_steps=14 \
--policy.max_state_dim=24 \
--policy.max_action_dim=24 \
--policy.num_inference_timesteps=32 \
--policy.postprocess_action_dim=7 \
--policy.binarize_gripper=true \
--policy.gripper_threshold=0.0 \
--policy.gripper_below_threshold_value=-1.0 \
--policy.gripper_above_threshold_value=1.0 \
--env.type=libero \
--env.task="${suite}" \
--env.task_ids="[${task_id}]" \
--env.camera_name=agentview_image,robot0_eye_in_hand_image \
--env.camera_name_mapping="{agentview_image: image, robot0_eye_in_hand_image: image2}" \
--env.control_mode=relative \
--env.obs_type=pixels_agent_pos \
--env.observation_width=448 \
--env.observation_height=448 \
--env.init_states=true \
--env.episode_length="${horizon}" \
--env.render_mode=rgb_array \
--env.max_parallel_tasks=1 \
--eval.n_episodes=50 \
--eval.batch_size=1 \
--eval.use_async_envs=false \
--eval.recording=false \
--seed=1000 \
--output_dir="./outputs/evo1-libero-stage2-70k-eval/${suite}/task-${task_id}" \
--job_name="evo1-libero-stage2-70k-${suite}-task-${task_id}"
done
```
Run all ten task IDs for each suite with these horizons:
| `env.task` | `env.episode_length` |
| ---------------- | -------------------: |
| `libero_spatial` | `280` |
| `libero_object` | `280` |
| `libero_goal` | `300` |
| `libero_10` | `520` |
Set `suite` and `horizon` for each row. This gives 500 episodes per suite and 2,000 episodes overall, while
the loop's fresh process per task matches the measured RNG-reset topology.
## References
- [EVO1 repository](https://github.com/MINT-SJTU/Evo-1)
-1
View File
@@ -59,7 +59,6 @@ The `lerobot-rollout --strategy.type=dagger` mode requires **teleoperators with
- `bi_openarm_mini` - Bimanual OpenArm Mini
- `so_leader` - SO100 / SO101 leader arm
- `bi_so_leader` - Bimanual SO100 / SO101 leader arms
> [!IMPORTANT]
> The provided commands default to `bi_openarm_follower` + `bi_openarm_mini`.
+1 -1
View File
@@ -98,7 +98,7 @@ The teleoperate command will automatically:
## Cameras
To add cameras to your setup, follow this [Guide](./cameras).
To add cameras to your setup, follow this [Guide](./cameras#setup-cameras).
## Teleoperate with cameras
+9 -163
View File
@@ -1,177 +1,23 @@
# LeRobot
<div class="flex justify-center">
<a target="_blank" href="https://huggingface.co/lerobot">
<img
alt="LeRobot, Hugging Face Robotics Library"
alt="HuggingFace Expert Acceleration Program"
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/lerobot-logo-thumbnail.png"
style="width: 100%"
></img>
</a>
</div>
# LeRobot
**State-of-the-art machine learning for real-world robotics**
🤗 LeRobot provides a hardware-agnostic, Python-native interface for controlling real robots - from affordable arms like the SO-ARM101 to full humanoids. Plus the tools to record, store, and share the datasets they generate. Every dataset uses the standardized **LeRobotDataset** format (synchronized video + action/state data) and can be streamed directly from the [Hugging Face Hub](https://huggingface.co/lerobot).
🤗 LeRobot aims to provide models, datasets, and tools for real-world robotics in PyTorch. The goal is to lower the barrier for entry to robotics so that everyone can contribute and benefit from sharing datasets and pretrained models.
🤗 On top of that data, LeRobot implements state-of-the-art policies - from lightweight imitation-learning models like ACT to large vision-language-action models like π₀ and SmolVLA - all trainable, shareable, and deployable with the same handful of CLI commands.
🤗 LeRobot contains state-of-the-art approaches that have been shown to transfer to the real-world with a focus on imitation learning and reinforcement learning.
The goal: lower the barrier to entry for robotics, so that everyone can contribute to, and benefit from, shared datasets and pretrained models.
🤗 LeRobot already provides a set of pretrained models, datasets with human collected demonstrations, and simulated environments so that everyone can get started.
<div align="center" style="display: flex; justify-content: center; gap: 8px; flex-wrap: wrap; margin: 20px 0;">
<a href="https://discord.gg/s3KuuzsPFb" target="_blank">
<img alt="Discord" src="https://img.shields.io/badge/Discord-Join_the_Community-5865F2?style=flat&logo=discord&logoColor=white">
</a>
<a href="https://x.com/LeRobotHF" target="_blank">
<img alt="X (Twitter)" src="https://img.shields.io/badge/X-Follow_%40LeRobotHF-black?style=flat&logo=x&logoColor=white">
</a>
<a href="https://huggingface.co/lerobot" target="_blank">
<img alt="Hugging Face Hub" src="https://img.shields.io/badge/HF_Hub-Models_%26_Datasets-FFD21E?style=flat">
</a>
</div>
🤗 LeRobot hosts pretrained models and datasets on the LeRobot HuggingFace page.
<div align="center">
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/robots_control_video.webp" width="640px" alt="Reachy 2 Demo">
</div>
## How It Works
**Teleoperate → Record → Train → Deploy**
1. **Teleoperate** - control the robot yourself (with a leader arm, keyboard, or phone) so it can learn from your movements.
2. **Record** - each demonstration is saved as a dataset: synchronized camera video plus the actions you took.
3. **Train** - a policy (the neural network that will control the robot) learns to imitate your demonstrations.
4. **Deploy** - run the trained policy on the robot and watch it complete the task on its own.
## Get Started
New here? [Install LeRobot](./installation), then pick your path:
<div class="grid grid-cols-1 md:grid-cols-3 gap-4 my-6">
<div class="border dark:border-gray-700 rounded-lg p-4 shadow">
<div class="text-lg font-semibold mb-2">🔧 I have a robot</div>
<p class="text-gray-700 dark:text-gray-300 text-sm">
LeRobot supports a wide range of arms and mobile robots. Popular picks:
</p>
<ul class="text-gray-700 dark:text-gray-300 text-sm list-disc pl-5 mb-2">
<li>
<a href="./so101">SO-101</a> - our flagship, low-cost arm
</li>
<li>
<a href="./lekiwi">LeKiwi</a> - a mobile base with an arm on top
</li>
<li>
<a href="./koch">Koch v1.1</a> - a long-time community favorite
</li>
<li>
or find yours under <strong>Robots</strong> in the sidebar
</li>
</ul>
<p class="text-gray-700 dark:text-gray-300 text-sm">
Once it's assembled and calibrated, record a dataset and train your first
policy with the <a href="./il_robots">imitation learning tutorial</a> - or
skip the CLI entirely with <a href="./lelab">LeLab</a>, a browser GUI for
the same workflow.
</p>
</div>
<div class="border dark:border-gray-700 rounded-lg p-4 shadow">
<div class="text-lg font-semibold mb-2">💻 No hardware yet</div>
<p class="text-gray-700 dark:text-gray-300 text-sm">
You can still train and evaluate policies without owning a robot:
</p>
<ul class="text-gray-700 dark:text-gray-300 text-sm list-disc pl-5 mb-2">
<li>
train on an existing
<a href="https://huggingface.co/datasets?other=LeRobot">
LeRobot dataset
</a>
from the Hub
</li>
<li>
evaluate in <a href="./envhub">simulation</a>, against benchmarks like
LIBERO or Meta-World
</li>
<li>
try the free <a href="./notebooks">Colab notebooks</a> - nothing to
install
</li>
</ul>
</div>
<div class="border dark:border-gray-700 rounded-lg p-4 shadow">
<div class="text-lg font-semibold mb-2">🤝 I want to contribute</div>
<p class="text-gray-700 dark:text-gray-300 text-sm">
Start with the <a href="./contributing">Contributing guide</a>, then
<a href="./bring_your_own_policies">add a new policy</a> or
<a href="./integrate_hardware">bring your own hardware</a>.
</p>
</div>
</div>
## Explore the Docs
<div class="grid grid-cols-1 md:grid-cols-3 gap-4 my-6">
<a
class="!no-underline border dark:border-gray-700 rounded-lg p-4 shadow hover:shadow-lg"
href="./cheat-sheet"
>
<div class="font-semibold mb-1">📋 Cheat Sheet</div>
<p class="text-gray-700 dark:text-gray-300 text-sm">
Every LeRobot CLI command, copy-paste ready.
</p>
</a>
<a
class="!no-underline border dark:border-gray-700 rounded-lg p-4 shadow hover:shadow-lg"
href="./hardware_guide"
>
<div class="font-semibold mb-1">🖥️ Compute & Hardware Guide</div>
<p class="text-gray-700 dark:text-gray-300 text-sm">
Which policy fits your GPU, and how long training takes.
</p>
</a>
<a
class="!no-underline border dark:border-gray-700 rounded-lg p-4 shadow hover:shadow-lg"
href="./lerobot-dataset-v3"
>
<div class="font-semibold mb-1">🗂️ LeRobotDataset</div>
<p class="text-gray-700 dark:text-gray-300 text-sm">
Load, stream, and visualize robot datasets from the Hub.
</p>
</a>
<a
class="!no-underline border dark:border-gray-700 rounded-lg p-4 shadow hover:shadow-lg"
href="./lelab"
>
<div class="font-semibold mb-1">🖼 LeLab</div>
<p class="text-gray-700 dark:text-gray-300 text-sm">
A browser GUI for calibrating, recording, and training - no CLI required.
</p>
</a>
<a
class="!no-underline border dark:border-gray-700 rounded-lg p-4 shadow hover:shadow-lg"
href="./act"
>
<div class="font-semibold mb-1">🧠 Policies</div>
<p class="text-gray-700 dark:text-gray-300 text-sm">
Start with ACT, our recommended first policy - or browse SmolVLA, π₀, and
more in the sidebar.
</p>
</a>
<a
class="!no-underline border dark:border-gray-700 rounded-lg p-4 shadow hover:shadow-lg"
href="./envhub"
>
<div class="font-semibold mb-1">🎮 Simulation & Benchmarks</div>
<p class="text-gray-700 dark:text-gray-300 text-sm">
Train and evaluate in simulated environments before touching real
hardware.
</p>
</a>
</div>
## Common Problems
Running into issues? A few of the most frequent ones:
- **Blurry or unusable camera footage** - lighting matters more than resolution. See the [Cameras](./cameras) guide.
- **Build or install errors** (`cmake`, `ffmpeg`, CUDA) - see the Troubleshooting section of the [Installation guide](./installation#troubleshooting).
- **Not sure which policy fits your GPU** - check the [Compute & Hardware Guide](./hardware_guide).
- **Still stuck?** Ask on [Discord](https://discord.gg/s3KuuzsPFb) - the community (and the LeRobot team) is there to help.
Join the LeRobot community on [Discord](https://discord.gg/s3KuuzsPFb)
+15 -17
View File
@@ -149,14 +149,13 @@ lerobot-rollout \
Foot pedal input is also supported via `--strategy.input_device=pedal`. Configure pedal codes with `--strategy.pedal.*` flags.
| Flag | Description |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--strategy.num_episodes` | Number of correction episodes to record (default: 10) |
| `--strategy.record_autonomous` | Record autonomous frames too (default: false) |
| `--strategy.upload_every_n_episodes` | Push to Hub every N episodes (default: 5) |
| `--strategy.input_device` | Input device: `keyboard` or `pedal` (default: keyboard) |
| `--strategy.smooth_handover` | Smoothly hand control over at pause / correction start (default: true). Disable for clutch-style teleops that re-reference at the current robot pose on engage |
| `--teleop.type` | **Required.** Teleoperator type |
| Flag | Description |
| ------------------------------------ | ------------------------------------------------------- |
| `--strategy.num_episodes` | Number of correction episodes to record (default: 10) |
| `--strategy.record_autonomous` | Record autonomous frames too (default: false) |
| `--strategy.upload_every_n_episodes` | Push to Hub every N episodes (default: 5) |
| `--strategy.input_device` | Input device: `keyboard` or `pedal` (default: keyboard) |
| `--teleop.type` | **Required.** Teleoperator type |
### Episodic (`--strategy.type=episodic`)
@@ -187,15 +186,14 @@ Teleop is optional — if omitted the robot holds its position during the reset
| `←` (left) | Discard episode and re-record it |
| `ESC` | Stop the recording session |
| Flag | Description |
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--dataset.num_episodes` | Number of episodes to record |
| `--dataset.episode_time_s` | Duration of each recording episode in seconds |
| `--dataset.reset_time_s` | Duration of the reset phase between episodes in seconds |
| `--teleop.type` | Optional. Teleoperator to drive the robot during resets |
| `--strategy.reset_to_initial_position` | Whether to reset the robot to its initial position between episodes |
| `--strategy.smooth_leader_to_follower_handover` | Whether to turn on or off the leader -> follower smooth handover behavior. |
| `--strategy.smooth_handover` | Smoothly hand control to the teleop at reset start (default: true). Disable for clutch-style teleops that re-reference at the current robot pose on engage |
| Flag | Description |
| ----------------------------------------------- | -------------------------------------------------------------------------- |
| `--dataset.num_episodes` | Number of episodes to record |
| `--dataset.episode_time_s` | Duration of each recording episode in seconds |
| `--dataset.reset_time_s` | Duration of the reset phase between episodes in seconds |
| `--teleop.type` | Optional. Teleoperator to drive the robot during resets |
| `--strategy.reset_to_initial_position` | Whether to reset the robot to its initial position between episodes |
| `--strategy.smooth_leader_to_follower_handover` | Whether to turn on or off the leader -> follower smooth handover behavior. |
---
+3 -19
View File
@@ -108,7 +108,6 @@ own binding plus a matching image block, e.g.
```yaml
ask_vqa_top:
route: vqa
bindings:
vqa_query: "emitted_at(t, style=vqa, role=user, camera=observation.images.top)"
vqa: "emitted_at(t, style=vqa, role=assistant, camera=observation.images.top)"
@@ -128,9 +127,7 @@ ask_vqa_top:
}
```
Add one such sub-recipe per camera the dataset records. The explicit
`route: vqa` marker makes a matching sparse VQA annotation take precedence
over normal weighted blend selection; component names are purely descriptive.
Add one such sub-recipe per camera the dataset records.
## Layer 3 — training format
@@ -144,20 +141,7 @@ sample["target_message_indices"]
The renderer does not apply a tokenizer chat template. Policy processors decide how to serialize the messages for their backbone, which keeps the same dataset usable across SmolVLA, Pi0.5, and any future VLM that expects OpenAI-style chat messages.
## Blends
Blend recipes select one weighted sub-recipe deterministically from the sample index.
`recipes/subtask_mem.yaml` trains the compact core blend — high-level subtask prediction, low-level execution, and memory. `recipes/subtask_mem_vqa_speech.yaml` is the fuller variant that also adds VQA and spoken interjection responses.
`recipes/subtask_joint.yaml` demonstrates joint sequence training rather than a
weighted blend. For the same sample, its assistant subtask is supervised with
text cross-entropy on the `low_level` stream while action prediction remains
active, matching the joint setup from the π0.5 paper. Enable
`--policy.joint_subtask_conditioning=true` to use that subtask conditioning at inference.
## Graceful absence
If both language columns are missing, `None`, or empty, `RenderMessagesStep` uses
the task string as low-level supervision when available and otherwise leaves the
sample unchanged. For an annotated sample, if no recipe branch applies and no
task fallback exists, rendering returns `None`, allowing a loader to retry another sample.
If both language columns are missing, `None`, or empty, `RenderMessagesStep` is a no-op.
If an event-scoped branch is selected on a frame without the required event row, rendering returns `None`, allowing a loader to retry another sample.
-16
View File
@@ -142,22 +142,6 @@ repo_id = "yaak-ai/L2D-v3"
dataset = StreamingLeRobotDataset(repo_id) # streams directly from the Hub
```
Datasets stored in an [HF Storage Bucket](https://huggingface.co/docs/hub/storage-buckets) (`hf://buckets/`) can be streamed the same way by passing `repo_type="bucket"`:
```python
dataset = StreamingLeRobotDataset("my-org/my-bucket", repo_type="bucket")
```
Both options are available in `lerobot-train` through `--dataset.streaming=true`, and `--dataset.repo_type=bucket` to stream from a bucket instead of a Hub dataset repo:
```bash
lerobot-train \
--dataset.repo_id=my-org/my-bucket \
--dataset.repo_type=bucket \
--dataset.streaming=true \
...
```
<div style="display:flex; justify-content:center; gap:12px; flex-wrap:wrap;">
<figure style="margin:0; text-align:center;">
<img
+12 -46
View File
@@ -92,20 +92,6 @@ LIBERO supports two control modes — `relative` (default) and `absolute`. Diffe
--env.control_mode=relative # or "absolute"
```
### Reset performance
By default, LeRobot preserves LIBERO's hard-reset behavior. With fixed initial
states enabled, you can opt into soft resets to skip rebuilding the simulator
model and renderer on every episode:
```bash
--env.init_states=true --env.hard_reset=false
```
Soft resets are faster but are not bit-identical to hard resets after the
environment's settling steps, so camera observations and policy results may
differ slightly. Use hard resets when reproducing benchmark results.
### Policy inputs and outputs
**Observations:**
@@ -128,58 +114,38 @@ differ slightly. Use hard resets when reproducing benchmark results.
### Recommended evaluation episodes
For reproducible benchmarking, use **10 episodes per task** across all four standard suites (Spatial, Object, Goal, Long). This gives 400 total episodes and matches the protocol used for published results. Success rates may vary by a few percent across evaluation seeds, so we recommend averaging over 3 seeds.
<Tip>
To compare two policies on the same episodes, use the same `--seed`, keep
`--env.init_states=true`, and run each task in a single batch
(`--eval.batch_size` equal to episodes per task).
</Tip>
For reproducible benchmarking, use **10 episodes per task** across all four standard suites (Spatial, Object, Goal, Long). This gives 400 total episodes and matches the protocol used for published results.
## Training
### Dataset
Two preprocessed LIBERO datasets are fully compatible with LeRobot. They contain the same demonstrations with the same schema and differ in how camera frames are stored:
We provide a preprocessed LIBERO dataset fully compatible with LeRobot:
| | [lerobot/libero](https://huggingface.co/datasets/lerobot/libero) | [HuggingFaceVLA/libero](https://huggingface.co/datasets/HuggingFaceVLA/libero) |
| ------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| episodes / frames / tasks | 1,693 / 273,465 / 40 | 1,693 / 273,465 / 40 |
| cameras | 2× 256×256×3 | 2× 256×256×3 |
| state / action dims | 8 / 7 | 8 / 7 |
| dataset format | v3.0 | v3.0 |
| camera encoding | MP4 video | PNG in parquet |
| download size | **1.9 GB** | 69.9 GB |
| extra dependency | video backend (`torchcodec` or `pyav`) | none |
**We recommend [lerobot/libero](https://huggingface.co/datasets/lerobot/libero)**: **37× smaller download** with **equivalent loading speed** (~330 samples/s per worker). Video re-encoding is slightly lossy; use the image-based variant if you cannot install a video decoding backend.
- [HuggingFaceVLA/libero](https://huggingface.co/datasets/HuggingFaceVLA/libero)
For reference, the original dataset published by Physical Intelligence:
- [physical-intelligence/libero](https://huggingface.co/datasets/physical-intelligence/libero)
<Tip>
Pin `--dataset.revision=<commit-sha>` when reporting results — Hub datasets can be re-uploaded, and success rates are only comparable against the same data revision.
</Tip>
### Example training command
Train SmolVLA on the recommended dataset:
```bash
lerobot-train \
--policy.type=smolvla \
--policy.repo_id=${HF_USER}/libero-test \
--policy.load_vlm_weights=true \
--policy.push_to_hub=false \
--dataset.repo_id=lerobot/libero \
--dataset.video_backend=torchcodec \
--output_dir=./outputs/libero_smolvla \
--dataset.repo_id=HuggingFaceVLA/libero \
--env.type=libero \
--env.task=libero_10 \
--output_dir=./outputs/ \
--steps=100000 \
--batch_size=64
--batch_size=4 \
--eval.batch_size=1 \
--eval.n_episodes=1 \
--env_eval_freq=1000
```
To share the result on the Hub, replace `--policy.push_to_hub=false` with `--policy.repo_id=${HF_USER}/libero-smolvla`. Evaluate saved checkpoints with `lerobot-eval` as shown in the [Evaluation](#evaluation) section.
## Reproducing published results
We reproduce the results of Pi0.5 on the LIBERO benchmark. We take the Physical Intelligence LIBERO base model (`pi05_libero`) and finetune for an additional 6k steps in bfloat16, with batch size of 256 on 8 H100 GPUs using the [HuggingFace LIBERO dataset](https://huggingface.co/datasets/HuggingFaceVLA/libero).
-14
View File
@@ -134,20 +134,6 @@ LIBERO-plus supports two control modes — `relative` (default) and `absolute`.
--env.control_mode=relative # or "absolute"
```
### Reset performance
By default, LeRobot preserves LIBERO's hard-reset behavior. With fixed initial
states enabled, you can opt into soft resets to skip rebuilding the simulator
model and renderer on every episode:
```bash
--env.init_states=true --env.hard_reset=false
```
Soft resets are faster but are not bit-identical to hard resets after the
environment's settling steps, so camera observations and policy results may
differ slightly. Use hard resets when reproducing benchmark results.
### Policy inputs and outputs
**Observations:**
+129 -125
View File
@@ -1,29 +1,28 @@
# Multi-GPU Training
LeRobot trains on multiple GPUs through [Hugging Face Accelerate](https://huggingface.co/docs/accelerate). Three data-parallel layouts are supported:
| Layout | What it does | Config |
| -------- | ------------------------------------------------------------- | ------------------------------------------------------- |
| **DDP** | Replicates the full model on every GPU | default on any multi-GPU launch |
| **FSDP** | Shards parameters, gradients, and optimizer state across GPUs | `--parallelism.dp_shard=N` |
| **HSDP** | Shards within groups of GPUs, replicates across groups | `--parallelism.dp_replicate=R --parallelism.dp_shard=S` |
This guide shows you how to train policies on multiple GPUs using [Hugging Face Accelerate](https://huggingface.co/docs/accelerate).
## Installation
`accelerate` is included in the `training` extra:
`accelerate` is included in the `training` extra. Install it with:
```bash
pip install 'lerobot[training]'
```
## Launching
## Training with Multiple GPUs
Distributed training can be launched through both `torchrun` and `accelerate launch`. Accelerate is used as a plain launcher: it does not manage the training configuration, and every distributed training setting lives in LeRobot's own config system.
You can launch training in two ways:
With `torchrun`:
### Option 1: Without config (specify parameters directly)
You can specify all parameters directly in the command without running `accelerate config`:
```bash
torchrun --nproc-per-node=2 $(which lerobot-train) \
accelerate launch \
--multi_gpu \
--num_processes=2 \
$(which lerobot-train) \
--dataset.repo_id=${HF_USER}/my_dataset \
--policy.type=act \
--policy.repo_id=${HF_USER}/my_trained_policy \
@@ -32,145 +31,150 @@ torchrun --nproc-per-node=2 $(which lerobot-train) \
--wandb.enable=true
```
With `accelerate launch` (as a plain launcher):
**Key accelerate parameters:**
- `--multi_gpu`: Enable multi-GPU training
- `--num_processes=2`: Number of GPUs to use
- `--mixed_precision=fp16`: Use fp16 mixed precision (or `bf16` if supported)
### Option 2: Using accelerate config
If you prefer to save your configuration, you can optionally configure accelerate for your hardware setup by running:
```bash
accelerate config
```
This interactive setup will ask you questions about your training environment (number of GPUs, mixed precision settings, etc.) and saves the configuration for future use. For a simple multi-GPU setup on a single machine, you can use these recommended settings:
- Compute environment: This machine
- Number of machines: 1
- Number of processes: (number of GPUs you want to use)
- GPU ids to use: (leave empty to use all)
- Mixed precision: fp16 or bf16 (recommended for faster training)
Then launch training with:
```bash
accelerate launch $(which lerobot-train) \
--dataset.repo_id=${HF_USER}/my_dataset \
--policy.type=act \
--policy.repo_id=${HF_USER}/my_trained_policy \
--output_dir=outputs/train/act_multi_gpu \
--job_name=act_multi_gpu \
--wandb.enable=true
```
## How It Works
When you launch training with accelerate:
1. **Automatic detection**: LeRobot automatically detects if it's running under accelerate
2. **Data distribution**: Your batch is automatically split across GPUs
3. **Gradient synchronization**: Gradients are synchronized across GPUs during backpropagation
4. **Single process logging**: Only the main process logs to wandb and saves checkpoints
## Learning Rate and Training Steps Scaling
**Important:** LeRobot does **NOT** automatically scale learning rates or training steps based on the number of GPUs. This gives you full control over your training hyperparameters.
### Why No Automatic Scaling?
Many distributed training frameworks automatically scale the learning rate by the number of GPUs (e.g., `lr = base_lr × num_gpus`).
However, LeRobot keeps the learning rate exactly as you specify it.
### When and How to Scale
If you want to scale your hyperparameters when using multiple GPUs, you should do it manually:
**Learning Rate Scaling:**
```bash
# Example: 2 GPUs with linear LR scaling
# Base LR: 1e-4, with 2 GPUs -> 2e-4
accelerate launch --num_processes=2 $(which lerobot-train) \
--dataset.repo_id=${HF_USER}/my_dataset \
--policy.type=act \
--policy.repo_id=${HF_USER}/my_trained_policy \
--output_dir=outputs/train/act_multi_gpu \
--job_name=act_multi_gpu \
--wandb.enable=true
--optimizer.lr=2e-4 \
--dataset.repo_id=lerobot/pusht \
--policy.type=act
```
With no `--parallelism.*` flags, a multi-process launch runs plain DDP. Multi-node runs use the standard `torchrun --nnodes/--node-rank/--rdzv-endpoint` flags (or `accelerate launch --num_machines/--machine_rank/--main_process_ip`).
**Training Steps Scaling:**
> [!WARNING]
> Accelerate's YAML config files (`accelerate launch --config_file some.yaml`, `accelerate config`) are not supported. They configure the engine through environment variables, bypassing LeRobot's configuration system, so `train_config.json` would no longer describe the settings a run actually used. `lerobot-train` therefore refuses to start when [accelerate environment variables](https://huggingface.co/docs/accelerate/usage_guides/fsdp) are set. Put the settings in `--parallelism.*` / `--accelerator.*` flags instead, or set `LEROBOT_ALLOW_ACCELERATE_ENV=1` to acknowledge the override and proceed anyway.
## Batch semantics, learning rate, and steps
Each of the `dp_replicate × dp_shard` data-parallel workers loads its own `--batch_size` micro-batch every step, so one training step consumes `batch_size × dp_world_size` samples, and `× gradient_accumulation_steps` of those go into each optimizer update:
```
effective_batch_size = batch_size × dp_world_size × gradient_accumulation_steps
```
The training banner prints this factorization at startup. `--steps` counts loop steps (micro-batches per worker), not optimizer updates.
Gradient accumulation is a first-class flag:
Since the effective batch size `bs` increases with multiple GPUs (batch_size × num_gpus), you may want to reduce the number of training steps proportionally:
```bash
torchrun --nproc-per-node=2 $(which lerobot-train) \
--batch_size=8 --accelerator.gradient_accumulation.steps=4 ...
# Example: 2 GPUs with effective batch size 2x larger
# Original: batch_size=8, steps=100000
# With 2 GPUs: batch_size=8 (16 in total), steps=50000
accelerate launch --num_processes=2 $(which lerobot-train) \
--batch_size=8 \
--steps=50000 \
--dataset.repo_id=lerobot/pusht \
--policy.type=act
```
**LeRobot does not auto-scale the learning rate or the number of steps** when the effective batch size grows. If you scale out and want equivalent training, please adjust manually, e.g. with 2 GPUs: double `--optimizer.lr` (linear scaling), or halve `--steps`.
## Training Large Models with FSDP
## Sharded training (FSDP)
DDP replicates the full model on every GPU, so a model that doesn't fit on one GPU won't fit under
DDP either. For large models, use **FSDP** (Fully Sharded Data Parallel), which shards parameters,
gradients, and optimizer state across GPUs. See the [accelerate FSDP guide](https://huggingface.co/docs/accelerate/usage_guides/fsdp) for background.
If a model is too large to train with DDP, shard it with FSDP2:
An example on how to launch LeRobot training with FSDP across 4 GPUs (1 machine):
```bash
torchrun --nproc-per-node=4 $(which lerobot-train) \
accelerate launch --config_file fsdp.yaml --num_processes=4 $(which lerobot-train) \
--dataset.repo_id=${HF_USER}/my_dataset \
--policy.type=<your_policy> \
--parallelism.dp_shard=4 \
--accelerator.mixed_precision=bf16 \
--output_dir=outputs/train/my_policy_fsdp
```
`--parallelism.dp_shard=-1` shards over however many processes the launcher started.
A minimal `fsdp.yaml` (FSDP1; shards params/grads/optimizer — ZeRO-3-equivalent):
### Wrap units
FSDP shards the model in units (typically the repeated transformer block) and gathers one unit at a time during forward/backward. Policies declare their wrap units via `_fsdp_wrap_modules` on the policy class. For example, ACT declares `["ACTEncoderLayer", "ACTDecoderLayer"]` and FastWAM declares `["MoTLayer"]`. For a policy without a `_fsdp_wrap_modules` declaration, pass one of the flags below. You can specify the module class name explicitly, or use a size-based policy instead:
```bash
--accelerator.fsdp.wrap_modules='["MyTransformerBlock"]' # explicit class names
--accelerator.fsdp.min_num_params=1000000 # or: wrap every submodule above 1M params
```yaml
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
mixed_precision: bf16
num_machines: 1
num_processes: 4
fsdp_config:
fsdp_version: 1
fsdp_sharding_strategy: FULL_SHARD # params + grads + optimizer (ZeRO-3)
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
fsdp_transformer_layer_cls_to_wrap: <YourTransformerBlock> # repeated block class to shard
fsdp_use_orig_params: true # required: optimizer is built pre-prepare
fsdp_state_dict_type: FULL_STATE_DICT
```
If a policy doesn't declare `_fsdp_wrap_modules` and no `--accelerator.fsdp.wrap_modules` or `--accelerator.fsdp.min_num_params` is passed, the run fails at startup rather than silently wrapping only the root module (which would forfeit all sharding memory savings).
Set `fsdp_transformer_layer_cls_to_wrap` to your model's repeated transformer-block class so each
block is sharded as its own unit. `fsdp_use_orig_params: true` is required because LeRobot builds the
optimizer before `accelerator.prepare()`.
Other sharding settings:
### FSDP checkpoints
- `--accelerator.fsdp.reshard_after_forward`: whether to keep each unit's parameters resident after forward.
- `--accelerator.fsdp.cpu_offload`: keeps parameters, gradients and optimizer states on CPU.
- `--accelerator.fsdp.ignored_modules`: a regex of module paths to keep unsharded.
LeRobot gathers the full state dict across all ranks and the main process writes it as a single
`model.safetensors`, loadable as usual with `Policy.from_pretrained(...)`. Two things to look out for:
### HSDP
Hybrid Sharded Data Parallel: parameters, gradients and optimizer states are sharded across `dp_shard` ranks, and that sharding is replicated `dp_replicate` times. Parameter all-gathers and gradient reduce-scatters stay inside a shard group; only the all-reduce that synchronizes the replicas crosses between groups. The two degrees must multiply to the world size:
```bash
# 16 GPUs = 2 nodes × 8: shard within each node, replicate across nodes
torchrun --nnodes=2 --nproc-per-node=8 ... $(which lerobot-train) \
--parallelism.dp_replicate=2 --parallelism.dp_shard=8 ...
```
## Checkpoints
Every checkpoint contains a `pretrained_model/` directory and a `training_state/` directory:
```text
005000/ # the training step at that checkpoint
├── pretrained_model/
│ ├── config.json # policy config
│ ├── train_config.json # the full training config
│ ├── model.safetensors # full weights (checkpoint_format ∈ {safetensors, safetensors_dcp}, or any non-sharded run)
│ ├── pytorch_model_fsdp_0/ # DCP weight shards (checkpoint_format ∈ {dcp, safetensors_dcp})
│ ├── policy_preprocessor.json # preprocessor config (when the run has a preprocessor)
│ ├── policy_preprocessor_step_*.safetensors # state of the stateful preprocessor steps
│ ├── policy_postprocessor.json # postprocessor config (when the run has a postprocessor)
│ └── policy_postprocessor_step_*.safetensors # state of the stateful postprocessor steps
└── training_state/
├── training_step.json # step counter, topology, and batch semantics
├── rng_state.safetensors # rng states
├── scheduler_state.json # scheduler state (when the run has a scheduler)
├── optimizer_state.safetensors # full optimizer state (non-sharded runs)
├── optimizer_param_groups.json # optimizer param groups (non-sharded runs)
└── optimizer_0/ # DCP optimizer shards (sharded runs)
```
During single-GPU or DDP training, the pipeline serializes each state dict into a single file: `model.safetensors` for the model and `optimizer_state.safetensors` for the optimizer.
During sharded training, the optimizer state is saved as DCP shards under `training_state/optimizer_0/`, and the layout of the model under `pretrained_model/` can be configured through `--checkpoint_format`:
| `--checkpoint_format` | Weights artifact | Use when |
| ------------------------- | -------------------------------------------- | --------------------------------------------------------------------- |
| `safetensors` _(default)_ | single `model.safetensors` only | you want every checkpoint immediately loadable with `from_pretrained` |
| `dcp` | `pytorch_model_fsdp_0/` shard directory only | gathering the full weights makes saves and resumes too slow |
| `safetensors_dcp` | both | you want fast resume _and_ immediately loadable checkpoints |
Two things to know about gathered (`safetensors`) checkpoints from sharded runs:
- **They store fp32 weights.** Under mixed precision training, FSDP keeps an fp32 master copy, and the checkpoint saves the master copy to make sure training resumes consistently.
- The gather is collective (all ranks participate) but only the main process writes.
### Converting DCP checkpoints
`lerobot-convert-dcp` merges a DCP shard directory into a regular `model.safetensors`, offline and without GPUs:
```bash
lerobot-convert-dcp --checkpoint_dir=outputs/train/run/checkpoints/005000
lerobot-convert-dcp --checkpoint_dir=... --delete_dcp=true --push_to_hub=${HF_USER}/my_policy
```
`--push_to_hub` publishes the converted directory as a model repo.
### Resuming
Resume with `--resume=true --config_path=.../checkpoints/last/pretrained_model/train_config.json`. Resuming from a DCP checkpoint supports resharding the model and optimizer state to the _current_ topology, which means you can resume with a different `dp_replicate/dp_shard` split. The data sampler can always resume at the right epoch and offset, but is only _sample-exact_ when the world size and batch size match the original run (a warning is logged otherwise).
> [!NOTE]
> FSDP checkpoints written by LeRobot 0.6.x and earlier used a different on-disk layout (a gathered full optimizer state) and **cannot be resumed**.
- **Checkpoints store fp32 weights.** Under mixed precision (`bf16`/`fp16`) FSDP keeps an fp32 master
copy, and the checkpoint saves it (~2× the bf16 size on disk) so training can resume consistently
with the fp32 optimizer state; `from_pretrained` casts back to the policy dtype on load. FSDP-specific
caveat: an fp32 checkpoint is materialized in full precision on the target device _before_ casting,
so loading it for inference on a tight GPU can OOM even when the bf16 model would fit — load on CPU
first, or cast `model.safetensors` to the deployment dtype offline.
- The sharded optimizer state is gathered into a full (world-size-independent) state dict and saved
alongside the model in the same `optimizer_state.safetensors` / `optimizer_param_groups.json`
format as single-GPU training, so **resume-from-checkpoint is supported** with `--resume=true`.
Resume reshards both the model and the optimizer state to the _current_ FSDP topology, so you can
resume an FSDP checkpoint on a different number of GPUs. Note that the data sampler is only
sample-exact when the world size and batch size match the original run (a warning is logged
otherwise); the optimizer/model state itself is unaffected.
## Notes
- Checkpoint saves and end-of-training publishes are collective (every rank enters them). Gathered weights, sidecar files and Hub uploads are written by the main process alone.
- Metrics are reduced across ranks before logging: losses are averaged, and `samples/s` reports cluster-wide throughput.
- Learning-rate scheduling is stepped once per training step regardless of the number of processes (`step_scheduler_with_optimizer=False` is baked in).
- The `--policy.use_amp` flag in `lerobot-train` is only used when **not** running with accelerate. When using accelerate, mixed precision is controlled by accelerate's configuration.
- Training logs, checkpoints, and hub uploads are only done by the main process to avoid conflicts. Non-main processes have console logging disabled to prevent duplicate output.
- The effective batch size is `batch_size × num_gpus`. If you use 4 GPUs with `--batch_size=8`, your effective batch size is 32.
- Learning rate scheduling is handled correctly across multiple processes—LeRobot sets `step_scheduler_with_optimizer=False` to prevent accelerate from adjusting scheduler steps based on the number of processes.
- When saving or pushing models, LeRobot automatically unwraps the model from accelerate's distributed wrapper to ensure compatibility.
- WandB integration automatically initializes only on the main process, preventing multiple runs from being created.
For background on the underlying machinery, see the [Accelerate FSDP guide](https://huggingface.co/docs/accelerate/usage_guides/fsdp). To go deeper on large-scale training, check out the [Ultrascale Playbook](https://huggingface.co/spaces/nanotron/ultrascale-playbook).
For more advanced configurations and troubleshooting, see the [Accelerate documentation](https://huggingface.co/docs/accelerate). If you want to learn more about how to train on a large number of GPUs, checkout this awesome guide: [Ultrascale Playbook](https://huggingface.co/spaces/nanotron/ultrascale-playbook).
+23 -113
View File
@@ -36,12 +36,6 @@ This diverse training mixture creates a "curriculum" that enables generalization
pip install -e ".[pi]"
```
If you installed LeRobot from PyPI:
```bash
pip install 'lerobot[pi]'
```
## Usage
To use π₀.₅ in your LeRobot configuration, specify the policy type as:
@@ -52,106 +46,27 @@ policy.type=pi05
## Training
### Quickstart on LIBERO
Finetune the LIBERO base model on [lerobot/libero](https://huggingface.co/datasets/lerobot/libero), a ~1.9 GB video-encoded copy of the demonstrations behind the [results below](#libero-benchmark-results).
It carries the keys π₀.₅ reads, which are also the ones the LIBERO environment produces at evaluation time:
| Feature | Shape in the dataset | How π₀.₅ consumes it |
| --------------------------- | -------------------- | ------------------------------------------------------- |
| `observation.images.image` | 256×256×3, agentview | resized to 224×224 |
| `observation.images.image2` | 256×256×3, wrist | resized to 224×224 |
| `observation.state` | 8 | discretized into 256 bins and written into the prompt |
| `action` | 7 | padded to 32 internally; the loss uses the first 7 dims |
**No `--rename_map` is needed here** — the keys already match; see [Rename Map and Empty Cameras](./rename_map) if yours differ.
<Tip>
π₀.₅ uses the gated
[google/paligemma-3b-pt-224](https://huggingface.co/google/paligemma-3b-pt-224)
tokenizer — accept its license on the Hub and log in with `hf auth login`
before training.
</Tip>
Sized for a single 80 GB GPU:
```bash
lerobot-train \
--dataset.repo_id=lerobot/libero \
--policy.type=pi05 \
--policy.pretrained_path=lerobot/pi05_libero_base \
--policy.normalization_mapping='{"ACTION": "MEAN_STD", "STATE": "MEAN_STD", "VISUAL": "IDENTITY"}' \
--policy.n_action_steps=10 \
--policy.empty_cameras=1 \
--policy.freeze_vision_encoder=false \
--policy.train_expert_only=false \
--policy.gradient_checkpointing=true \
--policy.dtype=bfloat16 \
--policy.device=cuda \
--policy.push_to_hub=false \
--output_dir=./outputs/pi05_libero \
--job_name=pi05_libero \
--batch_size=64 \
--num_workers=8 \
--steps=30000 \
--save_freq=5000 \
--seed=1000
```
**Mean/std normalization, not π₀.₅'s [quantile default](#quantile-statistics)** — matching [pi05_libero_finetuned_v044](https://huggingface.co/lerobot/pi05_libero_finetuned_v044), the checkpoint the results below were measured on.
**`--policy.n_action_steps=10` and `--policy.empty_cameras=1` are explicit** because `--policy.pretrained_path` loads weights only — `lerobot/pi05_libero_base` stores both, and they would otherwise fall back to `50` and `0` (see [Loading a checkpoint](#loading-a-checkpoint)).
Then evaluate a checkpoint with `lerobot-eval` and compare against the reference success rates — see [LIBERO](./libero).
### Quantile statistics
π₀.₅ normalizes `STATE` and `ACTION` with quantiles, so your dataset's `meta/stats.json` needs `q01` and `q99`. Older datasets carry only `min`/`max`/`mean`/`std` and fail on the first batch:
```
ValueError: QUANTILES normalization mode requires q01 and q99 stats
```
Recompute them:
```bash
lerobot-edit-dataset \
--repo_id your_dataset \
--new_repo_id your_dataset \
--operation.type recompute_stats \
--operation.overwrite true
```
**The result lands in `$HF_LEROBOT_HOME/your_dataset`**, not the cache `--dataset.repo_id` reads — so train with `--dataset.root=$HF_LEROBOT_HOME/your_dataset`, or add `--push_to_hub true` above.
Or keep the dataset as-is and pass `--policy.normalization_mapping='{"ACTION": "MEAN_STD", "STATE": "MEAN_STD", "VISUAL": "IDENTITY"}'`.
### Training Command Example
The same finetune with the VLM frozen: less memory, at some cost in success rate. Swap `--dataset.repo_id` for your own dataset.
Here's a complete training command for finetuning the base π₀.₅ model on your own dataset:
```bash
lerobot-train \
--dataset.repo_id=lerobot/libero \
--dataset.repo_id=your_dataset \
--policy.type=pi05 \
--policy.pretrained_path=lerobot/pi05_libero_base \
--policy.normalization_mapping='{"ACTION": "MEAN_STD", "STATE": "MEAN_STD", "VISUAL": "IDENTITY"}' \
--policy.n_action_steps=10 \
--policy.empty_cameras=1 \
--policy.freeze_vision_encoder=true \
--policy.train_expert_only=true \
--output_dir=./outputs/pi05_training \
--job_name=pi05_training \
--policy.repo_id=your_repo_id \
--policy.pretrained_path=lerobot/pi05_base \
--policy.compile_model=true \
--policy.gradient_checkpointing=true \
--wandb.enable=true \
--policy.dtype=bfloat16 \
--policy.freeze_vision_encoder=false \
--policy.train_expert_only=false \
--steps=3000 \
--policy.device=cuda \
--policy.push_to_hub=false \
--output_dir=./outputs/pi05_libero_expert \
--job_name=pi05_libero_expert \
--batch_size=64 \
--num_workers=8 \
--steps=30000 \
--save_freq=5000 \
--seed=1000
--batch_size=32
```
### Key Training Parameters
@@ -159,24 +74,10 @@ lerobot-train \
- **`--policy.compile_model=true`**: Enables model compilation for faster training
- **`--policy.gradient_checkpointing=true`**: Reduces memory usage significantly during training
- **`--policy.dtype=bfloat16`**: Use mixed precision training for efficiency
- **`--batch_size=64`**: Batch size for training, adapt this based on your GPU memory
- **`--batch_size=32`**: Batch size for training, adapt this based on your GPU memory
- **`--policy.pretrained_path=lerobot/pi05_base`**: The base π₀.₅ model you want to finetune, options are:
- [lerobot/pi05_base](https://huggingface.co/lerobot/pi05_base)
- [lerobot/pi05_libero_base](https://huggingface.co/lerobot/pi05_libero_base) (specifically trained on the Libero dataset)
### Loading a checkpoint
The two forms are not interchangeable:
| | `--policy.path` | `--policy.pretrained_path` |
| -------------------------------------- | ---------------------------------------------- | ------------------------------------ |
| Loads | weights **and** the checkpoint's `config.json` | weights only |
| Feature names | from the checkpoint | from your dataset |
| Stored settings, e.g. `n_action_steps` | inherited | reset to the defaults |
| `--policy.type` | must be omitted | required |
| `--rename_map` | needed when your camera keys differ | never — the keys come from your data |
Passing a `--rename_map` alongside `--policy.pretrained_path` renames the batch away from those names, and the first batch fails with `All image features are missing from the batch`.
- [lerobot/pi05_libero](https://huggingface.co/lerobot/pi05_libero) (specifically trained on the Libero dataset)
### Training Parameters Explained
@@ -187,6 +88,15 @@ Passing a `--rename_map` alongside `--policy.pretrained_path` renames the batch
**💡 Tip**: Setting `train_expert_only=true` freezes the VLM and trains only the action expert and projections, allowing finetuning with reduced memory usage.
If your dataset is not converted with `quantiles`, you can convert it with the following command:
```bash
python src/lerobot/scripts/augment_dataset_quantile_stats.py \
--repo-id=your_dataset \
```
Or train pi05 with this normalization mapping: `--policy.normalization_mapping='{"ACTION": "MEAN_STD", "STATE": "MEAN_STD", "VISUAL": "IDENTITY"}'`
## Relative Actions
By default, π₀.₅ predicts absolute actions. You can enable **relative actions** so the model predicts offsets relative to the current robot state. This can improve training stability for certain setups.
-2
View File
@@ -82,8 +82,6 @@ By default the env samples objects only from the `lightwheel` registry (what `--
All eval snippets below mirror the CI command (see `.github/workflows/benchmark_tests.yml`). The `--rename_map` argument maps RoboCasa's native camera keys (`robot0_agentview_left` / `robot0_eye_in_hand` / `robot0_agentview_right`) onto the three-camera (`camera1` / `camera2` / `camera3`) input layout the released `smolvla_robocasa` policy was trained on.
By default, each task uses the rollout horizon registered by RoboCasa. Set `--env.episode_length=<steps>` to apply the same explicit horizon to every selected task.
### Single-task evaluation (recommended for quick iteration)
```bash
+6 -3
View File
@@ -35,11 +35,14 @@ pip install --override <(printf 'gymnasium==0.29.1\nnumpy==1.26.4\n') \
### Docker (recommended)
```bash
# Build the RoboMME evaluation image from the repo root
docker build -f docker/Dockerfile.benchmark.robomme -t lerobot-benchmark-robomme .
# Build base image first (from repo root)
docker build -f docker/Dockerfile.eval-base -t lerobot-eval-base .
# Build RoboMME eval image (applies gymnasium + numpy pin overrides)
docker build -f docker/Dockerfile.benchmark.robomme -t lerobot-robomme .
```
The benchmark Dockerfile extends the published `huggingface/lerobot-gpu:latest` image, then overrides `gymnasium==0.29.1` and `numpy==1.26.4`. Both versions are runtime-safe for lerobot's actual API usage.
The `docker/Dockerfile.benchmark.robomme` image overrides `gymnasium==0.29.1` and `numpy==1.26.4` after lerobot's install. Both versions are runtime-safe for lerobot's actual API usage.
## Running Evaluation
+1 -1
View File
@@ -338,7 +338,7 @@ It is advisable to install one 3-pin cable in the motor after placing them befor
<hfoption id="Leader">
- Mount the leader holder onto the wrist and secure it with 4 M3x6mm screws.
- Attach the handle to the leader holder using 1 M2x6mm screw.
- Attach the handle to motor 5 using 1 M2x6mm screw.
- Insert the gripper motor, secure it with 2 M2x6mm screws on each side, attach a motor horn using a M3x6mm horn screw.
- Attach the follower trigger with 4 M3x6mm screws.
-339
View File
@@ -1,339 +0,0 @@
# Third-Party Robots & Teleoperators
The LeRobot ecosystem extends far beyond its officially supported hardware. Thanks to LeRobot's plugin architecture, the community has built integrations for a wide range of robot arms and teleoperation devices — from industrial manipulators to affordable hobbyist platforms, VR headsets, haptic devices, and full arm-plus-teleoperator kits. This page showcases community-maintained integrations you can use for teleoperation, data collection, and policy deployment.
> [!IMPORTANT]
> These projects are developed and maintained by third parties. Please refer to each repository for installation instructions, hardware requirements, and support.
Drop-in plugins are auto-discovered by package name: LeRobot imports any installed package prefixed with `lerobot_robot_` or `lerobot_teleoperator_`. Once installed, reference the `type` the plugin registers (see its README — it may differ from the package name) directly from any LeRobot command:
```bash
pip install lerobot_robot_<name> lerobot_teleoperator_<name>
lerobot-record \
--robot.type=<robot_name> \
--teleop.type=<teleoperator_name> \
--dataset.repo_id=${HF_USER}/my-dataset \
--dataset.num_episodes=5
```
> [!TIP]
> ⚠️ marks projects that are forks/extensions of LeRobot. They may require custom setup rather than working with an unmodified install. All other entries are drop-in plugins.
## Industrial & Collaborative Arms
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/SpesRobotics/lerobot-robot-xarm">lerobot-robot-xarm</a></td>
<td>Plugin for the xArm collaborative arm series from <a href="https://www.ufactory.cc/">UFACTORY</a>.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/lebai-robotics/lerobot_lebai">lerobot_lebai</a></td>
<td>Plugin for the six-axis collaborative arms from <a href="https://lebai.ltd/en/">Lebai</a>.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/wengmister/LeFranX">LeFranX</a> ⚠️</td>
<td>LeRobot extension for the <a href="https://franka.de/">Franka</a> research arm, paired with the <a href="https://www.robotera.com/">RobotEra XHand</a> hand for VR teleoperation.</td>
</tr>
</tbody>
</table>
#### Universal Robots UR5e
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/yechen056/UR5e-LeRobot">UR5e-LeRobot</a> ⚠️</td>
<td>LeRobot extension for the <a href="https://www.universal-robots.com/">Universal Robots UR5e</a>, with single-arm and bimanual support.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/scy-v/lerobot_ur5e_auto">lerobot_ur5e_auto</a> ⚠️</td>
<td>LeRobot extension for a mobile <a href="https://www.universal-robots.com/">Universal Robots UR5e</a>, adding automated recording at scale with minimal supervision.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/F-Fer/lerobot_ur5e_gello">lerobot_robot_ur5e</a></td>
<td>Plugin for the <a href="https://www.universal-robots.com/">Universal Robots UR5e</a> with a <a href="https://robotiq.com/">Robotiq</a> gripper, over RTDE control.</td>
</tr>
</tbody>
</table>
## Research & Learning Arms
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/TrossenRobotics/lerobot_trossen">lerobot_trossen</a></td>
<td>Plugin for the WidowX and ALOHA-style arms from <a href="https://www.trossenrobotics.com/">Trossen Robotics</a>.</td>
</tr>
</tbody>
</table>
#### AgileX Piper
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/AgRoboticsResearch/lerobot_robot_piper">lerobot_robot_piper (AgRobotics Research)</a></td>
<td>Plugin for the <a href="https://global.agilex.ai/">AgileX Piper</a> arm.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/WeGo-Robotics/lerobot_robot_piper">lerobot_robot_piper (WeGo Robotics)</a></td>
<td>Plugin for the <a href="https://global.agilex.ai/">AgileX Piper</a> arm, with multi-arm teleoperation, safety limits, and GUI tools.</td>
</tr>
</tbody>
</table>
## Affordable & Hobbyist Arms
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/servodevelop/fashionstar-lerobot-robot-cello">fashionstar-lerobot-robot-cello</a></td>
<td>Plugin for the StarAI Cello 6+1 degrees of freedom robot arm from <a href="https://fashionstar.com.hk/">FashionStar</a>.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/servodevelop/fashionstar-lerobot-robot-viola">fashionstar-lerobot-robot-viola</a></td>
<td>Plugin for the compact StarAI Viola 6+1 degrees of freedom robot arm from <a href="https://fashionstar.com.hk/">FashionStar</a>.</td>
</tr>
</tbody>
</table>
## Service, Mobile & Utility Robots
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/ugo-plus/lerobot-robot-ugo-pro">lerobot-robot-ugo-pro</a></td>
<td>Plugin for the ugo Pro dual-arm service robot from <a href="https://ugo.plus/products/ugo-pro/">ugo</a>.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/zuoxingdong/lerobot_robot_lekiwi_pincopen">lerobot_robot_lekiwi_pincopen</a></td>
<td>Plugin for a LeKiwi mobile manipulator with a <a href="https://github.com/pollen-robotics/PincOpen">PincOpen</a> gripper.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/KillingJacky/lerobot-robot-dummy">lerobot-robot-dummy</a></td>
<td>Plugin simulating a robot for recording without hardware. Useful for debugging !</td>
</tr>
</tbody>
</table>
## Teleoperators
### VR & Motion Controllers
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/SpesRobotics/lerobot-teleoperator-teleop">lerobot-teleoperator-teleop</a></td>
<td>Plugin turning a phone or VR headset into a teleoperator via <a href="https://immersiveweb.dev">WebXR</a>, wrapping the open-source <a href="https://github.com/SpesRobotics/teleop"><code>teleop</code></a> library.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/Jas000n/lerobot-teleoperator-spacemouse">lerobot-teleoperator-spacemouse</a></td>
<td>Plugin for the <a href="https://3dconnexion.com/">3Dconnexion SpaceMouse</a>, with inverse kinematics for SO-ARMS robots.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/Dream-Machines-Robotics/vr-teleop-kit">vr-teleop-kit</a></td>
<td>Plugin teleoperating arms from a <a href="https://www.meta.com/quest/">Meta Quest</a> (WebXR), relying on URDF descriptions for inverse kinematics.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/xensedyl/lerobot-teleoperator-pico4">lerobot-teleoperator-pico4</a></td>
<td>Plugin for the <a href="https://www.picoxr.com/">PICO 4</a> VR headset, with a companion controller-free <a href="https://github.com/xensedyl/lerobot-teleoperator-pico4-hand">hand-tracking variant</a>.</td>
</tr>
</tbody>
</table>
### Leader Arms
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/F-Fer/lerobot_ur5e_gello">lerobot_teleoperator_gello</a></td>
<td>Plugin for the 7 degrees of freedom <a href="https://wuphilipp.github.io/gello_site/">GELLO</a> teleoperator.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/uynitsuj/lerobot_teleoperator_yamactiveleader">lerobot_teleoperator_yamactiveleader</a></td>
<td>Plugin for the active YAM teleoperator from <a href="https://i2rt.com/">I2RT</a>, a bilateral force-feedback arm.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/charlie8612/lerobot_teleoperator_omy">lerobot_teleoperator_omy</a></td>
<td>Plugin for the OMY-L100 6 degrees of freedom teleoperator from <a href="https://www.robotis.com/">ROBOTIS</a>.</td>
</tr>
<tr style="border:0">
<td><a href="https://pypi.org/project/lerobot-teleoperator-pipermate/">lerobot-teleoperator-pipermate</a></td>
<td>Plugin for the PiperMate teleoperator (<a href="https://fashionstar.com.hk/">FashionStar</a> UART servos), driving the <a href="https://global.agilex.ai/">AgileX Piper</a> arm.</td>
</tr>
</tbody>
</table>
### Haptic Devices
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/chohh7391/lerobot_teleoperator_inverse3">lerobot_teleoperator_inverse3</a></td>
<td>Plugin for the <a href="https://www.haply.co/">Haply Inverse3</a> haptic device, adding force-feedback teleoperation.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/hzhz112/lerobot_teleoperator_omega7">lerobot_teleoperator_omega7</a></td>
<td>Plugin for the <a href="https://www.forcedimension.com/">Force Dimension omega.7</a> haptic device, adding force-feedback teleoperation.</td>
</tr>
</tbody>
</table>
### Networked & Remote
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://pypi.org/project/lerobot-teleoperator-livekit/">lerobot-teleoperator-livekit</a></td>
<td>Plugin receiving teleoperation commands over a <a href="https://livekit.io/">LiveKit</a> Portal (WebRTC) for remote control.</td>
</tr>
</tbody>
</table>
## Full Kits (Robot + Teleoperator)
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/villekuosmanen/lerobot-arx5">lerobot-arx5</a></td>
<td>Plugin for the <a href="https://www.arx-x.com/">ARX5</a> arm: <a href="https://pypi.org/project/lerobot-robot-arx5/"><code>lerobot-arx5</code></a> robot arm with its <a href="https://pypi.org/project/lerobot-teleoperator-arx5/"><code>lerobot-teleoperator-arx5</code></a> teleoperator arm.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/robertorobotics/Nextis-AIRA-3D">Nextis-AIRA-3D</a></td>
<td>Plugin for the 7 degrees of freedom arm from <a href="https://www.nextis.tech">Nextis</a>: robot arm <code>aira_follower</code> and teleoperator arm <code>aira_leader</code>.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/pravsels/lerobot_yam">lerobot_yam</a></td>
<td>Plugin suite for the YAM arm from <a href="https://i2rt.com/">I2RT</a>: robot arm <code>yam_follower</code> and teleoperator arm <code>yam_leader</code>.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/robot-learning-co/trlc-dk1">trlc-dk1</a></td>
<td>Plugin for the development kit from <a href="https://www.robot-learning.co/">The Robot Learning Company</a>: single and bimanual arms follower/teleoperator types.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/hexfellow/hex_lerobot_drivers">hex_lerobot_drivers</a></td>
<td>Plugin suite for <a href="https://hexfellow.com/">HEXFELLOW</a> devices: robots, teleoperators, and cameras (see <a href="./third_party_sensors">Cameras &amp; Sensors</a>).</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/Hiwonder-official/lerobot-robot-nexarm-follower">lerobot-robot-nexarm-follower</a></td>
<td>Plugin for the NexArm from <a href="https://www.hiwonder.com/">Hiwonder</a>: the <a href="https://github.com/Hiwonder-official/lerobot-robot-nexarm-follower">robot arm</a> and its matching <a href="https://github.com/Hiwonder-official/lerobot-teleoperator-nexarm-leader">teleoperator arm</a>.</td>
</tr>
</tbody>
</table>
## ROS 2 Bridges
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/ngres/leros2">leros2</a></td>
<td>Plugin bridging ROS 2 topics and actions to LeRobot robots and teleoperators.</td>
</tr>
<tr style="border:0">
<td><a href="https://github.com/ROBOTIS-GIT/lerobot_robot_ros2_zenoh">lerobot_robot_ros2_zenoh</a></td>
<td>Plugin bridging ROS 2 robots to LeRobot over <a href="https://zenoh.io">Zenoh</a> pub/sub transport.</td>
</tr>
</tbody>
</table>
## Contributing
Built your own LeRobot hardware integration? The plugin system makes it straightforward to add new robots and teleoperators — check out the [Bring Your Own Hardware](./integrate_hardware) guide to get started, and share your project with the community!
-99
View File
@@ -1,99 +0,0 @@
# Third-Party Cameras & Sensors
The LeRobot ecosystem extends far beyond its natively supported cameras (OpenCV, Intel RealSense, ZMQ, Reachy 2). Thanks to LeRobot's plugin architecture, the community has built drop-in camera and sensor integrations — from depth cameras to vision-based tactile sensors. This page showcases community-maintained camera and sensor integrations you can use for teleoperation, data collection, and policy deployment.
> [!IMPORTANT]
> These projects are developed and maintained by third parties. Please refer to each repository for installation instructions, hardware requirements, and support.
Drop-in plugins are auto-discovered by package name: LeRobot imports any installed package prefixed with `lerobot_camera_`. Once installed, reference the camera `type` the plugin registers (see its README — it may differ from the package name) directly from any LeRobot command:
```bash
pip install lerobot_camera_<name>
lerobot-record \
--robot.type=so101_follower \
--robot.port=/dev/ttyACM0 \
--robot.cameras="{ front: {type: <name>, width: 640, height: 480, fps: 30} }" \
--dataset.repo_id=${HF_USER}/my-dataset \
--dataset.num_episodes=5
```
## Tactile Sensors
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/xensedyl/lerobot-camera-xense">lerobot-camera-xense</a></td>
<td>Plugin for <a href="https://www.xenserobotics.com/">Xense</a> vision-based tactile sensors, exposing rectified/difference images, depth, and 2D markers.</td>
</tr>
</tbody>
</table>
## Depth Cameras
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/hexfellow/hex_lerobot_drivers/tree/main/lerobot_camera_berxel">lerobot_camera_berxel</a></td>
<td>Plugin for the <a href="https://www.berxel.com/">Berxel</a> depth camera, part of the broader <a href="https://hexfellow.com/">HEXFELLOW</a> driver suite.</td>
</tr>
</tbody>
</table>
## Networked Cameras
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/F-Fer/lerobot_ur5e_gello">lerobot_camera_zmq</a></td>
<td>Plugin streaming <a href="https://www.stereolabs.com/">Stereolabs ZED</a> and USB camera frames from a Raspberry Pi over the network.</td>
</tr>
</tbody>
</table>
## Virtual Cameras
<!-- prettier-ignore -->
<table width="100%" style="display:table; width:100%; table-layout:fixed;">
<colgroup>
<col width="30%" />
<col width="70%" />
</colgroup>
<thead style="border:0">
<tr style="border:0"><th>Project</th><th>Description</th></tr>
</thead>
<tbody>
<tr style="border:0">
<td><a href="https://github.com/hexfellow/hex_lerobot_drivers/tree/main/lerobot_camera_dummy">lerobot_camera_dummy</a></td>
<td>Plugin simulating a camera for recording without hardware. Useful for debugging !</td>
</tr>
</tbody>
</table>
## Contributing
Built your own LeRobot camera or sensor integration? Package it as an installable `lerobot_camera_<name>` plugin and it will be auto-discovered by the LeRobot CLI — see the [Bring Your Own Hardware](./integrate_hardware) guide and the [Cameras](./cameras) reference to get started, then share your project with the community!
-12
View File
@@ -40,15 +40,3 @@ lerobot-eval \
```
However, in most cases, presence of an accelerator is detected automatically and `policy.device` parameter can be omitted from CLI commands.
## Mixed precision
Training precision is owned by `--accelerator.mixed_precision`, which accepts `no` (default) and `bf16`:
```bash
lerobot-train \
--policy.type=act \
--accelerator.mixed_precision=bf16 ...
```
`bf16` requires an accelerator that supports it.
+42 -1
View File
@@ -290,6 +290,47 @@ lerobot-rollout \
---
## Part 5: SONIC Whole-Body Control (Latent-Token Policies)
LeRobot now ports the **decoder** half of NVIDIA's [SONIC](https://github.com/NVIDIA/sonic) whole-body deploy stack as the `SonicWholeBodyController`. Instead of commanding joints, the policy outputs a 64-D **latent motion token** (`motion_token.{i}.pos`) each tick; the SONIC decoder maps that token plus recent proprioception history onto a residual action that is scaled and added to the standing pose, producing 50 Hz joint-position targets for all 29 DoF.
The controller loads its ONNX decoder and all deploy constants (PD gains, `default_angles`, `action_scale`, and the neutral token) from [`lerobot/sonic_decoder`](https://huggingface.co/lerobot/sonic_decoder). The repo also ships a distilled low-latency decoder, selectable with the controller's `policy_type="low_latency"` argument (defaults to `"default"`).
### Test in Simulation
Any token-output policy trained on SONIC motion tokens (e.g. [`nepyope/sonic_walk`](https://huggingface.co/nepyope/sonic_walk)) can drive it:
```bash
lerobot-rollout \
--strategy.type=base \
--policy.path=nepyope/sonic_walk \
--policy.device=cuda \
--robot.type=unitree_g1 \
--robot.is_simulation=true \
--robot.controller=SonicWholeBodyController \
--robot.cameras='{"ego_view": {"type": "zmq", "server_address": "localhost", "port": 5555, "camera_name": "head_camera", "width": 640, "height": 480, "fps": 30, "warmup_s": 15}}'
```
### Run on the Physical Robot
Start the robot server (see Part 2), then point the rollout at the robot instead of the sim:
```bash
lerobot-rollout \
--strategy.type=base \
--policy.path=nepyope/sonic_walk \
--policy.device=cuda \
--robot.type=unitree_g1 \
--robot.is_simulation=false \
--robot.robot_ip=<ROBOT_IP> \
--robot.controller=SonicWholeBodyController \
--robot.cameras='{"ego_view": {"type": "zmq", "server_address": "<ROBOT_IP>", "port": 5555, "camera_name": "head_camera", "width": 640, "height": 480, "fps": 30, "warmup_s": 5}}'
```
On connect, the robot eases every joint to the decoder's `default_angles` before the controller takes over, so the first policy commands don't snap from the connect-time pose.
---
## Additional Resources
- [Unitree SDK Documentation](https://github.com/unitreerobotics/unitree_sdk2_python)
@@ -300,4 +341,4 @@ lerobot-rollout \
---
_Last updated: March 2026_
_Last updated: July 2026_
+3 -51
View File
@@ -11,10 +11,9 @@ LeRobot provides several utilities for manipulating datasets:
3. **Merge Datasets** - Combine multiple datasets into one. The datasets must have identical features, and episodes are concatenated in the order specified in `repo_ids`
4. **Add Features** - Add new features to a dataset
5. **Remove Features** - Remove features from a dataset
6. **Modify Tasks** - Change the natural-language task descriptions associated with episodes
7. **Convert to Video** - Convert image-based datasets to video format for efficient storage (RGB and depth cameras are encoded with separate encoders)
8. **Re-encode Videos** - Re-encode an existing video dataset's RGB and/or depth streams with new encoder settings
9. **Show the Info of Datasets** - Show the summary of datasets information such as number of episode etc.
6. **Convert to Video** - Convert image-based datasets to video format for efficient storage (RGB and depth cameras are encoded with separate encoders)
7. **Re-encode Videos** - Re-encode an existing video dataset's RGB and/or depth streams with new encoder settings
8. **Show the Info of Datasets** - Show the summary of datasets information such as number of episode etc.
The core implementation is in `lerobot.datasets.dataset_tools`.
An example script detailing how to use the tools API is available in `examples/dataset/use_dataset_tools.py`.
@@ -90,53 +89,6 @@ lerobot-edit-dataset \
--operation.feature_names "['observation.images.top']"
```
#### Modify Tasks
Change the natural-language task descriptions attached to episodes. This is useful for fixing typos, standardizing wording, or re-labeling episodes.
> [!WARNING]
> `modify_tasks` modifies the dataset **in-place** (updating `meta/tasks.parquet`, the `task_index` column in the data files, the `tasks` column in the episode metadata, and `total_tasks` in `meta/info.json`). The `--new_repo_id` and `--new_root` parameters are ignored for this operation.
```bash
# Set a single task for all episodes
lerobot-edit-dataset \
--repo_id lerobot/pusht \
--operation.type modify_tasks \
--operation.new_task "Pick up the cube and place it"
# Set different tasks for specific episodes
lerobot-edit-dataset \
--repo_id lerobot/pusht \
--operation.type modify_tasks \
--operation.episode_tasks '{"0": "Task A", "1": "Task B", "2": "Task A"}'
# Replace existing task strings wherever they appear
lerobot-edit-dataset \
--repo_id lerobot/pusht \
--operation.type modify_tasks \
--operation.task_replacements '{"Pick up the red cube": "Lift the red cube"}'
# Combine modes in a single run
lerobot-edit-dataset \
--repo_id lerobot/pusht \
--operation.type modify_tasks \
--operation.new_task "Default task" \
--operation.task_replacements '{"Pick up the red cube": "Lift the red cube"}' \
--operation.episode_tasks '{"5": "Special task for episode 5"}'
```
**Parameters:**
- `new_task`: A single task string used as the default for episodes not otherwise covered.
- `episode_tasks`: Mapping from episode index to task string.
- `task_replacements`: Mapping from existing task strings to their replacements, applied to episodes whose current task matches a key. Every key must be an existing task in the dataset.
The modes can be combined in a single run. Per episode, the task is resolved with the following precedence:
`episode_tasks` > `task_replacements` > `new_task` > original task
At least one of `new_task`, `episode_tasks`, or `task_replacements` must be specified. An episode that ends up with no task raises an error.
#### Convert to Video
Convert an image-based dataset to video format, creating a new LeRobotDataset where images are stored as videos. This is useful for reducing storage requirements and improving data loading performance. The new dataset will have the exact same structure as the original, but with images encoded as MP4 videos in the proper LeRobot format.
+10 -10
View File
@@ -49,16 +49,16 @@ lerobot-record \
All flags below are prefixed with `--dataset.rgb_encoder.` on the CLI.
| Parameter | Type | Default | Description |
| --------------- | ---------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `vcodec` | `str` | `"libsvtav1"` | Video codec name. `"auto"` picks the first available hardware encoder from a fixed preference list, falling back to `libsvtav1`. |
| `pix_fmt` | `str` | `"yuv420p"` | Output pixel format. Must be supported by the chosen codec in your FFmpeg build. |
| `g` | `int` | `2` | GOP size — a keyframe every `g` frames. Emitted as FFmpeg option `g`. |
| `crf` | `int` or `float` | `30` | Abstract quality value, mapped per codec (see the [mapping](https://github.com/huggingface/lerobot/blob/main/src/lerobot/configs/video.py#L197)). Lower → higher quality / larger output where the mapping is monotone. |
| `preset` | `int` or `str` | `12` \* | Encoder speed preset; meaning depends on the codec. <br/>\* When unset and `vcodec=libsvtav1`, LeRobot defaults to `12`. |
| `fast_decode` | `int` | `0` | `libsvtav1`: `02`, passed via `svtav1-params`. <br/>`h264` / `hevc` (software): if `>0`, sets `tune=fastdecode`. <br/>Other codecs: usually unused. |
| `video_backend` | `str` | `"pyav"` | Only `"pyav"` is currently implemented for video encoding. |
| `extra_options` | `dict` | `{}` | Extra FFmpeg or codec specific options merged after the structured fields above. Cannot override keys already set by those fields. |
| Parameter | Type | Default | Description |
| --------------- | ---------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `vcodec` | `str` | `"libsvtav1"` | Video codec name. `"auto"` picks the first available hardware encoder from a fixed preference list, falling back to `libsvtav1`. |
| `pix_fmt` | `str` | `"yuv420p"` | Output pixel format. Must be supported by the chosen codec in your FFmpeg build. |
| `g` | `int` | `2` | GOP size — a keyframe every `g` frames. Emitted as FFmpeg option `g`. |
| `crf` | `int` or `float` | `30` | Abstract quality value, mapped per codec (see the [mapping](#mapping-videoencoderconfig--ffmpeg-options) below). Lower → higher quality / larger output where the mapping is monotone. |
| `preset` | `int` or `str` | `12` \* | Encoder speed preset; meaning depends on the codec. <br/>\* When unset and `vcodec=libsvtav1`, LeRobot defaults to `12`. |
| `fast_decode` | `int` | `0` | `libsvtav1`: `02`, passed via `svtav1-params`. <br/>`h264` / `hevc` (software): if `>0`, sets `tune=fastdecode`. <br/>Other codecs: usually unused. |
| `video_backend` | `str` | `"pyav"` | Only `"pyav"` is currently implemented for video encoding. |
| `extra_options` | `dict` | `{}` | Extra FFmpeg or codec specific options merged after the structured fields above. Cannot override keys already set by those fields. |
---
-287
View File
@@ -1,287 +0,0 @@
# Writing docstrings
LeRobot's API reference is generated directly from the docstrings in `src/lerobot/`. A docstring is not a
comment — it is the published documentation for that object, and the format below is what the renderer and
the CI checks parse.
This page is the contract. If you are adding or editing anything public in `src/lerobot/`, follow it.
> [!IMPORTANT]
> **An undocumented public method is an invisible one.** `[[autodoc]]` silently skips members that have no
> docstring — no warning, no error, it simply does not appear on the rendered page. Coverage and
> API-reference completeness are the same problem.
## The format in one example
Google section headers, Hugging Face type formatting. Both, not one or the other.
````python
def send_action(self, action: RobotAction, rate_hz: float = 30.0) -> RobotAction:
"""Command the robot to move to a target joint configuration.
Values are clipped by the configured maximum relative target before reaching the motors, so the
returned action may differ from the requested one.
Args:
action (`dict[str, float]`):
Target values keyed by motor name, e.g. `{"shoulder_pan.pos": 0.0}`. Keys must match the
robot's action features.
rate_hz (`float`, *optional*, defaults to `30.0`):
Control loop frequency.
Returns:
`dict[str, float]`: The action actually written to the motors after safety clipping.
Raises:
DeviceNotConnectedError: If the robot has not been connected.
Example:
```python
>>> from lerobot.robots.so_follower import SO101Follower, SO101FollowerConfig
>>> robot = SO101Follower(SO101FollowerConfig(port="/dev/ttyACM0")) # doctest: +SKIP
>>> robot.connect() # doctest: +SKIP
>>> robot.send_action({"shoulder_pan.pos": 0.0}) # doctest: +SKIP
```
"""
````
Cross-references are omitted from the examples on this page — see [Cross-references](#cross-references) for
their syntax and why they cannot be shown inside a code block.
## Rules
### Sections
`Args:` · `Returns:` · `Raises:` · `Yields:` · `Example:` · `Note:`
In that order. No other section headers. A one-line summary comes first, then an optional free-form
description, then the sections.
### The `Args:` line is machine-parsed
```
name (`type`, *optional*, defaults to `X`):
Description, indented on its own line.
```
The `*optional*, defaults to` clause is **checked against the real signature default** by
`make check-docstrings`. It is not decorative — if you write a default that has drifted from the code, CI
fails. Omit the clause entirely for required parameters:
```python
Args:
port (`str`):
Serial port the arm is connected to, e.g. `/dev/ttyACM0`.
max_relative_target (`float | dict[str, float]`, *optional*):
Caps the magnitude of the relative positional target vector. `None` disables clipping.
use_degrees (`bool`, *optional*, defaults to `True`):
Keep `True` for backward compatibility with existing policies and datasets.
```
Types go in backticks. Use `*optional*` with no `defaults to` when the default is `None` or is otherwise not
worth restating.
### `Returns:` is type-first
One indented line, type first, then a colon, then the description:
```python
Returns:
`dict[str, float]`: The action actually written to the motors after safety clipping.
```
`Yields:` takes the same shape.
### `**Attributes**:`, never `Attributes:`
doc-builder parses a bare `Attributes:` as a **synonym for `Parameters:`**, so your attributes get rendered
as constructor arguments. This is silent and wrong. Whenever the attributes differ from the constructor
parameters, use the bold form with a `--` separator:
```python
class Robot(abc.ABC):
"""The base abstract class for all LeRobot-compatible robots.
**Attributes**:
- **config_class** (`type[RobotConfig]`) -- The expected configuration class for this robot.
- **name** (`str`) -- The unique robot name used to identify this robot type.
"""
```
Note `--`, not `:`.
### Cross-references
Use doc-builder's bracket syntax: a square-bracketed backtick-quoted path. **Sphinx roles (`:pymeth:`,
`:pyattr:`) are not supported** and render as literal text on the page.
| Want | Write |
| ---------------------------- | ----------------------------------- |
| Class in the main package | &#91;`Robot`&#93; |
| Method, show the full path | &#91;`Robot.connect`&#93; |
| Method, show the bare name | &#91;`~Robot.connect`&#93; |
| Nested path | &#91;`~robots.Robot.connect`&#93; |
| Object in another HF library | &#91;`~accelerate.Accelerator`&#93; |
The `~` strips the path from the **link text only**; the link still resolves to the full path.
> [!NOTE]
> doc-builder resolves this syntax everywhere in a page — including inside fenced code blocks. That is why
> the docstring examples on this page use plain prose instead of cross-references: a code block containing
> one would render the resolved link rather than the syntax you need to type. In your own docstrings, use
> cross-references freely; this restriction only affects documentation _about_ the syntax.
### Callouts
Use GitHub-style blockquotes:
```markdown
> [!TIP]
> Call this once at startup — it takes about two seconds.
> [!WARNING]
> Torque is disabled on disconnect. The arm will drop if it is holding a load.
```
The `<Tip>` component is legacy per doc-builder; don't add new ones.
### Examples must be fenced
An example lives inside a fenced ` ```python ` block containing `>>> `. The fence is what makes it render
as a code block, and it is what the doctest preprocessor's regex looks for:
````python
Example:
```python
>>> from lerobot.robots.so_follower import SO101FollowerConfig
>>> cfg = SO101FollowerConfig(port="/dev/ttyACM0")
>>> cfg.use_degrees
True
```
````
> [!WARNING]
> An unfenced `>>>` is still collected — doctest finds prompts anywhere in a docstring. What you lose is the
> rendering, so it shows up as a wall of prose on the page. Every example needs the fence.
Every example either executes in CI or carries `# doctest: +SKIP`. Anything that touches hardware, a GPU, or
downloads from the Hub gets `+SKIP`:
````python
Example:
```python
>>> robot.connect() # doctest: +SKIP
>>> policy = ACTPolicy.from_pretrained("lerobot/act_aloha_sim_transfer_cube_human") # doctest: +SKIP
```
````
Add files containing runnable examples to `utils/documentation_tests.txt`.
Put examples on the three to five genuine entry points of a module. Examples on trivial accessors are noise.
## Three patterns you will hit constantly
### Config dataclasses
Configuration fields are historically documented with `#` comments above each field. **doc-builder cannot
see inline comments** — such a class renders with every field listed and not a single description. Move them
into an `Args:` block on the class docstring:
```python
@dataclass
class SOFollowerConfig:
"""Configuration for SO-family follower arms.
Args:
port (`str`):
Serial port the arm is connected to, e.g. `/dev/ttyACM0`.
max_relative_target (`float | dict[str, float]`, *optional*):
Caps the magnitude of the relative positional target vector. A scalar applies to all motors;
a dict maps motor name to a per-motor cap. `None` disables clipping.
use_degrees (`bool`, *optional*, defaults to `True`):
Keep `True` for backward compatibility with existing policies and datasets.
"""
port: str
max_relative_target: float | dict[str, float] | None = None
use_degrees: bool = True
```
> [!IMPORTANT]
> **doc-builder does not inherit docstrings from base classes.** LeRobot's registered config classes are
> often thin multiple-inheritance shims:
>
> ```python
> @RobotConfig.register_subclass("so101_follower")
> @dataclass
> class SOFollowerRobotConfig(RobotConfig, SOFollowerConfig):
> pass
> ```
>
> That class renders **every** field — including the ones it inherits — with no descriptions at all, no
> matter how well the bases are documented. The `Args:` block must live on the concrete class that
> `[[autodoc]]` names, and it must cover inherited fields too.
### Base class, then concrete subclass
The abstract base carries the canonical contract. Subclasses document only what deviates — port semantics,
calibration quirks, motor layout, supported feature keys. Do not copy the base contract into every subclass.
`Robot`, `Teleoperator`, `Camera`, `MotorsBus`, `ProcessorStep`, and `PreTrainedPolicy` all follow this
shape.
### Module-level aliases
Several public names are aliases rather than distinct classes:
```python
SO100FollowerConfig = SOFollowerRobotConfig
SO101FollowerConfig = SOFollowerRobotConfig
```
`[[autodoc]]` resolves the alias and renders the **canonical** class name, so a `## SO101FollowerConfig`
heading will show `class lerobot.robots.so_follower.SOFollowerRobotConfig` in the body. Document the
canonical class once, and mention the aliases in the page's prose rather than giving each alias its own
autodoc block.
## What not to document
- **Private members.** Anything starting with `_` is not part of the public API.
- **The type annotation restated as prose.** `port (`str`): A string.` adds nothing. Say what it is for.
- **Vendored upstream code.** `src/lerobot/policies/molmoact2/molmoact2_hf_model/` is vendored from
`transformers` and already carries upstream-style docstrings. Leave it alone — restyling it only creates
conflicts on the next sync. It is excluded from the API reference and from the docstring checks.
## How this is enforced
| Check | What it catches |
| ------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `make check-docstrings` | An `Args:` entry that doesn't match the signature; a documented default that has drifted from the real one |
| `make doctest` | Examples that no longer run |
| `make check-doctest-list` | Stale or unsorted entries in `utils/documentation_tests.txt` |
| `ruff` (`D` rules) | Google-convention style violations |
| `interrogate` | Docstring coverage falling below the current threshold |
| doc-builder | A `[[autodoc]]` path that points at something that doesn't exist — this breaks the docs build |
Run them together before opening a PR:
```bash
make check-docstrings && make doctest && pre-commit run --all-files
```
Then render the page and actually look at it:
```bash
doc-builder build lerobot docs/source/ --build_dir /tmp/doc-build
```
## Checklist
- [ ] Every public member you touched has a docstring.
- [ ] Every `Args:` entry matches the signature, including the `*optional*, defaults to` clause.
- [ ] `Returns:` is type-first on one indented line.
- [ ] No bare `Attributes:` — use `**Attributes**:` with `--` separators.
- [ ] No Sphinx roles — cross-references use &#91;`~module.Class.method`&#93;.
- [ ] Examples are inside a fenced ` ```python ` block, and either run in CI or carry `# doctest: +SKIP`.
- [ ] Config dataclass fields are in an `Args:` block on the concrete class, not `#` comments.
- [ ] The rendered page has been eyeballed.
+1 -1
View File
@@ -44,7 +44,6 @@ from typing import Protocol
import numpy as np
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.processor import (
RobotProcessorPipeline,
@@ -57,6 +56,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
EEBoundsAndSafety,
InverseKinematicsEEToJoints,
)
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.constants import HF_LEROBOT_CALIBRATION, HF_LEROBOT_HOME, TELEOPERATORS
from lerobot.utils.robot_utils import precise_sleep
@@ -38,7 +38,7 @@ from typing import TYPE_CHECKING
import numpy as np
from lerobot.lerobot_types import RobotAction
from lerobot.types import RobotAction
from .base import _GRIPPER_MOTOR_SCALE, IsaacTeleopTeleoperator, _isaacteleop_available
from .config_isaac_teleop import SO101LeaderArmConfig
@@ -32,7 +32,7 @@ from typing import TYPE_CHECKING, Any
import numpy as np
from lerobot.lerobot_types import RobotAction
from lerobot.types import RobotAction
from .base import IsaacTeleopTeleoperator, _isaacteleop_available
from .config_isaac_teleop import XRControllerConfig
@@ -26,8 +26,8 @@ from __future__ import annotations
from dataclasses import dataclass
from lerobot.configs.types import FeatureType, PipelineFeatureType, PolicyFeature
from lerobot.lerobot_types import RobotAction
from lerobot.processor import ProcessorStepRegistry, RobotActionProcessorStep
from lerobot.types import RobotAction
from lerobot.utils.rotation import Rotation
from .base import _GRIPPER_MOTOR_SCALE
+1 -1
View File
@@ -21,7 +21,6 @@ from lerobot.cameras.opencv import OpenCVCameraConfig
from lerobot.common.control_utils import predict_action
from lerobot.configs import FeatureType, PolicyFeature
from lerobot.datasets import LeRobotDataset, aggregate_pipeline_dataset_features, create_initial_features
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.policies import make_pre_post_processors
from lerobot.policies.act import ACTPolicy
@@ -39,6 +38,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
ForwardKinematicsJointsToEE,
InverseKinematicsEEToJoints,
)
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.constants import ACTION, OBS_STR
from lerobot.utils.feature_utils import build_dataset_frame, combine_feature_dicts
from lerobot.utils.keyboard_input import init_keyboard_listener
+1 -1
View File
@@ -16,7 +16,6 @@
from lerobot.cameras.opencv import OpenCVCameraConfig
from lerobot.datasets import LeRobotDataset, aggregate_pipeline_dataset_features, create_initial_features
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.processor import (
RobotProcessorPipeline,
@@ -37,6 +36,7 @@ from lerobot.scripts.lerobot_record import record_loop
from lerobot.teleoperators.phone import Phone, PhoneConfig
from lerobot.teleoperators.phone.config_phone import PhoneOS
from lerobot.teleoperators.phone.phone_processor import MapPhoneActionToRobotAction
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.feature_utils import combine_feature_dicts
from lerobot.utils.keyboard_input import init_keyboard_listener
from lerobot.utils.utils import log_say
+1 -1
View File
@@ -17,7 +17,6 @@
import time
from lerobot.datasets import LeRobotDataset
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.processor import (
RobotProcessorPipeline,
@@ -28,6 +27,7 @@ from lerobot.robots.so_follower import SO100Follower, SO100FollowerConfig
from lerobot.robots.so_follower.robot_kinematic_processor import (
InverseKinematicsEEToJoints,
)
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.constants import ACTION
from lerobot.utils.robot_utils import precise_sleep
from lerobot.utils.utils import log_say
+1 -1
View File
@@ -27,7 +27,6 @@ Highlight, or DAgger via ``lerobot-rollout --strategy.type=...``.
from lerobot.cameras.opencv import OpenCVCameraConfig
from lerobot.configs import PreTrainedConfig
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.processor import (
RobotProcessorPipeline,
@@ -44,6 +43,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
from lerobot.rollout import BaseStrategyConfig, RolloutConfig, build_rollout_context
from lerobot.rollout.inference import SyncInferenceConfig
from lerobot.rollout.strategies import BaseStrategy
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.process import ProcessSignalHandler
from lerobot.utils.utils import init_logging
+1 -1
View File
@@ -15,7 +15,6 @@
import time
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.processor import (
RobotProcessorPipeline,
@@ -32,6 +31,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
from lerobot.teleoperators.phone import Phone, PhoneConfig
from lerobot.teleoperators.phone.config_phone import PhoneOS
from lerobot.teleoperators.phone.phone_processor import MapPhoneActionToRobotAction
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.robot_utils import precise_sleep
from lerobot.utils.visualization_utils import init_rerun, log_rerun_data
+1 -1
View File
@@ -417,7 +417,7 @@ class RTCEvaluator:
def run_evaluation(self):
"""Run evaluation on two random dataset samples using three separate policies.
Note: Policies are deinitialized after each step to free memory. Large models
Note: Policies are deinitalized after each step to free memory. Large models
(e.g., VLA models with billions of parameters) cannot fit three instances in
memory simultaneously. By deleting and garbage collecting after each step,
we ensure only one policy is loaded at a time.
+1 -1
View File
@@ -21,7 +21,6 @@ from lerobot.cameras.opencv import OpenCVCameraConfig
from lerobot.common.control_utils import predict_action
from lerobot.configs import FeatureType, PolicyFeature
from lerobot.datasets import LeRobotDataset, aggregate_pipeline_dataset_features, create_initial_features
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.policies import make_pre_post_processors
from lerobot.policies.act import ACTPolicy
@@ -39,6 +38,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
ForwardKinematicsJointsToEE,
InverseKinematicsEEToJoints,
)
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.constants import ACTION, OBS_STR
from lerobot.utils.feature_utils import build_dataset_frame, combine_feature_dicts
from lerobot.utils.keyboard_input import init_keyboard_listener
+1 -1
View File
@@ -17,7 +17,6 @@
from lerobot.cameras.opencv import OpenCVCameraConfig
from lerobot.datasets import LeRobotDataset, aggregate_pipeline_dataset_features, create_initial_features
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.processor import (
RobotProcessorPipeline,
@@ -34,6 +33,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
)
from lerobot.scripts.lerobot_record import record_loop
from lerobot.teleoperators.so_leader import SO100Leader, SO100LeaderConfig
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.feature_utils import combine_feature_dicts
from lerobot.utils.keyboard_input import init_keyboard_listener
from lerobot.utils.utils import log_say
+1 -1
View File
@@ -18,7 +18,6 @@
import time
from lerobot.datasets import LeRobotDataset
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.processor import (
RobotProcessorPipeline,
@@ -29,6 +28,7 @@ from lerobot.robots.so_follower import SO100Follower, SO100FollowerConfig
from lerobot.robots.so_follower.robot_kinematic_processor import (
InverseKinematicsEEToJoints,
)
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.constants import ACTION
from lerobot.utils.robot_utils import precise_sleep
from lerobot.utils.utils import log_say
+1 -1
View File
@@ -25,7 +25,6 @@ forward/inverse kinematics.
from lerobot.cameras.opencv import OpenCVCameraConfig
from lerobot.configs import PreTrainedConfig
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.processor import (
RobotProcessorPipeline,
@@ -42,6 +41,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
from lerobot.rollout import BaseStrategyConfig, RolloutConfig, build_rollout_context
from lerobot.rollout.inference import SyncInferenceConfig
from lerobot.rollout.strategies import BaseStrategy
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.process import ProcessSignalHandler
from lerobot.utils.utils import init_logging
+1 -1
View File
@@ -16,7 +16,6 @@
import time
from lerobot.lerobot_types import RobotAction, RobotObservation
from lerobot.model.kinematics import RobotKinematics
from lerobot.processor import (
RobotProcessorPipeline,
@@ -31,6 +30,7 @@ from lerobot.robots.so_follower.robot_kinematic_processor import (
InverseKinematicsEEToJoints,
)
from lerobot.teleoperators.so_leader import SO100Leader, SO100LeaderConfig
from lerobot.types import RobotAction, RobotObservation
from lerobot.utils.robot_utils import precise_sleep
from lerobot.utils.visualization_utils import init_rerun, log_rerun_data
+23 -86
View File
@@ -25,7 +25,7 @@ discord = "https://discord.gg/s3KuuzsPFb"
[project]
name = "lerobot"
version = "0.6.2"
version = "0.6.1"
description = "🤗 LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch"
dynamic = ["readme"]
license = { text = "Apache-2.0" }
@@ -67,8 +67,8 @@ dependencies = [
"einops>=0.8.0,<0.9.0",
# Config & Hub
"draccus>=0.11.6,<0.12.0",
"huggingface-hub>=1.6.0,<2.0.0",
"draccus==0.10.0", # TODO: Relax version constraint
"huggingface-hub>=1.0.0,<2.0.0",
"requests>=2.32.0,<3.0.0",
# Environments
@@ -87,7 +87,7 @@ dependencies = [
# Build tools (required by opencv-python-headless on some platforms)
"cmake>=3.29.0.1,<4.2.0",
"setuptools>=71.0.0,<82.0.0", # torch 2.11 requires setuptools<82; a higher cap makes the resolver downgrade torch
"setuptools>=71.0.0,<81.0.0",
]
# Optional dependencies
@@ -95,7 +95,7 @@ dependencies = [
# ── Feature-scoped extras ──────────────────────────────────
dataset = [
"datasets>=4.8.0,<5.0.0",
"datasets>=4.7.0,<5.0.0",
"pandas>=2.0.0,<3.0.0", # NOTE: Transitive dependency of datasets
"pyarrow>=21.0.0,<30.0.0", # NOTE: Transitive dependency of datasets
"lerobot[av-dep]",
@@ -261,7 +261,7 @@ annotations = [
# Development
dev = ["pre-commit>=3.7.0,<5.0.0", "debugpy>=1.8.1,<1.9.0", "lerobot[grpcio-dep]", "grpcio-tools>=1.73.1,<2.0.0", "mypy>=1.19.1", "ruff>=0.14.1", "lerobot[notebook]"]
notebook = ["jupyter>=1.0.0,<2.0.0", "ipykernel>=6.0.0,<7.0.0"]
test = ["pytest>=8.1.0,<10.0.0", "pytest-timeout>=2.4.0,<3.0.0", "pytest-cov>=5.0.0,<8.0.0", "mock-serial>=0.0.1,<0.1.0 ; sys_platform != 'win32'"]
test = ["pytest>=8.1.0,<9.0.0", "pytest-timeout>=2.4.0,<3.0.0", "pytest-cov>=5.0.0,<8.0.0", "mock-serial>=0.0.1,<0.1.0 ; sys_platform != 'win32'"]
video_benchmark = ["scikit-image>=0.23.2,<0.26.0", "pandas>=2.2.2,<2.4.0"]
# Simulation
@@ -346,7 +346,6 @@ lerobot-record="lerobot.scripts.lerobot_record:main"
lerobot-replay="lerobot.scripts.lerobot_replay:main"
lerobot-setup-motors="lerobot.scripts.lerobot_setup_motors:main"
lerobot-teleoperate="lerobot.scripts.lerobot_teleoperate:main"
lerobot-convert-dcp="lerobot.scripts.lerobot_convert_dcp:main"
lerobot-eval="lerobot.scripts.lerobot_eval:main"
lerobot-train="lerobot.scripts.lerobot_train:main"
lerobot-train-tokenizer="lerobot.scripts.lerobot_train_tokenizer:main"
@@ -401,63 +400,19 @@ exclude = ["tests/artifacts/**/*.safetensors", "*_pb2.py", "*_pb2_grpc.py"]
# N: pep8-naming
# TODO: Uncomment rules when ready to use
select = [
"E", "W", "F", "I", "B", "C4", "T20", "N", "UP", "SIM", "D" #, "A", "S", "RUF"
"E", "W", "F", "I", "B", "C4", "T20", "N", "UP", "SIM" #, "A", "S", "D", "RUF"
]
ignore = [
"E501", # Line too long
"T201", # Print statement found
"T203", # Pprint statement found
"B008", # Perform function call in argument defaults
# D100/D104: module- and package-level docstrings. The API reference is generated from class and
# function docstrings; a banner at the top of every file and every __init__.py would not appear on any
# rendered page. Coverage of the things that do get rendered is enforced by interrogate instead.
"D100",
"D104",
]
[tool.ruff.lint.per-file-ignores]
"__init__.py" = ["F401", "F403", "E402", "D104"]
"__init__.py" = ["F401", "F403", "E402"]
# E402: conditional-import guards (TYPE_CHECKING / is_package_available) must precede the imports they protect
"src/lerobot/scripts/convert_dataset_v21_to_v30.py" = ["E402"]
# D (pydocstyle) is enabled globally, but only holds for code that has been converted to the docstring
# standard in docs/source/writing_docstrings.mdx. Every module below is still on the old style; each entry
# is deleted as that module is converted, and this block can be removed once it is empty.
#
# Not part of the API reference and not planned for conversion: tests, examples, benchmarks, templates,
# CI helper scripts and the packaging shim.
"tests/**" = ["D"]
"examples/**" = ["D"]
"benchmarks/**" = ["D"]
"scripts/**" = ["D"]
"setup.py" = ["D"]
"src/lerobot/templates/**" = ["D"]
# Vendored from transformers; keeps its upstream docstring style so syncs stay clean.
"src/lerobot/policies/molmoact2/molmoact2_hf_model/**" = ["D"]
# Awaiting conversion, one PR per module.
"src/lerobot/annotations/**" = ["D"]
"src/lerobot/async_inference/**" = ["D"]
"src/lerobot/common/**" = ["D"]
"src/lerobot/configs/**" = ["D"]
"src/lerobot/data_processing/**" = ["D"]
"src/lerobot/datasets/**" = ["D"]
"src/lerobot/distributed/**" = ["D"]
"src/lerobot/envs/**" = ["D"]
"src/lerobot/jobs/**" = ["D"]
"src/lerobot/model/**" = ["D"]
"src/lerobot/motors/**" = ["D"]
"src/lerobot/optim/**" = ["D"]
"src/lerobot/policies/**" = ["D"]
"src/lerobot/processor/**" = ["D"]
"src/lerobot/rewards/**" = ["D"]
"src/lerobot/rl/**" = ["D"]
"src/lerobot/rollout/**" = ["D"]
"src/lerobot/scripts/**" = ["D"]
"src/lerobot/teleoperators/**" = ["D"]
"src/lerobot/transforms/**" = ["D"]
"src/lerobot/transport/**" = ["D"]
"src/lerobot/utils/**" = ["D"]
"src/lerobot/lerobot_types.py" = ["D"]
[tool.ruff.lint.isort]
combine-as-imports = true
known-first-party = ["lerobot"]
@@ -501,34 +456,25 @@ default.extend-ignore-identifiers-re = [
"seperated_timestep",
]
# Docstring coverage gate. `fail-under` is a RATCHET, not a target: it is set just below the currently
# measured coverage so it passes today, and is raised in the same PR that documents a module. Never set it
# to a value that fails on main. The destination is 100; see docs/source/writing_docstrings.mdx.
[tool.interrogate]
ignore-init-module = true
ignore-init-method = true
ignore-nested-functions = false
ignore-magic = false
ignore-semiprivate = false
ignore-private = false
ignore-property-decorators = false
ignore-module = false
ignore-setters = false
fail-under = 55
output-format = "term-missing"
color = true
paths = ["src/lerobot"]
exclude = ["src/lerobot/policies/molmoact2/molmoact2_hf_model"]
# TODO: Uncomment when ready to use
# [tool.interrogate]
# ignore-init-module = true
# ignore-init-method = true
# ignore-nested-functions = false
# ignore-magic = false
# ignore-semiprivate = false
# ignore-private = false
# ignore-property-decorators = false
# ignore-module = false
# ignore-setters = false
# fail-under = 80
# output-format = "term-missing"
# color = true
# paths = ["src/lerobot"]
# TODO: Enable mypy gradually module by module across multiple PRs
# Uncomment [tool.mypy] first, then uncomment individual module overrides as they get proper type annotations
[tool.pytest.ini_options]
markers = [
"multigpu: distributed tests needing 2-4 GPUs (CI: docker_publish.yml lane)",
"multigpu_heavy: 8-GPU sweeps and soak tests; never run in CI",
]
[tool.mypy]
python_version = "3.12"
ignore_missing_imports = true
@@ -575,15 +521,6 @@ disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
[[tool.mypy.overrides]]
module = "lerobot.distributed.*"
ignore_errors = false
# extra strictness for the distributed engine
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
[[tool.mypy.overrides]]
module = "lerobot.optim.*"
ignore_errors = false
+2 -1
View File
@@ -14,7 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""LeRobot -- PyTorch library for real-world robotics.
"""
LeRobot -- PyTorch library for real-world robotics.
Provides datasets, pretrained policies, and tools for training, evaluation,
data collection, and robot control. Integrates with Hugging Face Hub for
+1 -1
View File
@@ -13,7 +13,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""To enable `lerobot.__version__`."""
"""To enable `lerobot.__version__`"""
from importlib.metadata import PackageNotFoundError, version
+1 -1
View File
@@ -38,7 +38,6 @@ import draccus
import grpc
import torch
from lerobot.lerobot_types import PolicyAction
from lerobot.policies import get_policy_class, make_pre_post_processors
from lerobot.processor import PolicyProcessorPipeline
from lerobot.transport import (
@@ -46,6 +45,7 @@ from lerobot.transport import (
services_pb2_grpc, # type: ignore
)
from lerobot.transport.utils import receive_bytes_in_chunks
from lerobot.types import PolicyAction
from .configs import PolicyServerConfig
from .constants import SUPPORTED_POLICIES
+10 -11
View File
@@ -33,10 +33,10 @@ class Camera(abc.ABC):
- Connection/disconnection
- Frame capture (sync/async/latest)
**Attributes**:
- **fps** (`int | None`) -- Configured frames per second.
- **width** (`int | None`) -- Frame width in pixels.
- **height** (`int | None`) -- Frame height in pixels.
Attributes:
fps (int | None): Configured frames per second
width (int | None): Frame width in pixels
height (int | None): Frame height in pixels
"""
def __init__(self, config: CameraConfig):
@@ -50,23 +50,23 @@ class Camera(abc.ABC):
self.height: int | None = config.height
def __enter__(self):
"""Context manager entry.
"""
Context manager entry.
Automatically connects to the camera.
"""
self.connect()
return self
def __exit__(self, exc_type, exc_value, traceback) -> None:
"""Context manager exit.
"""
Context manager exit.
Automatically disconnects, ensuring resources are released even on error.
"""
self.disconnect()
def __del__(self) -> None:
"""Destructor safety net.
"""
Destructor safety net.
Attempts to disconnect if the object is garbage collected without cleanup.
"""
try:
@@ -90,7 +90,6 @@ class Camera(abc.ABC):
@abc.abstractmethod
def find_cameras() -> list[dict[str, Any]]:
"""Detects available cameras connected to the system.
Returns:
List[Dict[str, Any]]: A list of dictionaries,
where each dictionary contains information about a detected camera.
-66
View File
@@ -22,36 +22,15 @@ import draccus # type: ignore # TODO: add type stubs for draccus
class ColorMode(str, Enum):
"""Color channel order for frames returned by a camera.
**Attributes**:
- **RGB** -- Red-green-blue channel order.
- **BGR** -- Blue-green-red channel order, OpenCV's native order.
"""
RGB = "rgb"
BGR = "bgr"
@classmethod
def _missing_(cls, value: object) -> None:
"""Reject a value that is not a valid `ColorMode`.
Raises:
ValueError: Always, naming the invalid value and the valid choices.
"""
raise ValueError(f"`color_mode` is expected to be in {list(cls)}, but {value} is provided.")
class Cv2Rotation(int, Enum):
"""Clockwise rotation to apply to a frame after capture, in degrees.
**Attributes**:
- **NO_ROTATION** -- No rotation.
- **ROTATE_90** -- Rotate 90° clockwise.
- **ROTATE_180** -- Rotate 180°.
- **ROTATE_270** -- Rotate 270° clockwise (90° counter-clockwise).
"""
NO_ROTATION = 0
ROTATE_90 = 90
ROTATE_180 = 180
@@ -59,31 +38,11 @@ class Cv2Rotation(int, Enum):
@classmethod
def _missing_(cls, value: object) -> None:
"""Reject a value that is not a valid `Cv2Rotation`.
Raises:
ValueError: Always, naming the invalid value and the valid choices.
"""
raise ValueError(f"`rotation` is expected to be in {list(cls)}, but {value} is provided.")
# Subset from https://docs.opencv.org/3.4/d4/d15/group__videoio__flags__base.html
class Cv2Backends(int, Enum):
"""OpenCV capture backend to request when opening a device.
See the [OpenCV `VideoCaptureAPIs` reference](https://docs.opencv.org/3.4/d4/d15/group__videoio__flags__base.html)
for the full list this is a subset of.
**Attributes**:
- **ANY** -- Let OpenCV auto-detect the backend.
- **V4L2** -- Video4Linux2, the usual choice on Linux.
- **DSHOW** -- DirectShow, a Windows backend.
- **PVAPI** -- PvAPI, for Prosilica GigE cameras.
- **ANDROID** -- Android's native camera API.
- **AVFOUNDATION** -- AVFoundation, the usual choice on macOS.
- **MSMF** -- Microsoft Media Foundation, a Windows backend.
"""
ANY = 0
V4L2 = 200
DSHOW = 700
@@ -94,40 +53,15 @@ class Cv2Backends(int, Enum):
@classmethod
def _missing_(cls, value: object) -> None:
"""Reject a value that is not a valid `Cv2Backends`.
Raises:
ValueError: Always, naming the invalid value and the valid choices.
"""
raise ValueError(f"`backend` is expected to be in {list(cls)}, but {value} is provided.")
@dataclass(kw_only=True)
class CameraConfig(draccus.ChoiceRegistry, abc.ABC): # type: ignore # TODO: add type stubs for draccus
"""Base configuration shared by every camera backend.
Concrete backends subclass this and register themselves with
`@CameraConfig.register_subclass("name")`, which is what makes `--camera.type=name` work on the
command line. Subclasses inherit the three fields below and must document them alongside their own.
Args:
fps (`int`, *optional*):
Requested frames per second for the color stream. `None` leaves it at the backend's default.
width (`int`, *optional*):
Requested frame width in pixels. `None` leaves it at the backend's default.
height (`int`, *optional*):
Requested frame height in pixels. `None` leaves it at the backend's default.
"""
fps: int | None = None
width: int | None = None
height: int | None = None
@property
def type(self) -> str:
"""Return the registered name this config was registered under.
Returns:
`str`: The name passed to `@CameraConfig.register_subclass`, e.g. `"opencv"`.
"""
return str(self.get_choice_name(self.__class__))
+32 -12
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Provides the OpenCVCamera class for capturing frames from cameras using OpenCV."""
"""
Provides the OpenCVCamera class for capturing frames from cameras using OpenCV.
"""
import logging
import math
@@ -48,7 +50,8 @@ logger = logging.getLogger(__name__)
class OpenCVCamera(Camera):
"""Manages camera interactions using OpenCV for efficient frame recording.
"""
Manages camera interactions using OpenCV for efficient frame recording.
This class provides a high-level interface to connect to, configure, and read
frames from cameras compatible with OpenCV's VideoCapture. It supports both
@@ -90,7 +93,8 @@ class OpenCVCamera(Camera):
"""
def __init__(self, config: OpenCVCameraConfig):
"""Initializes the OpenCVCamera instance.
"""
Initializes the OpenCVCamera instance.
Args:
config: The configuration settings for the camera.
@@ -121,7 +125,6 @@ class OpenCVCamera(Camera):
self._reset_connection_settings()
def __str__(self) -> str:
"""Return a short representation naming the class and its `index_or_path`."""
return f"{self.__class__.__name__}({self.index_or_path})"
def _reset_connection_settings(self) -> None:
@@ -140,7 +143,8 @@ class OpenCVCamera(Camera):
@check_if_already_connected
def connect(self, warmup: bool = True) -> None:
"""Connects to the OpenCV camera specified in the configuration.
"""
Connects to the OpenCV camera specified in the configuration.
Initializes the OpenCV VideoCapture object, sets desired camera properties
(FPS, width, height), starts the background reading thread and performs initial checks.
@@ -154,6 +158,7 @@ class OpenCVCamera(Camera):
ConnectionError: If the specified camera index/path is not found or fails to open.
RuntimeError: If the camera opens but fails to apply requested settings.
"""
# Use 1 thread for OpenCV operations to avoid potential conflicts or
# blocking in multi-threaded applications, especially during data collection.
cv2.setNumThreads(1)
@@ -191,7 +196,8 @@ class OpenCVCamera(Camera):
@check_if_not_connected
def _configure_capture_settings(self) -> None:
"""Applies the specified FOURCC, FPS, width, and height settings to the connected camera.
"""
Applies the specified FOURCC, FPS, width, and height settings to the connected camera.
This method attempts to set the camera properties via OpenCV. It checks if
the camera successfully applied the settings and raises an error if not.
@@ -208,6 +214,7 @@ class OpenCVCamera(Camera):
to the requested value.
DeviceNotConnectedError: If the camera is not connected.
"""
if self.videocapture is None:
raise DeviceNotConnectedError(f"{self} videocapture is not initialized")
@@ -239,6 +246,7 @@ class OpenCVCamera(Camera):
def _validate_fps(self) -> None:
"""Validates and sets the camera's frames per second (FPS)."""
if self.videocapture is None:
raise DeviceNotConnectedError(f"{self} videocapture is not initialized")
@@ -253,6 +261,7 @@ class OpenCVCamera(Camera):
def _validate_fourcc(self) -> None:
"""Validates and sets the camera's FOURCC code."""
fourcc_code = cv2.VideoWriter_fourcc(*self.config.fourcc)
if self.videocapture is None:
@@ -273,6 +282,7 @@ class OpenCVCamera(Camera):
def _validate_width_and_height(self) -> None:
"""Validates and sets the camera's frame capture width and height."""
if self.videocapture is None:
raise DeviceNotConnectedError(f"{self} videocapture is not initialized")
@@ -296,7 +306,8 @@ class OpenCVCamera(Camera):
@staticmethod
def find_cameras() -> list[dict[str, Any]]:
"""Detects available OpenCV cameras connected to the system.
"""
Detects available OpenCV cameras connected to the system.
On Linux, it scans '/dev/video*' paths. On other systems (like macOS, Windows),
it checks indices from 0 up to `MAX_OPENCV_INDEX`.
@@ -364,7 +375,8 @@ class OpenCVCamera(Camera):
@check_if_not_connected
def read(self, color_mode: ColorMode | None = None) -> NDArray[Any]:
"""Reads a single frame synchronously from the camera.
"""
Reads a single frame synchronously from the camera.
This is a blocking call. It waits for the next available frame from the
camera hardware via OpenCV.
@@ -380,6 +392,7 @@ class OpenCVCamera(Camera):
received frame dimensions don't match expectations before rotation.
ValueError: If an invalid `color_mode` is requested.
"""
start_time = time.perf_counter()
if color_mode is not None:
@@ -399,7 +412,8 @@ class OpenCVCamera(Camera):
return frame
def _postprocess_image(self, image: NDArray[Any]) -> NDArray[Any]:
"""Applies color conversion, dimension validation, and rotation to a raw frame.
"""
Applies color conversion, dimension validation, and rotation to a raw frame.
Args:
image (np.ndarray): The raw image frame (expected BGR format from OpenCV).
@@ -412,6 +426,7 @@ class OpenCVCamera(Camera):
RuntimeError: If the raw frame dimensions do not match the configured
`width` and `height`.
"""
if self.color_mode not in (ColorMode.RGB, ColorMode.BGR):
raise ValueError(
f"Invalid color mode '{self.color_mode}'. Expected {ColorMode.RGB} or {ColorMode.BGR}."
@@ -437,7 +452,8 @@ class OpenCVCamera(Camera):
return processed_image
def _read_loop(self) -> None:
"""Internal loop run by the background thread for asynchronous reading.
"""
Internal loop run by the background thread for asynchronous reading.
On each iteration:
1. Reads a color frame (blocking call)
@@ -522,7 +538,8 @@ class OpenCVCamera(Camera):
@check_if_not_connected
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
"""Reads the latest available frame asynchronously.
"""
Reads the latest available frame asynchronously.
This method retrieves the most recent frame captured by the background
read thread. It does not block waiting for the camera hardware directly,
@@ -542,6 +559,7 @@ class OpenCVCamera(Camera):
TimeoutError: If no frame becomes available within the specified timeout.
RuntimeError: If an unexpected error occurs.
"""
if self.thread is None or not self.thread.is_alive():
raise RuntimeError(f"{self} read thread is not running.")
@@ -576,6 +594,7 @@ class OpenCVCamera(Camera):
DeviceNotConnectedError: If the camera is not connected.
RuntimeError: If the camera is connected but has not captured any frames yet.
"""
if self.thread is None or not self.thread.is_alive():
raise RuntimeError(f"{self} read thread is not running.")
@@ -595,7 +614,8 @@ class OpenCVCamera(Camera):
return frame
def disconnect(self) -> None:
"""Disconnects from the camera and cleans up resources.
"""
Disconnects from the camera and cleans up resources.
Stops the background read thread (if running) and releases the OpenCV
VideoCapture object.
@@ -40,28 +40,17 @@ class OpenCVCameraConfig(CameraConfig):
OpenCVCameraConfig(0, 30, 1280, 720, fourcc="YUYV") # With YUYV format
```
Args:
index_or_path (`int | Path`):
Either an integer representing the camera device index, or a Path object pointing to a video
file.
color_mode (`ColorMode`, *optional*, defaults to `ColorMode.RGB`):
Color mode for image output.
rotation (`Cv2Rotation`, *optional*, defaults to `Cv2Rotation.NO_ROTATION`):
Image rotation setting (0°, 90°, 180°, or 270°).
warmup_s (`int`, *optional*, defaults to 1):
Time reading frames before returning from connect (in seconds).
fourcc (`str`, *optional*):
FOURCC code for video format (e.g., `"MJPG"`, `"YUYV"`, `"I420"`). `None` auto-detects.
backend (`Cv2Backends`, *optional*, defaults to `Cv2Backends.ANY`):
OpenCV backend identifier. See [`~cameras.Cv2Backends`] for the supported values.
fps (`int`, *optional*):
Requested frames per second for the color stream. `None` leaves it at the backend's default.
width (`int`, *optional*):
Requested frame width in pixels for the color stream. `None` leaves it at the backend's
default.
height (`int`, *optional*):
Requested frame height in pixels for the color stream. `None` leaves it at the backend's
default.
Attributes:
index_or_path: Either an integer representing the camera device index,
or a Path object pointing to a video file.
fps: Requested frames per second for the color stream.
width: Requested frame width in pixels for the color stream.
height: Requested frame height in pixels for the color stream.
color_mode: Color mode for image output (RGB or BGR). Defaults to RGB.
rotation: Image rotation setting (0°, 90°, 180°, or 270°). Defaults to no rotation.
warmup_s: Time reading frames before returning from connect (in seconds)
fourcc: FOURCC code for video format (e.g., "MJPG", "YUYV", "I420"). Defaults to None (auto-detect).
backend: OpenCV backend identifier (https://docs.opencv.org/3.4/d4/d15/group__videoio__flags__base.html). Defaults to ANY.
Note:
- Only 3-channel color output (RGB/BGR) is currently supported.
@@ -77,11 +66,6 @@ class OpenCVCameraConfig(CameraConfig):
backend: Cv2Backends = Cv2Backends.ANY
def __post_init__(self) -> None:
"""Normalize `color_mode`, `rotation`, and `backend`, and validate `fourcc`.
Raises:
ValueError: If `fourcc` is set and is not a 4-character string.
"""
self.color_mode = ColorMode(self.color_mode)
self.rotation = Cv2Rotation(self.rotation)
self.backend = Cv2Backends(self.backend)
@@ -43,24 +43,16 @@ class Reachy2CameraConfig(CameraConfig):
) # Left teleop camera, 640x480 @ 30FPS
```
Args:
name (`str`):
Name of the camera device. Either `"teleop"` or `"depth"`.
image_type (`str`):
Type of image stream. For the `"teleop"` camera, either `"left"` or `"right"`. For the
`"depth"` camera, either `"rgb"` or `"depth"` (depth is not supported yet).
color_mode (`ColorMode`, *optional*, defaults to `ColorMode.RGB`):
Color mode for image output.
ip_address (`str`, *optional*, defaults to `"localhost"`):
IP address of the robot.
port (`int`, *optional*, defaults to 50065):
Port number for the camera server.
fps (`int`, *optional*):
Requested frames per second for the color stream. Not configurable for Reachy 2 cameras.
width (`int`, *optional*):
Requested frame width in pixels for the color stream.
height (`int`, *optional*):
Requested frame height in pixels for the color stream.
Attributes:
name: Name of the camera device. Can be "teleop" or "depth".
image_type: Type of image stream. For "teleop" camera, can be "left" or "right".
For "depth" camera, can be "rgb" or "depth". (depth is not supported yet)
fps: Requested frames per second for the color stream. Not configurable for Reachy 2 cameras.
width: Requested frame width in pixels for the color stream.
height: Requested frame height in pixels for the color stream.
color_mode: Color mode for image output (RGB or BGR). Defaults to RGB.
ip_address: IP address of the robot. Defaults to "localhost".
port: Port number for the camera server. Defaults to 50065.
Note:
- Only 3-channel color output (RGB/BGR) is currently supported.
@@ -73,11 +65,6 @@ class Reachy2CameraConfig(CameraConfig):
port: int = 50065
def __post_init__(self) -> None:
"""Normalize `color_mode` and validate `name`/`image_type`.
Raises:
ValueError: If `name` is not `"teleop"`/`"depth"`, or `image_type` is not valid for `name`.
"""
if self.name not in ["teleop", "depth"]:
raise ValueError(f"`name` is expected to be 'teleop' or 'depth', but {self.name} is provided.")
if (self.name == "teleop" and self.image_type not in ["left", "right"]) or (
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Provides the Reachy2Camera class for capturing frames from Reachy 2 cameras using Reachy 2's CameraManager."""
"""
Provides the Reachy2Camera class for capturing frames from Reachy 2 cameras using Reachy 2's CameraManager.
"""
from __future__ import annotations
@@ -40,8 +42,6 @@ else:
CameraManager = None
class CameraView:
"""Fallback stand-in for `reachy2_sdk`'s `CameraView` when the SDK is not installed."""
LEFT = 0
RIGHT = 1
@@ -55,7 +55,8 @@ logger = logging.getLogger(__name__)
class Reachy2Camera(Camera):
"""Manages Reachy 2 camera using Reachy 2 CameraManager.
"""
Manages Reachy 2 camera using Reachy 2 CameraManager.
This class provides a high-level interface to connect to, configure, and read
frames from Reachy 2 cameras. It supports both synchronous and asynchronous
@@ -69,7 +70,8 @@ class Reachy2Camera(Camera):
"""
def __init__(self, config: Reachy2CameraConfig):
"""Initializes the Reachy2Camera instance.
"""
Initializes the Reachy2Camera instance.
Args:
config: The configuration settings for the camera.
@@ -86,7 +88,6 @@ class Reachy2Camera(Camera):
self.cam_manager: CameraManager | None = None
def __str__(self) -> str:
"""Return a short representation naming the class, camera name, and image type."""
return f"{self.__class__.__name__}({self.config.name}, {self.config.image_type})"
@property
@@ -104,7 +105,8 @@ class Reachy2Camera(Camera):
raise ValueError(f"Invalid camera name '{self.config.name}'. Expected 'teleop' or 'depth'.")
def connect(self, warmup: bool = True) -> None:
"""Connects to the Reachy2 CameraManager as specified in the configuration.
"""
Connects to the Reachy2 CameraManager as specified in the configuration.
Raises:
DeviceNotConnectedError: If the camera is not connected.
@@ -118,12 +120,15 @@ class Reachy2Camera(Camera):
@staticmethod
def find_cameras() -> list[dict[str, Any]]:
"""Detection not implemented for Reachy2 cameras."""
"""
Detection not implemented for Reachy2 cameras.
"""
raise NotImplementedError("Camera detection is not implemented for Reachy2 cameras.")
@check_if_not_connected
def read(self, color_mode: ColorMode | None = None) -> NDArray[Any]:
"""Reads a single frame synchronously from the camera.
"""
Reads a single frame synchronously from the camera.
This method retrieves the most recent frame available in Reachy 2's low-level software.
@@ -182,7 +187,8 @@ class Reachy2Camera(Camera):
@check_if_not_connected
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
"""Same as [`~cameras.reachy2_camera.Reachy2Camera.read`]; Reachy 2 has no separate async path.
"""
Same as read()
Returns:
np.ndarray: The latest captured frame as a NumPy array in the format
@@ -193,6 +199,7 @@ class Reachy2Camera(Camera):
TimeoutError: If no frame becomes available within the specified timeout.
RuntimeError: If an unexpected error occurs.
"""
return self.read()
@check_if_not_connected
@@ -213,6 +220,7 @@ class Reachy2Camera(Camera):
DeviceNotConnectedError: If the camera is not connected.
RuntimeError: If the camera is connected but has not captured any frames yet.
"""
if self.latest_frame is None or self.latest_timestamp is None:
raise RuntimeError(f"{self} has not captured any frames yet.")
@@ -226,11 +234,13 @@ class Reachy2Camera(Camera):
@check_if_not_connected
def disconnect(self) -> None:
"""Stops the background read thread (if running).
"""
Stops the background read thread (if running).
Raises:
DeviceNotConnectedError: If the camera is already disconnected.
"""
if self.cam_manager is not None:
self.cam_manager.disconnect()
+73 -135
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Provides the RealSenseCamera class for capturing frames from Intel RealSense cameras."""
"""
Provides the RealSenseCamera class for capturing frames from Intel RealSense cameras.
"""
import logging
import sys
@@ -44,7 +46,8 @@ pkg_name = "pyrealsense2-macosx" if sys.platform == "darwin" else "pyrealsense2"
class RealSenseCamera(Camera):
"""Manages interactions with Intel RealSense cameras for frame and depth recording.
"""
Manages interactions with Intel RealSense cameras for frame and depth recording.
This class provides an interface similar to `OpenCVCamera` but tailored for
RealSense devices, leveraging the `pyrealsense2` library. It uses the camera's
@@ -106,13 +109,9 @@ class RealSenseCamera(Camera):
```
"""
# Maximum number of warmup attempts made by connect(). A failed attempt is first
# retried with a plain pipeline stop/start, which is usually enough to recover the
# stream; a USB hardware reset is performed before the final attempt as a last resort.
_MAX_CONNECT_ATTEMPTS = 3
def __init__(self, config: RealSenseCameraConfig):
"""Initializes the RealSenseCamera instance.
"""
Initializes the RealSenseCamera instance.
Args:
config: The configuration settings for the camera.
@@ -157,7 +156,6 @@ class RealSenseCamera(Camera):
self._reset_connection_settings()
def __str__(self) -> str:
"""Return a short representation naming the class and its `serial_number`."""
return f"{self.__class__.__name__}({self.serial_number})"
def _reset_connection_settings(self) -> None:
@@ -175,133 +173,71 @@ class RealSenseCamera(Camera):
"""Checks if the camera pipeline is started and streams are active."""
return self.rs_pipeline is not None and self.rs_profile is not None
def _hardware_reset(self, wait_s: float = 5.0) -> None:
"""Issue a USB hardware reset to recover an unresponsive device (common on D405)."""
context = rs.context()
for device in context.query_devices():
if device.get_info(rs.camera_info.serial_number) == self.serial_number:
logger.info(f"{self} performing hardware reset.")
device.hardware_reset()
time.sleep(wait_s)
return
logger.warning(f"{self} device not found for hardware reset, skipping.")
def _open_pipeline(self) -> None:
"""Initializes the RealSense pipeline, starts it, and starts the background read thread.
Raises:
ValueError: If the configuration is invalid, a requested sensor option is unsupported,
or a requested sensor value is invalid.
ConnectionError: If the camera is found but fails to start the pipeline or no RealSense devices are detected at all.
RuntimeError: If the pipeline starts but fails to apply requested settings.
"""
rs_pipeline = rs.pipeline()
rs_config = rs.config()
self._configure_rs_pipeline_config(rs_config)
try:
rs_profile = rs_pipeline.start(rs_config)
except RuntimeError as e:
raise ConnectionError(
f"Failed to open {self}.Run `lerobot-find-cameras realsense` to find available cameras."
) from e
self.rs_pipeline = rs_pipeline
self.rs_profile = rs_profile
try:
self._configure_capture_settings()
self._configure_sensor_options()
self._start_read_thread()
except BaseException:
self._release_after_failed_setup()
raise
def _run_warmup(self) -> None:
"""Blocks until at least one valid frame has been captured by the background thread.
Raises:
ConnectionError: If no frame arrives before ``warmup_s`` elapses.
"""
# NOTE(Steven/Caroline): Enforcing at least one second of warmup as RS cameras need a bit of time before the first read. If we don't wait, the first read from the warmup will raise.
self.warmup_s = max(self.warmup_s, 1)
warmup_read = self.async_read if self.use_rgb else self.async_read_depth
start_time = time.time()
while time.time() - start_time < self.warmup_s:
warmup_read(timeout_ms=self.warmup_s * 1000)
time.sleep(0.1)
with self.frame_lock:
if (self.use_rgb and self.latest_color_frame is None) or (
self.use_depth and self.latest_depth_frame is None
):
raise ConnectionError(f"{self} failed to capture frames during warmup.")
def _release_after_failed_setup(self) -> None:
"""Releases the device handle and restores auto-detected settings after a failed attempt."""
try:
self._cleanup_resources()
except Exception:
logger.exception(f"Failed to fully clean up {self} after connect() failed.")
self._reset_connection_settings()
@check_if_already_connected
def connect(self, warmup: bool = True) -> None:
"""Connects to the RealSense camera specified in the configuration.
"""
Connects to the RealSense camera specified in the configuration.
Initializes the RealSense pipeline, configures the required streams (color
and optionally depth), starts the pipeline, and validates the actual stream settings.
If the pipeline starts but no frames arrive during warmup, retries up to
``_MAX_CONNECT_ATTEMPTS`` times, performing a USB hardware reset before the
final attempt.
Args:
warmup (bool): If True, waits at connect() time until at least one valid frame
has been captured by the background thread. Defaults to True.
Raises:
DeviceAlreadyConnectedError: If the camera is already connected.
ValueError: If the configuration is invalid (e.g., missing serial/name, name not unique).
ValueError: If the configuration is invalid, a requested sensor option is unsupported,
or a requested sensor value is invalid.
ConnectionError: If the camera is found but fails to start the pipeline or no RealSense devices are detected at all.
RuntimeError: If the pipeline starts but fails to apply requested settings.
"""
if not warmup:
self._open_pipeline()
logger.info(f"{self} connected.")
return
last_error: Exception | None = None
self.rs_pipeline = rs.pipeline()
rs_config = rs.config()
self._configure_rs_pipeline_config(rs_config)
for attempt in range(1, self._MAX_CONNECT_ATTEMPTS + 1):
if attempt == self._MAX_CONNECT_ATTEMPTS:
self._hardware_reset()
try:
self.rs_profile = self.rs_pipeline.start(rs_config)
except RuntimeError as e:
self.rs_profile = None
self.rs_pipeline = None
raise ConnectionError(
f"Failed to open {self}.Run `lerobot-find-cameras realsense` to find available cameras."
) from e
self._open_pipeline()
try:
self._configure_capture_settings()
self._configure_sensor_options()
self._start_read_thread()
connected = False
# NOTE(Steven/Caroline): Enforcing at least one second of warmup as RS cameras need a bit of time before the first read. If we don't wait, the first read from the warmup will raise.
self.warmup_s = max(self.warmup_s, 1)
warmup_read = self.async_read if self.use_rgb else self.async_read_depth
start_time = time.time()
while time.time() - start_time < self.warmup_s:
warmup_read(timeout_ms=self.warmup_s * 1000)
time.sleep(0.1)
with self.frame_lock:
if (self.use_rgb and self.latest_color_frame is None) or (
self.use_depth and self.latest_depth_frame is None
):
raise ConnectionError(f"{self} failed to capture frames during warmup.")
except BaseException:
try:
self._run_warmup()
connected = True
except (TimeoutError, ConnectionError) as e:
last_error = e
finally:
if not connected:
self._release_after_failed_setup()
self._cleanup_resources()
except Exception:
logger.exception(f"Failed to fully clean up {self} after connect() failed.")
self._reset_connection_settings()
raise
if connected:
logger.info(f"{self} connected.")
return
logger.warning(f"{self} warmup failed (attempt {attempt}/{self._MAX_CONNECT_ATTEMPTS}).")
raise ConnectionError(
f"{self} failed to capture frames after {self._MAX_CONNECT_ATTEMPTS} attempts."
) from last_error
logger.info(f"{self} connected.")
@staticmethod
def find_cameras() -> list[dict[str, Any]]:
"""Detects available Intel RealSense cameras connected to the system.
"""
Detects available Intel RealSense cameras connected to the system.
Returns:
List[Dict[str, Any]]: A list of dictionaries,
@@ -400,6 +336,7 @@ class RealSenseCamera(Camera):
Raises:
DeviceNotConnectedError: If device is not connected.
"""
if self.rs_profile is None:
raise RuntimeError(f"{self}: rs_profile must be initialized before use.")
@@ -428,12 +365,11 @@ class RealSenseCamera(Camera):
return self._async_read(timeout_ms=10000, read_depth=read_depth)
def _get_color_sensor(self) -> "rs.sensor":
"""Returns the dedicated "RGB Camera" sensor that controls the color stream.
"""Returns the sensor that controls the color stream.
Manual color controls are only applied to a dedicated RGB module. Cameras
without one (e.g. the D405, whose color stream comes from the shared
"Stereo Module") are unsupported, so we never fall back to another sensor
to avoid altering the depth stream.
Most RealSense cameras expose "RGB Camera" for color. The D405 has no
separate RGB module its color stream comes from "Stereo Module".
We try RGB Camera first, then fall back to Stereo Module.
"""
if self.rs_profile is None:
raise RuntimeError(f"{self}: rs_profile must be initialized before use.")
@@ -441,14 +377,12 @@ class RealSenseCamera(Camera):
device = self.rs_profile.get_device()
sensors = {s.get_info(rs.camera_info.name): s for s in device.query_sensors()}
if "RGB Camera" in sensors:
return sensors["RGB Camera"]
for name in ("RGB Camera", "Stereo Module"):
if name in sensors:
return sensors[name]
available = list(sensors.keys())
raise RuntimeError(
f"{self}: manual color controls require a dedicated 'RGB Camera' module, which this camera does not have. ",
f"Available sensors: {available}.",
)
raise RuntimeError(f"{self}: no color sensor found. Available sensors: {available}")
def _set_sensor_option(self, sensor: "rs.sensor", option: "rs.option", value: float, label: str) -> None:
"""Sets a sensor option, re-raising range errors with actionable diagnostics."""
@@ -537,7 +471,8 @@ class RealSenseCamera(Camera):
@check_if_not_connected
def read_depth(self, timeout_ms: int = 200) -> NDArray[Any]:
"""Reads a single frame (depth) synchronously from the camera.
"""
Reads a single frame (depth) synchronously from the camera.
This is a blocking call. It waits for a coherent set of frames (depth)
from the camera hardware via the RealSense pipeline.
@@ -575,7 +510,8 @@ class RealSenseCamera(Camera):
@check_if_not_connected
def read(self, color_mode: ColorMode | None = None, timeout_ms: int = 0) -> NDArray[Any]:
"""Reads a single frame (color) synchronously from the camera.
"""
Reads a single frame (color) synchronously from the camera.
This is a blocking call. It waits for a coherent set of frames (color)
from the camera hardware via the RealSense pipeline.
@@ -589,6 +525,7 @@ class RealSenseCamera(Camera):
RuntimeError: If reading frames from the pipeline fails or frames are invalid.
ValueError: If an invalid `color_mode` is requested.
"""
start_time = time.perf_counter()
if color_mode is not None:
@@ -612,12 +549,11 @@ class RealSenseCamera(Camera):
return frame
def _postprocess_image(self, image: NDArray[Any], depth_frame: bool = False) -> NDArray[Any]:
"""Applies color conversion, dimension validation, and rotation to a raw color frame.
"""
Applies color conversion, dimension validation, and rotation to a raw color frame.
Args:
image (np.ndarray): The raw image frame (expected RGB format from RealSense).
depth_frame (bool): Whether `image` is a single-channel depth frame rather than a 3-channel
color frame.
Returns:
np.ndarray: The processed image frame according to `self.color_mode` and `self.rotation`.
@@ -627,6 +563,7 @@ class RealSenseCamera(Camera):
RuntimeError: If the raw frame dimensions do not match the configured
`width` and `height`.
"""
if self.color_mode and self.color_mode not in (ColorMode.RGB, ColorMode.BGR):
raise ValueError(
f"Invalid requested color mode '{self.color_mode}'. Expected {ColorMode.RGB} or {ColorMode.BGR}."
@@ -655,7 +592,8 @@ class RealSenseCamera(Camera):
return processed_image
def _read_loop(self) -> None:
"""Internal loop run by the background thread for asynchronous reading.
"""
Internal loop run by the background thread for asynchronous reading.
On each iteration:
1. Reads a color/depth frame (blocking call with 10s timeout)
@@ -688,9 +626,6 @@ class RealSenseCamera(Camera):
capture_time = time.perf_counter()
with self.frame_lock:
# Under the lock, so a late frame cannot resurrect the buffer _stop_read_thread() cleared.
if stop_event.is_set():
break
if self.use_rgb:
self.latest_color_frame = processed_color_frame
if self.use_depth:
@@ -779,7 +714,8 @@ class RealSenseCamera(Camera):
@check_if_not_connected
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
"""Reads the latest available frame data (color) asynchronously.
"""
Reads the latest available frame data (color) asynchronously.
This method retrieves the most recent color frame captured by the background
read thread. It does not block waiting for the camera hardware directly,
@@ -799,6 +735,7 @@ class RealSenseCamera(Camera):
TimeoutError: If no frame data becomes available within the specified timeout.
RuntimeError: If the background thread died unexpectedly or another error occurs.
"""
if not self.use_rgb:
raise RuntimeError(f"{self}: cannot read color — camera was configured with use_rgb=False.")
@@ -884,18 +821,19 @@ class RealSenseCamera(Camera):
return self._read_latest(max_age_ms=max_age_ms, read_depth=True)
def disconnect(self) -> None:
"""Disconnects from the camera, stops the pipeline, and cleans up resources.
"""
Disconnects from the camera, stops the pipeline, and cleans up resources.
Stops the background read thread (if running) and stops the RealSense pipeline.
Raises:
DeviceNotConnectedError: If the camera is already disconnected (pipeline not running).
"""
if not self.is_connected and self.thread is None:
raise DeviceNotConnectedError(
f"Attempted to disconnect {self}, but it appears already disconnected."
)
self._cleanup_resources()
logger.info(f"{self} disconnected.")
@@ -36,39 +36,27 @@ class RealSenseCameraConfig(CameraConfig):
RealSenseCameraConfig("0123456789", 30, 640, 480, rotation=Cv2Rotation.ROTATE_90) # With 90° rotation
```
Args:
serial_number_or_name (`str`):
Unique serial number or human-readable name to identify the camera.
color_mode (`ColorMode`, *optional*, defaults to `ColorMode.RGB`):
Color mode for image output.
use_rgb (`bool`, *optional*, defaults to `True`):
Whether to enable the color stream.
use_depth (`bool`, *optional*, defaults to `False`):
Whether to enable the depth stream.
rotation (`Cv2Rotation`, *optional*, defaults to `Cv2Rotation.NO_ROTATION`):
Image rotation setting (0°, 90°, 180°, or 270°).
warmup_s (`int`, *optional*, defaults to 1):
Time reading frames before returning from connect (in seconds).
exposure (`int`, *optional*):
Manual exposure value for the color sensor. When set, auto-exposure is disabled and this fixed
value is used. Valid ranges are camera-model specific and reported if the value is rejected.
`None` leaves auto-exposure unchanged.
gain (`int`, *optional*):
Manual gain value for the color sensor. When set, auto-exposure is disabled and this fixed
gain is used, which also freezes exposure at its current value when no exposure is configured.
Valid ranges are camera-model specific and reported if the value is rejected. `None` leaves it
unchanged.
white_balance (`int`, *optional*):
Manual white balance value for the color sensor. When set, auto white balance is disabled and
this fixed value is used. Valid ranges are camera-model specific and reported if the value is
rejected. `None` leaves it unchanged.
fps (`int`, *optional*):
Requested frames per second for the color stream. The depth stream, when enabled, uses the
same FPS. Either all of `fps`, `width`, and `height` must be set, or none of them.
width (`int`, *optional*):
Requested frame width in pixels for the color stream.
height (`int`, *optional*):
Requested frame height in pixels for the color stream.
Attributes:
fps: Requested frames per second for the color stream.
width: Requested frame width in pixels for the color stream.
height: Requested frame height in pixels for the color stream.
serial_number_or_name: Unique serial number or human-readable name to identify the camera.
color_mode: Color mode for image output (RGB or BGR). Defaults to RGB.
use_rgb: Whether to enable the color stream. Defaults to True.
use_depth: Whether to enable depth stream. Defaults to False.
rotation: Image rotation setting (0°, 90°, 180°, or 270°). Defaults to no rotation.
warmup_s: Time reading frames before returning from connect (in seconds)
exposure: Manual exposure value for the color sensor. When set, auto-exposure is
disabled and this fixed value is used. Valid ranges are camera-model specific
and reported if the value is rejected. Defaults to None (leave unchanged).
gain: Manual gain value for the color sensor. When set, auto-exposure is disabled
and this fixed gain is used, which also freezes exposure at its current value
when no exposure is configured. Valid ranges are camera-model specific and
reported if the value is rejected. Defaults to None (leave unchanged).
white_balance: Manual white balance value for the color sensor. When set, auto
white balance is disabled and this fixed value is used. Valid ranges are
camera-model specific and reported if the value is rejected. Defaults to None
(leave unchanged).
Note:
- Either name or serial_number must be specified.
@@ -89,12 +77,6 @@ class RealSenseCameraConfig(CameraConfig):
white_balance: int | None = None
def __post_init__(self) -> None:
"""Normalize `color_mode`/`rotation` and validate the stream and sensor-option settings.
Raises:
ValueError: If neither `use_rgb` nor `use_depth` is enabled, if a manual color sensor option
is set without `use_rgb=True`, or if only some of `fps`/`width`/`height` are set.
"""
self.color_mode = ColorMode(self.color_mode)
self.rotation = Cv2Rotation(self.rotation)
-36
View File
@@ -23,33 +23,6 @@ from .configs import CameraConfig, Cv2Rotation
def make_cameras_from_configs(camera_configs: dict[str, CameraConfig]) -> dict[str, Camera]:
"""Instantiate one [`~cameras.Camera`] per entry in a mapping of configs.
Dispatches on each config's registered `type` to build the matching backend class. This only
constructs the camera objects; call [`~cameras.Camera.connect`] on each before use.
Args:
camera_configs (`dict[str, CameraConfig]`):
Camera configs keyed by the name each camera should be identified by, e.g. in a robot's
observation features.
Returns:
`dict[str, Camera]`: A camera instance per key, in the same order as `camera_configs`.
Raises:
ValueError: If a config's type is not a known backend and building it via the generic device
factory also fails.
Example:
```python
>>> from lerobot.cameras.opencv import OpenCVCameraConfig
>>> from lerobot.cameras.utils import make_cameras_from_configs
>>> configs = {"top": OpenCVCameraConfig(index_or_path=0, fps=30, width=640, height=480)}
>>> cameras = make_cameras_from_configs(configs)
>>> list(cameras.keys())
['top']
```
"""
cameras: dict[str, Camera] = {}
for key, cfg in camera_configs.items():
@@ -84,15 +57,6 @@ def make_cameras_from_configs(camera_configs: dict[str, CameraConfig]) -> dict[s
def get_cv2_rotation(rotation: Cv2Rotation) -> int | None:
"""Map a [`~cameras.Cv2Rotation`] to the OpenCV rotation flag `cv2.rotate` expects.
Args:
rotation (`Cv2Rotation`):
The configured rotation.
Returns:
`int | None`: The matching `cv2.ROTATE_*` constant, or `None` for [`~cameras.Cv2Rotation.NO_ROTATION`].
"""
import cv2 # type: ignore # TODO: add type stubs for OpenCV
if rotation == Cv2Rotation.ROTATE_90:
+21 -15
View File
@@ -14,9 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Captures frames from remote cameras via ZeroMQ.
Uses a JSON protocol of the following form:
"""
ZMQCamera - Captures frames from remote cameras via ZeroMQ using JSON protocol in the
following format:
{
"timestamps": {"camera_name": float},
"images": {"camera_name": "<base64-jpeg>"}
@@ -52,7 +52,8 @@ logger = logging.getLogger(__name__)
class ZMQCamera(Camera):
"""Manages camera interactions via ZeroMQ for receiving frames from a remote server.
"""
Manages camera interactions via ZeroMQ for receiving frames from a remote server.
This class connects to a ZMQ Publisher, subscribes to frame topics, and decodes
incoming JSON messages containing Base64 encoded images. It supports both
@@ -80,11 +81,6 @@ class ZMQCamera(Camera):
"""
def __init__(self, config: ZMQCameraConfig):
"""Initialize the camera with the given configuration.
Args:
config: Camera configuration, including the image server's address and port.
"""
require_package("pyzmq", extra="pyzmq-dep", import_name="zmq")
super().__init__(config)
@@ -109,7 +105,6 @@ class ZMQCamera(Camera):
self.new_frame_event: Event = Event()
def __str__(self) -> str:
"""Return a short representation naming the camera and its server address."""
return f"ZMQCamera({self.camera_name}@{self.server_address}:{self.port})"
@property
@@ -125,6 +120,7 @@ class ZMQCamera(Camera):
warmup (bool): If True, waits for the camera to provide at least one
valid frame before returning. Defaults to True.
"""
logger.info(f"Connecting to {self}...")
try:
@@ -175,11 +171,15 @@ class ZMQCamera(Camera):
@staticmethod
def find_cameras() -> list[dict[str, Any]]:
"""Detection not implemented for ZMQ cameras; they require manual server address/port configuration."""
"""
Detection not implemented for ZMQ cameras. These cameras require manual configuration (server address/port).
"""
raise NotImplementedError("Camera detection is not implemented for ZMQ cameras.")
def _read_from_hardware(self) -> NDArray[Any]:
"""Reads a single frame directly from the ZMQ socket."""
"""
Reads a single frame directly from the ZMQ socket.
"""
if not self.is_connected or self.socket is None:
raise DeviceNotConnectedError(f"{self} is not connected.")
@@ -215,7 +215,8 @@ class ZMQCamera(Camera):
@check_if_not_connected
def read(self, color_mode: ColorMode | None = None) -> NDArray[Any]:
"""Reads a single frame synchronously from the camera.
"""
Reads a single frame synchronously from the camera.
This is a blocking call. It waits for the next available frame from the
camera background thread.
@@ -242,7 +243,9 @@ class ZMQCamera(Camera):
return frame
def _read_loop(self) -> None:
"""Internal loop run by the background thread for asynchronous reading."""
"""
Internal loop run by the background thread for asynchronous reading.
"""
stop_event = self.stop_event
if stop_event is None:
raise RuntimeError(f"{self}: stop_event is not initialized.")
@@ -303,7 +306,8 @@ class ZMQCamera(Camera):
@check_if_not_connected
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
"""Reads the latest available frame asynchronously.
"""
Reads the latest available frame asynchronously.
Args:
timeout_ms (float): Maximum time in milliseconds to wait for a frame
@@ -317,6 +321,7 @@ class ZMQCamera(Camera):
TimeoutError: If no frame data becomes available within the specified timeout.
RuntimeError: If the background thread is not running.
"""
if self.thread is None or not self.thread.is_alive():
raise RuntimeError(f"{self} read thread is not running.")
@@ -348,6 +353,7 @@ class ZMQCamera(Camera):
DeviceNotConnectedError: If the camera is not connected.
RuntimeError: If the camera is connected but has not captured any frames yet.
"""
if self.thread is None or not self.thread.is_alive():
raise RuntimeError(f"{self} read thread is not running.")
@@ -24,32 +24,6 @@ __all__ = ["ZMQCameraConfig", "ColorMode"]
@CameraConfig.register_subclass("zmq")
@dataclass
class ZMQCameraConfig(CameraConfig):
"""Configuration for a camera served over a ZeroMQ socket by `ImageServer`.
Use this to read frames from a camera attached to a different machine (e.g. a Raspberry Pi on a
robot), which streams JPEG-encoded frames to this config's `server_address`/`port` over ZMQ.
Args:
server_address (`str`):
Address of the machine running the image server, e.g. `"192.168.1.50"`.
port (`int`, *optional*, defaults to 5555):
TCP port the image server is publishing on. Must be between 1 and 65535.
camera_name (`str`, *optional*, defaults to `"zmq_camera"`):
Name used to identify this camera in logs and observation keys.
color_mode (`ColorMode`, *optional*, defaults to `ColorMode.RGB`):
Color mode for the decoded frames.
timeout_ms (`int`, *optional*, defaults to 5000):
How long to wait for a frame before raising a timeout, in milliseconds. Must be positive.
warmup_s (`int`, *optional*, defaults to 1):
Time spent reading frames before returning from connect, in seconds.
fps (`int`, *optional*):
Requested frames per second. `None` leaves it at the server's own rate.
width (`int`, *optional*):
Requested frame width in pixels. `None` leaves it at the server's own resolution.
height (`int`, *optional*):
Requested frame height in pixels. `None` leaves it at the server's own resolution.
"""
server_address: str
port: int = 5555
camera_name: str = "zmq_camera"
@@ -58,12 +32,6 @@ class ZMQCameraConfig(CameraConfig):
warmup_s: int = 1
def __post_init__(self) -> None:
"""Normalize `color_mode` and validate the socket settings.
Raises:
ValueError: If `timeout_ms` is not positive, `server_address` is empty, or `port` is outside
the 1-65535 range.
"""
self.color_mode = ColorMode(self.color_mode)
if self.timeout_ms <= 0:
+2 -28
View File
@@ -14,8 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Streams camera images over ZMQ.
"""
Streams camera images over ZMQ.
Uses lerobot's OpenCVCamera for capture, encodes images to base64 and sends them over ZMQ.
"""
@@ -47,12 +47,6 @@ class CameraCaptureThread:
"""Background thread that continuously captures and encodes frames from a camera."""
def __init__(self, camera: OpenCVCamera, name: str):
"""Initialize the capture thread for one camera.
Args:
camera: The already-connected camera to read frames from.
name: Name used to identify this camera in log messages.
"""
self.camera = camera
self.name = name
self.latest_encoded: str | None = None # Pre-encoded JPEG as base64
@@ -95,21 +89,7 @@ class CameraCaptureThread:
class ImageServer:
"""Reads from one or more local OpenCV cameras and publishes their frames over a ZMQ `PUB` socket.
Runs on the machine physically connected to the cameras (e.g. a Raspberry Pi on a robot); pair with
[`~cameras.zmq.ZMQCamera`] on the machine that consumes the stream.
"""
def __init__(self, config: dict, port: int = 5555):
"""Connect every camera listed in `config` and open the publishing socket.
Args:
config: A mapping with an optional `"fps"` (the publish loop rate, not the camera capture
rate) and a `"cameras"` mapping of camera name to a dict with `"device_id"`, `"shape"`
(`[height, width]`), and `"fourcc"` keys.
port: TCP port to publish on.
"""
# fps controls the publish loop rate (how often frames are sent over ZMQ), not the camera capture rate
self.fps = config.get("fps", 30)
self.cameras: dict[str, OpenCVCamera] = {}
@@ -122,7 +102,6 @@ class ImageServer:
fps=self.fps,
width=shape[1],
height=shape[0],
fourcc=cfg.get("fourcc", "MJPG"),
color_mode=ColorMode.RGB,
)
camera = OpenCVCamera(cam_config)
@@ -144,11 +123,6 @@ class ImageServer:
logger.info(f"ImageServer running on port {port}")
def run(self):
"""Start each camera's capture thread and publish frames until interrupted.
Blocks until `KeyboardInterrupt`, then stops the capture threads, disconnects the cameras, and
closes the socket.
"""
frame_count = 0
frame_times = deque(maxlen=60)
last_published_ts: dict[str, float] = {}
+1 -1
View File
@@ -35,9 +35,9 @@ else:
if TYPE_CHECKING:
from lerobot.datasets import LeRobotDataset
from lerobot.lerobot_types import PolicyAction
from lerobot.processor import PolicyProcessorPipeline
from lerobot.robots import Robot
from lerobot.types import PolicyAction
def predict_action(
+171 -612
View File
@@ -13,41 +13,16 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Training-output persistence: checkpoints, two-phase resume, and hub publishing.
Rank discipline: every function here that can
contain a collective is documented as such and must run on ALL ranks; rank-0-only file writes
sit under one grouped ``is_main_process()`` gate per contiguous region, placed below all
collectives. The leaf save/load helpers carry no rank gates of their own the exception is
``PreTrainedPolicy._save_pretrained``, whose gate is internal because its collective gather and
its writes live in the same method.
"""
import logging
from importlib.resources import files
from pathlib import Path
from tempfile import TemporaryDirectory
from typing import TYPE_CHECKING, Any
import torch.distributed as dist
from huggingface_hub import HfApi, ModelCard, ModelCardData, snapshot_download
from huggingface_hub import HfApi, snapshot_download
from torch.optim import Optimizer
from torch.optim.lr_scheduler import LRScheduler
from lerobot.__version__ import __version__
from lerobot.configs.policies import PreTrainedConfig
from lerobot.configs.rewards import RewardModelConfig
from lerobot.configs.train import TrainPipelineConfig
from lerobot.distributed.checkpoint import (
is_sharded_module,
load_sharded_model,
load_sharded_optimizer,
save_sharded_model,
save_sharded_optimizer,
)
from lerobot.distributed.utils import is_main_process
from lerobot.optim import (
load_optimizer_state,
load_optimizer_state_dict,
load_scheduler_state,
save_optimizer_state,
save_scheduler_state,
@@ -65,63 +40,49 @@ from lerobot.utils.hub import find_latest_hub_checkpoint
from lerobot.utils.io_utils import load_json, write_json
from lerobot.utils.random_utils import load_rng_state, save_rng_state
if TYPE_CHECKING:
from accelerate import Accelerator
from lerobot.datasets.dataset_metadata import LeRobotDatasetMetadata
from lerobot.rewards.pretrained import PreTrainedRewardModel
def get_step_identifier(step: int, total_steps: int) -> str:
"""Format a step number as the zero-padded identifier used for checkpoint directory names.
Args:
step (int): The training step to format.
total_steps (int): The total number of training steps; sets the padding width
(minimum 6 digits).
Returns:
str: The zero-padded step identifier, e.g. `"005000"`.
"""
num_digits = max(6, len(str(total_steps)))
return f"{step:0{num_digits}d}"
def get_step_checkpoint_dir(output_dir: Path, total_steps: int, step: int) -> Path:
"""Returns the checkpoint sub-directory corresponding to the step number.
Args:
output_dir (Path): The training run's output directory.
total_steps (int): The total number of training steps; sets the identifier padding.
step (int): The training step of the checkpoint.
Returns:
Path: The checkpoint step directory, `output_dir/checkpoints/<step-identifier>`.
"""
"""Returns the checkpoint sub-directory corresponding to the step number."""
step_identifier = get_step_identifier(step, total_steps)
return output_dir / CHECKPOINTS_DIR / step_identifier
def should_save_checkpoint(step: int, save_freq: int, total_steps: int) -> bool:
"""Whether a checkpoint should be saved at ``step``.
A checkpoint is saved every ``save_freq`` steps and always after the final step. A
non-positive ``save_freq`` disables periodic saving (only the final checkpoint is
written), mirroring how ``log_freq``/``eval_freq`` treat non-positive values and
avoiding a ``ZeroDivisionError`` from ``step % 0``.
"""
return (save_freq > 0 and step % save_freq == 0) or step == total_steps
def save_training_step(
step: int, save_dir: Path, num_processes: int | None = None, batch_size: int | None = None
) -> None:
state: dict = {"step": step}
# num_processes and batch_size are recorded so a resumed run can detect a changed world size or
# batch size: the sampler's resume offset is computed from the (num_processes, batch_size) that
# produced `step`, since both scale how many sampler positions a step consumes (see
# compute_sampler_state).
if num_processes is not None:
state["num_processes"] = num_processes
if batch_size is not None:
state["batch_size"] = batch_size
write_json(state, save_dir / TRAINING_STEP)
def update_last_checkpoint(checkpoint_dir: Path) -> None:
"""Point the `last` symlink in the checkpoints directory at the given checkpoint.
def load_training_step(save_dir: Path) -> int:
training_step = load_json(save_dir / TRAINING_STEP)
return training_step["step"]
Any existing `last` symlink is replaced. The link target is relative to the checkpoints
directory, so the tree stays valid when the run directory is moved.
Args:
checkpoint_dir (Path): The checkpoint step directory the `last` link should target.
"""
def load_training_num_processes(checkpoint_dir: Path) -> int | None:
"""World size recorded at checkpoint time, or None for checkpoints written before it was stored."""
return load_json(checkpoint_dir / TRAINING_STATE_DIR / TRAINING_STEP).get("num_processes")
def load_training_batch_size(checkpoint_dir: Path) -> int | None:
"""Per-process batch size recorded at checkpoint time, or None for older checkpoints."""
return load_json(checkpoint_dir / TRAINING_STATE_DIR / TRAINING_STEP).get("batch_size")
def update_last_checkpoint(checkpoint_dir: Path) -> Path:
last_checkpoint_dir = checkpoint_dir.parent / LAST_CHECKPOINT_LINK
if last_checkpoint_dir.is_symlink():
last_checkpoint_dir.unlink()
@@ -129,68 +90,6 @@ def update_last_checkpoint(checkpoint_dir: Path) -> None:
last_checkpoint_dir.symlink_to(relative_target)
# ---------------------------------------------------------------------------------------------
# training_step.json
# ---------------------------------------------------------------------------------------------
def save_training_metadata(step: int, save_dir: Path, cfg: TrainPipelineConfig) -> None:
"""Record the step counter plus everything a resume needs to reason about topology changes.
`step` counts loop iterations (= micro-batches), so
the sampler resume offset is `step x batch_size x dp_world_size` with no grad-accum factor.
`grad_accum_steps` and the parallelism snapshot are recorded so a resume can warn precisely
when the optimizer-update cadence or the sharding topology changed.
Args:
step (int): The training step (micro-batch counter) to record.
save_dir (Path): The `training_state/` directory to write `training_step.json` into.
cfg (TrainPipelineConfig): The training config whose batch size, gradient-accumulation,
and parallelism settings are snapshotted alongside the step.
"""
state: dict[str, Any] = {
"step": step,
"dp_world_size": cfg.parallelism.dp_world_size,
"batch_size": cfg.batch_size,
"grad_accum_steps": cfg.accelerator.gradient_accumulation.steps,
"parallelism": {
"dp_replicate": cfg.parallelism.dp_replicate,
"dp_shard": cfg.parallelism.dp_shard,
"ring_degree": cfg.parallelism.context_parallel.ring_degree,
"ulysses_degree": cfg.parallelism.context_parallel.ulysses_degree,
},
}
write_json(state, save_dir / TRAINING_STEP)
def load_training_metadata(training_state_dir: Path) -> dict[str, Any]:
"""Read everything `save_training_metadata` recorded, in a single pass.
Every key is always present: fields a checkpoint predates come back as None, so a caller
reading `metadata["batch_size"]` gets a KeyError on a typo rather than a silent None.
Args:
training_state_dir (Path): The checkpoint's `training_state/` directory.
Returns:
dict[str, Any]: `step` plus the `dp_world_size`, `batch_size`, `grad_accum_steps` and
`parallelism` snapshot recorded alongside it (None where not recorded).
"""
state = load_json(training_state_dir / TRAINING_STEP)
return {
"step": int(state["step"]),
"dp_world_size": state.get("dp_world_size", state.get("num_processes")),
"batch_size": state.get("batch_size"),
"grad_accum_steps": state.get("grad_accum_steps"),
"parallelism": state.get("parallelism"),
}
# ---------------------------------------------------------------------------------------------
# Checkpoint save
# ---------------------------------------------------------------------------------------------
def save_checkpoint(
checkpoint_dir: Path,
step: int,
@@ -200,301 +99,192 @@ def save_checkpoint(
scheduler: LRScheduler | None = None,
preprocessor: PolicyProcessorPipeline | None = None,
postprocessor: PolicyProcessorPipeline | None = None,
accelerator: "Accelerator | None" = None,
num_processes: int | None = None,
batch_size: int | None = None,
model_state_dict: dict | None = None,
optim_state_dict: dict | None = None,
) -> None:
"""This function creates the following directory structure:
005000/ # training step at checkpoint
pretrained_model/
config.json # policy config
model.safetensors # policy weights (checkpoint_format ∈ {safetensors, safetensors_dcp}, or any non-sharded run)
pytorch_model_fsdp_0/ # DCP model shards (checkpoint_format ∈ {dcp, safetensors_dcp})
model.safetensors # policy weights
train_config.json # train config
policy_preprocessor.json # preprocessor config (if preprocessor provided)
policy_preprocessor_step_*.safetensors # state of the stateful preprocessor steps
policy_postprocessor.json # postprocessor config (if postprocessor provided)
policy_postprocessor_step_*.safetensors # state of the stateful postprocessor steps
processor.json # processor config (if preprocessor provided)
step_*.safetensors # processor state files (if any)
training_state/
optimizer_param_groups.json # optimizer param groups (non-sharded runs)
optimizer_state.safetensors # optimizer state (non-sharded runs)
optimizer_0/ # DCP optimizer shards (sharded runs)
optimizer_param_groups.json # optimizer param groups
optimizer_state.safetensors # optimizer state
rng_state.safetensors # rng states
scheduler_state.json # scheduler state (if scheduler provided)
training_step.json # training step + dp_world_size/batch_size/grad_accum + topology
Collective: MUST be called on every rank. Rank-0-only writes are gated internally, so the
call site needs no rank branches.
scheduler_state.json # scheduler state
training_step.json # training step
Args:
checkpoint_dir (Path): The checkpoint step directory to write (e.g. `.../checkpoints/005000`).
step (int): The training step at that checkpoint.
cfg (TrainPipelineConfig): The training config used for this run.
step (int): The training step at that checkpoint.
policy (PreTrainedPolicy): The policy to save.
optimizer (Optimizer): The optimizer to save the state from.
optimizer (Optimizer | None, optional): The optimizer to save the state from. Defaults to None.
scheduler (LRScheduler | None, optional): The scheduler to save the state from. Defaults to None.
preprocessor (PolicyProcessorPipeline | None, optional): The preprocessor/pipeline to save.
preprocessor: The preprocessor/pipeline to save. Defaults to None.
postprocessor: The postprocessor/pipeline to save. Defaults to None.
num_processes (int | None, optional): Distributed world size to record for sample-exact
resume. Defaults to None (not recorded).
batch_size (int | None, optional): Per-process batch size to record for sample-exact
resume. Defaults to None (not recorded).
model_state_dict: Pre-gathered full (unsharded) model state dict. Required under FSDP,
where `policy.state_dict()` would return sharded tensors; the caller gathers it via a
cross-rank collective and passes it here so rank 0 can write it directly. It holds
FSDP's fp32 master weights and is saved as-is (the loader casts to the policy dtype on
read). When None (DDP / single-GPU), the model is saved the normal way. Defaults to None.
optim_state_dict: Pre-gathered full (unsharded) optimizer state dict. Required under FSDP
(gathered alongside `model_state_dict` via `gather_fsdp_state_dicts`); saved in the same
safetensors format as the single-GPU path. When None, `optimizer.state_dict()` is used.
Defaults to None.
postprocessor (PolicyProcessorPipeline | None, optional): The postprocessor/pipeline to save.
Defaults to None.
accelerator (Accelerator | None, optional): The accelerator the policy was prepared with;
used to unwrap the model and required on sharded runs, where it owns the DCP save
channels. Defaults to None (plain single-process saves).
"""
pretrained_dir = checkpoint_dir / PRETRAINED_MODEL_DIR
fmt = cfg.checkpoint_format
policy_to_save = accelerator.unwrap_model(policy) if accelerator is not None else policy
sharded = is_sharded_module(policy_to_save)
# -- model artifact(s): the two collective-capable calls ----------------------------------
policy.save_pretrained(pretrained_dir, state_dict=model_state_dict)
cfg.save_pretrained(pretrained_dir)
if cfg.peft is not None:
# PeftModel.save_pretrained is an external API with no internal rank gate, and the
# adapters are replicated (PEFT x sharded is rejected at validation): main rank writes.
if is_main_process():
policy_to_save.save_pretrained(pretrained_dir)
elif fmt.wants_safetensors or not sharded:
# Collective when sharded (full gather); writes happen on the main process only in all
# multi-rank layouts (the gate lives inside _save_pretrained, next to its collective gather).
policy_to_save.save_pretrained(pretrained_dir)
if fmt.wants_dcp and sharded:
save_sharded_model(accelerator, policy_to_save, pretrained_dir)
# -- sidecar configs: ONE gate for the whole contiguous rank-0-only region ----------------
if is_main_process():
if fmt.wants_dcp and not fmt.wants_safetensors:
# save_pretrained did not run: keep the DCP-only checkpoint self-describing.
policy_to_save.config.save_pretrained(pretrained_dir)
cfg.save_pretrained(pretrained_dir)
if cfg.peft is not None:
# PEFT's save_pretrained writes only adapter weights + config; the policy config
# needed to reload the base model is written explicitly.
policy_to_save.config.save_pretrained(pretrained_dir)
if preprocessor is not None:
preprocessor.save_pretrained(pretrained_dir)
if postprocessor is not None:
postprocessor.save_pretrained(pretrained_dir)
# When using PEFT, policy.save_pretrained will only write the adapter weights + config, not the
# policy config which we need for loading the model. In this case we'll write it ourselves.
policy.config.save_pretrained(pretrained_dir)
if preprocessor is not None:
preprocessor.save_pretrained(pretrained_dir)
if postprocessor is not None:
postprocessor.save_pretrained(pretrained_dir)
save_training_state(
checkpoint_dir, step, cfg, optimizer, scheduler, accelerator, sharded=sharded, model=policy_to_save
checkpoint_dir,
step,
optimizer,
scheduler,
num_processes=num_processes,
batch_size=batch_size,
optim_state_dict=optim_state_dict,
)
if accelerator is not None:
accelerator.wait_for_everyone()
def save_training_state(
checkpoint_dir: Path,
step: int,
cfg: TrainPipelineConfig,
optimizer: Optimizer | dict[str, Optimizer] | None = None,
train_step: int,
optimizer: Optimizer | None = None,
scheduler: LRScheduler | None = None,
accelerator: "Accelerator | None" = None,
*,
sharded: bool = False,
model: PreTrainedPolicy | None = None,
num_processes: int | None = None,
batch_size: int | None = None,
optim_state_dict: dict | None = None,
) -> None:
"""Write training_state/. Collective under sharding: call on every rank.
"""
Saves the training step, optimizer state, scheduler state, and rng state.
Args:
checkpoint_dir (Path): The checkpoint step directory; `training_state/` is created inside it.
step (int): The training step at that checkpoint.
cfg (TrainPipelineConfig): The training config used for this run (its topology and
accumulation settings are recorded in `training_step.json`).
optimizer (Optimizer | dict[str, Optimizer] | None, optional): The optimizer(s) to save
the state from. Defaults to None.
scheduler (LRScheduler | None, optional): The scheduler to save the state from.
save_dir (Path): The directory to save artifacts to.
train_step (int): Current training step.
optimizer (Optimizer | None, optional): The optimizer from which to save the state_dict.
Defaults to None.
accelerator (Accelerator | None, optional): Required when `sharded` is True it owns
the DCP optimizer save channel. Defaults to None.
sharded (bool): The model's sharding state, computed once in `save_checkpoint` and
threaded here so the two sites cannot disagree. Defaults to False.
model (PreTrainedPolicy | None, optional): Required only for the sharded optimizer
channel: torch's optimizer DCP APIs are model-coupled (the state dict is keyed by
model FQNs), so accelerate's `save_fsdp_optimizer` needs the sharded module
alongside the optimizer. Defaults to None.
scheduler (LRScheduler | None, optional): The scheduler from which to save the state_dict.
Defaults to None.
num_processes (int | None, optional): Distributed world size to record. Defaults to None.
batch_size (int | None, optional): Per-process batch size to record. Defaults to None.
optim_state_dict: Pre-gathered full optimizer state dict (for FSDP). Saved instead of
`optimizer.state_dict()` when provided. Defaults to None.
"""
save_dir = checkpoint_dir / TRAINING_STATE_DIR
# All ranks: the directory must exist before the DCP optimizer collective writes into it
# (exist_ok makes the concurrent mkdir race-free on shared filesystems).
save_dir.mkdir(parents=True, exist_ok=True)
if optimizer is not None and sharded:
if accelerator is None or model is None:
raise ValueError("Saving a sharded optimizer state requires the accelerator and model.")
# Collective — all ranks write their DCP shards into optimizer_0/.
save_sharded_optimizer(accelerator, optimizer, model, save_dir)
if is_main_process(): # ONE grouped gate for the whole rank-0-only region
save_training_metadata(step, save_dir, cfg)
save_rng_state(save_dir)
if scheduler is not None:
save_scheduler_state(scheduler, save_dir)
if optimizer is not None and not sharded:
save_optimizer_state(optimizer, save_dir)
save_training_step(train_step, save_dir, num_processes=num_processes, batch_size=batch_size)
save_rng_state(save_dir)
if optimizer is not None:
save_optimizer_state(optimizer, save_dir, optim_state_dict=optim_state_dict)
if scheduler is not None:
save_scheduler_state(scheduler, save_dir)
# ---------------------------------------------------------------------------------------------
# Two-phase resume
# ---------------------------------------------------------------------------------------------
def resume_before_prepare(cfg: TrainPipelineConfig) -> int:
"""Phase 1 — before `accelerator.prepare()`: restore RNG and return the step counter.
Pure loaders only. The sampler resume offset is *derived* from the returned step inside the
dataloader factory, and everything bound to sharded objects (model DCP shards, optimizer,
scheduler) loads in `resume_after_prepare`.
def load_training_state(
checkpoint_dir: Path, optimizer: Optimizer, scheduler: LRScheduler | None, load_optimizer: bool = True
) -> tuple[int, Optimizer, LRScheduler | None]:
"""
Loads the training step, optimizer state, scheduler state, and rng state.
This is used to resume a training run.
Args:
cfg (TrainPipelineConfig): The resumed training config; `cfg.checkpoint_path` locates
the checkpoint to restore from.
checkpoint_dir (Path): The checkpoint directory. Should contain a 'training_state' dir.
optimizer (Optimizer): The optimizer to load the state_dict to.
scheduler (LRScheduler | None): The scheduler to load the state_dict to (can be None).
load_optimizer (bool, optional): Whether to load the optimizer state from disk. Defaults to
True. Set to False under FSDP, where the sharded optimizer state must be loaded after
`accelerator.prepare()` via `load_fsdp_optimizer_state` (the optimizer is returned
untouched here).
Raises:
NotADirectoryError: If 'checkpoint_dir' doesn't contain a 'training_state' dir
Returns:
int: The training step recorded in the checkpoint (micro-batch counter).
Raises:
NotADirectoryError: If the checkpoint has no `training_state/` directory.
ValueError: If the resumed topology crosses the sharded/non-sharded boundary relative
to the one recorded in the checkpoint.
tuple[int, Optimizer, LRScheduler | None]: training step, optimizer and scheduler with their
state_dict loaded.
"""
training_state_dir = cfg.checkpoint_path / TRAINING_STATE_DIR
training_state_dir = checkpoint_dir / TRAINING_STATE_DIR
if not training_state_dir.is_dir():
raise NotADirectoryError(training_state_dir)
metadata = load_training_metadata(training_state_dir)
_guard_resume_changes(cfg, metadata)
load_rng_state(training_state_dir)
return metadata["step"]
def _guard_resume_changes(cfg: TrainPipelineConfig, metadata: dict[str, Any]) -> None:
"""Check the resumed run settings against the ones recorded in the checkpoint.
Two tiers, both driven by the checkpoint's recorded parallelism snapshot:
- **Hard error** when the resume crosses the sharded/non-sharded boundary in either
direction: the checkpoint's training-state artifacts only support resuming on the same
kind of topology (resharding works across sizes, not across kinds). Checkpoints without
a recorded snapshot skip this check.
- **One warning** naming every other recorded setting that differs those changes are
legal (DCP reshards weights and optimizer state across topologies and the sampler offset
adapts), but a changed ``grad_accum_steps`` shifts the optimizer-update cadence, so the
resume says precisely what differs. The sampler-exactness warnings
(``dp_world_size``/``batch_size``) live with the sampler math in the dataloader factory.
Args:
cfg (TrainPipelineConfig): The resumed training config, compared against the settings
recorded in the checkpoint.
metadata (dict[str, Any]): The checkpoint's recorded training metadata, as returned by
`load_training_metadata`.
Raises:
ValueError: If the checkpoint records a sharded topology and the resumed run is
non-sharded, or vice versa.
"""
snapshot = metadata["parallelism"]
if snapshot is not None:
recorded_sharded = (
snapshot.get("dp_shard", 1) != 1
or snapshot.get("ring_degree", 1) * snapshot.get("ulysses_degree", 1) > 1
)
if recorded_sharded != cfg.parallelism.is_sharded:
raise ValueError(
f"Cannot resume: the checkpoint was written with a "
f"{'sharded' if recorded_sharded else 'non-sharded'} topology "
f"(dp_replicate={snapshot.get('dp_replicate')}, dp_shard={snapshot.get('dp_shard')}) "
f"but this run is {'sharded' if cfg.parallelism.is_sharded else 'non-sharded'} "
f"(dp_replicate={cfg.parallelism.dp_replicate}, dp_shard={cfg.parallelism.dp_shard})."
)
recorded = {
"grad_accum_steps": (
metadata["grad_accum_steps"],
cfg.accelerator.gradient_accumulation.steps,
),
}
if snapshot is not None:
recorded.update(
{
"dp_replicate": (snapshot.get("dp_replicate"), cfg.parallelism.dp_replicate),
"dp_shard": (snapshot.get("dp_shard"), cfg.parallelism.dp_shard),
"ring_degree": (
snapshot.get("ring_degree"),
cfg.parallelism.context_parallel.ring_degree,
),
"ulysses_degree": (
snapshot.get("ulysses_degree"),
cfg.parallelism.context_parallel.ulysses_degree,
),
}
)
changed = [f"{key}: {was} -> {now}" for key, (was, now) in recorded.items() if was not in (None, now)]
if changed and is_main_process():
logging.warning(
"Resuming with settings that differ from the checkpoint: " + "; ".join(changed) + ". "
"Topology changes reshard safely via DCP; a changed grad_accum_steps shifts the "
"optimizer-update cadence (the step counter keeps counting micro-batches)."
)
def resume_after_prepare(
cfg: TrainPipelineConfig,
accelerator: "Accelerator",
policy: PreTrainedPolicy,
optimizer: Optimizer | dict[str, Optimizer],
scheduler: LRScheduler | None,
) -> None:
"""Phase 2 — after `accelerator.prepare()`: model (DCP) -> optimizer -> scheduler.
Collective under sharding: call on every rank. The model-weight source follows the
checkpoint's own recorded `checkpoint_format` (on resume, `cfg` was parsed from the
checkpoint's train_config.json): DCP-bearing formats load shards here into the prepared
model (whose construction skipped the safetensors load); the safetensors format was already
loaded by `from_pretrained` before sharding no model step here.
Args:
cfg (TrainPipelineConfig): The resumed training config; `cfg.checkpoint_path` locates
the checkpoint and `cfg.checkpoint_format` selects the model-weight source.
accelerator (Accelerator): The accelerator the policy was prepared with; it unwraps the
model and owns the DCP load channels.
policy (PreTrainedPolicy): The prepared (possibly sharded) policy to load weights into.
optimizer (Optimizer | dict[str, Optimizer]): The prepared optimizer(s) to restore.
scheduler (LRScheduler | None): The scheduler to restore, or None if the run has none.
Raises:
FileNotFoundError: If the checkpoint format declares DCP model shards but the shard
directory is missing (e.g. it was pruned before upload).
"""
checkpoint_dir = cfg.checkpoint_path
pretrained_dir = checkpoint_dir / PRETRAINED_MODEL_DIR
training_state_dir = checkpoint_dir / TRAINING_STATE_DIR
unwrapped = accelerator.unwrap_model(policy)
sharded = is_sharded_module(unwrapped)
if cfg.checkpoint_format.wants_dcp:
from accelerate.utils.constants import FSDP_MODEL_NAME
dcp_dir = pretrained_dir / f"{FSDP_MODEL_NAME}_0"
if not dcp_dir.is_dir():
raise FileNotFoundError(
f"checkpoint_format={cfg.checkpoint_format.value} declares DCP model shards, "
f"but {dcp_dir} is missing. If the shards were pruned, convert what remains "
"with `lerobot-convert-dcp` or resume from a safetensors checkpoint."
)
load_sharded_model(accelerator, unwrapped, pretrained_dir)
if sharded:
# Requires the prepared optimizer: FSDP2's prepare rebinds param groups to DTensors but
# never migrates optimizer.state — DCP reshards it here (works across topology changes).
load_sharded_optimizer(accelerator, optimizer, unwrapped, training_state_dir)
else:
load_optimizer_state(optimizer, training_state_dir)
step = load_training_step(training_state_dir)
if load_optimizer:
optimizer = load_optimizer_state(optimizer, training_state_dir)
if scheduler is not None:
load_scheduler_state(scheduler, training_state_dir)
scheduler = load_scheduler_state(scheduler, training_state_dir)
return step, optimizer, scheduler
# ---------------------------------------------------------------------------------------------
# Hub: checkpoint push (resume artifact) and publishing (distribution artifact)
# ---------------------------------------------------------------------------------------------
def gather_fsdp_state_dicts(model, optimizer) -> tuple[dict, dict]:
"""Gather the full (unsharded) model and optimizer state dicts under FSDP.
`model.state_dict()` and `FSDP.optim_state_dict(...)` are cross-rank collectives, so this must be
called on *every* rank with the prepared (FSDP-wrapped) `model` and `optimizer`. With
`rank0_only=True` and `offload_to_cpu=True`, every rank runs the all-gather but only rank 0
materializes the full dicts (the others get empty dicts) and they are kept on CPU to bound GPU
memory. The returned optimizer state dict is keyed by parameter FQNs and is world-size
independent; `load_fsdp_optimizer_state` reshards it on resume.
Returns:
(model_state_dict, optim_state_dict): full dicts on rank 0, empty dicts on other ranks.
"""
from torch.distributed.fsdp import (
FullOptimStateDictConfig,
FullStateDictConfig,
FullyShardedDataParallel as FSDP, # noqa F401
StateDictType,
)
state_cfg = FullStateDictConfig(offload_to_cpu=True, rank0_only=True)
optim_cfg = FullOptimStateDictConfig(offload_to_cpu=True, rank0_only=True)
with FSDP.state_dict_type(model, StateDictType.FULL_STATE_DICT, state_cfg, optim_cfg):
model_state_dict = model.state_dict()
optim_state_dict = FSDP.optim_state_dict(model, optimizer)
return model_state_dict, optim_state_dict
def load_fsdp_optimizer_state(model, optimizer, checkpoint_dir: Path) -> None:
"""Load the FSDP optimizer state (saved as safetensors) and reshard it into the optimizer.
This is a cross-rank collective and must be called on every rank *after* `accelerator.prepare()`
with the prepared (FSDP-wrapped) `model` and `optimizer`. The saved state is the full,
world-size-independent optimizer state (keyed by parameter FQNs); `FSDP.optim_state_dict_to_load`
reshards it to the current FSDP topology, so resume on a different number of GPUs works.
"""
from torch.distributed.fsdp import (
FullOptimStateDictConfig,
FullStateDictConfig,
FullyShardedDataParallel as FSDP, # noqa F401
StateDictType,
)
# Every rank reads the same full state from the (shared) checkpoint dir, so rank0_only=False.
full_osd = load_optimizer_state_dict(checkpoint_dir / TRAINING_STATE_DIR)
state_cfg = FullStateDictConfig(rank0_only=False)
optim_cfg = FullOptimStateDictConfig(rank0_only=False)
with FSDP.state_dict_type(model, StateDictType.FULL_STATE_DICT, state_cfg, optim_cfg):
sharded_osd = FSDP.optim_state_dict_to_load(model=model, optim=optimizer, optim_state_dict=full_osd)
optimizer.load_state_dict(sharded_osd)
def push_checkpoint_to_hub(
@@ -510,16 +300,6 @@ def push_checkpoint_to_hub(
The model repo is created idempotently, and the commit is tagged with the
checkpoint step so a checkpoint can be recovered with
--policy.pretrained_revision=<step> instead of a commit sha.
The directory is uploaded verbatim including DCP shards under the DCP formats: this tree
exists for *resume*, not distribution, and `resolve_resume_checkpoint` downloads it back
symmetrically.
Args:
checkpoint_dir (Path): The local checkpoint step directory to upload.
repo_id (str): The Hub model repo to push to (created idempotently if missing).
private (bool | None): Whether a newly created repo should be private. Defaults to
None (public unless the organization's default is private).
"""
api = HfApi()
api.create_repo(repo_id=repo_id, repo_type="model", private=private, exist_ok=True)
@@ -547,16 +327,6 @@ def resolve_resume_checkpoint(repo_id: str, output_dir: Path) -> Path:
into `output_dir/checkpoints/<step>/`, recreate the local `last` symlink, and return that local
checkpoint dir. Used to resume training from the Hub on a machine (or HF Jobs pod) that does not
have the original local run dir.
Args:
repo_id (str): The Hub model repo holding `checkpoints/<step>/` subtrees.
output_dir (Path): The local run directory to download the checkpoint into.
Returns:
Path: The local checkpoint step directory, `output_dir/checkpoints/<step>`.
Raises:
FileNotFoundError: If the repo contains no checkpoints under `checkpoints/`.
"""
latest = find_latest_hub_checkpoint(repo_id)
if latest is None:
@@ -573,214 +343,3 @@ def resolve_resume_checkpoint(repo_id: str, output_dir: Path) -> Path:
checkpoint_dir = output_dir / latest
update_last_checkpoint(checkpoint_dir)
return checkpoint_dir
def publish_trained_model(
cfg: TrainPipelineConfig,
model: "PreTrainedPolicy | PreTrainedRewardModel",
preprocessor: PolicyProcessorPipeline | None,
postprocessor: PolicyProcessorPipeline | None,
dataset_meta: "LeRobotDatasetMetadata | None",
*,
peft_model: Any | None = None,
) -> None:
"""Publish the complete training bundle as a distributable model repo.
Collective-safe: call on ALL ranks the model commit gathers sharded weights through
`save_pretrained`; uploads happen on the main process only (gated inside
`HubMixin.push_to_hub` and here). Commits, in order: (1) the model (skipped for PEFT
adapters replace full weights), (2) the preprocessor, (3) the postprocessor, (4) the bundle
sidecar: README.md model card + train_config.json (+ adapter weights and the wrapped
policy's config in the PEFT case). Every commit uploads a freshly assembled directory, so
a published repo carries only the distributable artifacts.
Args:
cfg (TrainPipelineConfig): The training config; saved as `train_config.json` and used
to render the model card.
model (PreTrainedPolicy | PreTrainedRewardModel): The trained model to publish; its
config supplies the target repo id, visibility, license, and tags.
preprocessor (PolicyProcessorPipeline | None): The preprocessor pipeline to publish
alongside the model, if any.
postprocessor (PolicyProcessorPipeline | None): The postprocessor pipeline to publish
alongside the model, if any.
dataset_meta (LeRobotDatasetMetadata | None): Dataset metadata for the model card, if
available.
peft_model (Any | None): The PEFT wrapper when training adapters; its adapter weights
replace the full model weights in the published repo. Defaults to None.
Raises:
ValueError: If the model config carries no repo id (`--policy.repo_id`).
"""
model_cfg = model.config
repo_id = model_cfg.repo_id
if not repo_id:
raise ValueError("Publishing requires a repo id (--policy.repo_id).")
ignore = ["*.tmp", "*.log"]
if peft_model is None:
# Calls are made on the exact objects that own each method (never through PEFT's
# attribute forwarding), so the peft branch below never touches this path.
model.push_to_hub(repo_id, private=model_cfg.private, ignore_patterns=ignore)
if preprocessor is not None:
preprocessor.push_to_hub(repo_id, private=model_cfg.private)
if postprocessor is not None:
postprocessor.push_to_hub(repo_id, private=model_cfg.private)
if is_main_process():
api = HfApi()
repo_id = api.create_repo(repo_id=repo_id, private=model_cfg.private, exist_ok=True).repo_id
with TemporaryDirectory(ignore_cleanup_errors=True) as tmp:
saved_path = Path(tmp) / repo_id
saved_path.mkdir(parents=True, exist_ok=True)
if peft_model is not None:
peft_model.save_pretrained(saved_path) # adapter weights + adapter config
model.config.save_pretrained(saved_path) # PEFT cannot write the policy config
card = generate_model_card(model_cfg, cfg=cfg, dataset_meta=dataset_meta)
card.save(str(saved_path / "README.md"))
cfg.save_pretrained(saved_path) # train_config.json
commit_info = api.upload_folder(
repo_id=repo_id,
repo_type="model",
folder_path=saved_path,
commit_message="Upload model card and train config",
allow_patterns=["*.safetensors", "*.json", "*.yaml", "*.md"],
ignore_patterns=ignore,
)
# Contract: lerobot.jobs.hf.submit_to_hf watches for this exact "Model pushed to <url>"
# line to end a remote run early. Keep the wording and URL format in sync.
logging.info(f"Model pushed to {commit_info.repo_url.url}")
if dist.is_initialized():
dist.barrier()
# ---------------------------------------------------------------------------------------------
# Model card
# ---------------------------------------------------------------------------------------------
_BASE_MODEL_MAPPING = {
"smolvla": "lerobot/smolvla_base",
"pi0": "lerobot/pi0_base",
"pi05": "lerobot/pi05_base",
"pi0_fast": "lerobot/pi0fast-base",
"xvla": "lerobot/xvla-base",
}
def build_card_context(
cfg: TrainPipelineConfig | None,
dataset_meta: "LeRobotDatasetMetadata | None",
input_features: dict | None,
output_features: dict | None,
) -> dict:
"""Collect optional data for the model-card template.
Returns plain values only (no Markdown) the template in
``lerobot/templates/lerobot_modelcard_template.md`` decides how and whether to show
each one. Everything is best-effort: anything unavailable is left empty/None and the
template simply skips that section, so this never breaks a Hub push.
Args:
cfg (TrainPipelineConfig | None): The training config supplying the training section,
if available.
dataset_meta (LeRobotDatasetMetadata | None): Dataset metadata supplying the dataset,
robot-type, and camera sections, if available.
input_features (dict | None): The policy's input feature declarations, if any.
output_features (dict | None): The policy's output feature declarations, if any.
Returns:
dict: Template context with `training`, `input_features`, `output_features`,
`dataset`, `robot_type`, and `cameras` entries; unavailable pieces stay
empty/None.
"""
context = {
"training": None,
"input_features": input_features or {},
"output_features": output_features or {},
"dataset": None,
"robot_type": None,
"cameras": [],
}
if cfg is not None:
optimizer = getattr(cfg, "optimizer", None)
context["training"] = {
"steps": cfg.steps,
"batch_size": cfg.batch_size,
"seed": cfg.seed,
"optimizer": getattr(optimizer, "type", None) if optimizer else None,
"lr": getattr(optimizer, "lr", None) if optimizer else None,
"lerobot_version": __version__,
}
if dataset_meta is not None:
context["dataset"] = {
"repo_id": dataset_meta.repo_id,
"episodes": dataset_meta.total_episodes,
"frames": dataset_meta.total_frames,
"fps": dataset_meta.fps,
"tasks": [str(task) for task in dataset_meta.tasks.index],
}
context["robot_type"] = dataset_meta.robot_type
context["cameras"] = [key.split(".")[-1] for key in dataset_meta.camera_keys]
return context
def generate_model_card(
model_cfg: PreTrainedConfig | RewardModelConfig,
cfg: TrainPipelineConfig | None = None,
dataset_meta: "LeRobotDatasetMetadata | None" = None,
) -> ModelCard:
"""Render the LeRobot model card for a trained policy or reward model.
A free function on purpose: every template variable comes from arguments the model
config, the training config, and the dataset metadata none from a live model, so a card
can also be rendered from a checkpoint's `config.json` alone (see `lerobot-convert-dcp`).
The config type selects the template: reward models get the reward-model card, policies the
policy card with the training/dataset sections.
Args:
model_cfg (PreTrainedConfig | RewardModelConfig): The model config providing type,
license, tags, repo id, and for policies the feature declarations.
cfg (TrainPipelineConfig | None, optional): The training config for the training and
dataset card sections. Defaults to None.
dataset_meta (LeRobotDatasetMetadata | None, optional): Dataset metadata for the
dataset card sections. Defaults to None.
Returns:
ModelCard: The rendered and validated LeRobot model card.
"""
model_type = model_cfg.type
base_model = _BASE_MODEL_MAPPING.get(model_type)
if isinstance(model_cfg, RewardModelConfig):
tags = {"robotics", "lerobot", "reward-model", model_type}
template_card = (
files("lerobot.templates")
.joinpath("lerobot_rewardmodel_modelcard_template.md")
.read_text("utf-8")
)
context: dict[str, Any] = {} # the reward template renders from card_data alone
else:
tags = {"robotics", "lerobot", model_type}
template_card = (
files("lerobot.templates").joinpath("lerobot_modelcard_template.md").read_text("utf-8")
)
context = build_card_context(cfg, dataset_meta, model_cfg.input_features, model_cfg.output_features)
# Used by the template to pre-fill commands and the "Fine-tuned from" line.
context["policy_repo_id"] = model_cfg.repo_id
context["base_model"] = base_model
card_data = ModelCardData(
license=model_cfg.license or "apache-2.0",
library_name="lerobot",
pipeline_tag="robotics",
tags=list(tags.union(model_cfg.tags or [])),
model_name=model_type,
datasets=cfg.dataset.repo_id if cfg is not None else None,
base_model=base_model,
)
card = ModelCard.from_template(card_data, template_str=template_card, **context)
card.validate()
return card
-273
View File
@@ -1,273 +0,0 @@
#!/usr/bin/env python
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Execution-runtime configuration: everything handed to (or applied by) the `Accelerator`.
Each sub-config mirrors the plain-typed subset of the corresponding accelerate object and
builds it at runtime (the way ``OptimizerConfig.build()`` constructs a ``torch.optim.Optimizer``),
so the whole tree round-trips through the CLI and ``train_config.json`` and parsing a config
never imports accelerate.
"""
from dataclasses import dataclass, field
from enum import Enum
from typing import TYPE_CHECKING
from lerobot.configs.parallelism import ParallelismConfig
if TYPE_CHECKING:
from accelerate import Accelerator
from accelerate.utils import (
DistributedDataParallelKwargs,
FullyShardedDataParallelPlugin,
GradientAccumulationPlugin,
)
@dataclass
class FSDPConfig:
"""Mirror of the `FullyShardedDataParallelPlugin` subset LeRobot supports (FSDP2 only).
Exactly one wrap policy applies: `wrap_modules` (module *class names* forming the FSDP
units and, later, the activation-checkpointing units) or `min_num_params` (size-based).
When both are None, the policy's own `_fsdp_wrap_modules` declaration is used; a run where
no wrap source exists at all fails loudly rather than silently wrapping only the root.
"""
reshard_after_forward: bool = True
wrap_modules: list[str] | None = None
min_num_params: int | None = None
cpu_offload: bool = False
# Regex matched against module FQNs to exclude their parameters from sharding.
ignored_modules: str | None = None
def __post_init__(self) -> None:
"""Validate the wrap-policy fields.
Raises:
ValueError: If both ``wrap_modules`` and ``min_num_params`` are set (they are
mutually exclusive wrap policies), or if ``min_num_params`` is < 1.
"""
if self.wrap_modules is not None and self.min_num_params is not None:
raise ValueError(
"fsdp.wrap_modules and fsdp.min_num_params are mutually exclusive wrap policies."
)
if self.min_num_params is not None and self.min_num_params < 1:
raise ValueError(f"fsdp.min_num_params must be >= 1, got {self.min_num_params}.")
def build_plugin(self) -> "FullyShardedDataParallelPlugin":
"""Build the FSDP2 plugin for `Accelerator(fsdp_plugin=...)`.
Returns:
FullyShardedDataParallelPlugin: FSDP2 (`fsdp_version=2`) plugin carrying the
mirrored wrap policy, resharding, CPU-offload, and ignored-modules settings.
"""
from accelerate.utils import FullyShardedDataParallelPlugin
use_size_policy = self.min_num_params is not None
return FullyShardedDataParallelPlugin(
fsdp_version=2,
reshard_after_forward=self.reshard_after_forward,
auto_wrap_policy="size_based_wrap" if use_size_policy else "transformer_based_wrap",
# May legitimately still be None here: the policy-declared default is applied right
# before `accelerator.prepare()` (see lerobot.distributed.factory.set_fsdp_wrap_modules).
transformer_cls_names_to_wrap=list(self.wrap_modules) if self.wrap_modules else None,
min_num_params=self.min_num_params,
cpu_offload=self.cpu_offload,
ignored_modules=self.ignored_modules,
# state_dict_type stays at the FSDP2 default (SHARDED_STATE_DICT) and is never
# switched: full gathers go through torch's state-dict API, which does not consult
# the plugin. activation_checkpointing stays False: AC is LeRobot-owned.
)
@dataclass
class DDPConfig:
"""Mirror of the `DistributedDataParallelKwargs` subset LeRobot exposes."""
# Today's in-script default, kept for models with conditional computation.
find_unused_parameters: bool = True
gradient_as_bucket_view: bool = False
static_graph: bool = False
def build_kwargs_handler(self) -> "DistributedDataParallelKwargs":
"""Build the DDP kwargs handler for `Accelerator(kwargs_handlers=[...])`.
Returns:
DistributedDataParallelKwargs: Handler carrying the mirrored DDP fields, applied
by accelerate when it wraps the model in `DistributedDataParallel`.
"""
from accelerate.utils import DistributedDataParallelKwargs
return DistributedDataParallelKwargs(
find_unused_parameters=self.find_unused_parameters,
gradient_as_bucket_view=self.gradient_as_bucket_view,
static_graph=self.static_graph,
)
@dataclass
class GradientAccumulationConfig:
"""Mirror of the `GradientAccumulationPlugin` subset LeRobot supports.
Only the step count is a knob. ``sync_with_dataloader`` is pinned to False by
:meth:`build_plugin`: the training loop cycles a finite dataloader, so accelerate's default
of syncing at every dataloader end would force an optimizer step at every dataset epoch
boundary instead of every ``steps`` micro-batches.
"""
steps: int = 1
def __post_init__(self) -> None:
"""Validate the accumulation step count.
Raises:
ValueError: If ``steps`` is < 1.
"""
if self.steps < 1:
raise ValueError(f"gradient_accumulation.steps must be >= 1, got {self.steps}.")
def build_plugin(self) -> "GradientAccumulationPlugin":
"""Build the plugin for `Accelerator(gradient_accumulation_plugin=...)`.
A named plugin argument, not a `kwargs_handlers` entry: accelerate consumes this object
through its dedicated constructor parameter the `KwargsHandler` base class only lends
it `to_kwargs()`, so the consumption site, not the inheritance, decides its role.
Returns:
GradientAccumulationPlugin: Carrying the mirrored step count, with
``sync_with_dataloader=False`` pinned (see the class docstring).
"""
from accelerate.utils import GradientAccumulationPlugin
return GradientAccumulationPlugin(num_steps=self.steps, sync_with_dataloader=False)
@dataclass
class CompileConfig:
"""torch.compile knobs — a configured placeholder: wiring lands in a later round.
The setup-order contract it will follow is already fixed: compile applies
after CP dispatch install and activation checkpointing, before `fully_shard`, regionally
(per wrap unit) the only combination proven with FSDP2.
"""
enabled: bool = False
backend: str = "inductor"
mode: str | None = None
regional: bool = True
class ActivationCheckpointingMode(str, Enum):
NONE = "none"
FULL = "full"
@dataclass
class ActivationCheckpointingConfig:
"""Activation-checkpointing knobs — a configured placeholder: wiring lands in a later round.
AC units will coincide with the FSDP wrap units (one declaration drives both), applied
before torch.compile and `fully_shard` (the same ordering contract as CompileConfig).
"""
mode: ActivationCheckpointingMode = ActivationCheckpointingMode.NONE
@dataclass
class AcceleratorConfig:
"""Builds the `Accelerator` — the runtime counterpart of the `parallelism` topology.
`mixed_precision` selects accelerate-native AMP for DDP/single-GPU runs and the FSDP2
`MixedPrecisionPolicy` for sharded runs (accelerate derives it). Sharded runs support
"no" and "bf16" only; fp16's GradScaler-over-DTensor path is unverified and fails fast
at config validation.
"""
mixed_precision: str = "no"
gradient_accumulation: GradientAccumulationConfig = field(default_factory=GradientAccumulationConfig)
fsdp: FSDPConfig = field(default_factory=FSDPConfig)
ddp: DDPConfig = field(default_factory=DDPConfig)
compile: CompileConfig = field(default_factory=CompileConfig)
activation_checkpointing: ActivationCheckpointingConfig = field(
default_factory=ActivationCheckpointingConfig
)
def __post_init__(self) -> None:
"""Validate the accelerate-facing scalar fields.
Raises:
ValueError: If ``mixed_precision`` is not one of ``"no"``, ``"fp16"``, ``"bf16"``.
"""
if self.mixed_precision not in ("no", "fp16", "bf16"):
raise ValueError(
f"mixed_precision must be one of 'no', 'fp16', 'bf16', got {self.mixed_precision!r}."
)
def build(self, parallelism: ParallelismConfig, *, cpu: bool = False) -> "Accelerator":
"""Translate the mirrored fields into a ready `Accelerator` (call once per process).
`parallelism` must already be resolved against the world size. The degradation matrix
is encoded here and nowhere else: sharded -> FSDP2 (+HSDP via the accelerate
`ParallelismConfig` mesh), replicated-only -> DDP kwargs, single process -> plain.
Args:
parallelism (ParallelismConfig): The resolved process topology; selects which
accelerate path (FSDP2 mesh, DDP kwargs handler, or plain) is configured.
cpu (bool): Force CPU execution even when CUDA is available. Defaults to False.
Returns:
Accelerator: The configured accelerate entry point for this process.
"""
from accelerate import Accelerator
kwargs: dict = {
# LeRobot steps its scheduler manually once per training step; accelerate must not
# rescale scheduler stepping by num_processes.
"step_scheduler_with_optimizer": False,
"gradient_accumulation_plugin": self.gradient_accumulation.build_plugin(),
"mixed_precision": self.mixed_precision,
"cpu": cpu,
}
if parallelism.is_sharded:
kwargs["fsdp_plugin"] = self.fsdp.build_plugin()
kwargs["parallelism_config"] = _accelerate_parallelism_config(parallelism)
elif parallelism.is_replicated_only:
kwargs["kwargs_handlers"] = [self.ddp.build_kwargs_handler()]
return Accelerator(**kwargs)
def _accelerate_parallelism_config(parallelism: ParallelismConfig) -> object:
"""LeRobot topology -> accelerate `ParallelismConfig`.
CP is declared honestly (`cp_size = ring x ulysses`) so accelerate builds the canonical
mesh, folds CP into the FSDP shard group (`dp_shard_cp`), and duplicates batches within CP
groups. The ring/ulysses sub-structure stays private to `lerobot.distributed.ParallelDims`.
Args:
parallelism (ParallelismConfig): The resolved LeRobot topology to translate.
Returns:
object: The accelerate `ParallelismConfig` mirroring `dp_replicate`, `dp_shard`, and
the collapsed `cp_size` (annotated as `object` so importing this module never
imports accelerate).
"""
from accelerate.parallelism_config import ParallelismConfig as AccelerateParallelismConfig
return AccelerateParallelismConfig(
dp_replicate_size=parallelism.dp_replicate,
dp_shard_size=parallelism.dp_shard,
cp_size=parallelism.cp_size,
)
-26
View File
@@ -14,7 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
from dataclasses import dataclass, field
from lerobot.transforms import ImageTransformsConfig
@@ -22,8 +21,6 @@ from lerobot.utils.import_utils import get_safe_default_video_backend
from .video import DEFAULT_DEPTH_UNIT, DEPTH_METER_UNIT, DEPTH_MILLIMETER_UNIT
logger = logging.getLogger(__name__)
@dataclass
class DatasetConfig:
@@ -32,15 +29,10 @@ class DatasetConfig:
# "dataset_index" into the returned item. The index mapping is made according to the order in which the
# datasets are provided.
repo_id: str
# Hub repository type: "dataset" (default) or "bucket" for an HF Storage Bucket streamed over
# hf://buckets/. Buckets are streaming-only, so "bucket" requires streaming=true.
repo_type: str = "dataset"
# Root directory for a concrete local dataset tree (e.g. 'dataset/path'). If None, local datasets are
# looked up under $HF_LEROBOT_HOME/repo_id and Hub downloads use a revision-safe cache under $HF_LEROBOT_HOME/hub.
root: str | None = None
episodes: list[int] | None = None
# Episode indices to drop (e.g. corrupt or heterogeneous ones). Applied on top of `episodes`.
exclude_episodes: list[int] | None = None
image_transforms: ImageTransformsConfig = field(default_factory=ImageTransformsConfig)
revision: str | None = None
use_imagenet_stats: bool = True
@@ -56,16 +48,6 @@ class DatasetConfig:
eval_split: float = 0.0
def __post_init__(self) -> None:
if self.repo_type not in ("dataset", "bucket"):
raise ValueError(f"repo_type must be 'dataset' or 'bucket', got {self.repo_type!r}")
if self.repo_type == "bucket" and not self.streaming:
raise ValueError(
"repo_type='bucket' is streaming-only: set streaming=true to train from an HF Storage Bucket."
)
if self.repo_type == "bucket" and self.eval_split != 0.0:
raise ValueError(
"eval_split requires map-style datasets and is not supported with repo_type='bucket'."
)
if self.depth_output_unit not in (DEPTH_METER_UNIT, DEPTH_MILLIMETER_UNIT):
raise ValueError(
f"depth_output_unit must be '{DEPTH_METER_UNIT}' or '{DEPTH_MILLIMETER_UNIT}', got {self.depth_output_unit!r}"
@@ -80,14 +62,6 @@ class DatasetConfig:
if len(self.episodes) != len(set(self.episodes)):
duplicates = sorted({ep for ep in self.episodes if self.episodes.count(ep) > 1})
raise ValueError(f"Episode indices contain duplicates: {duplicates}")
if self.exclude_episodes is not None:
negative_episodes = [episode for episode in self.exclude_episodes if episode < 0]
if negative_episodes:
logger.warning(
"Ignoring negative exclude_episodes entries: %s",
negative_episodes,
)
self.exclude_episodes = [episode for episode in self.exclude_episodes if episode >= 0]
@dataclass
-190
View File
@@ -1,190 +0,0 @@
#!/usr/bin/env python
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Declarative process topology for distributed training and inference.
The mesh convention (canonical row-major rank layout, outermost first)::
(dp_replicate, dp_shard, ring, ulysses)
- ``dp_replicate x dp_shard`` is the data-parallel world: HSDP replicates over
``dp_replicate`` and shards parameters over ``dp_shard``. FSDP2's actual shard
group folds context parallelism in (``dp_shard x ring x ulysses``), matching
accelerate's ``dp_shard_cp`` flattening and torchtitan's ``fsdp`` axis.
- ``ring`` is the outer and ``ulysses`` the inner context-parallel dim
(diffusers convention: ulysses all-to-all exchanges run over adjacent, typically
NVLink-connected ranks).
- ``cfg_parallel`` (classifier-free-guidance parallelism) is a branch-parallel,
inference-only dim that sits between dp and the sequence dims. It never
affects weight sharding or checkpoints.
This module is pure configuration: plain-typed dataclasses that draccus can
round-trip through the CLI and ``train_config.json``. Runtime objects (device
meshes, process groups) live in :mod:`lerobot.distributed`.
"""
import os
from dataclasses import dataclass, field
@dataclass
class ContextParallelConfig:
"""Ring x Ulysses context parallelism (sequence parallelism for attention).
Both degrees are configured placeholders in this release: the CP engine is not implemented
yet, and enabling either degree > 1 fails fast at config validation. The fields exist now so
that the CLI surface, checkpoint metadata, and mesh math are stable when the engine lands.
"""
ring_degree: int = 1
ulysses_degree: int = 1
def __post_init__(self) -> None:
"""Validate the declared context-parallel degrees.
Raises:
ValueError: If ``ring_degree`` or ``ulysses_degree`` is < 1.
"""
if self.ring_degree < 1 or self.ulysses_degree < 1:
raise ValueError(
f"Context-parallel degrees must be >= 1, got ring_degree={self.ring_degree}, "
f"ulysses_degree={self.ulysses_degree}."
)
@property
def size(self) -> int:
"""Total number of ranks a full sequence is sharded across."""
return self.ring_degree * self.ulysses_degree
@dataclass
class ParallelismConfig:
"""Degrees of every parallelism dim. Invariant: their product equals the world size.
Degradations are expressed purely through the degrees (no mode flags):
- single process: all degrees 1;
- DDP: ``dp_replicate == world_size`` (auto-filled when every sharding field is left at its
default plain ``torchrun`` keeps today's out-of-the-box behavior);
- FSDP: ``dp_shard > 1`` (or ``-1`` to fill the remaining world into the shard dim);
- HSDP: ``dp_replicate > 1`` and ``dp_shard > 1``.
``resolve()`` turns the declared degrees into concrete ones once the world size is known and
is the single place the world-size equation is enforced. It is called by
:func:`lerobot.distributed.factory.make_accelerator`; the config is inert until then.
"""
dp_replicate: int = 1
# -1 is an explicit opt-in sentinel: shard over world_size // (dp_replicate * cp).
dp_shard: int = 1
context_parallel: ContextParallelConfig = field(default_factory=ContextParallelConfig)
# Classifier-free-guidance parallelism — inference-only (cosmos/vllm-omni precedent:
# cond/uncond branches on different ranks). Reserved for the serving round; training
# validates it to 1. Meaningful values are 1 or 2 (Cosmos3 has two CFG branches).
cfg_parallel: int = 1
def __post_init__(self) -> None:
"""Validate the declared degrees (world-size-independent checks only).
Raises:
ValueError: If ``dp_replicate`` is < 1, ``dp_shard`` is neither >= 1 nor the
``-1`` infer sentinel, or ``cfg_parallel`` is not 1 or 2.
"""
if self.dp_replicate < 1:
raise ValueError(f"dp_replicate must be >= 1, got {self.dp_replicate}.")
if self.dp_shard < 1 and self.dp_shard != -1:
raise ValueError(f"dp_shard must be >= 1, or -1 to infer, got {self.dp_shard}.")
if self.cfg_parallel not in (1, 2):
raise ValueError(f"cfg_parallel must be 1 or 2, got {self.cfg_parallel}.")
@property
def cp_size(self) -> int:
"""Total context-parallel size (``ring_degree * ulysses_degree``)."""
return self.context_parallel.size
@property
def is_sharded(self) -> bool:
"""True when the run uses FSDP2 (parameters sharded); selects the sharded engine path."""
return self.dp_shard != 1 or self.cp_size > 1
@property
def is_replicated_only(self) -> bool:
"""True for plain DDP (weights replicated, no sharding)."""
return not self.is_sharded and self.dp_replicate > 1
@property
def dp_world_size(self) -> int:
"""Number of distinct data-parallel workers (batches are sharded this many ways).
Returns:
int: ``dp_replicate * dp_shard``.
Raises:
RuntimeError: If accessed while ``dp_shard`` is still the ``-1`` sentinel, i.e.
before :meth:`resolve` has bound the degrees to a world size.
"""
if self.dp_shard == -1:
raise RuntimeError("dp_world_size is undefined before resolve() fills dp_shard=-1.")
return self.dp_replicate * self.dp_shard
def resolve(self, world_size: int) -> None:
"""Bind the declared degrees to a concrete world size (idempotent).
Fills the ``dp_shard=-1`` sentinel, auto-fills ``dp_replicate`` for the DDP degradation,
and enforces ``dp_replicate * dp_shard * cp == world_size`` with every degree echoed on
failure.
Args:
world_size (int): Total number of launched processes (torchrun's ``WORLD_SIZE``).
Raises:
ValueError: If a context-parallel degree is > 1 (the CP engine is not implemented
yet), if ``dp_shard=-1`` cannot be inferred because ``world_size`` is not
divisible by ``dp_replicate * cp``, or if the resolved degrees do not multiply
to ``world_size``.
"""
if self.cp_size > 1:
raise ValueError(
"Context parallelism is not implemented yet: ring_degree and ulysses_degree "
"must be 1. The fields are reserved for the CP engine round."
)
if self.is_sharded:
if self.dp_shard == -1:
self.dp_shard, remainder = divmod(world_size, self.dp_replicate * self.cp_size)
if remainder or self.dp_shard < 1:
raise ValueError(
f"Cannot infer dp_shard: world_size={world_size} is not divisible by "
f"dp_replicate={self.dp_replicate} * cp={self.cp_size}."
)
elif self.dp_replicate == 1:
# Untouched config on a multi-process launch: fill the DDP degradation.
self.dp_replicate = world_size
total = self.dp_replicate * self.dp_shard * self.cp_size
if total != world_size:
raise ValueError(
f"Parallelism degrees do not multiply to the world size: dp_replicate="
f"{self.dp_replicate} * dp_shard={self.dp_shard} * ring="
f"{self.context_parallel.ring_degree} * ulysses="
f"{self.context_parallel.ulysses_degree} = {total} != WORLD_SIZE={world_size}."
)
def world_size_from_env() -> int:
"""World size as set by torchrun (or 1 outside distributed launches).
Returns:
int: The ``WORLD_SIZE`` environment variable, or 1 when unset.
"""
return int(os.environ.get("WORLD_SIZE", "1"))
+29 -145
View File
@@ -27,13 +27,6 @@ from typing import Any, TypeVar, cast
import draccus
import yaml # type: ignore[import-untyped]
from draccus.help_formatter import SimpleHelpFormatter
from draccus.utils import DecodingError
from draccus.wrappers import DataclassWrapper
from draccus.wrappers.choice_wrapper import ChoiceWrapper, UnionWrapper
from draccus.wrappers.field_wrapper import FieldWrapper
from draccus.wrappers.suppressing_argparse import SuppressingArgumentParser
from draccus.wrappers.wrapper import AggregateWrapper, Wrapper
from lerobot.utils.utils import has_method
@@ -79,18 +72,11 @@ def get_cli_overrides(field_name: str, args: Sequence[str] | None = None) -> lis
args = sys.argv[1:]
attr_level_args = []
detect_string = f"--{field_name}."
excluded_names = (draccus.CHOICE_TYPE_KEY, PATH_KEY)
for index, arg in enumerate(args):
if not arg.startswith(detect_string):
continue
denested_arg = arg.removeprefix(detect_string)
if denested_arg.split("=", maxsplit=1)[0] in excluded_names:
continue
attr_level_args.append(f"--{denested_arg}")
if "=" not in arg and index + 1 < len(args) and not args[index + 1].startswith("--"):
attr_level_args.append(args[index + 1])
exclude_strings = (f"--{field_name}.{draccus.CHOICE_TYPE_KEY}=", f"--{field_name}.{PATH_KEY}=")
for arg in args:
if arg.startswith(detect_string) and not arg.startswith(exclude_strings):
denested_arg = f"--{arg.removeprefix(detect_string)}"
attr_level_args.append(denested_arg)
return attr_level_args
@@ -98,12 +84,10 @@ def get_cli_overrides(field_name: str, args: Sequence[str] | None = None) -> lis
def parse_arg(arg_name: str, args: Sequence[str] | None = None) -> str | None:
if args is None:
args = sys.argv[1:]
option = f"--{arg_name}"
for index, arg in enumerate(args):
if arg.startswith(f"{option}="):
return arg.removeprefix(f"{option}=")
if arg == option and index + 1 < len(args) and not args[index + 1].startswith("--"):
return args[index + 1]
prefix = f"--{arg_name}="
for arg in args:
if arg.startswith(prefix):
return arg[len(prefix) :]
return None
@@ -111,7 +95,7 @@ def parse_plugin_args(plugin_arg_suffix: str, args: Sequence[str]) -> dict[str,
"""Parse plugin-related arguments from command-line arguments.
This function extracts arguments from command-line arguments that match a specified suffix pattern.
It accepts arguments in the formats '--key=value' and '--key value' and returns them as a dictionary.
It processes arguments in the format '--key=value' and returns them as a dictionary.
Args:
plugin_arg_suffix (str): The suffix to identify plugin-related arguments.
@@ -128,18 +112,13 @@ def parse_plugin_args(plugin_arg_suffix: str, args: Sequence[str]) -> dict[str,
{'env.discover_packages_path': 'my_package'}
"""
plugin_args = {}
for index, arg in enumerate(args):
if not arg.startswith("--"):
continue
key, separator, value = arg[2:].partition("=")
if plugin_arg_suffix not in key:
continue
if not separator:
if index + 1 >= len(args) or args[index + 1].startswith("--"):
continue
value = args[index + 1]
plugin_args[key] = value
for arg in args:
if "=" in arg and plugin_arg_suffix in arg:
key, value = arg.split("=", 1)
# Remove leading '--' if present
if key.startswith("--"):
key = key[2:]
plugin_args[key] = value
return plugin_args
@@ -206,82 +185,10 @@ def get_type_arg(field_name: str, args: Sequence[str] | None = None) -> str | No
return parse_arg(f"{field_name}.{draccus.CHOICE_TYPE_KEY}", args)
def _register_scoped_actions(
wrapper: Wrapper, parser: SuppressingArgumentParser, cli_args: Sequence[str]
) -> None:
"""Like draccus's own Wrapper.register_actions, but for a ChoiceType field only recurses into
the already-selected subclass (per CLI `.type` args), instead of every registered choice.
This mirrors draccus 0.11.x's internal wrapper traversal because its public parser eagerly registers
every choice before parsing the command line. Keep this in sync when updating draccus.
"""
if isinstance(wrapper, ChoiceWrapper):
group = parser.add_argument_group(title=wrapper.title, description=wrapper.description)
children = wrapper._children
arg_name = f"{wrapper.dest}.{draccus.CHOICE_TYPE_KEY}" if wrapper.dest else draccus.CHOICE_TYPE_KEY
group.add_argument(
f"--{arg_name}",
choices=list(children.keys()),
help=f"Which type of {wrapper.title} to use",
required=wrapper.required,
)
selected = get_type_arg(wrapper.dest, cli_args) if wrapper.dest else None
if selected in children:
_register_scoped_actions(children[selected], parser, cli_args)
elif isinstance(wrapper, DataclassWrapper):
group = parser.add_argument_group(title=wrapper.title, description=wrapper.description)
for child in wrapper._children:
if isinstance(child, AggregateWrapper):
parser.add_argument(
f"--{child.name}", type=str, required=False, help=f"Config file for {child.name}"
)
_register_scoped_actions(child, parser, cli_args)
elif isinstance(child, FieldWrapper):
child.add_action(group)
elif isinstance(wrapper, UnionWrapper):
group = parser.add_argument_group(title=wrapper.title, description=wrapper.description)
has_field_wrapper = False
for child in wrapper._children:
if isinstance(child, (DataclassWrapper, ChoiceWrapper)):
_register_scoped_actions(child, parser, cli_args)
elif isinstance(child, FieldWrapper):
has_field_wrapper = True
if has_field_wrapper:
group.add_argument(f"--{wrapper.dest}", required=False)
else:
wrapper.register_actions(parser)
def print_scoped_help(config_class: type, cli_args: Sequence[str]) -> None:
"""Prints --help output scoped to the choices already resolved on the CLI (e.g. --env.type=pusht),
instead of draccus's default of expanding every registered subclass of every ChoiceType field."""
parser = SuppressingArgumentParser(formatter_class=SimpleHelpFormatter)
parser.add_argument(
f"--{draccus.utils.CONFIG_ARG}", type=str, help="Path for a config file to parse with draccus"
)
_register_scoped_actions(DataclassWrapper(config_class), parser, cli_args)
parser.print_help()
def filter_arg(field_to_filter: str, args: Sequence[str] | None = None) -> list[str]:
if args is None:
return []
option = f"--{field_to_filter}"
filtered_args = []
index = 0
while index < len(args):
arg = args[index]
if arg == option:
index += 1
if index < len(args) and not args[index].startswith("--"):
index += 1
continue
if arg.startswith(f"{option}="):
index += 1
continue
filtered_args.append(arg)
index += 1
return filtered_args
return [arg for arg in args if not arg.startswith(f"--{field_to_filter}=")]
def filter_path_args(fields_to_filter: str | list[str], args: Sequence[str] | None = None) -> list[str]:
@@ -313,23 +220,7 @@ def filter_path_args(fields_to_filter: str | list[str], args: Sequence[str] | No
argument=None,
message=f"Cannot specify both --{field}.{PATH_KEY} and --{field}.{draccus.CHOICE_TYPE_KEY}",
)
option_prefix = f"--{field}."
retained_args = []
index = 0
while index < len(filtered_args):
arg = filtered_args[index]
if arg.startswith(option_prefix):
index += 1
if (
"=" not in arg
and index < len(filtered_args)
and not filtered_args[index].startswith("--")
):
index += 1
continue
retained_args.append(arg)
index += 1
filtered_args = retained_args
filtered_args = [arg for arg in filtered_args if not arg.startswith(f"--{field}.")]
return filtered_args
@@ -408,9 +299,6 @@ def wrap(config_path: Path | None = None) -> Callable[[F], F]:
# add the relevant CLI arg to the error message
raise PluginLoadError(f"{e}\nFailed plugin CLI Arg: {plugin_cli_arg}") from e
cli_args = filter_arg(plugin_cli_arg, cli_args)
if "--help" in cli_args or "-h" in cli_args:
print_scoped_help(argtype, cli_args)
sys.exit(0)
config_path_cli = parse_arg("config_path", cli_args)
if has_method(argtype, "__get_path_fields__"):
path_fields = argtype.__get_path_fields__()
@@ -418,21 +306,17 @@ def wrap(config_path: Path | None = None) -> Callable[[F], F]:
# Also extract path fields from the YAML/JSON config file
if config_path_cli:
config_path_cli = extract_path_fields_from_config(config_path_cli, path_fields)
try:
if has_method(argtype, "from_pretrained") and config_path_cli:
if has_method(argtype, "from_pretrained") and config_path_cli:
cli_args = filter_arg("config_path", cli_args)
cfg = argtype.from_pretrained(config_path_cli, cli_args=cli_args)
else:
if config_path_cli:
cli_args = filter_arg("config_path", cli_args)
cfg = argtype.from_pretrained(config_path_cli, cli_args=cli_args)
else:
if config_path_cli:
cli_args = filter_arg("config_path", cli_args)
cfg = draccus.parse(
config_class=argtype,
config_path=config_path_cli or config_path,
args=cli_args,
)
except DecodingError as e:
print(f"error: {e}", file=sys.stderr)
sys.exit(1)
cfg = draccus.parse(
config_class=argtype,
config_path=config_path_cli or config_path,
args=cli_args,
)
response = fn(cfg, *args, **kwargs)
return response
+2 -4
View File
@@ -163,10 +163,8 @@ class PreTrainedConfig(draccus.ChoiceRegistry, HubMixin, abc.ABC): # type: igno
return None
def _save_pretrained(self, save_directory: Path) -> None:
# Encode against the base class so draccus includes the choice "type" key,
# which `from_pretrained` needs to resolve the concrete subclass.
with open(save_directory / CONFIG_NAME, "w") as f:
json.dump(draccus.encode(self, PreTrainedConfig), f, indent=4)
with open(save_directory / CONFIG_NAME, "w") as f, draccus.config_type("json"):
draccus.dump(self, f, indent=4)
@classmethod
def from_pretrained(
+7 -24
View File
@@ -23,7 +23,6 @@ from typing import Any, Literal, get_args
MessageRole = Literal["user", "assistant", "system", "tool"]
MessageStream = Literal["high_level", "low_level"]
RecipeRoute = Literal["vqa"]
DEFAULT_BINDINGS = {
"subtask": "active_at(t, style=subtask)",
@@ -41,7 +40,6 @@ discovery (here) and rendered-message substitution (in ``language_render``)."""
_VALID_ROLES = frozenset(get_args(MessageRole))
_VALID_STREAMS = frozenset(get_args(MessageStream))
_VALID_ROUTES = frozenset(get_args(RecipeRoute))
@dataclass
@@ -80,7 +78,7 @@ class MessageTurn:
raise ValueError(f"Unsupported message stream: {self.stream!r}")
if self.content is None and self.tool_calls_from is None:
raise ValueError("MessageTurn.content is required unless tool_calls_from is set.")
if self.content is not None and not isinstance(self.content, str | list):
if self.content is not None and not isinstance(self.content, (str, list)):
raise TypeError("MessageTurn.content must be a string, a list of HF-style blocks, or None.")
if isinstance(self.content, list):
for block in self.content:
@@ -101,16 +99,13 @@ class TrainingRecipe:
A recipe is either a *message recipe* (``messages`` plus optional
``bindings``) or a *blend recipe* (``blend`` mapping names to weighted
sub-recipes). ``weight`` and ``route`` are only meaningful inside a blend;
``route: vqa`` gives sparse VQA annotations priority over normal weighted
selection.
sub-recipes). ``weight`` is only meaningful inside a blend.
"""
messages: list[MessageTurn] | None = None
bindings: dict[str, str] | None = None
blend: dict[str, TrainingRecipe] | None = None
weight: float | None = None
route: RecipeRoute | None = None
def __post_init__(self) -> None:
"""Validate that exactly one of ``messages`` or ``blend`` is set."""
@@ -118,10 +113,6 @@ class TrainingRecipe:
raise ValueError("TrainingRecipe must set only one of messages or blend.")
if self.messages is None and self.blend is None:
raise ValueError("TrainingRecipe must set one of messages or blend.")
if self.route is not None and self.route not in _VALID_ROUTES:
raise ValueError(f"Unsupported recipe route: {self.route!r}")
if self.blend is not None and self.route is not None:
raise ValueError("TrainingRecipe.route may only be set on a message recipe inside a blend.")
if self.messages is not None:
self._validate_message_recipe()
@@ -156,9 +147,8 @@ class TrainingRecipe:
return cls.from_dict(data)
def _validate_message_recipe(self) -> None:
"""Validate bindings and require text or low-level action supervision."""
if self.messages is None:
raise ValueError("Cannot validate a message recipe without messages.")
"""Ensure every templated binding is known and at least one turn is a target."""
assert self.messages is not None
known_bindings = set(DEFAULT_BINDINGS) | set(self.bindings or {}) | {"task"}
for turn in self.messages:
@@ -166,19 +156,12 @@ class TrainingRecipe:
if missing:
raise ValueError(f"MessageTurn references unknown binding(s): {sorted(missing)}")
has_target = any(turn.target for turn in self.messages)
has_low_level = any(turn.stream == "low_level" for turn in self.messages)
if not (has_target or has_low_level):
raise ValueError(
"Message recipes must contain at least one supervised turn — "
"either ``target: true`` (text CE) or ``stream: low_level`` "
"(flow/action loss)."
)
if not any(turn.target for turn in self.messages):
raise ValueError("Message recipes must contain at least one target turn.")
def _validate_blend_recipe(self) -> None:
"""Ensure each blend component is a non-empty, weighted message recipe."""
if self.blend is None:
raise ValueError("Cannot validate a blend recipe without blend components.")
assert self.blend is not None
if not self.blend:
raise ValueError("Blend recipes must contain at least one component.")
-16
View File
@@ -1,16 +0,0 @@
# Predicts subtasks from tasks and trains subtask-conditioned action flow without memory or plans.
# Requires `subtask` annotations; samples with missing `if_present` bindings do not render.
blend:
high_level_subtask:
weight: 0.30
messages:
- {role: user, content: "${task}", stream: high_level}
- {role: assistant, content: "${subtask}", stream: high_level, target: true, if_present: subtask}
low_level_execution:
weight: 0.70
messages:
# The low-level stream trains action flow on the generated or annotated subtask.
- {role: user, content: "${subtask}", stream: low_level, if_present: subtask}
@@ -1,13 +0,0 @@
# Paper-style joint sequence (pi0.5 §IV-B): one sample supervises the subtask
# text with CE and, because the assistant turn is part of the prefix, conditions
# the FAST and flow action losses on the same annotated subtask in one forward.
# The supervised span is attended causally; the action losses see task + subtask.
#
# Pair with `--policy.joint_subtask_conditioning=true` at inference so the flow
# prefix reproduces this layout (task turn with state + causal generated subtask).
# Samples without a `subtask` annotation fall back to a plain task-prompt
# low-level sample via `if_present`.
messages:
- {role: user, content: "${task}", stream: low_level}
- {role: assistant, content: "${subtask}", stream: low_level, target: true, if_present: subtask}
@@ -1,30 +0,0 @@
# Trains subtask prediction, subtask-conditioned action flow, and memory updates without plans.
# Requires `subtask` and `memory`; missing `if_present` bindings skip the affected sub-recipe.
blend:
high_level_subtask:
weight: 0.25
messages:
- {role: user, content: "${task}", stream: high_level}
- {role: assistant, content: "${subtask}", stream: high_level, target: true, if_present: subtask}
low_level_execution:
weight: 0.60
messages:
# The low-level stream trains action flow on the generated or annotated subtask.
- {role: user, content: "${subtask}", stream: low_level, if_present: subtask}
memory_update:
# `active_at` densifies sparse boundaries while preserving the prior-memory/subtask mapping.
# Inference controls update timing through `subtask_change` events.
weight: 0.15
bindings:
prior_memory: "nth_prev(style=memory, offset=1)"
current_memory: "active_at(t, style=memory)"
completed_subtask: "nth_prev(style=subtask, offset=1)"
messages:
- {role: user, content: "${task}", stream: high_level}
- {role: assistant, content: "Previous memory: ${prior_memory}", stream: high_level, if_present: prior_memory}
- {role: user, content: "Completed subtask: ${completed_subtask}", stream: high_level, if_present: completed_subtask}
- {role: assistant, content: "${current_memory}", stream: high_level, target: true, if_present: current_memory}
@@ -1,72 +0,0 @@
# Adds memory, spoken interjection responses, and camera-grounded VQA to subtask/action training.
# Missing optional annotations skip only their sub-recipe; `say` tool calls tokenize as `<say>...</say>`.
blend:
high_level_subtask:
weight: 0.25
messages:
- {role: user, content: "${task}", stream: high_level}
- {role: assistant, content: "${subtask}", stream: high_level, target: true, if_present: subtask}
low_level_execution:
weight: 0.40
messages:
# The low-level stream trains action flow on the generated or annotated subtask.
- {role: user, content: "${subtask}", stream: low_level, if_present: subtask}
memory_update:
# `active_at` densifies sparse boundaries while preserving the prior-memory/subtask mapping.
# Inference controls update timing through `subtask_change` events.
weight: 0.10
bindings:
prior_memory: "nth_prev(style=memory, offset=1)"
current_memory: "active_at(t, style=memory)"
completed_subtask: "nth_prev(style=subtask, offset=1)"
messages:
- {role: user, content: "${task}", stream: high_level}
- {role: assistant, content: "Previous memory: ${prior_memory}", stream: high_level, if_present: prior_memory}
- {role: user, content: "Completed subtask: ${completed_subtask}", stream: high_level, if_present: completed_subtask}
- {role: assistant, content: "${current_memory}", stream: high_level, target: true, if_present: current_memory}
user_interjection_response:
weight: 0.10
bindings:
interjection: "emitted_at(t, style=interjection)"
speech: "emitted_at(t, role=assistant, tool_name=say)"
messages:
- {role: user, content: "${task}", stream: high_level}
- {role: user, content: "${interjection}", stream: high_level, if_present: interjection}
# The assistant target is a `say` tool call flattened to a `<say>...</say>` marker.
- {role: assistant, stream: high_level, target: true, if_present: speech, tool_calls_from: speech}
# Each camera uses a separate VQA sub-recipe for view-specific binding.
ask_vqa_top:
weight: 0.075
route: vqa
bindings:
vqa_query: "emitted_at(t, style=vqa, role=user, camera=observation.images.front)"
vqa: "emitted_at(t, style=vqa, role=assistant, camera=observation.images.front)"
messages:
- role: user
stream: high_level
if_present: vqa_query
content:
- {type: image, feature: observation.images.front}
- {type: text, text: "${vqa_query}"}
- {role: assistant, content: "${vqa}", stream: high_level, target: true, if_present: vqa}
ask_vqa_wrist:
weight: 0.075
route: vqa
bindings:
vqa_query: "emitted_at(t, style=vqa, role=user, camera=observation.images.wrist)"
vqa: "emitted_at(t, style=vqa, role=assistant, camera=observation.images.wrist)"
messages:
- role: user
stream: high_level
if_present: vqa_query
content:
- {type: image, feature: observation.images.wrist}
- {type: text, text: "${vqa_query}"}
- {role: assistant, content: "${vqa}", stream: high_level, target: true, if_present: vqa}
+2 -4
View File
@@ -103,10 +103,8 @@ class RewardModelConfig(draccus.ChoiceRegistry, HubMixin, abc.ABC):
pass
def _save_pretrained(self, save_directory: Path) -> None:
# Encode against the base class so draccus includes the choice "type" key,
# which `from_pretrained` needs to resolve the concrete subclass.
with open(save_directory / CONFIG_NAME, "w") as f:
json.dump(draccus.encode(self, RewardModelConfig), f, indent=4)
with open(save_directory / CONFIG_NAME, "w") as f, draccus.config_type("json"):
draccus.dump(self, f, indent=4)
@classmethod
def from_pretrained(
+1 -98
View File
@@ -18,7 +18,6 @@ import multiprocessing
import os
import tempfile
from dataclasses import dataclass, field
from enum import Enum
from pathlib import Path
from typing import Any
@@ -27,8 +26,6 @@ from huggingface_hub import hf_hub_download
from huggingface_hub.errors import HfHubHTTPError
from lerobot import envs
from lerobot.configs.accelerator import AcceleratorConfig, ActivationCheckpointingMode
from lerobot.configs.parallelism import ParallelismConfig
from lerobot.optim import LRSchedulerConfig, OptimizerConfig
from lerobot.utils.constants import PRETRAINED_MODEL_DIR
from lerobot.utils.hub import HubMixin, find_latest_hub_checkpoint
@@ -42,34 +39,6 @@ from .rewards import RewardModelConfig
TRAIN_CONFIG_NAME = "train_config.json"
class CheckpointFormat(str, Enum):
"""Model-artifact format inside training checkpoints.
Selects only the *model* artifact; the training_state layout is format-independent (the
optimizer channel is always DCP under sharded runs, safetensors+json otherwise).
- SAFETENSORS (default): a full `model.safetensors` maximum compatibility, one gather per
save under sharding.
- DCP: sharded `pytorch_model_fsdp_0/*.distcp` only fastest save/resume; convert with
`lerobot-convert-dcp` before distributing.
- SAFETENSORS_AND_DCP: both artifacts, written independently.
"""
SAFETENSORS = "safetensors"
DCP = "dcp"
SAFETENSORS_AND_DCP = "safetensors_dcp"
@property
def wants_safetensors(self) -> bool:
"""True when a full `model.safetensors` artifact should be written."""
return self in (CheckpointFormat.SAFETENSORS, CheckpointFormat.SAFETENSORS_AND_DCP)
@property
def wants_dcp(self) -> bool:
"""True when sharded DCP model shards (`pytorch_model_fsdp_0/`) should be written."""
return self in (CheckpointFormat.DCP, CheckpointFormat.SAFETENSORS_AND_DCP)
def _migrate_legacy_rabc_fields(config: dict[str, Any]) -> dict[str, Any] | None:
"""Return migrated payload for legacy RA-BC fields, or None when no migration is needed."""
legacy_fields = (
@@ -150,18 +119,10 @@ class TrainPipelineConfig(HubMixin):
tolerance_s: float = 1e-4
save_checkpoint: bool = True
# Checkpoint is saved every `save_freq` training iterations and after the last training step.
# A non-positive value disables periodic saving, keeping only the final checkpoint.
save_freq: int = 20_000
# Model-artifact format inside checkpoints; non-default values require a sharded run.
checkpoint_format: CheckpointFormat = CheckpointFormat.SAFETENSORS
use_policy_training_preset: bool = True
optimizer: OptimizerConfig | None = None
scheduler: LRSchedulerConfig | None = None
# Process topology: dp_replicate / dp_shard (HSDP) and context-parallel degree placeholders.
parallelism: ParallelismConfig = field(default_factory=ParallelismConfig)
# Execution runtime handed to the Accelerator: mixed precision, gradient accumulation,
# FSDP/DDP tuning knobs, compile & activation-checkpointing placeholders.
accelerator: AcceleratorConfig = field(default_factory=AcceleratorConfig)
eval: EvalConfig = field(default_factory=EvalConfig)
wandb: WandBConfig = field(default_factory=WandBConfig)
peft: PeftConfig | None = None
@@ -233,11 +194,7 @@ class TrainPipelineConfig(HubMixin):
)
if Path(config_path).resolve().exists():
# `config_path` may point at the checkpoint's train_config.json or at its
# pretrained_model/ directory (both documented above) — resolve either to
# the pretrained_model/ directory.
config_path_obj = Path(config_path)
policy_dir = config_path_obj.parent if config_path_obj.is_file() else config_path_obj
policy_dir = Path(config_path).parent
self.checkpoint_path = policy_dir.parent
elif self.job.is_remote:
return
@@ -329,60 +286,6 @@ class TrainPipelineConfig(HubMixin):
if self.save_checkpoint_to_hub and not (self.policy is not None and self.policy.repo_id):
raise ValueError("save_checkpoint_to_hub requires --policy.repo_id.")
self._validate_distributed()
def _validate_distributed(self) -> None:
"""Fail-fasts for the distributed-training scope.
Raises:
ValueError: If the config requests anything outside the verified scope: context
parallelism or CFG parallelism (reserved placeholders), the compile or
activation-checkpointing placeholders, a DCP checkpoint format on a
non-sharded run, or under sharded training fp16 mixed precision, PEFT,
reward-model training, in-training environment evaluation, or multi-optimizer
configs.
"""
if self.parallelism.cp_size > 1:
raise ValueError(
"Context parallelism is not implemented yet: --parallelism.context_parallel.* "
"degrees must be 1 (reserved for the CP engine round)."
)
if self.parallelism.cfg_parallel != 1:
raise ValueError(
"CFG parallelism is inference-only and must be 1 for training "
"(cfg_parallel is reserved for the serving round)."
)
if self.accelerator.compile.enabled:
raise ValueError("--accelerator.compile is a placeholder and not wired yet.")
if self.accelerator.activation_checkpointing.mode is not ActivationCheckpointingMode.NONE:
raise ValueError("--accelerator.activation_checkpointing is a placeholder and not wired yet.")
if self.checkpoint_format is not CheckpointFormat.SAFETENSORS and not self.parallelism.is_sharded:
raise ValueError(
f"checkpoint_format={self.checkpoint_format.value} requires a sharded run "
"(--parallelism.dp_shard != 1); non-sharded checkpoints are always safetensors."
)
if self.parallelism.is_sharded:
if self.accelerator.mixed_precision == "fp16":
raise ValueError(
"fp16 is not supported under sharded training (GradScaler over DTensor "
"gradients is unverified); use bf16 or full precision."
)
if self.peft is not None:
raise ValueError("PEFT is not supported under sharded training yet.")
if self.is_reward_model_training:
raise ValueError(
"Reward-model training is not supported under sharded training yet "
"(reward models declare no FSDP wrap units and have no sharded save path)."
)
if self.env is not None and self.env_eval_freq > 0:
raise ValueError(
"In-training environment evaluation is not supported under sharded training "
"(a rank-0-only rollout of a sharded model deadlocks on collectives); set "
"--env_eval_freq=0 and evaluate with lerobot-eval on saved checkpoints."
)
if self.optimizer is not None and self.optimizer.builds_multiple_optimizers:
raise ValueError("Multi-optimizer configs are not supported under sharded training.")
@classmethod
def __get_path_fields__(cls) -> list[str]:
"""Keys for draccus pretrained-path loading."""
@@ -76,7 +76,7 @@ import torch
from pydantic import BaseModel, Field
from transformers import AutoProcessor, Qwen3VLMoeForConditionalGeneration
from lerobot.datasets import LeRobotDataset, resolve_episode_indices
from lerobot.datasets import LeRobotDataset
# Pydantic Models for SARM Subtask Annotation
@@ -1049,10 +1049,7 @@ def main():
torch_dtype = {"bfloat16": torch.bfloat16, "float16": torch.float16, "float32": torch.float32}[args.dtype]
# Determine episodes
resolved_episodes = resolve_episode_indices(args.episodes, dataset.meta.total_episodes)
episode_indices = (
resolved_episodes if resolved_episodes is not None else list(range(dataset.meta.total_episodes))
)
episode_indices = args.episodes or list(range(dataset.meta.total_episodes))
existing_annotations = load_annotations_from_dataset(dataset.root, prefix="sparse")
if args.skip_existing:
+1 -2
View File
@@ -52,7 +52,7 @@ from .pipeline_features import aggregate_pipeline_dataset_features, create_initi
from .pyav_utils import check_video_encoder_parameters_pyav, detect_available_encoders_pyav
from .sampler import EpisodeAwareSampler, compute_sampler_state
from .streaming_dataset import StreamingLeRobotDataset
from .utils import DEFAULT_EPISODES_PATH, create_lerobot_dataset_card, resolve_episode_indices
from .utils import DEFAULT_EPISODES_PATH, create_lerobot_dataset_card
from .video_utils import VideoEncodingManager
# NOTE: Low-level I/O functions (cast_stats_to_numpy, get_parquet_file_size_in_mb, etc.)
@@ -97,7 +97,6 @@ __all__ = [
"reencode_dataset",
"remove_feature",
"resolve_delta_timestamps",
"resolve_episode_indices",
"safe_stop_image_writer",
"split_dataset",
"write_stats",

Some files were not shown because too many files have changed in this diff Show More