mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-26 03:06:01 +00:00
Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0d383d09f2 | |||
| ab2b5b04dd | |||
| ac5c7b8600 | |||
| a6befef0ba | |||
| 53843007ea | |||
| d3bed0feee | |||
| a0eb860d1e | |||
| cfd9ff969c | |||
| f59eae4e27 | |||
| a993af9c51 | |||
| 392246feaf | |||
| 19dcbc19f1 | |||
| 679faeaafc | |||
| 228cb5ddb9 | |||
| ad176c6d41 | |||
| d6c605e8c5 | |||
| 9c82c39c7b | |||
| 73dbb6f43a |
@@ -51,6 +51,7 @@ pre-commit run --all-files # Lint + format (ruff, typo
|
||||
## Notes
|
||||
|
||||
- **Mypy is gradual**: strict only for `lerobot.envs`, `lerobot.configs`, `lerobot.optim`, `lerobot.model`, `lerobot.cameras`, `lerobot.motors`, `lerobot.transport`. Add type annotations when modifying these modules.
|
||||
- **Optional dependencies**: many policies, envs, and robots are behind extras (e.g., `lerobot[aloha]`). New imports for optional packages must be guarded or lazy. See `pyproject.toml [project.optional-dependencies]`.
|
||||
- **Imports**: prefer top-level imports; relative (`from .sibling import X`) across sibling files within a module, absolute (`from lerobot.module import X`) across modules.
|
||||
- **Optional dependencies**: many policies, envs, and robots are behind extras (e.g., `lerobot[aloha]`, see `pyproject.toml`). Guard optional imports with `TYPE_CHECKING or _foo_available` at module top + a `require_package(...)` check at use time. Reuse the `_foo_available` flags in `utils/import_utils.py`; don't call `is_package_available`.
|
||||
- **Video decoding**: datasets can store observations as video files. `LeRobotDataset` handles frame extraction, but tests need ffmpeg installed.
|
||||
- **Prioritize use of `uv run`** to execute Python commands (not raw `python` or `pip`).
|
||||
|
||||
@@ -83,7 +83,7 @@ episode_index=0
|
||||
print(f"{dataset[episode_index]['action'].shape=}\n")
|
||||
```
|
||||
|
||||
Learn more about it in the [LeRobotDataset Documentation](https://huggingface.co/docs/lerobot/lerobot-dataset-v3)
|
||||
Learn more about it in the [LeRobotDataset Documentation](https://huggingface.co/docs/lerobot/lerobot-dataset-v3).
|
||||
|
||||
## SoTA Models
|
||||
|
||||
@@ -109,7 +109,7 @@ lerobot-train \
|
||||
| **World Models** | [VLA-JEPA](./docs/source/vla_jepa.mdx), [LingBot-VA](./docs/source/lingbot_va.mdx), [FastWAM](./docs/source/fastwam.mdx) |
|
||||
| **Reward Models** | [SARM](./docs/source/sarm.mdx), [TOPReward](./docs/source/topreward.mdx), [Robometer](./docs/source/robometer.mdx) |
|
||||
|
||||
Similarly to the hardware, you can easily implement your own policy & leverage LeRobot's data collection, training, and visualization tools, and share your model to the HF Hub
|
||||
Similarly to the hardware, you can easily implement your own policy & leverage LeRobot's data collection, training, and visualization tools, and share your model to the HF Hub.
|
||||
|
||||
For detailed policy setup guides, see the [Policy Documentation](https://huggingface.co/docs/lerobot/bring_your_own_policies). For GPU/RAM requirements and expected training time per policy, see the [Compute Hardware Guide](https://huggingface.co/docs/lerobot/hardware_guide).
|
||||
|
||||
@@ -126,7 +126,7 @@ lerobot-eval \
|
||||
--eval.n_episodes=10
|
||||
```
|
||||
|
||||
Learn how to implement your own simulation environment or benchmark and distribute it from the HF Hub by following the [EnvHub Documentation](https://huggingface.co/docs/lerobot/envhub)
|
||||
Learn how to implement your own simulation environment or benchmark and distribute it from the HF Hub by following the [EnvHub Documentation](https://huggingface.co/docs/lerobot/envhub).
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@@ -89,8 +89,8 @@ subtask.
|
||||
|
||||
The resulting spans are then stitched into a gap-free, full-episode
|
||||
cover, so **every frame has exactly one active subtask**. See
|
||||
[`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py)
|
||||
for the production settings (single camera, timestamped contact sheets,
|
||||
[Running on Hugging Face Jobs](#running-on-hugging-face-jobs) for the
|
||||
production settings (single camera, timestamped contact sheets,
|
||||
auto-windowed subtask generation).
|
||||
|
||||
### Tools
|
||||
@@ -110,28 +110,67 @@ not-yet-implemented.
|
||||
|
||||
## Running on Hugging Face Jobs
|
||||
|
||||
Annotation runs on [Hugging Face Jobs](https://huggingface.co/docs/hub/en/jobs).
|
||||
The repo ships a launcher script you copy and tweak for your dataset:
|
||||
Annotating a real dataset needs a GPU big enough to serve the VLM, so
|
||||
`lerobot-annotate` can dispatch itself to
|
||||
[Hugging Face Jobs](https://huggingface.co/docs/hub/en/jobs) — same as
|
||||
`lerobot-train`. Add `--job.target=<flavor>` to the exact command you'd
|
||||
run locally and it runs on that hardware instead:
|
||||
|
||||
```bash
|
||||
HF_TOKEN=hf_... uv run python examples/annotations/run_hf_job.py
|
||||
hf auth login # once
|
||||
|
||||
uv run lerobot-annotate \
|
||||
--repo_id=user/my_dataset \
|
||||
--new_repo_id=user/my_dataset_annotated \
|
||||
--push_to_hub=true \
|
||||
--vlm.model_id=Qwen/Qwen3.6-27B \
|
||||
--vlm.num_gpus=1 \
|
||||
--vlm.serve_command="vllm serve Qwen/Qwen3.6-27B --tensor-parallel-size 1 \
|
||||
--max-model-len 32768 --gpu-memory-utilization 0.8 \
|
||||
--uvicorn-log-level warning --port {port}" \
|
||||
--vlm.serve_ready_timeout_s=1800 \
|
||||
--vlm.chat_template_kwargs='{"enable_thinking": false}' \
|
||||
--job.target=h200
|
||||
```
|
||||
|
||||
[`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py)
|
||||
starts a single-GPU `h200` job (bump it to `h200x4` for big datasets)
|
||||
that:
|
||||
That submits a single-GPU `h200` job that:
|
||||
|
||||
1. installs `lerobot` (from `main`) plus the annotation extras,
|
||||
2. boots one vLLM server per GPU (using the `vllm/vllm-openai` image) and
|
||||
drives it over the OpenAI-compatible API,
|
||||
3. runs the `plan` / `interjections` / `vqa` modules across the dataset
|
||||
with `lerobot-annotate`,
|
||||
1. starts from the `vllm/vllm-openai` image and installs `lerobot` on top,
|
||||
2. boots one vLLM server per GPU and drives it over the OpenAI-compatible API,
|
||||
3. runs the `plan` / `interjections` / `vqa` modules across the dataset,
|
||||
4. with `--push_to_hub=true`, uploads the result to `--new_repo_id` (or
|
||||
back to `--repo_id` in place if you leave that unset).
|
||||
|
||||
To use a different dataset, model, or hub repo, edit the `CMD` block in
|
||||
the script. Every flag there maps directly to a `lerobot-annotate` flag
|
||||
(run `lerobot-annotate --help` for the full list).
|
||||
The command streams the job's logs; `Ctrl-C` detaches without cancelling
|
||||
it. List the available flavors and their pricing with `hf jobs hardware`.
|
||||
|
||||
<Tip warning={true}>
|
||||
|
||||
Qwen3.6 ships with thinking enabled, which eats the token budget the
|
||||
annotator needs for its JSON answer — `--vlm.chat_template_kwargs='{"enable_thinking": false}'`
|
||||
turns it off. Without `--push_to_hub=true` the annotated dataset is
|
||||
discarded when the pod exits.
|
||||
|
||||
</Tip>
|
||||
|
||||
### Job options
|
||||
|
||||
| Flag | Default | What it does |
|
||||
| ------------------- | ------------------------- | ------------------------------------------------------------------------------- |
|
||||
| `--job.target` | `local` | HF Jobs flavor to run on (e.g. `h200`, `h200x4`). Omitted/`local` runs here. |
|
||||
| `--job.image` | `vllm/vllm-openai:latest` | Runtime image for the pod. |
|
||||
| `--job.timeout` | `2h` | Wall-clock cap. Raise it for large datasets. |
|
||||
| `--job.detach` | `false` | Submit and exit instead of streaming logs. |
|
||||
| `--job.lerobot_ref` | `main` | Git ref of lerobot installed on the pod — point it at a branch to test changes. |
|
||||
| `--job.tags` | `[]` | Extra tags on the job and on any dataset it pushes (`lerobot` is always added). |
|
||||
|
||||
For a bigger dataset, scale to `h200x4` and raise
|
||||
`--vlm.parallel_servers` / `--vlm.num_gpus` to match, and give the job
|
||||
more headroom with e.g. `--job.timeout=8h`.
|
||||
|
||||
Remote runs need `--repo_id` (the pod pulls the dataset from the Hub;
|
||||
`--root` names a directory only your machine has). A dataset that exists
|
||||
only in your local cache is pushed to a **private** repo first.
|
||||
|
||||
## Key options
|
||||
|
||||
|
||||
@@ -165,6 +165,8 @@ Batches are flat dictionaries keyed by the constants in [`lerobot.utils.constant
|
||||
|
||||
LeRobot uses `PolicyProcessorPipeline`s to normalize inputs and de-normalize outputs around your policy. For a concrete reference, see [`processor_act.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/act/processor_act.py) or [`processor_diffusion.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/diffusion/processor_diffusion.py).
|
||||
|
||||
Pay close attention here: processors are the most common reproducibility pain point. A mismatch in normalization mode (`IDENTITY` vs `MEAN_STD` vs `MIN_MAX` vs `QUANTILES`/`QUANTILE10`) or in which features get normalized will train and eval without erroring, yet silently wreck results. Make sure the modes match how the checkpoint was trained, that the required stats exist (e.g. `QUANTILES` needs `q01`/`q99`), and that the pre- and post-processors stay consistent.
|
||||
|
||||
```python
|
||||
# processor_my_policy.py
|
||||
from typing import Any
|
||||
@@ -304,7 +306,9 @@ Mirror an existing policy that's structurally similar to yours; the diff is smal
|
||||
|
||||
### Heavy / optional dependencies
|
||||
|
||||
Most policies need a heavy backbone (transformers, diffusers, a specific VLM SDK). The convention is **two-step gating**: a `TYPE_CHECKING`-guarded import at module top, and a `require_package` runtime check in the constructor. [`modeling_diffusion.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/diffusion/modeling_diffusion.py) is the canonical reference:
|
||||
Most policies need a heavy backbone (transformers, diffusers, a specific VLM SDK). Wherever one exists, prefer loading it e.g from `transformers` or `diffusers` rather than re-implementing the architecture in-tree.
|
||||
|
||||
The convention is **two-step gating**: a `TYPE_CHECKING`-guarded import at module top, and a `require_package` runtime check in the constructor. [`modeling_diffusion.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/diffusion/modeling_diffusion.py) is the canonical reference:
|
||||
|
||||
```python
|
||||
from typing import TYPE_CHECKING
|
||||
@@ -374,6 +378,7 @@ The general expectations are in [`CONTRIBUTING.md`](https://github.com/huggingfa
|
||||
- [ ] Optional deps live behind a `[project.optional-dependencies]` extra and the `TYPE_CHECKING + require_package` guard.
|
||||
- [ ] `tests/policies/` updated; backward-compat artifact committed & policy-specific tests.
|
||||
- [ ] `src/lerobot/policies/<name>/README.md` symlinked into `docs/source/policy_<name>_README.md`; user-facing `docs/source/<name>.mdx` written and added to `_toctree.yml`.
|
||||
- [ ] `lerobot-train --policy.type my_policy ...` runs end-to-end for at least a few steps + save a checkpoint that can be loaded and run by `lerobot-eval` or `lerobot-rollout`.
|
||||
- [ ] `templates/lerobot_modelcard_template.md` has a description entry and a `policy_docs` link for your policy.
|
||||
- [ ] The models table in the root `README.md` lists your policy in the right category, linking to your doc page.
|
||||
- [ ] At least one reproducible benchmark eval in the policy MDX with a published checkpoint (sim benchmark, or real-robot dataset + checkpoint).
|
||||
|
||||
@@ -1,3 +1,11 @@
|
||||
# OMX
|
||||
|
||||
<img
|
||||
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/omx_mainimage.png"
|
||||
alt="OMX"
|
||||
width=600
|
||||
/>
|
||||
|
||||
## Order and Assemble the parts
|
||||
|
||||
First, assemble the OMX hardware following the official assembly guide.
|
||||
|
||||
@@ -252,6 +252,10 @@ lerobot-dataset-viz \
|
||||
--episode-index 0
|
||||
```
|
||||
|
||||
For a private or gated dataset, authenticate first with `hf auth login`, or set the
|
||||
`HF_TOKEN` environment variable. The Hub client then discovers the credential
|
||||
automatically; no token argument is needed.
|
||||
|
||||
**From a local folder:**
|
||||
Add the `--root` option and set `--mode local`. For example, to search in `./my_local_data_dir/lerobot/pusht`:
|
||||
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""Launch ``lerobot-annotate`` on a Hugging Face job (vllm + Qwen3.6-27B VLM).
|
||||
|
||||
Spawns one single-GPU ``h200`` job that:
|
||||
|
||||
1. installs ``lerobot`` from ``main`` plus the annotation extras,
|
||||
2. boots one vllm server with Qwen3.6-27B (dense VLM),
|
||||
3. runs the plan / interjections / vqa modules across the dataset
|
||||
in free-form mode (each episode generates its own subtasks +
|
||||
memory),
|
||||
4. uploads the annotated dataset to ``--new_repo_id`` (when set)
|
||||
or back to ``--repo_id``.
|
||||
|
||||
Usage:
|
||||
|
||||
HF_TOKEN=hf_... uv run python examples/annotations/run_hf_job.py
|
||||
|
||||
Adjust ``CMD`` (dataset, model, hub repo) and ``flavor`` below for your
|
||||
run. For larger datasets, scale to ``h200x4`` and raise
|
||||
``--vlm.parallel_servers`` / ``--vlm.num_gpus`` to match.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from huggingface_hub import get_token, run_job
|
||||
|
||||
token = os.environ.get("HF_TOKEN") or get_token()
|
||||
if not token:
|
||||
raise RuntimeError("No HF token. Run `huggingface-cli login` or `export HF_TOKEN=hf_...`")
|
||||
|
||||
CMD = (
|
||||
"apt-get update -qq && apt-get install -y -qq git ffmpeg && "
|
||||
"pip install --no-deps "
|
||||
"'lerobot @ git+https://github.com/huggingface/lerobot.git@main' && "
|
||||
# Pins mirror pyproject.toml — unpinned installs pull av 18 / datasets 5 /
|
||||
# draccus 0.11, which break lerobot at import time.
|
||||
"pip install --upgrade-strategy only-if-needed "
|
||||
"'datasets>=4.7.0,<5.0.0' 'pyarrow>=21.0.0,<30.0.0' 'av>=15.0.0,<16.0.0' 'draccus==0.10.0' "
|
||||
"'pandas>=2.0.0,<3.0.0' jsonlines gymnasium torchcodec mergedeep pyyaml-include toml typing-inspect "
|
||||
"openai && "
|
||||
"export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 && "
|
||||
"export VLLM_VIDEO_BACKEND=pyav && "
|
||||
"lerobot-annotate "
|
||||
"--repo_id=pepijn223/robocasa_pretrain_human300_v4 "
|
||||
"--new_repo_id=pepijn223/robocasa_pretrain_human300_v4_annotated "
|
||||
"--push_to_hub=true "
|
||||
"--vlm.backend=openai "
|
||||
"--vlm.model_id=Qwen/Qwen3.6-27B "
|
||||
"--vlm.num_gpus=1 "
|
||||
'--vlm.serve_command="vllm serve Qwen/Qwen3.6-27B '
|
||||
"--tensor-parallel-size 1 --max-model-len 32768 "
|
||||
'--gpu-memory-utilization 0.8 --uvicorn-log-level warning --port {port}" '
|
||||
"--vlm.serve_ready_timeout_s=1800 "
|
||||
# Qwen3.6 ships with thinking on; annotation wants plain JSON answers.
|
||||
"--vlm.chat_template_kwargs='{\"enable_thinking\": false}'"
|
||||
)
|
||||
|
||||
job = run_job(
|
||||
image="vllm/vllm-openai:latest",
|
||||
command=["bash", "-c", CMD],
|
||||
flavor="h200",
|
||||
secrets={"HF_TOKEN": token},
|
||||
timeout="2h",
|
||||
)
|
||||
print(f"Job URL: {job.url}")
|
||||
print(f"Job ID: {job.id}")
|
||||
+1
-1
@@ -155,7 +155,7 @@ accelerate-dep = ["accelerate>=1.14.0,<2.0.0"]
|
||||
can-dep = ["python-can>=4.2.0,<5.0.0"]
|
||||
peft-dep = ["peft>=0.18.0,<1.0.0"]
|
||||
scipy-dep = ["scipy>=1.14.0,<2.0.0"]
|
||||
diffusers-dep = ["diffusers>=0.27.2,<0.36.0"]
|
||||
diffusers-dep = ["diffusers>=0.38.0,<0.40.0"]
|
||||
qwen-vl-utils-dep = ["qwen-vl-utils>=0.0.11,<0.1.0"]
|
||||
matplotlib-dep = ["matplotlib>=3.10.3,<4.0.0", "contourpy>=1.3.0,<2.0.0"] # NOTE: Explicitly listing contourpy helps the resolver converge faster.
|
||||
pyserial-dep = ["pyserial>=3.5,<4.0"]
|
||||
|
||||
@@ -20,6 +20,29 @@ from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from lerobot.configs.default import JobConfig
|
||||
|
||||
# The annotation pipeline boots its own vLLM server, so the pod starts from the
|
||||
# official vLLM runtime rather than the prebuilt `lerobot-gpu` training image;
|
||||
# `lerobot` is pip-installed on top (see `lerobot.jobs.annotate`).
|
||||
DEFAULT_ANNOTATE_JOB_IMAGE = "vllm/vllm-openai:latest"
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnnotationJobConfig(JobConfig):
|
||||
"""`JobConfig` with the annotation runtime's defaults.
|
||||
|
||||
Adds `lerobot_ref` because the vLLM image ships no lerobot: the pod installs
|
||||
it from git, and the ref decides which code actually annotates. Point it at a
|
||||
branch/tag/SHA to try unmerged changes remotely.
|
||||
"""
|
||||
|
||||
image: str = DEFAULT_ANNOTATE_JOB_IMAGE
|
||||
# Annotation is a bounded pass over a dataset; a tighter cap than training's
|
||||
# "2d" keeps a wedged vLLM server from burning a day of GPU time.
|
||||
timeout: str | None = "2h"
|
||||
lerobot_ref: str = "main"
|
||||
|
||||
|
||||
@dataclass
|
||||
class PlanConfig:
|
||||
@@ -207,6 +230,11 @@ class AnnotationPipelineConfig:
|
||||
vlm: VlmConfig = field(default_factory=VlmConfig)
|
||||
executor: ExecutorConfig = field(default_factory=ExecutorConfig)
|
||||
|
||||
# Where the annotation runs: omitted / "local" annotates on this machine, any
|
||||
# other value is an HF Jobs flavor (e.g. "h200") and submits the run there.
|
||||
# List flavors + pricing with `hf jobs hardware`.
|
||||
job: AnnotationJobConfig = field(default_factory=AnnotationJobConfig)
|
||||
|
||||
skip_validation: bool = False
|
||||
only_episodes: tuple[int, ...] | None = None
|
||||
|
||||
|
||||
@@ -30,8 +30,8 @@ Phase 3 is why the ``plan`` module must be re-entered after the
|
||||
timestamps.
|
||||
|
||||
Distributed execution is provided by Hugging Face Jobs (see
|
||||
``examples/annotations/run_hf_job.py``); the runner inside the job
|
||||
invokes ``lerobot-annotate`` which uses this in-process executor.
|
||||
``lerobot.jobs.annotate``, reached via ``--job.target=<flavor>``); the pod
|
||||
inside the job invokes ``lerobot-annotate`` which uses this in-process executor.
|
||||
Episode-level concurrency is controlled by
|
||||
``ExecutorConfig.episode_parallelism``.
|
||||
"""
|
||||
|
||||
@@ -194,12 +194,13 @@ def make_vlm_client(config: VlmConfig) -> VlmClient:
|
||||
"""Build the shared VLM client.
|
||||
|
||||
Only the ``openai`` backend is supported for now. The shipped workflow
|
||||
is Hugging Face Jobs (``examples/annotations/run_hf_job.py``): it boots
|
||||
a vLLM server inside the ``vllm/vllm-openai`` image and the pipeline
|
||||
talks to it over the OpenAI-compatible API (``--vlm.backend=openai``,
|
||||
optionally auto-spawning the server via ``auto_serve`` /
|
||||
``serve_command``). The former in-process ``vllm`` / ``transformers``
|
||||
backends were removed to keep the support surface to the HF Jobs path.
|
||||
is Hugging Face Jobs (``lerobot-annotate --job.target=<flavor>``): it
|
||||
boots a vLLM server inside the ``vllm/vllm-openai`` image and the
|
||||
pipeline talks to it over the OpenAI-compatible API
|
||||
(``--vlm.backend=openai``, optionally auto-spawning the server via
|
||||
``auto_serve`` / ``serve_command``). The former in-process ``vllm`` /
|
||||
``transformers`` backends were removed to keep the support surface to
|
||||
the HF Jobs path.
|
||||
|
||||
For ``stub``, construct :class:`StubVlmClient` directly with a responder
|
||||
callable; it is rejected here to make accidental misuse obvious.
|
||||
@@ -213,8 +214,8 @@ def make_vlm_client(config: VlmConfig) -> VlmClient:
|
||||
if config.backend in {"vllm", "transformers"}:
|
||||
raise ValueError(
|
||||
f"backend={config.backend!r} (in-process local model) is not supported for now — "
|
||||
"only backend='openai' (the Hugging Face Jobs flow) is. Run the pipeline via "
|
||||
"examples/annotations/run_hf_job.py, which serves the model with vLLM in the "
|
||||
"only backend='openai' (the Hugging Face Jobs flow) is. Run the pipeline with "
|
||||
"`lerobot-annotate --job.target=<flavor>`, which serves the model with vLLM in the "
|
||||
"vllm/vllm-openai image and talks to it over the OpenAI-compatible API."
|
||||
)
|
||||
raise ValueError(f"Unknown VLM backend: {config.backend!r}")
|
||||
|
||||
@@ -173,7 +173,8 @@ class Reachy2Camera(Camera):
|
||||
raise ValueError(
|
||||
f"Invalid color mode '{self.color_mode}'. Expected {ColorMode.RGB} or {ColorMode.BGR}."
|
||||
)
|
||||
if self.color_mode == ColorMode.RGB:
|
||||
is_depth_frame = self.config.name == "depth" and self.config.image_type == "depth"
|
||||
if not is_depth_frame and self.color_mode == ColorMode.RGB:
|
||||
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
||||
|
||||
self.latest_frame = frame
|
||||
|
||||
@@ -453,7 +453,7 @@ class RealSenseCamera(Camera):
|
||||
)
|
||||
|
||||
processed_image = image
|
||||
if self.color_mode == ColorMode.BGR:
|
||||
if not depth_frame and self.color_mode == ColorMode.BGR:
|
||||
processed_image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
|
||||
|
||||
if self.rotation in [cv2.ROTATE_90_CLOCKWISE, cv2.ROTATE_90_COUNTERCLOCKWISE, cv2.ROTATE_180]:
|
||||
|
||||
@@ -73,6 +73,8 @@ class LeRobotDatasetMetadata:
|
||||
revision: str | None = None,
|
||||
force_cache_sync: bool = False,
|
||||
metadata_buffer_size: int = 10,
|
||||
*,
|
||||
token: str | bool | None = None,
|
||||
):
|
||||
"""Load or download metadata for an existing LeRobot dataset.
|
||||
|
||||
@@ -94,6 +96,10 @@ class LeRobotDatasetMetadata:
|
||||
even when local files exist.
|
||||
metadata_buffer_size: Number of episode metadata records to buffer
|
||||
in memory before flushing to parquet.
|
||||
token: Authentication token used for Hub requests. Pass a string
|
||||
token, ``True`` to require the locally stored token, ``False``
|
||||
to disable authentication, or ``None`` to use the Hugging Face
|
||||
Hub default.
|
||||
"""
|
||||
self.repo_id = repo_id
|
||||
self.revision = revision if revision else CODEBASE_VERSION
|
||||
@@ -113,9 +119,12 @@ class LeRobotDatasetMetadata:
|
||||
self._load_metadata()
|
||||
except (FileNotFoundError, NotADirectoryError):
|
||||
if is_valid_version(self.revision):
|
||||
self.revision = get_safe_version(self.repo_id, self.revision)
|
||||
if token is None:
|
||||
self.revision = get_safe_version(self.repo_id, self.revision)
|
||||
else:
|
||||
self.revision = get_safe_version(self.repo_id, self.revision, token=token)
|
||||
|
||||
self._pull_from_repo(allow_patterns="meta/")
|
||||
self._pull_from_repo(allow_patterns="meta/", token=token)
|
||||
self._load_metadata()
|
||||
|
||||
def _flush_metadata_buffer(self) -> None:
|
||||
@@ -220,7 +229,10 @@ class LeRobotDatasetMetadata:
|
||||
self,
|
||||
allow_patterns: list[str] | str | None = None,
|
||||
ignore_patterns: list[str] | str | None = None,
|
||||
*,
|
||||
token: str | bool | None = None,
|
||||
) -> None:
|
||||
token_kwargs = {} if token is None else {"token": token}
|
||||
if self._requested_root is None:
|
||||
self.root = Path(
|
||||
snapshot_download(
|
||||
@@ -230,6 +242,7 @@ class LeRobotDatasetMetadata:
|
||||
cache_dir=HF_LEROBOT_HUB_CACHE,
|
||||
allow_patterns=allow_patterns,
|
||||
ignore_patterns=ignore_patterns,
|
||||
**token_kwargs,
|
||||
)
|
||||
)
|
||||
return
|
||||
@@ -242,6 +255,7 @@ class LeRobotDatasetMetadata:
|
||||
local_dir=self._requested_root,
|
||||
allow_patterns=allow_patterns,
|
||||
ignore_patterns=ignore_patterns,
|
||||
**token_kwargs,
|
||||
)
|
||||
self.root = self._requested_root
|
||||
|
||||
|
||||
@@ -65,6 +65,8 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
encoder_threads: int | None = None,
|
||||
streaming_encoding: bool = False,
|
||||
encoder_queue_maxsize: int = 30,
|
||||
*,
|
||||
token: str | bool | None = None,
|
||||
):
|
||||
"""
|
||||
2 modes are available for instantiating this class, depending on 2 different use cases:
|
||||
@@ -197,6 +199,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
instead of writing PNG images first. This makes save_episode() near-instant. Defaults to False.
|
||||
encoder_queue_maxsize (int, optional): Maximum number of frames to buffer per camera when using
|
||||
streaming encoding. Defaults to 30 (~1s at 30fps).
|
||||
token: Authentication token used while downloading this dataset
|
||||
from the Hub. Pass a string token, ``True`` to require the
|
||||
locally stored token, ``False`` to disable authentication, or
|
||||
``None`` to use the Hugging Face Hub default. The token is not
|
||||
retained on the dataset instance after initialization.
|
||||
|
||||
Note:
|
||||
Write-mode parameters (``streaming_encoding``, ``batch_encoding_size``) passed to
|
||||
@@ -220,7 +227,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
|
||||
# Load metadata (sets self.root once from the resolved metadata root)
|
||||
self.meta = LeRobotDatasetMetadata(
|
||||
self.repo_id, self._requested_root, self.revision, force_cache_sync=force_cache_sync
|
||||
self.repo_id,
|
||||
self._requested_root,
|
||||
self.revision,
|
||||
force_cache_sync=force_cache_sync,
|
||||
token=token,
|
||||
)
|
||||
self.root = self.meta.root
|
||||
self.revision = self.meta.revision
|
||||
@@ -260,8 +271,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
# Load actual data
|
||||
if force_cache_sync or not self.reader.try_load():
|
||||
if is_valid_version(self.revision):
|
||||
self.revision = get_safe_version(self.repo_id, self.revision)
|
||||
self._download(download_videos)
|
||||
if token is None:
|
||||
self.revision = get_safe_version(self.repo_id, self.revision)
|
||||
else:
|
||||
self.revision = get_safe_version(self.repo_id, self.revision, token=token)
|
||||
self._download(download_videos, token=token)
|
||||
self.reader.load_and_activate()
|
||||
|
||||
# Detect write-mode params for backward compatibility
|
||||
@@ -478,18 +492,19 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
"""Return the number of frames in the selected episodes."""
|
||||
return self.num_frames
|
||||
|
||||
def __getitem__(self, idx) -> dict:
|
||||
"""Return a single frame by index, with all transforms applied.
|
||||
def __getitem__(self, idx: int | slice) -> dict | list[dict]:
|
||||
"""Return one frame or a slice of frames, with all transforms applied.
|
||||
|
||||
Loads the frame from the underlying HF dataset, expands delta-timestamp
|
||||
windows, decodes video frames, and applies image transforms. Delegates
|
||||
the core logic to :meth:`DatasetReader.get_item`.
|
||||
the core logic to :class:`DatasetReader`.
|
||||
|
||||
Args:
|
||||
idx: Index into the (possibly episode-filtered) dataset.
|
||||
idx: Integer index or slice into the possibly episode-filtered dataset.
|
||||
|
||||
Returns:
|
||||
Dict mapping feature names to their tensor values for this frame.
|
||||
A frame dictionary for an integer index, or a list of frame
|
||||
dictionaries for a slice.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If the dataset is currently being recorded and
|
||||
@@ -499,6 +514,9 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
raise RuntimeError(
|
||||
"Cannot read from a dataset that is being recorded. Call finalize() first, then access items."
|
||||
)
|
||||
if isinstance(idx, slice):
|
||||
return [self[item_idx] for item_idx in range(*idx.indices(len(self)))]
|
||||
|
||||
reader = self._ensure_reader()
|
||||
if reader.hf_dataset is None:
|
||||
# One-shot load after finalize()
|
||||
@@ -622,10 +640,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
hub_api.delete_tag(self.repo_id, tag=CODEBASE_VERSION, repo_type="dataset")
|
||||
hub_api.create_tag(self.repo_id, tag=CODEBASE_VERSION, revision=branch, repo_type="dataset")
|
||||
|
||||
def _download(self, download_videos: bool = True) -> None:
|
||||
def _download(self, download_videos: bool = True, *, token: str | bool | None = None) -> None:
|
||||
"""Downloads the dataset from the given 'repo_id' at the provided version."""
|
||||
ignore_patterns = None if download_videos else "videos/"
|
||||
files = None
|
||||
token_kwargs = {} if token is None else {"token": token}
|
||||
if self.episodes is not None:
|
||||
# Reader is guaranteed to exist here (created in __init__ before _download)
|
||||
files = self.reader.get_episodes_file_paths()
|
||||
@@ -639,6 +658,7 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
cache_dir=HF_LEROBOT_HUB_CACHE,
|
||||
allow_patterns=files,
|
||||
ignore_patterns=ignore_patterns,
|
||||
**token_kwargs,
|
||||
)
|
||||
)
|
||||
else:
|
||||
@@ -650,6 +670,7 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
local_dir=self._requested_root,
|
||||
allow_patterns=files,
|
||||
ignore_patterns=ignore_patterns,
|
||||
**token_kwargs,
|
||||
)
|
||||
self.meta.root = self._requested_root
|
||||
|
||||
@@ -789,6 +810,8 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
image_writer_threads: int = 0,
|
||||
streaming_encoding: bool = False,
|
||||
encoder_queue_maxsize: int = 30,
|
||||
*,
|
||||
token: str | bool | None = None,
|
||||
) -> "LeRobotDataset":
|
||||
"""Resume recording on an existing dataset.
|
||||
|
||||
@@ -822,6 +845,8 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
streaming_encoding: If ``True``, encode video in real-time during
|
||||
capture.
|
||||
encoder_queue_maxsize: Max buffered frames per camera for streaming.
|
||||
token: Authentication token used if metadata must be downloaded
|
||||
from the Hub. The token is not retained on the dataset instance.
|
||||
|
||||
Returns:
|
||||
A :class:`LeRobotDataset` in write mode, ready to append episodes.
|
||||
@@ -850,7 +875,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
||||
|
||||
# Load metadata (revision-safe when root is not provided)
|
||||
obj.meta = LeRobotDatasetMetadata(
|
||||
obj.repo_id, obj._requested_root, obj.revision, force_cache_sync=force_cache_sync
|
||||
obj.repo_id,
|
||||
obj._requested_root,
|
||||
obj.revision,
|
||||
force_cache_sync=force_cache_sync,
|
||||
token=token,
|
||||
)
|
||||
|
||||
obj._encoder_threads = encoder_threads
|
||||
|
||||
@@ -48,6 +48,8 @@ class MultiLeRobotDataset(torch.utils.data.Dataset):
|
||||
tolerances_s: dict | None = None,
|
||||
download_videos: bool = True,
|
||||
video_backend: str | None = None,
|
||||
*,
|
||||
token: str | bool | None = None,
|
||||
):
|
||||
super().__init__()
|
||||
self.repo_ids = repo_ids
|
||||
@@ -65,6 +67,7 @@ class MultiLeRobotDataset(torch.utils.data.Dataset):
|
||||
tolerance_s=self.tolerances_s[repo_id],
|
||||
download_videos=download_videos,
|
||||
video_backend=video_backend,
|
||||
token=token,
|
||||
)
|
||||
for repo_id in repo_ids
|
||||
]
|
||||
|
||||
@@ -256,6 +256,8 @@ class StreamingLeRobotDataset(torch.utils.data.IterableDataset):
|
||||
shuffle: bool = True,
|
||||
return_uint8: bool = False,
|
||||
depth_output_unit: str = DEFAULT_DEPTH_UNIT,
|
||||
*,
|
||||
token: str | bool | None = None,
|
||||
):
|
||||
"""Initialize a StreamingLeRobotDataset.
|
||||
|
||||
@@ -278,6 +280,11 @@ class StreamingLeRobotDataset(torch.utils.data.IterableDataset):
|
||||
shuffle (bool, optional): Whether to shuffle the dataset across exhaustions. Defaults to True.
|
||||
depth_output_unit (str, optional): Physical unit depth maps are dequantized to ("m" or "mm").
|
||||
Defaults to "mm".
|
||||
token: Authentication token used while streaming this dataset from
|
||||
the Hub. Pass a string token, ``True`` to require the locally
|
||||
stored token, ``False`` to disable authentication, or ``None``
|
||||
to use the Hugging Face Hub default. The token is not retained
|
||||
on the dataset instance after initialization.
|
||||
"""
|
||||
super().__init__()
|
||||
self.repo_id = repo_id
|
||||
@@ -306,7 +313,11 @@ class StreamingLeRobotDataset(torch.utils.data.IterableDataset):
|
||||
|
||||
# Load metadata
|
||||
self.meta = LeRobotDatasetMetadata(
|
||||
self.repo_id, self._requested_root, self.revision, force_cache_sync=force_cache_sync
|
||||
self.repo_id,
|
||||
self._requested_root,
|
||||
self.revision,
|
||||
force_cache_sync=force_cache_sync,
|
||||
token=token,
|
||||
)
|
||||
self.root = self.meta.root
|
||||
self.revision = self.meta.revision
|
||||
@@ -334,12 +345,14 @@ class StreamingLeRobotDataset(torch.utils.data.IterableDataset):
|
||||
self.delta_timestamps = delta_timestamps
|
||||
self.delta_indices = get_delta_indices(self.delta_timestamps, self.fps)
|
||||
|
||||
token_kwargs = {} if token is None or self.streaming_from_local else {"token": token}
|
||||
self.hf_dataset: datasets.IterableDataset = load_dataset(
|
||||
self.repo_id if not self.streaming_from_local else str(self.root),
|
||||
split="train",
|
||||
streaming=self.streaming,
|
||||
data_files="data/*/*.parquet",
|
||||
revision=self.revision,
|
||||
**token_kwargs,
|
||||
)
|
||||
|
||||
self.num_shards = min(self.hf_dataset.num_shards, max_num_shards)
|
||||
|
||||
@@ -325,16 +325,19 @@ def check_version_compatibility(
|
||||
logging.warning(FUTURE_MESSAGE.format(repo_id=repo_id, version=v_check))
|
||||
|
||||
|
||||
def get_repo_versions(repo_id: str) -> list[packaging.version.Version]:
|
||||
def get_repo_versions(repo_id: str, *, token: str | bool | None = None) -> list[packaging.version.Version]:
|
||||
"""Return available valid versions (branches and tags) on a given Hub repo.
|
||||
|
||||
Args:
|
||||
repo_id (str): The repository ID on the Hugging Face Hub.
|
||||
token: Authentication token used for Hub requests. Pass a string token,
|
||||
``True`` to require the locally stored token, ``False`` to disable
|
||||
authentication, or ``None`` to use the Hugging Face Hub default.
|
||||
|
||||
Returns:
|
||||
list[packaging.version.Version]: A list of valid versions found.
|
||||
"""
|
||||
api = HfApi()
|
||||
api = HfApi() if token is None else HfApi(token=token)
|
||||
repo_refs = api.list_repo_refs(repo_id, repo_type="dataset")
|
||||
repo_refs = [b.name for b in repo_refs.branches + repo_refs.tags]
|
||||
repo_versions = []
|
||||
@@ -345,7 +348,12 @@ def get_repo_versions(repo_id: str) -> list[packaging.version.Version]:
|
||||
return repo_versions
|
||||
|
||||
|
||||
def get_safe_version(repo_id: str, version: str | packaging.version.Version) -> str:
|
||||
def get_safe_version(
|
||||
repo_id: str,
|
||||
version: str | packaging.version.Version,
|
||||
*,
|
||||
token: str | bool | None = None,
|
||||
) -> str:
|
||||
"""Return the specified version if available on repo, or the latest compatible one.
|
||||
|
||||
If the exact version is not found, it looks for the latest version with the
|
||||
@@ -354,6 +362,7 @@ def get_safe_version(repo_id: str, version: str | packaging.version.Version) ->
|
||||
Args:
|
||||
repo_id (str): The repository ID on the Hugging Face Hub.
|
||||
version (str | packaging.version.Version): The target version.
|
||||
token: Authentication token forwarded to the Hub version lookup.
|
||||
|
||||
Returns:
|
||||
str: The safe version string (e.g., "v1.2.3") to use as a revision.
|
||||
@@ -366,7 +375,7 @@ def get_safe_version(repo_id: str, version: str | packaging.version.Version) ->
|
||||
target_version = (
|
||||
packaging.version.parse(version) if not isinstance(version, packaging.version.Version) else version
|
||||
)
|
||||
hub_versions = get_repo_versions(repo_id)
|
||||
hub_versions = get_repo_versions(repo_id) if token is None else get_repo_versions(repo_id, token=token)
|
||||
|
||||
if not hub_versions:
|
||||
raise RevisionNotFoundError(
|
||||
|
||||
@@ -322,7 +322,7 @@ class HILSerlRobotEnvConfig(EnvConfig):
|
||||
class LiberoEnv(EnvConfig):
|
||||
task: str = "libero_10" # can also choose libero_spatial, libero_object, etc.
|
||||
task_ids: list[int] | None = None
|
||||
fps: int = 30
|
||||
fps: int = 20 # Must match robosuite's default control_freq (20 Hz)
|
||||
episode_length: int | None = None
|
||||
obs_type: str = "pixels_agent_pos"
|
||||
render_mode: str = "rgb_array"
|
||||
@@ -354,6 +354,9 @@ class LiberoEnv(EnvConfig):
|
||||
control_mode: str = "relative" # or "absolute"
|
||||
|
||||
def __post_init__(self):
|
||||
if self.fps <= 0:
|
||||
raise ValueError(f"fps must be positive, got {self.fps}")
|
||||
|
||||
if self.obs_type == "pixels":
|
||||
self.features[LIBERO_KEY_PIXELS_AGENTVIEW] = PolicyFeature(
|
||||
type=FeatureType.VISUAL, shape=(self.observation_height, self.observation_width, 3)
|
||||
@@ -412,6 +415,7 @@ class LiberoEnv(EnvConfig):
|
||||
"render_mode": self.render_mode,
|
||||
"observation_height": self.observation_height,
|
||||
"observation_width": self.observation_width,
|
||||
"control_freq": self.fps,
|
||||
}
|
||||
if self.task_ids is not None:
|
||||
kwargs["task_ids"] = self.task_ids
|
||||
|
||||
@@ -125,10 +125,13 @@ class LiberoEnv(gym.Env):
|
||||
n_envs: int = 1,
|
||||
camera_name_mapping: dict[str, str] | None = None,
|
||||
num_steps_wait: int = 10,
|
||||
control_freq: int = 20,
|
||||
control_mode: str = "relative",
|
||||
is_libero_plus: bool = False,
|
||||
):
|
||||
super().__init__()
|
||||
if control_freq <= 0:
|
||||
raise ValueError(f"control_freq must be positive, got {control_freq}")
|
||||
self.task_id = task_id
|
||||
self.is_libero_plus = is_libero_plus
|
||||
self.obs_type = obs_type
|
||||
@@ -154,6 +157,7 @@ class LiberoEnv(gym.Env):
|
||||
}
|
||||
self.camera_name_mapping = camera_name_mapping
|
||||
self.num_steps_wait = num_steps_wait
|
||||
self.control_freq = control_freq
|
||||
self.episode_index = episode_index
|
||||
self.episode_length = episode_length
|
||||
# Load once and keep
|
||||
@@ -260,6 +264,7 @@ class LiberoEnv(gym.Env):
|
||||
bddl_file_name=self._task_bddl_file,
|
||||
camera_heights=self.observation_height,
|
||||
camera_widths=self.observation_width,
|
||||
control_freq=self.control_freq,
|
||||
)
|
||||
env.reset()
|
||||
self._env = env
|
||||
|
||||
@@ -18,6 +18,7 @@ from lerobot.utils.import_utils import require_package
|
||||
# guard the optional dependency here so importing this package fails loudly if it's missing.
|
||||
require_package("datasets", extra="dataset")
|
||||
|
||||
from .annotate import submit_annotate_to_hf
|
||||
from .hf import submit_to_hf
|
||||
|
||||
__all__ = ["submit_to_hf"]
|
||||
__all__ = ["submit_annotate_to_hf", "submit_to_hf"]
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""Run ``lerobot-annotate`` on HF Jobs (HuggingFace GPUs).
|
||||
|
||||
Same shape as the training submitter in ``hf.py``, with one difference: the
|
||||
annotation pipeline serves its own VLM, so the pod starts from the official
|
||||
``vllm/vllm-openai`` image (which has no lerobot) instead of the prebuilt
|
||||
``lerobot-gpu`` image, and installs lerobot on top before running.
|
||||
|
||||
Because there is no config repo to stage, the pod replays the user's own CLI
|
||||
flags — everything except the client-only ``--job.*`` and the host-local
|
||||
``--root``, which is replaced by ``--repo_id`` so the pod pulls the dataset
|
||||
from the Hub.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shlex
|
||||
import sys
|
||||
from dataclasses import is_dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from huggingface_hub import HfApi, get_token, run_job
|
||||
|
||||
from .dataset import ensure_dataset_available
|
||||
|
||||
# Package-internal reuse of the training submitter's job plumbing: following a
|
||||
# submitted job and forwarding argv are identical for annotation runs.
|
||||
from .hf import _pod_forwarded_args, follow_job, resolve_job_tags
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from lerobot.annotations.steerable_pipeline.config import AnnotationPipelineConfig
|
||||
|
||||
LEROBOT_GIT_URL = "https://github.com/huggingface/lerobot.git"
|
||||
|
||||
# Mirrors the pins in pyproject.toml. The vLLM image resolves dependencies on its
|
||||
# own otherwise, and pulls av 18 / datasets 5 / draccus 0.11 — each of which breaks
|
||||
# lerobot at import time. `--upgrade-strategy only-if-needed` keeps vLLM's own
|
||||
# (torch, transformers, ...) pins intact.
|
||||
_RUNTIME_REQUIREMENTS = (
|
||||
"'datasets>=4.7.0,<5.0.0' 'pyarrow>=21.0.0,<30.0.0' 'av>=15.0.0,<16.0.0' 'draccus==0.10.0' "
|
||||
"'pandas>=2.0.0,<3.0.0' jsonlines gymnasium torchcodec mergedeep pyyaml-include toml typing-inspect "
|
||||
"openai"
|
||||
)
|
||||
|
||||
# Flags the submitter resolves itself instead of forwarding verbatim: `--root`
|
||||
# names a directory only this machine has, `--repo_id` is re-emitted from the
|
||||
# config, and the config-file args name local files (rejected up front by
|
||||
# `submit_annotate_to_hf`). `--job.*` is dropped separately, by prefix; bare
|
||||
# `--job` is not, hence its entry here — it is the one arg that could smuggle a
|
||||
# remote `target` onto the pod and have the job recursively submit itself.
|
||||
_SUBMITTER_OWNED_ARGS = ("--root", "--repo_id", "--config_path", "--job")
|
||||
|
||||
|
||||
def _local_config_file_args(cfg: AnnotationPipelineConfig) -> list[str]:
|
||||
"""The CLI args that name a config file on the client's disk.
|
||||
|
||||
draccus exposes ``--config_path`` for the whole config plus a ``--<field>``
|
||||
for every nested dataclass (``--vlm``, ``--plan``, ``--job``, ...). The pod has
|
||||
none of those files, so a remote run has to reject them rather than silently
|
||||
drop the settings they carry.
|
||||
"""
|
||||
return ["--config_path", *(f"--{name}" for name in vars(cfg) if is_dataclass(getattr(cfg, name)))]
|
||||
|
||||
|
||||
def build_pod_setup(lerobot_ref: str) -> str:
|
||||
"""Shell prelude that turns the vLLM image into a ``lerobot-annotate`` runtime."""
|
||||
spec = f"lerobot @ git+{LEROBOT_GIT_URL}@{lerobot_ref}"
|
||||
return (
|
||||
# git to install from the repo, ffmpeg to decode the dataset's videos.
|
||||
"apt-get update -qq && apt-get install -y -qq git ffmpeg && "
|
||||
f"pip install --no-deps {shlex.quote(spec)} && "
|
||||
f"pip install --upgrade-strategy only-if-needed {_RUNTIME_REQUIREMENTS} && "
|
||||
# vLLM's cudagraph memory estimate over-reserves and starves the KV cache;
|
||||
# PyAV is the video backend the server can decode our frames with.
|
||||
"export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 && "
|
||||
"export VLLM_VIDEO_BACKEND=pyav"
|
||||
)
|
||||
|
||||
|
||||
def build_pod_command(repo_id: str, lerobot_ref: str, argv: list[str]) -> list[str]:
|
||||
"""Build the ``bash -c`` command the pod runs: setup prelude, then annotation.
|
||||
|
||||
``argv`` is the user's CLI (``sys.argv[1:]``) minus the flags in
|
||||
``_SUBMITTER_OWNED_ARGS``; ``--repo_id`` is re-added from the config so the pod
|
||||
always annotates the dataset we just made sure is reachable on the Hub.
|
||||
``--job.target=local`` stops the pod from re-dispatching to itself.
|
||||
"""
|
||||
forwarded = _pod_forwarded_args(argv, drop_names=_SUBMITTER_OWNED_ARGS, drop_prefixes=("--job.",))
|
||||
annotate = shlex.join(["lerobot-annotate", f"--repo_id={repo_id}", *forwarded, "--job.target=local"])
|
||||
return ["bash", "-c", f"{build_pod_setup(lerobot_ref)} && {annotate}"]
|
||||
|
||||
|
||||
def submit_annotate_to_hf(cfg: AnnotationPipelineConfig) -> None:
|
||||
"""Submit an annotation run to HF Jobs infrastructure.
|
||||
|
||||
Resolves credentials, makes sure the source dataset is reachable from the pod,
|
||||
submits the job, then tails its logs until the job reaches a terminal stage —
|
||||
or returns immediately with ``--job.detach``. Ctrl-C detaches without
|
||||
cancelling the remote job.
|
||||
"""
|
||||
token = get_token()
|
||||
if not token:
|
||||
raise RuntimeError("Not logged in to Hugging Face. Run `hf auth login` first.")
|
||||
|
||||
if cfg.repo_id is None:
|
||||
raise ValueError(
|
||||
"Remote annotation requires --repo_id: the pod downloads the dataset from the Hub, "
|
||||
"and --root only names a directory on this machine."
|
||||
)
|
||||
|
||||
argv = sys.argv[1:]
|
||||
passed = {tok.split("=", 1)[0] for tok in argv}
|
||||
used_config_files = sorted(passed.intersection(_local_config_file_args(cfg)))
|
||||
if used_config_files:
|
||||
raise ValueError(
|
||||
f"{', '.join(used_config_files)} cannot be used with a remote --job.target: the pod "
|
||||
"cannot read config files from this machine. Pass the settings as CLI flags instead."
|
||||
)
|
||||
|
||||
if not cfg.push_to_hub:
|
||||
# The pod's filesystem is discarded when the job ends, so without a push the
|
||||
# run produces nothing. Warn rather than fail: a smoke test over
|
||||
# --only_episodes that only inspects the logs is a legitimate use.
|
||||
print(
|
||||
"WARNING: --push_to_hub is off. The annotated dataset lives only on the pod and is "
|
||||
"discarded when the job ends. Pass --push_to_hub=true to keep the result."
|
||||
)
|
||||
|
||||
api = HfApi(token=token)
|
||||
tags = resolve_job_tags(cfg.job.tags)
|
||||
ensure_dataset_available(cfg.repo_id, api=api, tags=tags)
|
||||
|
||||
command = build_pod_command(cfg.repo_id, cfg.job.lerobot_ref, argv)
|
||||
|
||||
print(f"Submitting job to HF Jobs (flavor={cfg.job.target}, image={cfg.job.image}) ...")
|
||||
job_info = run_job(
|
||||
image=cfg.job.image,
|
||||
command=command,
|
||||
flavor=cfg.job.target,
|
||||
secrets={"HF_TOKEN": token},
|
||||
timeout=cfg.job.timeout,
|
||||
# HF Jobs labels are key/value; expose each tag as a queryable label.
|
||||
labels=dict.fromkeys(tags, "true"),
|
||||
)
|
||||
job_id = job_info.id
|
||||
job_url = getattr(job_info, "url", None)
|
||||
print(f"Job submitted: {job_id}")
|
||||
if job_url:
|
||||
print(f" Job page: {job_url}")
|
||||
target_repo_id = cfg.new_repo_id or cfg.repo_id
|
||||
if cfg.push_to_hub:
|
||||
print(f" Dataset repo: https://huggingface.co/datasets/{target_repo_id}")
|
||||
print(f" Monitor: hf jobs logs {job_id}")
|
||||
print(f" Cancel: hf jobs cancel {job_id}")
|
||||
|
||||
# No success marker: `lerobot-annotate` keeps working after the upload log line
|
||||
# (dataset card, version tag), so completion has to be stage-based.
|
||||
if not follow_job(job_id, detach=cfg.job.detach):
|
||||
return
|
||||
|
||||
if cfg.push_to_hub:
|
||||
print(f"\nAnnotation complete — dataset pushed to https://huggingface.co/datasets/{target_repo_id}")
|
||||
else:
|
||||
print("\nAnnotation complete. Note: --push_to_hub was off, so the result stayed on the pod.")
|
||||
+69
-54
@@ -223,6 +223,74 @@ def _poll_until_done(
|
||||
return None
|
||||
|
||||
|
||||
def follow_job(job_id: str, *, detach: bool = False, success_marker: str | None = None) -> bool:
|
||||
"""Watch a submitted job to the end, streaming its logs to stdout.
|
||||
|
||||
Returns True when the job finished successfully and False when we stopped watching
|
||||
without a verdict — `detach`, or the user pressing Ctrl-C, which detaches rather than
|
||||
cancelling the remote job. Raises RuntimeError when the job reaches a terminal stage
|
||||
other than COMPLETED.
|
||||
|
||||
`success_marker` finishes as soon as that string appears in the logs instead of waiting
|
||||
out the platform's post-run finalization (~30s). Callers that have a log line meaning
|
||||
"the artifact is on the Hub" should pass it; without one, completion is stage-based.
|
||||
"""
|
||||
if detach:
|
||||
return False
|
||||
|
||||
done = threading.Event()
|
||||
detached = threading.Event()
|
||||
marker_seen = threading.Event()
|
||||
stage_holder: dict[str, str | None] = {}
|
||||
|
||||
def _poll() -> None:
|
||||
stage_holder["stage"] = _poll_until_done(job_id, done, status_holder=stage_holder)
|
||||
|
||||
poll_thread = threading.Thread(target=_poll, daemon=True)
|
||||
poll_thread.start()
|
||||
log_thread = threading.Thread(
|
||||
target=_tail_logs, args=(job_id, done, success_marker, marker_seen), daemon=True
|
||||
)
|
||||
log_thread.start()
|
||||
|
||||
def _detach(sig, frame):
|
||||
detached.set()
|
||||
done.set()
|
||||
print("\nDetached. Job is still running.")
|
||||
print(f" Monitor: hf jobs logs {job_id}")
|
||||
print(f" Cancel: hf jobs cancel {job_id}")
|
||||
|
||||
# signal.signal only works on the main thread; when called from a worker thread
|
||||
# (e.g. an orchestration framework) skip the Ctrl-C-detaches-instead-of-cancels
|
||||
# handler rather than crashing with ValueError.
|
||||
install_sigint = threading.current_thread() is threading.main_thread()
|
||||
original_sigint = signal.getsignal(signal.SIGINT) if install_sigint else None
|
||||
if install_sigint:
|
||||
signal.signal(signal.SIGINT, _detach)
|
||||
try:
|
||||
# Timeout-based join so SIGINT is delivered to the main thread promptly.
|
||||
while poll_thread.is_alive():
|
||||
poll_thread.join(timeout=0.5)
|
||||
log_thread.join(timeout=5)
|
||||
finally:
|
||||
if install_sigint:
|
||||
signal.signal(signal.SIGINT, original_sigint)
|
||||
|
||||
if detached.is_set():
|
||||
return False
|
||||
if marker_seen.is_set():
|
||||
return True
|
||||
|
||||
stage = stage_holder.get("stage")
|
||||
if stage != "COMPLETED":
|
||||
message = stage_holder.get("message")
|
||||
detail = f" ({message})" if message else ""
|
||||
raise RuntimeError(
|
||||
f"Job {job_id} ended with stage={stage}{detail}. Check logs: hf jobs logs {job_id}"
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def _pod_forwarded_args(
|
||||
argv: list[str], drop_names: tuple[str, ...] = (), drop_prefixes: tuple[str, ...] = ()
|
||||
) -> list[str]:
|
||||
@@ -362,64 +430,11 @@ def submit_to_hf(cfg: TrainPipelineConfig) -> None:
|
||||
print(f" Monitor: hf jobs logs {job_id}")
|
||||
print(f" Cancel: hf jobs cancel {job_id}")
|
||||
|
||||
if cfg.job.detach:
|
||||
return
|
||||
|
||||
done = threading.Event()
|
||||
detached = threading.Event()
|
||||
pushed_ok = threading.Event()
|
||||
stage_holder: dict[str, str | None] = {}
|
||||
|
||||
def _poll() -> None:
|
||||
stage_holder["stage"] = _poll_until_done(job_id, done, status_holder=stage_holder)
|
||||
|
||||
poll_thread = threading.Thread(target=_poll, daemon=True)
|
||||
poll_thread.start()
|
||||
# Finish as soon as the model is pushed, rather than waiting out the platform's
|
||||
# post-run finalization before the job stage flips to COMPLETED. This matches the
|
||||
# exact log line emitted by PreTrainedPolicy.push_model_to_hub — the two must stay
|
||||
# in sync. If it ever stops matching we just fall back to stage-based completion
|
||||
# (~30s slower), so the contract is an optimization, not a correctness requirement.
|
||||
success_marker = f"Model pushed to https://huggingface.co/{repo_id}"
|
||||
log_thread = threading.Thread(
|
||||
target=_tail_logs, args=(job_id, done, success_marker, pushed_ok), daemon=True
|
||||
)
|
||||
log_thread.start()
|
||||
|
||||
def _detach(sig, frame):
|
||||
detached.set()
|
||||
done.set()
|
||||
print("\nDetached. Job is still running.")
|
||||
print(f" Monitor: hf jobs logs {job_id}")
|
||||
print(f" Cancel: hf jobs cancel {job_id}")
|
||||
|
||||
# signal.signal only works on the main thread; when called from a worker thread
|
||||
# (e.g. an orchestration framework) skip the Ctrl-C-detaches-instead-of-cancels
|
||||
# handler rather than crashing with ValueError.
|
||||
install_sigint = threading.current_thread() is threading.main_thread()
|
||||
original_sigint = signal.getsignal(signal.SIGINT) if install_sigint else None
|
||||
if install_sigint:
|
||||
signal.signal(signal.SIGINT, _detach)
|
||||
try:
|
||||
# Timeout-based join so SIGINT is delivered to the main thread promptly.
|
||||
while poll_thread.is_alive():
|
||||
poll_thread.join(timeout=0.5)
|
||||
log_thread.join(timeout=5)
|
||||
finally:
|
||||
if install_sigint:
|
||||
signal.signal(signal.SIGINT, original_sigint)
|
||||
|
||||
if detached.is_set():
|
||||
return
|
||||
|
||||
if pushed_ok.is_set():
|
||||
if follow_job(job_id, detach=cfg.job.detach, success_marker=success_marker):
|
||||
print(f"\nTraining complete — model pushed to https://huggingface.co/{repo_id}")
|
||||
return
|
||||
|
||||
stage = stage_holder.get("stage")
|
||||
if stage != "COMPLETED":
|
||||
message = stage_holder.get("message")
|
||||
detail = f" ({message})" if message else ""
|
||||
raise RuntimeError(
|
||||
f"Job {job_id} ended with stage={stage}{detail}. Check logs: hf jobs logs {job_id}"
|
||||
)
|
||||
|
||||
@@ -20,7 +20,6 @@ import logging
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from copy import deepcopy
|
||||
from functools import cached_property
|
||||
from typing import TYPE_CHECKING, Any, TypedDict
|
||||
|
||||
from lerobot.utils.decorators import check_if_already_connected, check_if_not_connected
|
||||
@@ -854,7 +853,7 @@ class DamiaoMotorsBus(MotorsBusBase):
|
||||
else:
|
||||
raise ValueError(f"Motor {motor_obj} doesn't have a valid recv_id (None).")
|
||||
|
||||
@cached_property
|
||||
@property
|
||||
def is_calibrated(self) -> bool:
|
||||
"""Check if motors are calibrated."""
|
||||
return bool(self.calibration)
|
||||
|
||||
@@ -23,6 +23,7 @@ from __future__ import annotations
|
||||
|
||||
import abc
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Sequence
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
@@ -818,13 +819,13 @@ class SerialMotorsBus(MotorsBusBase):
|
||||
"""
|
||||
motor_names = self._get_motors_list(motors)
|
||||
|
||||
start_positions = self.sync_read("Present_Position", motor_names, normalize=False)
|
||||
start_positions = self.sync_read("Present_Position", motor_names, normalize=False, num_retry=5)
|
||||
mins = start_positions.copy()
|
||||
maxes = start_positions.copy()
|
||||
|
||||
user_pressed_enter = False
|
||||
while not user_pressed_enter:
|
||||
positions = self.sync_read("Present_Position", motor_names, normalize=False)
|
||||
positions = self.sync_read("Present_Position", motor_names, normalize=False, num_retry=5)
|
||||
mins = {motor: min(positions[motor], min_) for motor, min_ in mins.items()}
|
||||
maxes = {motor: max(positions[motor], max_) for motor, max_ in maxes.items()}
|
||||
|
||||
@@ -837,9 +838,12 @@ class SerialMotorsBus(MotorsBusBase):
|
||||
if enter_pressed():
|
||||
user_pressed_enter = True
|
||||
|
||||
if display_values and not user_pressed_enter:
|
||||
# Move cursor up to overwrite the previous output
|
||||
move_cursor_up(len(motor_names) + 3)
|
||||
if not user_pressed_enter:
|
||||
if display_values:
|
||||
# Move cursor up to overwrite the previous output
|
||||
move_cursor_up(len(motor_names) + 3)
|
||||
# Throttle reads even when the live table is disabled.
|
||||
time.sleep(0.02)
|
||||
|
||||
same_min_max = [motor for motor in motor_names if mins[motor] == maxes[motor]]
|
||||
if same_min_max:
|
||||
|
||||
@@ -79,6 +79,8 @@ class DiffusionConfig(PreTrainedConfig):
|
||||
use_film_scale_modulation: FiLM (https://huggingface.co/papers/1709.07871) is used for the Unet conditioning.
|
||||
Bias modulation is used be default, while this parameter indicates whether to also use scale
|
||||
modulation.
|
||||
gradient_checkpointing: Whether to checkpoint the Unet residual blocks during training. This reduces
|
||||
activation memory at the cost of recomputing those blocks during the backward pass.
|
||||
noise_scheduler_type: Name of the noise scheduler to use. Supported options: ["DDPM", "DDIM"].
|
||||
num_train_timesteps: Number of diffusion steps for the forward diffusion schedule.
|
||||
beta_schedule: Name of the diffusion beta schedule as per DDPMScheduler from Hugging Face diffusers.
|
||||
@@ -132,6 +134,7 @@ class DiffusionConfig(PreTrainedConfig):
|
||||
n_groups: int = 8
|
||||
diffusion_step_embed_dim: int = 128
|
||||
use_film_scale_modulation: bool = True
|
||||
gradient_checkpointing: bool = False
|
||||
# Noise scheduler.
|
||||
noise_scheduler_type: str = "DDPM"
|
||||
num_train_timesteps: int = 100
|
||||
|
||||
@@ -31,6 +31,7 @@ import torch
|
||||
import torch.nn.functional as F # noqa: N812
|
||||
import torchvision
|
||||
from torch import Tensor, nn
|
||||
from torch.utils.checkpoint import checkpoint
|
||||
|
||||
from lerobot.utils.constants import ACTION, OBS_ENV_STATE, OBS_IMAGES, OBS_STATE
|
||||
from lerobot.utils.import_utils import _diffusers_available, require_package
|
||||
@@ -727,22 +728,35 @@ class DiffusionConditionalUnet1d(nn.Module):
|
||||
else:
|
||||
global_feature = timesteps_embed
|
||||
|
||||
use_gc = self.config.gradient_checkpointing and self.training
|
||||
|
||||
# Run encoder, keeping track of skip features to pass to the decoder.
|
||||
encoder_skip_features: list[Tensor] = []
|
||||
for resnet, resnet2, downsample in self.down_modules:
|
||||
x = resnet(x, global_feature)
|
||||
x = resnet2(x, global_feature)
|
||||
if use_gc:
|
||||
x = checkpoint(resnet, x, global_feature, use_reentrant=False)
|
||||
x = checkpoint(resnet2, x, global_feature, use_reentrant=False)
|
||||
else:
|
||||
x = resnet(x, global_feature)
|
||||
x = resnet2(x, global_feature)
|
||||
encoder_skip_features.append(x)
|
||||
x = downsample(x)
|
||||
|
||||
for mid_module in self.mid_modules:
|
||||
x = mid_module(x, global_feature)
|
||||
if use_gc:
|
||||
x = checkpoint(mid_module, x, global_feature, use_reentrant=False)
|
||||
else:
|
||||
x = mid_module(x, global_feature)
|
||||
|
||||
# Run decoder, using the skip features from the encoder.
|
||||
for resnet, resnet2, upsample in self.up_modules:
|
||||
x = torch.cat((x, encoder_skip_features.pop()), dim=1)
|
||||
x = resnet(x, global_feature)
|
||||
x = resnet2(x, global_feature)
|
||||
if use_gc:
|
||||
x = checkpoint(resnet, x, global_feature, use_reentrant=False)
|
||||
x = checkpoint(resnet2, x, global_feature, use_reentrant=False)
|
||||
else:
|
||||
x = resnet(x, global_feature)
|
||||
x = resnet2(x, global_feature)
|
||||
x = upsample(x)
|
||||
|
||||
x = self.final_conv(x)
|
||||
|
||||
@@ -524,8 +524,6 @@ class PI05Pytorch(nn.Module): # see openpi `PI0Pytorch`
|
||||
|
||||
def embed_suffix(self, noisy_actions, timestep):
|
||||
"""Embed noisy_actions, timestep to prepare for Expert Gemma processing."""
|
||||
embs = []
|
||||
pad_masks = []
|
||||
att_masks = []
|
||||
|
||||
# Embed timestep using sine-cosine positional encoding
|
||||
@@ -551,23 +549,17 @@ class PI05Pytorch(nn.Module): # see openpi `PI0Pytorch`
|
||||
return F.silu(x)
|
||||
|
||||
time_emb = self._apply_checkpoint(time_mlp_func, time_emb)
|
||||
action_time_emb = action_emb
|
||||
adarms_cond = time_emb
|
||||
|
||||
embs.append(action_time_emb)
|
||||
bsize, action_time_dim = action_time_emb.shape[:2]
|
||||
action_time_mask = torch.ones(bsize, action_time_dim, dtype=torch.bool, device=timestep.device)
|
||||
pad_masks.append(action_time_mask)
|
||||
bsize, action_time_dim = action_emb.shape[:2]
|
||||
pad_masks = torch.ones(bsize, action_time_dim, dtype=torch.bool, device=timestep.device)
|
||||
|
||||
# Set attention masks so that image, language and state inputs do not attend to action tokens
|
||||
att_masks += [1] + ([0] * (self.config.chunk_size - 1))
|
||||
|
||||
embs = torch.cat(embs, dim=1)
|
||||
pad_masks = torch.cat(pad_masks, dim=1)
|
||||
att_masks = torch.tensor(att_masks, dtype=embs.dtype, device=embs.device)
|
||||
att_masks = torch.tensor(att_masks, dtype=action_emb.dtype, device=action_emb.device)
|
||||
att_masks = att_masks[None, :].expand(bsize, len(att_masks))
|
||||
|
||||
return embs, pad_masks, att_masks, adarms_cond
|
||||
return action_emb, pad_masks, att_masks, adarms_cond
|
||||
|
||||
def forward(self, images, img_masks, tokens, masks, actions, noise, time) -> Tensor:
|
||||
"""Do a full training forward pass and compute the loss."""
|
||||
|
||||
@@ -61,9 +61,15 @@ import torch.nn.functional as F # noqa: N812
|
||||
from torch import Tensor, nn
|
||||
|
||||
from lerobot.utils.constants import ACTION, OBS_LANGUAGE_ATTENTION_MASK, OBS_LANGUAGE_TOKENS, OBS_STATE
|
||||
from lerobot.utils.device_utils import get_safe_dtype
|
||||
from lerobot.utils.import_utils import require_package
|
||||
|
||||
from ..common.flow_matching import euler_integrate, sample_noise, sample_time_beta
|
||||
from ..common.vla_utils import (
|
||||
create_sinusoidal_pos_embedding,
|
||||
make_att_2d_masks,
|
||||
pad_vector,
|
||||
resize_with_pad,
|
||||
)
|
||||
from ..pretrained import PreTrainedPolicy
|
||||
from ..rtc.modeling_rtc import RTCProcessor
|
||||
from ..utils import (
|
||||
@@ -79,96 +85,6 @@ class ActionSelectKwargs(TypedDict, total=False):
|
||||
execution_horizon: int | None
|
||||
|
||||
|
||||
def create_sinusoidal_pos_embedding(
|
||||
time: torch.tensor, dimension: int, min_period: float, max_period: float, device="cpu"
|
||||
) -> Tensor:
|
||||
"""Computes sine-cosine positional embedding vectors for scalar positions."""
|
||||
if dimension % 2 != 0:
|
||||
raise ValueError(f"dimension ({dimension}) must be divisible by 2")
|
||||
|
||||
if time.ndim != 1:
|
||||
raise ValueError("The time tensor is expected to be of shape `(batch_size, )`.")
|
||||
|
||||
dtype = get_safe_dtype(torch.float64, device.type)
|
||||
fraction = torch.linspace(0.0, 1.0, dimension // 2, dtype=dtype, device=device)
|
||||
period = min_period * (max_period / min_period) ** fraction
|
||||
|
||||
# Compute the outer product
|
||||
scaling_factor = 1.0 / period * 2 * math.pi
|
||||
sin_input = scaling_factor[None, :] * time[:, None]
|
||||
pos_emb = torch.cat([torch.sin(sin_input), torch.cos(sin_input)], dim=1)
|
||||
return pos_emb
|
||||
|
||||
|
||||
def make_att_2d_masks(pad_masks, att_masks):
|
||||
"""Copied from big_vision.
|
||||
|
||||
Tokens can attend to valid inputs tokens which have a cumulative mask_ar
|
||||
smaller or equal to theirs. This way `mask_ar` int[B, N] can be used to
|
||||
setup several types of attention, for example:
|
||||
|
||||
[[1 1 1 1 1 1]]: pure causal attention.
|
||||
|
||||
[[0 0 0 1 1 1]]: prefix-lm attention. The first 3 tokens can attend between
|
||||
themselves and the last 3 tokens have a causal attention. The first
|
||||
entry could also be a 1 without changing behaviour.
|
||||
|
||||
[[1 0 1 0 1 0 0 1 0 0]]: causal attention between 4 blocks. Tokens of a
|
||||
block can attend all previous blocks and all tokens on the same block.
|
||||
|
||||
Args:
|
||||
input_mask: bool[B, N] true if its part of the input, false if padding.
|
||||
mask_ar: int32[B, N] mask that's 1 where previous tokens cannot depend on
|
||||
it and 0 where it shares the same attention mask as the previous token.
|
||||
"""
|
||||
if att_masks.ndim != 2:
|
||||
raise ValueError(att_masks.ndim)
|
||||
if pad_masks.ndim != 2:
|
||||
raise ValueError(pad_masks.ndim)
|
||||
|
||||
cumsum = torch.cumsum(att_masks, dim=1)
|
||||
att_2d_masks = cumsum[:, None, :] <= cumsum[:, :, None]
|
||||
pad_2d_masks = pad_masks[:, None, :] * pad_masks[:, :, None]
|
||||
att_2d_masks = att_2d_masks & pad_2d_masks
|
||||
return att_2d_masks
|
||||
|
||||
|
||||
def resize_with_pad(img, width, height, pad_value=-1):
|
||||
# assume no-op when width height fits already
|
||||
if img.ndim != 4:
|
||||
raise ValueError(f"(b,c,h,w) expected, but {img.shape}")
|
||||
|
||||
cur_height, cur_width = img.shape[2:]
|
||||
|
||||
ratio = max(cur_width / width, cur_height / height)
|
||||
resized_height = int(cur_height / ratio)
|
||||
resized_width = int(cur_width / ratio)
|
||||
resized_img = F.interpolate(
|
||||
img, size=(resized_height, resized_width), mode="bilinear", align_corners=False
|
||||
)
|
||||
|
||||
pad_height = max(0, int(height - resized_height))
|
||||
pad_width = max(0, int(width - resized_width))
|
||||
|
||||
# pad on left and top of image
|
||||
padded_img = F.pad(resized_img, (pad_width, 0, pad_height, 0), value=pad_value)
|
||||
return padded_img
|
||||
|
||||
|
||||
def pad_vector(vector, new_dim):
|
||||
"""Can be (batch_size x sequence_length x features_dimension)
|
||||
or (batch_size x features_dimension)
|
||||
"""
|
||||
if vector.shape[-1] == new_dim:
|
||||
return vector
|
||||
shape = list(vector.shape)
|
||||
current_dim = shape[-1]
|
||||
shape[-1] = new_dim
|
||||
new_vector = torch.zeros(*shape, dtype=vector.dtype, device=vector.device)
|
||||
new_vector[..., :current_dim] = vector
|
||||
return new_vector
|
||||
|
||||
|
||||
def normalize(x, min_val, max_val):
|
||||
return (x - min_val) / (max_val - min_val)
|
||||
|
||||
@@ -429,7 +345,13 @@ class SmolVLAPolicy(PreTrainedPolicy):
|
||||
for key in present_img_keys:
|
||||
img = batch[key][:, -1, :, :, :] if batch[key].ndim == 5 else batch[key]
|
||||
if self.config.resize_imgs_with_padding is not None:
|
||||
img = resize_with_pad(img, *self.config.resize_imgs_with_padding, pad_value=0)
|
||||
# SmolVLA stores the target as (width, height); the shared helper expects (height, width).
|
||||
img = resize_with_pad(
|
||||
img,
|
||||
self.config.resize_imgs_with_padding[1],
|
||||
self.config.resize_imgs_with_padding[0],
|
||||
pad_value=0,
|
||||
)
|
||||
|
||||
# Normalize from range [0,1] to [-1,1] as expacted by siglip
|
||||
img = img * 2.0 - 1.0
|
||||
@@ -619,20 +541,10 @@ class VLAFlowMatching(nn.Module):
|
||||
params.requires_grad = self.config.train_state_proj
|
||||
|
||||
def sample_noise(self, shape, device):
|
||||
noise = torch.normal(
|
||||
mean=0.0,
|
||||
std=1.0,
|
||||
size=shape,
|
||||
dtype=torch.float32,
|
||||
device=device,
|
||||
)
|
||||
return noise
|
||||
return sample_noise(shape, device)
|
||||
|
||||
def sample_time(self, bsize, device):
|
||||
beta_dist = torch.distributions.Beta(concentration1=1.5, concentration0=1.0)
|
||||
time_beta = beta_dist.sample((bsize,)).to(device=device, dtype=torch.float32)
|
||||
time = time_beta * 0.999 + 0.001
|
||||
return time
|
||||
return sample_time_beta(bsize, device, alpha=1.5, beta=1.0, scale=0.999, offset=0.001)
|
||||
|
||||
def embed_prefix(
|
||||
self, images, img_masks, lang_tokens, lang_masks, state: torch.Tensor = None
|
||||
@@ -800,7 +712,6 @@ class VLAFlowMatching(nn.Module):
|
||||
past_key_values=None,
|
||||
inputs_embeds=[prefix_embs, suffix_embs],
|
||||
use_cache=False,
|
||||
fill_kv_cache=False,
|
||||
)
|
||||
suffix_out = suffix_out[:, -self.config.chunk_size :]
|
||||
# Original openpi code, upcast attention output
|
||||
@@ -839,46 +750,24 @@ class VLAFlowMatching(nn.Module):
|
||||
past_key_values=None,
|
||||
inputs_embeds=[prefix_embs, None],
|
||||
use_cache=self.config.use_cache,
|
||||
fill_kv_cache=True,
|
||||
)
|
||||
num_steps = self.config.num_steps
|
||||
dt = -1.0 / num_steps
|
||||
|
||||
x_t = noise
|
||||
for step in range(num_steps):
|
||||
time = 1.0 + step * dt
|
||||
time_tensor = torch.tensor(time, dtype=torch.float32, device=device).expand(bsize)
|
||||
|
||||
def denoise_step_partial_call(input_x_t, current_timestep=time_tensor):
|
||||
return self.denoise_step(
|
||||
x_t=input_x_t,
|
||||
prefix_pad_masks=prefix_pad_masks,
|
||||
past_key_values=past_key_values,
|
||||
timestep=current_timestep,
|
||||
)
|
||||
|
||||
if self._rtc_enabled():
|
||||
inference_delay = kwargs.get("inference_delay")
|
||||
prev_chunk_left_over = kwargs.get("prev_chunk_left_over")
|
||||
execution_horizon = kwargs.get("execution_horizon")
|
||||
|
||||
v_t = self.rtc_processor.denoise_step(
|
||||
x_t=x_t,
|
||||
prev_chunk_left_over=prev_chunk_left_over,
|
||||
inference_delay=inference_delay,
|
||||
time=time,
|
||||
original_denoise_step_partial=denoise_step_partial_call,
|
||||
execution_horizon=execution_horizon,
|
||||
)
|
||||
else:
|
||||
v_t = denoise_step_partial_call(x_t)
|
||||
|
||||
x_t = x_t + dt * v_t
|
||||
|
||||
if self.rtc_processor is not None and self.rtc_processor.is_debug_enabled():
|
||||
self.rtc_processor.track(time=time, x_t=x_t, v_t=v_t)
|
||||
|
||||
return x_t
|
||||
return euler_integrate(
|
||||
lambda input_x_t, current_timestep: self.denoise_step(
|
||||
x_t=input_x_t,
|
||||
prefix_pad_masks=prefix_pad_masks,
|
||||
past_key_values=past_key_values,
|
||||
timestep=current_timestep,
|
||||
),
|
||||
noise,
|
||||
num_steps,
|
||||
rtc_processor=self.rtc_processor,
|
||||
rtc_enabled=self._rtc_enabled(),
|
||||
inference_delay=kwargs.get("inference_delay"),
|
||||
prev_chunk_left_over=kwargs.get("prev_chunk_left_over"),
|
||||
execution_horizon=kwargs.get("execution_horizon"),
|
||||
)
|
||||
|
||||
def denoise_step(
|
||||
self,
|
||||
@@ -907,8 +796,10 @@ class VLAFlowMatching(nn.Module):
|
||||
past_key_values=past_key_values,
|
||||
inputs_embeds=[None, suffix_embs],
|
||||
use_cache=self.config.use_cache,
|
||||
fill_kv_cache=False,
|
||||
)
|
||||
if past_key_values is not None:
|
||||
# Self-attention layers append suffix K/V in place; restore the prefix for the next step.
|
||||
past_key_values.crop(prefix_len)
|
||||
suffix_out = outputs_embeds[1]
|
||||
suffix_out = suffix_out[:, -self.config.chunk_size :]
|
||||
suffix_out = suffix_out.to(dtype=torch.float32)
|
||||
|
||||
@@ -26,6 +26,7 @@ if TYPE_CHECKING or _transformers_available:
|
||||
AutoModel,
|
||||
AutoModelForImageTextToText,
|
||||
AutoProcessor,
|
||||
DynamicCache,
|
||||
SmolVLMForConditionalGeneration,
|
||||
)
|
||||
else:
|
||||
@@ -33,6 +34,7 @@ else:
|
||||
AutoModel = None
|
||||
AutoModelForImageTextToText = None
|
||||
AutoProcessor = None
|
||||
DynamicCache = None
|
||||
SmolVLMForConditionalGeneration = None
|
||||
|
||||
|
||||
@@ -216,9 +218,8 @@ class SmolVLMWithExpertModel(nn.Module):
|
||||
batch_size,
|
||||
head_dim,
|
||||
use_cache: bool = True,
|
||||
fill_kv_cache: bool = True,
|
||||
past_key_values=None,
|
||||
) -> list[torch.Tensor]:
|
||||
past_key_values: "DynamicCache | None" = None,
|
||||
) -> "tuple[list[torch.Tensor], DynamicCache | None]":
|
||||
query_states = []
|
||||
key_states = []
|
||||
value_states = []
|
||||
@@ -259,22 +260,16 @@ class SmolVLMWithExpertModel(nn.Module):
|
||||
query_states = apply_rope(query_states, position_ids_)
|
||||
key_states = apply_rope(key_states, position_ids_)
|
||||
|
||||
if use_cache and past_key_values is None:
|
||||
past_key_values = {}
|
||||
|
||||
if use_cache:
|
||||
if fill_kv_cache:
|
||||
past_key_values[layer_idx] = {
|
||||
"key_states": key_states,
|
||||
"value_states": value_states,
|
||||
}
|
||||
else:
|
||||
# TODO here, some optimization can be done - similar to a `StaticCache` we can declare the `max_len` before.
|
||||
# so we create an empty cache, with just one cuda malloc, and if (in autoregressive case) we reach
|
||||
# the max len, then we (for instance) double the cache size. This implementation already exists
|
||||
# in `transformers`. (molbap)
|
||||
key_states = torch.cat([past_key_values[layer_idx]["key_states"], key_states], dim=1)
|
||||
value_states = torch.cat([past_key_values[layer_idx]["value_states"], value_states], dim=1)
|
||||
# `DynamicCache` stores tensors as [batch, heads, seq, head_dim]; this module works with
|
||||
# [batch, seq, heads, head_dim]. During prefix prefill this stores the (post-RoPE) K/V and
|
||||
# returns them unchanged; during denoising it appends the suffix K/V and returns
|
||||
# [prefix; suffix], exactly like the previous hand-rolled dict cache.
|
||||
key_states, value_states = past_key_values.update(
|
||||
key_states.transpose(1, 2), value_states.transpose(1, 2), layer_idx
|
||||
)
|
||||
key_states = key_states.transpose(1, 2)
|
||||
value_states = value_states.transpose(1, 2)
|
||||
|
||||
attention_interface = self.get_attention_interface()
|
||||
|
||||
@@ -293,13 +288,12 @@ class SmolVLMWithExpertModel(nn.Module):
|
||||
batch_size,
|
||||
head_dim,
|
||||
use_cache: bool = True,
|
||||
fill_kv_cache: bool = True,
|
||||
past_key_values=None,
|
||||
) -> list[torch.Tensor]:
|
||||
past_key_values: "DynamicCache | None" = None,
|
||||
) -> "tuple[list[torch.Tensor], DynamicCache | None]":
|
||||
attention_interface = self.get_attention_interface()
|
||||
|
||||
att_outputs = []
|
||||
assert len(inputs_embeds) == 2 or (use_cache and past_key_values is not None and not fill_kv_cache), (
|
||||
assert len(inputs_embeds) == 2 or (use_cache and past_key_values is not None), (
|
||||
f"Both len(inputs_embeds) == {len(inputs_embeds)} and past_key_values is {past_key_values}"
|
||||
)
|
||||
|
||||
@@ -332,22 +326,13 @@ class SmolVLMWithExpertModel(nn.Module):
|
||||
else:
|
||||
expert_position_id = position_ids
|
||||
|
||||
if use_cache and past_key_values is None:
|
||||
past_key_values = {}
|
||||
|
||||
if use_cache:
|
||||
if fill_kv_cache:
|
||||
past_key_values[layer_idx] = {
|
||||
"key_states": key_states,
|
||||
"value_states": value_states,
|
||||
}
|
||||
else:
|
||||
# TODO here, some optimization can be done - similar to a `StaticCache` we can declare the `max_len` before.
|
||||
# so we create an empty cache, with just one cuda malloc, and if (in autoregressive case) we reach
|
||||
# the max len, then we (for instance) double the cache size. This implementation already exists
|
||||
# in `transformers`. (molbap)
|
||||
key_states = past_key_values[layer_idx]["key_states"]
|
||||
value_states = past_key_values[layer_idx]["value_states"]
|
||||
if use_cache and past_key_values is not None:
|
||||
# Cross-attention layers never fill the cache themselves: during the prefix prefill every
|
||||
# layer goes through `forward_attn_layer`, which stores the (post-RoPE) VLM K/V for this
|
||||
# layer index. Here we only read them back (no concatenation: the expert cross-attends to
|
||||
# the fixed prefix). `DynamicCache` stores [batch, heads, seq, head_dim]; transpose back.
|
||||
key_states = past_key_values.layers[layer_idx].keys.transpose(1, 2)
|
||||
value_states = past_key_values.layers[layer_idx].values.transpose(1, 2)
|
||||
|
||||
# Expert
|
||||
expert_layer = model_layers[1][layer_idx]
|
||||
@@ -360,14 +345,15 @@ class SmolVLMWithExpertModel(nn.Module):
|
||||
expert_hidden_states = expert_hidden_states.to(dtype=expert_layer.self_attn.q_proj.weight.dtype)
|
||||
expert_query_state = expert_layer.self_attn.q_proj(expert_hidden_states).view(expert_hidden_shape)
|
||||
|
||||
_key_states = key_states.to(dtype=expert_layer.self_attn.k_proj.weight.dtype).view(
|
||||
# reshape (not view): K/V read back from the cache are transposed, hence non-contiguous
|
||||
_key_states = key_states.to(dtype=expert_layer.self_attn.k_proj.weight.dtype).reshape(
|
||||
*key_states.shape[:2], -1
|
||||
)
|
||||
expert_key_states = expert_layer.self_attn.k_proj(_key_states).view(
|
||||
*_key_states.shape[:-1], -1, expert_layer.self_attn.head_dim
|
||||
) # k_proj should have same dim as kv
|
||||
|
||||
_value_states = value_states.to(dtype=expert_layer.self_attn.v_proj.weight.dtype).view(
|
||||
_value_states = value_states.to(dtype=expert_layer.self_attn.v_proj.weight.dtype).reshape(
|
||||
*value_states.shape[:2], -1
|
||||
)
|
||||
expert_value_states = expert_layer.self_attn.v_proj(_value_states).view(
|
||||
@@ -416,10 +402,9 @@ class SmolVLMWithExpertModel(nn.Module):
|
||||
self,
|
||||
attention_mask: torch.Tensor | None = None,
|
||||
position_ids: torch.LongTensor | None = None,
|
||||
past_key_values: list[torch.FloatTensor] | None = None,
|
||||
past_key_values: "DynamicCache | None" = None,
|
||||
inputs_embeds: list[torch.FloatTensor] = None,
|
||||
use_cache: bool | None = None,
|
||||
fill_kv_cache: bool | None = None,
|
||||
):
|
||||
models = [self.get_vlm_model().text_model, self.lm_expert]
|
||||
model_layers = self.get_model_layers(models)
|
||||
@@ -431,6 +416,13 @@ class SmolVLMWithExpertModel(nn.Module):
|
||||
continue
|
||||
batch_size = hidden_states.shape[0]
|
||||
|
||||
# Prefix prefill: no cache was passed, so create one and fill it (every layer runs
|
||||
# self-attention over the prefix). When a filled cache is passed (denoising), layers
|
||||
# read from it instead.
|
||||
fill_kv_cache = use_cache and past_key_values is None
|
||||
if fill_kv_cache:
|
||||
past_key_values = DynamicCache()
|
||||
|
||||
# RMSNorm
|
||||
num_layers = self.num_vlm_layers
|
||||
head_dim = self.vlm.config.text_config.head_dim
|
||||
@@ -449,7 +441,6 @@ class SmolVLMWithExpertModel(nn.Module):
|
||||
batch_size,
|
||||
head_dim,
|
||||
use_cache=use_cache,
|
||||
fill_kv_cache=fill_kv_cache,
|
||||
past_key_values=past_key_values,
|
||||
)
|
||||
else:
|
||||
@@ -462,7 +453,6 @@ class SmolVLMWithExpertModel(nn.Module):
|
||||
batch_size,
|
||||
head_dim,
|
||||
use_cache=use_cache,
|
||||
fill_kv_cache=fill_kv_cache,
|
||||
past_key_values=past_key_values,
|
||||
)
|
||||
outputs_embeds = []
|
||||
|
||||
@@ -58,6 +58,9 @@ class BiSOFollower(BimanualMixin, Robot):
|
||||
port=config.left_arm_config.port,
|
||||
disable_torque_on_disconnect=config.left_arm_config.disable_torque_on_disconnect,
|
||||
max_relative_target=config.left_arm_config.max_relative_target,
|
||||
position_p_coefficient=config.left_arm_config.position_p_coefficient,
|
||||
position_i_coefficient=config.left_arm_config.position_i_coefficient,
|
||||
position_d_coefficient=config.left_arm_config.position_d_coefficient,
|
||||
use_degrees=config.left_arm_config.use_degrees,
|
||||
cameras=left_arm_cameras,
|
||||
)
|
||||
@@ -68,6 +71,9 @@ class BiSOFollower(BimanualMixin, Robot):
|
||||
port=config.right_arm_config.port,
|
||||
disable_torque_on_disconnect=config.right_arm_config.disable_torque_on_disconnect,
|
||||
max_relative_target=config.right_arm_config.max_relative_target,
|
||||
position_p_coefficient=config.right_arm_config.position_p_coefficient,
|
||||
position_i_coefficient=config.right_arm_config.position_i_coefficient,
|
||||
position_d_coefficient=config.right_arm_config.position_d_coefficient,
|
||||
use_degrees=config.right_arm_config.use_degrees,
|
||||
cameras=config.right_arm_config.cameras,
|
||||
)
|
||||
|
||||
@@ -150,9 +150,6 @@ class OpenArmFollower(Robot):
|
||||
|
||||
self.configure()
|
||||
|
||||
if self.is_calibrated:
|
||||
self.bus.set_zero_position()
|
||||
|
||||
self.bus.enable_torque()
|
||||
|
||||
logger.info(f"{self} connected.")
|
||||
|
||||
@@ -41,6 +41,11 @@ class SOFollowerConfig:
|
||||
# Set to `True` for backward compatibility with previous policies/dataset
|
||||
use_degrees: bool = True
|
||||
|
||||
# Position-mode PID gains written to Feetech STS3215 motors at connect time.
|
||||
position_p_coefficient: int = 16
|
||||
position_i_coefficient: int = 0
|
||||
position_d_coefficient: int = 32
|
||||
|
||||
|
||||
@RobotConfig.register_subclass("so101_follower")
|
||||
@RobotConfig.register_subclass("so100_follower")
|
||||
|
||||
@@ -161,11 +161,9 @@ class SOFollower(Robot):
|
||||
self.bus.configure_motors()
|
||||
for motor in self.bus.motors:
|
||||
self.bus.write("Operating_Mode", motor, OperatingMode.POSITION.value)
|
||||
# Set P_Coefficient to lower value to avoid shakiness (Default is 32)
|
||||
self.bus.write("P_Coefficient", motor, 16)
|
||||
# Set I_Coefficient and D_Coefficient to default value 0 and 32
|
||||
self.bus.write("I_Coefficient", motor, 0)
|
||||
self.bus.write("D_Coefficient", motor, 32)
|
||||
self.bus.write("P_Coefficient", motor, self.config.position_p_coefficient)
|
||||
self.bus.write("I_Coefficient", motor, self.config.position_i_coefficient)
|
||||
self.bus.write("D_Coefficient", motor, self.config.position_d_coefficient)
|
||||
|
||||
if motor == "gripper":
|
||||
self.bus.write("Max_Torque_Limit", motor, 500) # 50% of max torque to avoid burnout
|
||||
|
||||
@@ -24,7 +24,14 @@ Example:
|
||||
--root=/path/to/dataset \\
|
||||
--vlm.model_id=Qwen/Qwen2.5-VL-7B-Instruct
|
||||
|
||||
For distributed runs, see ``examples/annotations/run_hf_job.py``.
|
||||
Pass ``--job.target=<flavor>`` to run the same command on a Hugging Face
|
||||
Jobs GPU instead of this machine (see ``lerobot.jobs.annotate``):
|
||||
|
||||
uv run lerobot-annotate \\
|
||||
--repo_id=user/dataset \\
|
||||
--new_repo_id=user/dataset_annotated \\
|
||||
--push_to_hub=true \\
|
||||
--job.target=h200
|
||||
"""
|
||||
|
||||
import logging
|
||||
@@ -69,6 +76,14 @@ def _resolve_root(cfg: AnnotationPipelineConfig) -> Path:
|
||||
def annotate(cfg: AnnotationPipelineConfig) -> None:
|
||||
"""Run the steerable annotation pipeline against a dataset."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
|
||||
if cfg.job.is_remote:
|
||||
# Imported lazily: the submitter pulls in LeRobotDataset (the `dataset`
|
||||
# extra), which a local annotation run over --root doesn't need.
|
||||
from lerobot.jobs.annotate import submit_annotate_to_hf
|
||||
|
||||
return submit_annotate_to_hf(cfg)
|
||||
|
||||
root = _resolve_root(cfg)
|
||||
logger.info("annotate: root=%s", root)
|
||||
|
||||
|
||||
@@ -51,19 +51,7 @@ from lerobot.teleoperators import ( # noqa: F401
|
||||
rebot_102_leader,
|
||||
so_leader,
|
||||
)
|
||||
|
||||
COMPATIBLE_DEVICES = [
|
||||
"koch_follower",
|
||||
"koch_leader",
|
||||
"omx_follower",
|
||||
"omx_leader",
|
||||
"openarm_mini",
|
||||
"so100_follower",
|
||||
"so100_leader",
|
||||
"so101_follower",
|
||||
"so101_leader",
|
||||
"lekiwi",
|
||||
]
|
||||
from lerobot.utils.import_utils import register_third_party_plugins
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -80,18 +68,19 @@ class SetupConfig:
|
||||
|
||||
@draccus.wrap()
|
||||
def setup_motors(cfg: SetupConfig):
|
||||
if cfg.device.type not in COMPATIBLE_DEVICES:
|
||||
raise NotImplementedError
|
||||
|
||||
if isinstance(cfg.device, RobotConfig):
|
||||
device = make_robot_from_config(cfg.device)
|
||||
else:
|
||||
device = make_teleoperator_from_config(cfg.device)
|
||||
|
||||
device.setup_motors()
|
||||
setup = getattr(device, "setup_motors", None)
|
||||
if not callable(setup):
|
||||
raise NotImplementedError(f"Device type '{cfg.device.type}' does not support motor setup.")
|
||||
setup()
|
||||
|
||||
|
||||
def main():
|
||||
register_third_party_plugins()
|
||||
setup_motors()
|
||||
|
||||
|
||||
|
||||
@@ -23,3 +23,5 @@ from ..config import TeleoperatorConfig
|
||||
@dataclass
|
||||
class GamepadTeleopConfig(TeleoperatorConfig):
|
||||
use_gripper: bool = True
|
||||
# Use hidapi instead of pygame for controllers that pygame cannot detect reliably.
|
||||
hidapi_fallback: bool = False
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
import sys
|
||||
from enum import IntEnum
|
||||
from typing import Any
|
||||
@@ -27,6 +28,8 @@ from ..teleoperator import Teleoperator
|
||||
from ..utils import TeleopEvents
|
||||
from .configuration_gamepad import GamepadTeleopConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GripperAction(IntEnum):
|
||||
CLOSE = 0
|
||||
@@ -56,6 +59,13 @@ class GamepadTeleop(Teleoperator):
|
||||
|
||||
self.gamepad = None
|
||||
|
||||
self.hidapi_fallback = config.hidapi_fallback
|
||||
if sys.platform == "darwin" and not self.hidapi_fallback:
|
||||
logger.warning(
|
||||
"On macOS, pygame may not reliably detect input from some controllers. "
|
||||
"If you experience issues, set `hidapi_fallback=true`."
|
||||
)
|
||||
|
||||
@property
|
||||
def action_features(self) -> dict:
|
||||
if self.config.use_gripper:
|
||||
@@ -76,9 +86,7 @@ class GamepadTeleop(Teleoperator):
|
||||
return {}
|
||||
|
||||
def connect(self) -> None:
|
||||
# use HidApi for macos
|
||||
if sys.platform == "darwin":
|
||||
# NOTE: On macOS, pygame doesn’t reliably detect input from some controllers so we fall back to hidapi
|
||||
if self.hidapi_fallback:
|
||||
from .gamepad_utils import GamepadControllerHID as Gamepad
|
||||
else:
|
||||
from .gamepad_utils import GamepadController as Gamepad
|
||||
|
||||
@@ -26,7 +26,7 @@ import cv2
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from lerobot.cameras.configs import Cv2Rotation
|
||||
from lerobot.cameras.configs import ColorMode, Cv2Rotation
|
||||
from lerobot.cameras.opencv import OpenCVCamera, OpenCVCameraConfig
|
||||
from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError
|
||||
|
||||
@@ -132,6 +132,28 @@ def test_read(index_or_path):
|
||||
assert isinstance(img, np.ndarray)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("index_or_path", TEST_IMAGE_PATHS, ids=TEST_IMAGE_SIZES)
|
||||
def test_color_mode_conversion(index_or_path):
|
||||
"""RGB and BGR reads of the same frame must differ only by a channel-axis reversal."""
|
||||
rgb_config = OpenCVCameraConfig(index_or_path=index_or_path, color_mode=ColorMode.RGB, warmup_s=0)
|
||||
bgr_config = OpenCVCameraConfig(index_or_path=index_or_path, color_mode=ColorMode.BGR, warmup_s=0)
|
||||
with OpenCVCamera(rgb_config) as rgb_cam:
|
||||
rgb = rgb_cam.read()
|
||||
with OpenCVCamera(bgr_config) as bgr_cam:
|
||||
bgr = bgr_cam.read()
|
||||
|
||||
assert rgb.shape == bgr.shape
|
||||
np.testing.assert_array_equal(rgb, bgr[..., ::-1])
|
||||
|
||||
|
||||
def test_postprocess_invalid_color_mode():
|
||||
config = OpenCVCameraConfig(index_or_path=DEFAULT_PNG_FILE_PATH)
|
||||
camera = OpenCVCamera(config)
|
||||
camera.color_mode = "invalid"
|
||||
with pytest.raises(ValueError):
|
||||
camera._postprocess_image(np.zeros((120, 160, 3), dtype=np.uint8))
|
||||
|
||||
|
||||
def test_read_before_connect():
|
||||
config = OpenCVCameraConfig(index_or_path=DEFAULT_PNG_FILE_PATH)
|
||||
|
||||
|
||||
@@ -22,6 +22,7 @@ import pytest
|
||||
|
||||
pytest.importorskip("reachy2_sdk")
|
||||
|
||||
from lerobot.cameras.configs import ColorMode
|
||||
from lerobot.cameras.reachy2_camera import Reachy2Camera, Reachy2CameraConfig
|
||||
from lerobot.utils.errors import DeviceNotConnectedError
|
||||
|
||||
@@ -33,28 +34,19 @@ PARAMS = [
|
||||
]
|
||||
|
||||
|
||||
def _make_cam_manager_mock():
|
||||
def _make_cam_manager_mock(color_frame, depth_frame=None):
|
||||
c = MagicMock(name="CameraManagerMock")
|
||||
|
||||
teleop = MagicMock(name="TeleopCam")
|
||||
teleop.width = 640
|
||||
teleop.height = 480
|
||||
teleop.get_frame = MagicMock(
|
||||
side_effect=lambda *_, **__: (
|
||||
np.zeros((480, 640, 3), dtype=np.uint8),
|
||||
time.time(),
|
||||
)
|
||||
)
|
||||
teleop.get_frame = MagicMock(side_effect=lambda *_, **__: (color_frame, time.time()))
|
||||
|
||||
depth = MagicMock(name="DepthCam")
|
||||
depth.width = 640
|
||||
depth.height = 480
|
||||
depth.get_frame = MagicMock(
|
||||
side_effect=lambda *_, **__: (
|
||||
np.zeros((480, 640, 3), dtype=np.uint8),
|
||||
time.time(),
|
||||
)
|
||||
)
|
||||
depth.get_frame = MagicMock(side_effect=lambda *_, **__: (color_frame, time.time()))
|
||||
depth.get_depth_frame = MagicMock(side_effect=lambda *_, **__: (depth_frame, time.time()))
|
||||
|
||||
c.is_connected.return_value = True
|
||||
c.teleop = teleop
|
||||
@@ -84,12 +76,14 @@ def _make_cam_manager_mock():
|
||||
# ids=["teleop-left", "teleop-right", "torso-rgb", "torso-depth"],
|
||||
ids=["teleop-left", "teleop-right", "torso-rgb"],
|
||||
)
|
||||
def camera(request):
|
||||
def camera(request, img_array_factory):
|
||||
name, image_type = request.param
|
||||
color_frame = img_array_factory(height=480, width=640)
|
||||
depth_frame = img_array_factory(height=480, width=640, channels=1, dtype=np.uint16)[..., 0]
|
||||
with (
|
||||
patch(
|
||||
"lerobot.cameras.reachy2_camera.reachy2_camera.CameraManager",
|
||||
side_effect=lambda *a, **k: _make_cam_manager_mock(),
|
||||
side_effect=lambda *a, **k: _make_cam_manager_mock(color_frame, depth_frame),
|
||||
),
|
||||
):
|
||||
config = Reachy2CameraConfig(name=name, image_type=image_type)
|
||||
@@ -188,6 +182,41 @@ def test_read_latest_too_old(camera):
|
||||
_ = camera.read_latest(max_age_ms=0) # immediately too old
|
||||
|
||||
|
||||
def test_color_mode_conversion(img_array_factory):
|
||||
"""teleop frames are native BGR: RGB reverses the channel axis, BGR is passed through."""
|
||||
frame = img_array_factory(height=8, width=8)
|
||||
|
||||
outputs = {}
|
||||
for color_mode in (ColorMode.RGB, ColorMode.BGR):
|
||||
with patch(
|
||||
"lerobot.cameras.reachy2_camera.reachy2_camera.CameraManager",
|
||||
side_effect=lambda *a, **k: _make_cam_manager_mock(frame),
|
||||
):
|
||||
cam = Reachy2Camera(Reachy2CameraConfig(name="teleop", image_type="left", color_mode=color_mode))
|
||||
cam.connect()
|
||||
outputs[color_mode] = cam.read()
|
||||
cam.disconnect()
|
||||
|
||||
np.testing.assert_array_equal(outputs[ColorMode.BGR], frame)
|
||||
np.testing.assert_array_equal(outputs[ColorMode.RGB], frame[..., ::-1])
|
||||
|
||||
|
||||
def test_depth_frame_not_color_converted(img_array_factory):
|
||||
"""A depth/depth frame must be returned as-is, without BGR<->RGB conversion."""
|
||||
color_frame = img_array_factory(height=8, width=8)
|
||||
depth = img_array_factory(height=8, width=8, channels=1, dtype=np.uint16)[..., 0]
|
||||
with patch(
|
||||
"lerobot.cameras.reachy2_camera.reachy2_camera.CameraManager",
|
||||
side_effect=lambda *a, **k: _make_cam_manager_mock(color_frame, depth_frame=depth),
|
||||
):
|
||||
cam = Reachy2Camera(Reachy2CameraConfig(name="depth", image_type="depth"))
|
||||
cam.connect()
|
||||
out = cam.read()
|
||||
cam.disconnect()
|
||||
|
||||
np.testing.assert_array_equal(out, depth)
|
||||
|
||||
|
||||
def test_wrong_camera_name():
|
||||
with pytest.raises(ValueError):
|
||||
_ = Reachy2CameraConfig(name="wrong-name", image_type="left")
|
||||
|
||||
@@ -25,7 +25,7 @@ from unittest.mock import patch
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from lerobot.cameras.configs import Cv2Rotation
|
||||
from lerobot.cameras.configs import ColorMode, Cv2Rotation
|
||||
from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError
|
||||
|
||||
pytest.importorskip("pyrealsense2")
|
||||
@@ -109,6 +109,32 @@ def test_read_depth():
|
||||
assert isinstance(img, np.ndarray)
|
||||
|
||||
|
||||
# These exercise _postprocess_image directly rather than read(): the bag playback returns
|
||||
# non-deterministic frames we can't compare against, and the depth read() path is skipped
|
||||
# (see test_read_depth) with the current pyrealsense2 version.
|
||||
def test_color_mode_conversion(img_array_factory):
|
||||
"""RGB (native for RealSense) is passed through; BGR reverses the channel axis."""
|
||||
color = img_array_factory(height=3, width=4)
|
||||
|
||||
outputs = {}
|
||||
for color_mode in (ColorMode.RGB, ColorMode.BGR):
|
||||
camera = RealSenseCamera(RealSenseCameraConfig(serial_number_or_name="042", color_mode=color_mode))
|
||||
camera.capture_height, camera.capture_width = color.shape[:2]
|
||||
outputs[color_mode] = camera._postprocess_image(color)
|
||||
|
||||
np.testing.assert_array_equal(outputs[ColorMode.RGB], color)
|
||||
np.testing.assert_array_equal(outputs[ColorMode.BGR], color[..., ::-1])
|
||||
|
||||
|
||||
def test_depth_frame_not_color_converted(img_array_factory):
|
||||
"""Depth frames must bypass color conversion, even when a BGR color_mode is set."""
|
||||
camera = RealSenseCamera(RealSenseCameraConfig(serial_number_or_name="042", color_mode=ColorMode.BGR))
|
||||
depth = img_array_factory(height=3, width=4, channels=1, dtype=np.uint16)[..., 0]
|
||||
camera.capture_height, camera.capture_width = depth.shape
|
||||
|
||||
np.testing.assert_array_equal(camera._postprocess_image(depth, depth_frame=True), depth)
|
||||
|
||||
|
||||
def test_read_before_connect():
|
||||
config = RealSenseCameraConfig(serial_number_or_name="042")
|
||||
camera = RealSenseCamera(config)
|
||||
|
||||
@@ -14,16 +14,21 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import Mock
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from packaging.version import Version
|
||||
|
||||
pytest.importorskip("datasets", reason="datasets is required (install lerobot[dataset])")
|
||||
|
||||
from datasets import Dataset # noqa: E402
|
||||
from huggingface_hub import DatasetCard
|
||||
|
||||
import lerobot.datasets.utils as dataset_utils
|
||||
from lerobot.datasets.io_utils import hf_transform_to_torch
|
||||
from lerobot.datasets.utils import create_lerobot_dataset_card
|
||||
from lerobot.datasets.utils import create_lerobot_dataset_card, get_repo_versions, get_safe_version
|
||||
from lerobot.utils.constants import ACTION, OBS_IMAGES
|
||||
from lerobot.utils.feature_utils import combine_feature_dicts
|
||||
|
||||
@@ -57,6 +62,30 @@ def test_default_parameters():
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||
def test_get_repo_versions_forwards_token(monkeypatch, token):
|
||||
api = Mock()
|
||||
api.list_repo_refs.return_value = SimpleNamespace(
|
||||
branches=[SimpleNamespace(name="v3.0")],
|
||||
tags=[],
|
||||
)
|
||||
hf_api = Mock(return_value=api)
|
||||
monkeypatch.setattr(dataset_utils, "HfApi", hf_api)
|
||||
|
||||
assert get_repo_versions("private/repo", token=token) == [Version("3.0")]
|
||||
hf_api.assert_called_once_with(token=token)
|
||||
api.list_repo_refs.assert_called_once_with("private/repo", repo_type="dataset")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||
def test_get_safe_version_forwards_token(monkeypatch, token):
|
||||
get_versions = Mock(return_value=[Version("3.0")])
|
||||
monkeypatch.setattr(dataset_utils, "get_repo_versions", get_versions)
|
||||
|
||||
assert get_safe_version("private/repo", "v3.0", token=token) == "v3.0"
|
||||
get_versions.assert_called_once_with("private/repo", token=token)
|
||||
|
||||
|
||||
def test_with_tags():
|
||||
tags = ["tag1", "tag2"]
|
||||
card = create_lerobot_dataset_card(tags=tags)
|
||||
|
||||
@@ -114,6 +114,20 @@ def test_dataset_initialization(tmp_path, lerobot_dataset_factory):
|
||||
assert dataset.num_frames == len(dataset)
|
||||
|
||||
|
||||
def test_dataset_slice(tmp_path, lerobot_dataset_factory):
|
||||
dataset = lerobot_dataset_factory(
|
||||
root=tmp_path / "test", total_episodes=3, total_frames=30, use_videos=False
|
||||
)
|
||||
|
||||
assert len(dataset[:5]) == 5
|
||||
assert len(dataset[::2]) == (len(dataset) + 1) // 2
|
||||
assert [item["index"].item() for item in dataset[4::-1]] == [4, 3, 2, 1, 0]
|
||||
assert [item["index"].item() for item in dataset[-3:]] == list(range(len(dataset) - 3, len(dataset)))
|
||||
assert dataset[len(dataset) :] == []
|
||||
assert isinstance(dataset[0], dict)
|
||||
assert dataset[:1][0].keys() == dataset[0].keys()
|
||||
|
||||
|
||||
# TODO(rcadene, aliberts): do not run LeRobotDataset.create, instead refactor LeRobotDatasetMetadata.create
|
||||
# and test the small resulting function that validates the features
|
||||
def test_dataset_feature_with_forward_slash_raises_error():
|
||||
@@ -1741,6 +1755,38 @@ def test_delta_timestamps_query_returns_correct_values(tmp_path, empty_lerobot_d
|
||||
assert is_pad == [True, False], f"Expected [True, False], got {is_pad}"
|
||||
|
||||
|
||||
def test_dataset_slice_with_delta_timestamps(tmp_path, empty_lerobot_dataset_factory):
|
||||
features = {
|
||||
"observation.state": {"dtype": "float32", "shape": (1,), "names": ["x"]},
|
||||
}
|
||||
dataset = empty_lerobot_dataset_factory(
|
||||
root=tmp_path / "test_slice_delta", features=features, use_videos=False, fps=10
|
||||
)
|
||||
|
||||
for frame_idx in range(5):
|
||||
dataset.add_frame(
|
||||
{
|
||||
"observation.state": torch.tensor([frame_idx], dtype=torch.float32),
|
||||
"task": "task_0",
|
||||
}
|
||||
)
|
||||
dataset.save_episode()
|
||||
dataset.finalize()
|
||||
|
||||
sliced_dataset = LeRobotDataset(
|
||||
dataset.repo_id,
|
||||
root=dataset.root,
|
||||
delta_timestamps={"observation.state": [-0.1, 0.0]},
|
||||
tolerance_s=0.04,
|
||||
)
|
||||
|
||||
items = sliced_dataset[:2]
|
||||
|
||||
assert items[0]["observation.state"].tolist() == [0.0, 0.0]
|
||||
assert items[0]["observation.state_is_pad"].tolist() == [True, False]
|
||||
assert items[1]["observation.state"].tolist() == [0.0, 1.0]
|
||||
|
||||
|
||||
def test_episode_filter_filters_dataset(tmp_path, lerobot_dataset_factory):
|
||||
"""episode_filter on LeRobotDataset narrows the loaded dataset to matching episodes."""
|
||||
dataset = lerobot_dataset_factory(root=tmp_path / "test", total_episodes=8, total_frames=200)
|
||||
|
||||
@@ -20,6 +20,7 @@ property delegation, and the full create-record-finalize-read lifecycle.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import Mock
|
||||
|
||||
import pytest
|
||||
@@ -191,6 +192,48 @@ def test_metadata_without_root_uses_hub_cache_snapshot_download(
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||
def test_metadata_download_forwards_token(tmp_path, monkeypatch, token):
|
||||
snapshot_root = tmp_path / "snapshot"
|
||||
snapshot_download = Mock(return_value=str(snapshot_root))
|
||||
get_safe_version = Mock(return_value="v3.0")
|
||||
load_metadata = Mock(side_effect=[FileNotFoundError, None])
|
||||
monkeypatch.setattr(dataset_metadata_module, "snapshot_download", snapshot_download)
|
||||
monkeypatch.setattr(dataset_metadata_module, "get_safe_version", get_safe_version)
|
||||
monkeypatch.setattr(LeRobotDatasetMetadata, "_load_metadata", load_metadata)
|
||||
|
||||
meta = LeRobotDatasetMetadata(
|
||||
repo_id=DUMMY_REPO_ID,
|
||||
revision="v3.0",
|
||||
token=token,
|
||||
)
|
||||
|
||||
assert meta.root == snapshot_root
|
||||
assert not hasattr(meta, "_token")
|
||||
get_safe_version.assert_called_once_with(DUMMY_REPO_ID, "v3.0", token=token)
|
||||
assert snapshot_download.call_args.kwargs["token"] is token
|
||||
|
||||
|
||||
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||
def test_data_download_forwards_token(tmp_path, monkeypatch, token):
|
||||
snapshot_root = tmp_path / "snapshot"
|
||||
snapshot_download = Mock(return_value=str(snapshot_root))
|
||||
monkeypatch.setattr(lerobot_dataset_module, "snapshot_download", snapshot_download)
|
||||
|
||||
dataset = LeRobotDataset.__new__(LeRobotDataset)
|
||||
dataset.repo_id = DUMMY_REPO_ID
|
||||
dataset.revision = "main"
|
||||
dataset.episodes = None
|
||||
dataset._requested_root = None
|
||||
dataset.meta = SimpleNamespace(root=None)
|
||||
dataset.reader = SimpleNamespace(root=None)
|
||||
|
||||
dataset._download(token=token)
|
||||
|
||||
assert dataset.root == snapshot_root
|
||||
assert snapshot_download.call_args.kwargs["token"] is token
|
||||
|
||||
|
||||
def test_without_root_reads_different_revisions_from_distinct_snapshot_roots(
|
||||
tmp_path,
|
||||
info_factory,
|
||||
|
||||
@@ -13,12 +13,16 @@
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import Mock
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
pytest.importorskip("datasets", reason="datasets is required (install lerobot[dataset])")
|
||||
|
||||
import lerobot.datasets.streaming_dataset as streaming_dataset_module
|
||||
from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset
|
||||
from lerobot.datasets.utils import safe_shard
|
||||
from lerobot.utils.constants import ACTION
|
||||
@@ -71,6 +75,40 @@ def get_frames_expected_order(streaming_ds: StreamingLeRobotDataset) -> list[int
|
||||
return expected_indices
|
||||
|
||||
|
||||
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||
@pytest.mark.parametrize("from_local", [False, True])
|
||||
def test_streaming_dataset_forwards_hub_token_only_for_remote_data(tmp_path, monkeypatch, token, from_local):
|
||||
requested_root = tmp_path / "local" if from_local else None
|
||||
metadata = SimpleNamespace(
|
||||
root=requested_root or tmp_path / "snapshot",
|
||||
revision=streaming_dataset_module.CODEBASE_VERSION,
|
||||
_version=streaming_dataset_module.CODEBASE_VERSION,
|
||||
features={},
|
||||
depth_keys=[],
|
||||
image_keys=[],
|
||||
rescale_depth_stats=Mock(),
|
||||
)
|
||||
metadata_cls = Mock(return_value=metadata)
|
||||
load_dataset = Mock(return_value=SimpleNamespace(num_shards=1))
|
||||
monkeypatch.setattr(streaming_dataset_module, "LeRobotDatasetMetadata", metadata_cls)
|
||||
monkeypatch.setattr(streaming_dataset_module, "load_dataset", load_dataset)
|
||||
|
||||
dataset = StreamingLeRobotDataset(DUMMY_REPO_ID, root=requested_root, token=token)
|
||||
|
||||
metadata_cls.assert_called_once_with(
|
||||
DUMMY_REPO_ID,
|
||||
requested_root,
|
||||
streaming_dataset_module.CODEBASE_VERSION,
|
||||
force_cache_sync=False,
|
||||
token=token,
|
||||
)
|
||||
if from_local:
|
||||
assert "token" not in load_dataset.call_args.kwargs
|
||||
else:
|
||||
assert load_dataset.call_args.kwargs["token"] is token
|
||||
assert not hasattr(dataset, "_token")
|
||||
|
||||
|
||||
def test_single_frame_consistency(tmp_path, lerobot_dataset_factory):
|
||||
"""Test if are correctly accessed"""
|
||||
ds_num_frames = 400
|
||||
|
||||
@@ -35,6 +35,17 @@ def test_unknown_type():
|
||||
make_env_config("nonexistent")
|
||||
|
||||
|
||||
def test_libero_fps_controls_simulator_frequency():
|
||||
cfg = LiberoEnv(fps=17)
|
||||
|
||||
assert cfg.gym_kwargs["control_freq"] == 17
|
||||
|
||||
|
||||
def test_libero_rejects_nonpositive_fps():
|
||||
with pytest.raises(ValueError, match="fps must be positive"):
|
||||
LiberoEnv(fps=0)
|
||||
|
||||
|
||||
def test_identity_processors():
|
||||
"""Base class get_env_processors() returns identity pipelines."""
|
||||
cfg = make_env_config("aloha")
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import shlex
|
||||
import sys
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import draccus
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("datasets", reason="datasets is required (install lerobot[dataset])")
|
||||
|
||||
from lerobot.annotations.steerable_pipeline.config import (
|
||||
DEFAULT_ANNOTATE_JOB_IMAGE,
|
||||
AnnotationJobConfig,
|
||||
AnnotationPipelineConfig,
|
||||
)
|
||||
from lerobot.jobs.annotate import build_pod_command, build_pod_setup, submit_annotate_to_hf
|
||||
|
||||
|
||||
def _parse(*args):
|
||||
return draccus.parse(AnnotationPipelineConfig, args=list(args))
|
||||
|
||||
|
||||
def _set_argv(monkeypatch, *args):
|
||||
monkeypatch.setattr(sys, "argv", ["lerobot-annotate", *args])
|
||||
|
||||
|
||||
# --- config ----------------------------------------------------------------
|
||||
|
||||
|
||||
def test_annotation_job_defaults_are_local_with_vllm_image():
|
||||
cfg = AnnotationJobConfig()
|
||||
assert cfg.target is None
|
||||
assert cfg.is_remote is False
|
||||
assert cfg.image == DEFAULT_ANNOTATE_JOB_IMAGE
|
||||
assert cfg.timeout == "2h"
|
||||
assert cfg.lerobot_ref == "main"
|
||||
|
||||
|
||||
def test_annotation_config_parses_job_target():
|
||||
cfg = _parse("--repo_id", "u/d", "--job.target", "h200")
|
||||
assert cfg.job.target == "h200"
|
||||
assert cfg.job.is_remote is True
|
||||
|
||||
|
||||
def test_annotation_config_defaults_to_local():
|
||||
assert _parse("--repo_id", "u/d").job.is_remote is False
|
||||
|
||||
|
||||
# --- pod command -----------------------------------------------------------
|
||||
|
||||
|
||||
def test_pod_setup_installs_requested_ref():
|
||||
setup = build_pod_setup("my-branch")
|
||||
assert "git+https://github.com/huggingface/lerobot.git@my-branch" in setup
|
||||
# The vLLM image has neither ffmpeg (video decode) nor lerobot's pinned deps.
|
||||
assert "ffmpeg" in setup
|
||||
assert "'draccus==0.10.0'" in setup
|
||||
|
||||
|
||||
def _annotate_argv(command):
|
||||
"""Extract the `lerobot-annotate ...` argv from a `bash -c` pod command."""
|
||||
assert command[:2] == ["bash", "-c"]
|
||||
_setup, _, annotate = command[2].rpartition(" && ")
|
||||
return shlex.split(annotate)
|
||||
|
||||
|
||||
def test_pod_command_forwards_user_flags_and_pins_local_target():
|
||||
command = build_pod_command(
|
||||
"u/d",
|
||||
"main",
|
||||
["--repo_id=u/d", "--new_repo_id=u/d_annotated", "--push_to_hub=true", "--job.target=h200"],
|
||||
)
|
||||
argv = _annotate_argv(command)
|
||||
assert argv[0] == "lerobot-annotate"
|
||||
# --job.* is client-side orchestration; the pod must not re-dispatch itself.
|
||||
assert not any(a.startswith("--job.") for a in argv[1:-1])
|
||||
assert argv[-1] == "--job.target=local"
|
||||
assert "--new_repo_id=u/d_annotated" in argv
|
||||
assert "--push_to_hub=true" in argv
|
||||
|
||||
|
||||
def test_pod_command_replaces_host_local_root_with_repo_id():
|
||||
"""--root points at a directory only the client has; the pod resolves by repo_id."""
|
||||
command = build_pod_command("u/d", "main", ["--root", "/home/me/datasets/d", "--seed=7"])
|
||||
argv = _annotate_argv(command)
|
||||
assert "--root" not in argv
|
||||
assert "/home/me/datasets/d" not in argv
|
||||
assert argv.count("--repo_id=u/d") == 1
|
||||
assert "--seed=7" in argv
|
||||
|
||||
|
||||
def test_pod_command_does_not_duplicate_repo_id():
|
||||
command = build_pod_command("u/d", "main", ["--repo_id", "u/d"])
|
||||
assert _annotate_argv(command).count("--repo_id=u/d") == 1
|
||||
|
||||
|
||||
def test_pod_command_quotes_flags_containing_spaces_and_json():
|
||||
"""serve_command and chat_template_kwargs must survive the trip through `bash -c`."""
|
||||
serve = "--vlm.serve_command=vllm serve Qwen/Qwen3.6-27B --max-model-len 32768 --port {port}"
|
||||
kwargs = '--vlm.chat_template_kwargs={"enable_thinking": false}'
|
||||
command = build_pod_command("u/d", "main", [serve, kwargs])
|
||||
argv = _annotate_argv(command)
|
||||
assert serve in argv
|
||||
assert kwargs in argv
|
||||
|
||||
|
||||
# --- submission ------------------------------------------------------------
|
||||
|
||||
|
||||
def test_submit_requires_login(monkeypatch):
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: None)
|
||||
with pytest.raises(RuntimeError, match="hf auth login"):
|
||||
submit_annotate_to_hf(_parse("--repo_id", "u/d", "--job.target", "h200"))
|
||||
|
||||
|
||||
def test_submit_requires_repo_id(monkeypatch):
|
||||
"""A remote run over --root alone can't work: the pod can't see the client's disk."""
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||
cfg = _parse("--root", "/tmp/d", "--job.target", "h200")
|
||||
with pytest.raises(ValueError, match="--repo_id"):
|
||||
submit_annotate_to_hf(cfg)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("arg", ["--config_path=annotate.yaml", "--vlm=vlm.yaml", "--job=job.yaml"])
|
||||
def test_submit_rejects_local_config_files(monkeypatch, arg):
|
||||
"""draccus takes a config file for the whole config and for each nested one; the
|
||||
pod can read none of them, so a remote run must refuse rather than drop them."""
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||
_set_argv(monkeypatch, arg, "--job.target=h200")
|
||||
cfg = _parse("--repo_id", "u/d", "--job.target", "h200")
|
||||
with pytest.raises(ValueError, match="cannot read config files"):
|
||||
submit_annotate_to_hf(cfg)
|
||||
|
||||
|
||||
def test_pod_command_drops_bare_job_config_file_arg():
|
||||
"""`--job` isn't caught by the `--job.` prefix, and could carry a remote target
|
||||
that would make the pod submit a job of its own — recursively."""
|
||||
argv = _annotate_argv(build_pod_command("u/d", "main", ["--job", "job.yaml", "--seed=7"]))
|
||||
assert "--job" not in argv
|
||||
assert "job.yaml" not in argv
|
||||
assert argv[-1] == "--job.target=local"
|
||||
|
||||
|
||||
def test_submit_dispatches_job(monkeypatch):
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.HfApi", lambda token=None: MagicMock())
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.ensure_dataset_available", lambda *a, **kw: None)
|
||||
|
||||
run_job_calls = []
|
||||
|
||||
def fake_run_job(**kwargs):
|
||||
run_job_calls.append(kwargs)
|
||||
return MagicMock(id="job-123")
|
||||
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.run_job", fake_run_job)
|
||||
_set_argv(monkeypatch, "--repo_id=u/d", "--push_to_hub=true", "--job.target=h200", "--job.detach=true")
|
||||
|
||||
cfg = _parse("--repo_id", "u/d", "--push_to_hub", "true", "--job.target", "h200", "--job.detach", "true")
|
||||
submit_annotate_to_hf(cfg)
|
||||
|
||||
assert len(run_job_calls) == 1
|
||||
call = run_job_calls[0]
|
||||
assert call["flavor"] == "h200"
|
||||
assert call["image"] == DEFAULT_ANNOTATE_JOB_IMAGE
|
||||
assert call["timeout"] == "2h"
|
||||
# The Hub token is forwarded so the pod can pull a private dataset and push the result.
|
||||
assert call["secrets"]["HF_TOKEN"] == "tok"
|
||||
assert call["labels"].get("lerobot") == "true"
|
||||
argv = _annotate_argv(call["command"])
|
||||
assert argv[0] == "lerobot-annotate"
|
||||
assert "--push_to_hub=true" in argv
|
||||
|
||||
|
||||
@pytest.mark.timeout(15)
|
||||
def test_submit_follows_job_to_completion(monkeypatch, capsys):
|
||||
"""Non-detach path must stream logs and RETURN (not hang) once the job is terminal.
|
||||
|
||||
Exercises the `follow_job` helper shared with the training submitter from the
|
||||
annotation side, which is why the job-state patches target `lerobot.jobs.hf`.
|
||||
Asserting on the completion message and not merely on "didn't hang" is what makes
|
||||
this fail if `follow_job` ever reports detached-without-a-verdict instead.
|
||||
"""
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.HfApi", lambda token=None: MagicMock())
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.ensure_dataset_available", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.run_job", lambda **kw: MagicMock(id="job-1", url="http://x"))
|
||||
monkeypatch.setattr(
|
||||
"lerobot.jobs.hf.inspect_job",
|
||||
lambda job_id: MagicMock(status=MagicMock(stage=MagicMock(value="COMPLETED"), message=None)),
|
||||
)
|
||||
monkeypatch.setattr("lerobot.jobs.hf.fetch_job_logs", lambda job_id, follow=True: iter(()))
|
||||
_set_argv(monkeypatch, "--repo_id=u/d", "--job.target=h200")
|
||||
|
||||
submit_annotate_to_hf(_parse("--repo_id", "u/d", "--push_to_hub", "true", "--job.target", "h200"))
|
||||
assert "Annotation complete" in capsys.readouterr().out
|
||||
|
||||
|
||||
@pytest.mark.timeout(15)
|
||||
def test_submit_raises_when_job_fails(monkeypatch):
|
||||
"""A job that ends in a non-COMPLETED stage must surface as an error, not a silent return."""
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.HfApi", lambda token=None: MagicMock())
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.ensure_dataset_available", lambda *a, **kw: None)
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.run_job", lambda **kw: MagicMock(id="job-1", url=None))
|
||||
monkeypatch.setattr(
|
||||
"lerobot.jobs.hf.inspect_job",
|
||||
lambda job_id: MagicMock(status=MagicMock(stage=MagicMock(value="ERROR"), message="Job timeout")),
|
||||
)
|
||||
monkeypatch.setattr("lerobot.jobs.hf.fetch_job_logs", lambda job_id, follow=True: iter(()))
|
||||
_set_argv(monkeypatch, "--repo_id=u/d", "--job.target=h200")
|
||||
|
||||
with pytest.raises(RuntimeError, match="stage=ERROR .Job timeout."):
|
||||
submit_annotate_to_hf(_parse("--repo_id", "u/d", "--job.target", "h200"))
|
||||
|
||||
|
||||
def test_submit_ensures_dataset_is_on_the_hub(monkeypatch):
|
||||
"""A local-only dataset is pushed (privately) before the job can reach it by repo_id."""
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.HfApi", lambda token=None: MagicMock())
|
||||
monkeypatch.setattr("lerobot.jobs.annotate.run_job", lambda **kw: MagicMock(id="job-1"))
|
||||
|
||||
seen = []
|
||||
monkeypatch.setattr(
|
||||
"lerobot.jobs.annotate.ensure_dataset_available",
|
||||
lambda repo_id, *, api, tags=None: seen.append((repo_id, tags)),
|
||||
)
|
||||
_set_argv(monkeypatch, "--repo_id=u/d", "--job.target=h200", "--job.detach=true")
|
||||
|
||||
submit_annotate_to_hf(
|
||||
_parse("--repo_id", "u/d", "--job.target", "h200", "--job.detach", "true", "--job.tags", '["lelab"]')
|
||||
)
|
||||
assert seen == [("u/d", ["lerobot", "lelab"])]
|
||||
@@ -29,12 +29,26 @@ from lerobot.jobs.hf import (
|
||||
_poll_until_done,
|
||||
build_remote_config_file,
|
||||
build_repo_id,
|
||||
follow_job,
|
||||
resolve_job_tags,
|
||||
resolve_wandb_api_key,
|
||||
submit_to_hf,
|
||||
)
|
||||
|
||||
|
||||
def test_follow_job_detach_returns_without_watching(monkeypatch):
|
||||
"""`detach` must short-circuit before any polling or log streaming starts."""
|
||||
|
||||
def _boom(*a, **kw):
|
||||
raise AssertionError("detach must not touch the job")
|
||||
|
||||
monkeypatch.setattr("lerobot.jobs.hf.inspect_job", _boom)
|
||||
monkeypatch.setattr("lerobot.jobs.hf.fetch_job_logs", _boom)
|
||||
# False = "stopped watching without a verdict", so callers stay quiet rather than
|
||||
# claiming success for a job that is still running.
|
||||
assert follow_job("job-1", detach=True) is False
|
||||
|
||||
|
||||
def test_resolve_job_tags_always_includes_lerobot_and_dedups():
|
||||
assert resolve_job_tags(None) == ["lerobot"]
|
||||
assert resolve_job_tags([]) == ["lerobot"]
|
||||
|
||||
@@ -405,12 +405,18 @@ def test_record_ranges_of_motion(mock_motors, dummy_motors):
|
||||
read_pos_stub = mock_motors.build_sequential_sync_read_stub(
|
||||
*X_SERIES_CONTROL_TABLE["Present_Position"], positions
|
||||
)
|
||||
with patch("lerobot.motors.motors_bus.enter_pressed", side_effect=[False, True]):
|
||||
bus = DynamixelMotorsBus(port=mock_motors.port, motors=dummy_motors)
|
||||
bus.connect(handshake=False)
|
||||
bus = DynamixelMotorsBus(port=mock_motors.port, motors=dummy_motors)
|
||||
bus.connect(handshake=False)
|
||||
|
||||
with (
|
||||
patch("lerobot.motors.motors_bus.enter_pressed", side_effect=[False, True]),
|
||||
patch("lerobot.motors.motors_bus.time.sleep") as mock_sleep,
|
||||
patch.object(bus, "sync_read", wraps=bus.sync_read) as mock_sync_read,
|
||||
):
|
||||
mins, maxes = bus.record_ranges_of_motion(display_values=False)
|
||||
|
||||
assert mock_motors.stubs[read_pos_stub].calls == 3
|
||||
assert all(call.kwargs["num_retry"] == 5 for call in mock_sync_read.call_args_list)
|
||||
mock_sleep.assert_called_once_with(0.02)
|
||||
assert mins == expected_mins
|
||||
assert maxes == expected_maxes
|
||||
|
||||
@@ -509,12 +509,18 @@ def test_record_ranges_of_motion(mock_motors, dummy_motors):
|
||||
stub = mock_motors.build_sequential_sync_read_stub(
|
||||
*STS_SMS_SERIES_CONTROL_TABLE["Present_Position"], positions
|
||||
)
|
||||
with patch("lerobot.motors.motors_bus.enter_pressed", side_effect=[False, True]):
|
||||
bus = FeetechMotorsBus(port=mock_motors.port, motors=dummy_motors)
|
||||
bus.connect(handshake=False)
|
||||
bus = FeetechMotorsBus(port=mock_motors.port, motors=dummy_motors)
|
||||
bus.connect(handshake=False)
|
||||
|
||||
with (
|
||||
patch("lerobot.motors.motors_bus.enter_pressed", side_effect=[False, True]),
|
||||
patch("lerobot.motors.motors_bus.time.sleep") as mock_sleep,
|
||||
patch.object(bus, "sync_read", wraps=bus.sync_read) as mock_sync_read,
|
||||
):
|
||||
mins, maxes = bus.record_ranges_of_motion(display_values=False)
|
||||
|
||||
assert mock_motors.stubs[stub].calls == 3
|
||||
assert all(call.kwargs["num_retry"] == 5 for call in mock_sync_read.call_args_list)
|
||||
mock_sleep.assert_called_once_with(0.02)
|
||||
assert mins == expected_mins
|
||||
assert maxes == expected_maxes
|
||||
|
||||
@@ -109,3 +109,22 @@ def test_send_action(follower):
|
||||
|
||||
goal_pos = {m: (i + 1) * 10 for i, m in enumerate(follower.bus.motors)}
|
||||
follower.bus.sync_write.assert_called_once_with("Goal_Position", goal_pos)
|
||||
|
||||
|
||||
def test_configure_writes_position_pid_coefficients():
|
||||
bus_mock = _make_bus_mock()
|
||||
bus_mock.motors = ["shoulder_pan"]
|
||||
robot = MagicMock()
|
||||
robot.bus = bus_mock
|
||||
robot.config = SO100FollowerConfig(
|
||||
port="/dev/null",
|
||||
position_p_coefficient=32,
|
||||
position_i_coefficient=1,
|
||||
position_d_coefficient=16,
|
||||
)
|
||||
|
||||
SO100Follower.configure(robot)
|
||||
|
||||
bus_mock.write.assert_any_call("P_Coefficient", "shoulder_pan", 32)
|
||||
bus_mock.write.assert_any_call("I_Coefficient", "shoulder_pan", 1)
|
||||
bus_mock.write.assert_any_call("D_Coefficient", "shoulder_pan", 16)
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
import lerobot.scripts.lerobot_setup_motors as motors_module
|
||||
|
||||
|
||||
def test_main_registers_plugins_before_parsing(monkeypatch):
|
||||
calls = []
|
||||
monkeypatch.setattr(motors_module, "register_third_party_plugins", lambda: calls.append("register"))
|
||||
monkeypatch.setattr(motors_module, "setup_motors", lambda: calls.append("setup"))
|
||||
|
||||
motors_module.main()
|
||||
|
||||
assert calls == ["register", "setup"]
|
||||
|
||||
|
||||
def test_setup_motors_accepts_third_party_device(monkeypatch):
|
||||
device = MagicMock()
|
||||
monkeypatch.setattr(motors_module, "make_teleoperator_from_config", lambda _: device)
|
||||
cfg = SimpleNamespace(device=SimpleNamespace(type="third_party"))
|
||||
|
||||
motors_module.setup_motors.__wrapped__(cfg)
|
||||
|
||||
device.setup_motors.assert_called_once_with()
|
||||
|
||||
|
||||
def test_setup_motors_reports_unsupported_device(monkeypatch):
|
||||
device = object()
|
||||
monkeypatch.setattr(motors_module, "make_teleoperator_from_config", lambda _: device)
|
||||
cfg = SimpleNamespace(device=SimpleNamespace(type="third_party"))
|
||||
|
||||
with pytest.raises(NotImplementedError, match="third_party"):
|
||||
motors_module.setup_motors.__wrapped__(cfg)
|
||||
Reference in New Issue
Block a user