mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-29 12:39:41 +00:00
Compare commits
20 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 2de042690e | |||
| 124d03608c | |||
| 0d383d09f2 | |||
| ab2b5b04dd | |||
| ac5c7b8600 | |||
| a6befef0ba | |||
| 53843007ea | |||
| d3bed0feee | |||
| a0eb860d1e | |||
| cfd9ff969c | |||
| f59eae4e27 | |||
| a993af9c51 | |||
| 392246feaf | |||
| 19dcbc19f1 | |||
| 679faeaafc | |||
| 228cb5ddb9 | |||
| ad176c6d41 | |||
| d6c605e8c5 | |||
| 9c82c39c7b | |||
| 73dbb6f43a |
@@ -51,6 +51,7 @@ pre-commit run --all-files # Lint + format (ruff, typo
|
|||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
- **Mypy is gradual**: strict only for `lerobot.envs`, `lerobot.configs`, `lerobot.optim`, `lerobot.model`, `lerobot.cameras`, `lerobot.motors`, `lerobot.transport`. Add type annotations when modifying these modules.
|
- **Mypy is gradual**: strict only for `lerobot.envs`, `lerobot.configs`, `lerobot.optim`, `lerobot.model`, `lerobot.cameras`, `lerobot.motors`, `lerobot.transport`. Add type annotations when modifying these modules.
|
||||||
- **Optional dependencies**: many policies, envs, and robots are behind extras (e.g., `lerobot[aloha]`). New imports for optional packages must be guarded or lazy. See `pyproject.toml [project.optional-dependencies]`.
|
- **Imports**: prefer top-level imports; relative (`from .sibling import X`) across sibling files within a module, absolute (`from lerobot.module import X`) across modules.
|
||||||
|
- **Optional dependencies**: many policies, envs, and robots are behind extras (e.g., `lerobot[aloha]`, see `pyproject.toml`). Guard optional imports with `TYPE_CHECKING or _foo_available` at module top + a `require_package(...)` check at use time. Reuse the `_foo_available` flags in `utils/import_utils.py`; don't call `is_package_available`.
|
||||||
- **Video decoding**: datasets can store observations as video files. `LeRobotDataset` handles frame extraction, but tests need ffmpeg installed.
|
- **Video decoding**: datasets can store observations as video files. `LeRobotDataset` handles frame extraction, but tests need ffmpeg installed.
|
||||||
- **Prioritize use of `uv run`** to execute Python commands (not raw `python` or `pip`).
|
- **Prioritize use of `uv run`** to execute Python commands (not raw `python` or `pip`).
|
||||||
|
|||||||
@@ -83,7 +83,7 @@ episode_index=0
|
|||||||
print(f"{dataset[episode_index]['action'].shape=}\n")
|
print(f"{dataset[episode_index]['action'].shape=}\n")
|
||||||
```
|
```
|
||||||
|
|
||||||
Learn more about it in the [LeRobotDataset Documentation](https://huggingface.co/docs/lerobot/lerobot-dataset-v3)
|
Learn more about it in the [LeRobotDataset Documentation](https://huggingface.co/docs/lerobot/lerobot-dataset-v3).
|
||||||
|
|
||||||
## SoTA Models
|
## SoTA Models
|
||||||
|
|
||||||
@@ -109,7 +109,7 @@ lerobot-train \
|
|||||||
| **World Models** | [VLA-JEPA](./docs/source/vla_jepa.mdx), [LingBot-VA](./docs/source/lingbot_va.mdx), [FastWAM](./docs/source/fastwam.mdx) |
|
| **World Models** | [VLA-JEPA](./docs/source/vla_jepa.mdx), [LingBot-VA](./docs/source/lingbot_va.mdx), [FastWAM](./docs/source/fastwam.mdx) |
|
||||||
| **Reward Models** | [SARM](./docs/source/sarm.mdx), [TOPReward](./docs/source/topreward.mdx), [Robometer](./docs/source/robometer.mdx) |
|
| **Reward Models** | [SARM](./docs/source/sarm.mdx), [TOPReward](./docs/source/topreward.mdx), [Robometer](./docs/source/robometer.mdx) |
|
||||||
|
|
||||||
Similarly to the hardware, you can easily implement your own policy & leverage LeRobot's data collection, training, and visualization tools, and share your model to the HF Hub
|
Similarly to the hardware, you can easily implement your own policy & leverage LeRobot's data collection, training, and visualization tools, and share your model to the HF Hub.
|
||||||
|
|
||||||
For detailed policy setup guides, see the [Policy Documentation](https://huggingface.co/docs/lerobot/bring_your_own_policies). For GPU/RAM requirements and expected training time per policy, see the [Compute Hardware Guide](https://huggingface.co/docs/lerobot/hardware_guide).
|
For detailed policy setup guides, see the [Policy Documentation](https://huggingface.co/docs/lerobot/bring_your_own_policies). For GPU/RAM requirements and expected training time per policy, see the [Compute Hardware Guide](https://huggingface.co/docs/lerobot/hardware_guide).
|
||||||
|
|
||||||
@@ -126,7 +126,7 @@ lerobot-eval \
|
|||||||
--eval.n_episodes=10
|
--eval.n_episodes=10
|
||||||
```
|
```
|
||||||
|
|
||||||
Learn how to implement your own simulation environment or benchmark and distribute it from the HF Hub by following the [EnvHub Documentation](https://huggingface.co/docs/lerobot/envhub)
|
Learn how to implement your own simulation environment or benchmark and distribute it from the HF Hub by following the [EnvHub Documentation](https://huggingface.co/docs/lerobot/envhub).
|
||||||
|
|
||||||
## Resources
|
## Resources
|
||||||
|
|
||||||
|
|||||||
@@ -89,8 +89,8 @@ subtask.
|
|||||||
|
|
||||||
The resulting spans are then stitched into a gap-free, full-episode
|
The resulting spans are then stitched into a gap-free, full-episode
|
||||||
cover, so **every frame has exactly one active subtask**. See
|
cover, so **every frame has exactly one active subtask**. See
|
||||||
[`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py)
|
[Running on Hugging Face Jobs](#running-on-hugging-face-jobs) for the
|
||||||
for the production settings (single camera, timestamped contact sheets,
|
production settings (single camera, timestamped contact sheets,
|
||||||
auto-windowed subtask generation).
|
auto-windowed subtask generation).
|
||||||
|
|
||||||
### Tools
|
### Tools
|
||||||
@@ -110,28 +110,67 @@ not-yet-implemented.
|
|||||||
|
|
||||||
## Running on Hugging Face Jobs
|
## Running on Hugging Face Jobs
|
||||||
|
|
||||||
Annotation runs on [Hugging Face Jobs](https://huggingface.co/docs/hub/en/jobs).
|
Annotating a real dataset needs a GPU big enough to serve the VLM, so
|
||||||
The repo ships a launcher script you copy and tweak for your dataset:
|
`lerobot-annotate` can dispatch itself to
|
||||||
|
[Hugging Face Jobs](https://huggingface.co/docs/hub/en/jobs) — same as
|
||||||
|
`lerobot-train`. Add `--job.target=<flavor>` to the exact command you'd
|
||||||
|
run locally and it runs on that hardware instead:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
HF_TOKEN=hf_... uv run python examples/annotations/run_hf_job.py
|
hf auth login # once
|
||||||
|
|
||||||
|
uv run lerobot-annotate \
|
||||||
|
--repo_id=user/my_dataset \
|
||||||
|
--new_repo_id=user/my_dataset_annotated \
|
||||||
|
--push_to_hub=true \
|
||||||
|
--vlm.model_id=Qwen/Qwen3.6-27B \
|
||||||
|
--vlm.num_gpus=1 \
|
||||||
|
--vlm.serve_command="vllm serve Qwen/Qwen3.6-27B --tensor-parallel-size 1 \
|
||||||
|
--max-model-len 32768 --gpu-memory-utilization 0.8 \
|
||||||
|
--uvicorn-log-level warning --port {port}" \
|
||||||
|
--vlm.serve_ready_timeout_s=1800 \
|
||||||
|
--vlm.chat_template_kwargs='{"enable_thinking": false}' \
|
||||||
|
--job.target=h200
|
||||||
```
|
```
|
||||||
|
|
||||||
[`run_hf_job.py`](https://github.com/huggingface/lerobot/blob/main/examples/annotations/run_hf_job.py)
|
That submits a single-GPU `h200` job that:
|
||||||
starts a single-GPU `h200` job (bump it to `h200x4` for big datasets)
|
|
||||||
that:
|
|
||||||
|
|
||||||
1. installs `lerobot` (from `main`) plus the annotation extras,
|
1. starts from the `vllm/vllm-openai` image and installs `lerobot` on top,
|
||||||
2. boots one vLLM server per GPU (using the `vllm/vllm-openai` image) and
|
2. boots one vLLM server per GPU and drives it over the OpenAI-compatible API,
|
||||||
drives it over the OpenAI-compatible API,
|
3. runs the `plan` / `interjections` / `vqa` modules across the dataset,
|
||||||
3. runs the `plan` / `interjections` / `vqa` modules across the dataset
|
|
||||||
with `lerobot-annotate`,
|
|
||||||
4. with `--push_to_hub=true`, uploads the result to `--new_repo_id` (or
|
4. with `--push_to_hub=true`, uploads the result to `--new_repo_id` (or
|
||||||
back to `--repo_id` in place if you leave that unset).
|
back to `--repo_id` in place if you leave that unset).
|
||||||
|
|
||||||
To use a different dataset, model, or hub repo, edit the `CMD` block in
|
The command streams the job's logs; `Ctrl-C` detaches without cancelling
|
||||||
the script. Every flag there maps directly to a `lerobot-annotate` flag
|
it. List the available flavors and their pricing with `hf jobs hardware`.
|
||||||
(run `lerobot-annotate --help` for the full list).
|
|
||||||
|
<Tip warning={true}>
|
||||||
|
|
||||||
|
Qwen3.6 ships with thinking enabled, which eats the token budget the
|
||||||
|
annotator needs for its JSON answer — `--vlm.chat_template_kwargs='{"enable_thinking": false}'`
|
||||||
|
turns it off. Without `--push_to_hub=true` the annotated dataset is
|
||||||
|
discarded when the pod exits.
|
||||||
|
|
||||||
|
</Tip>
|
||||||
|
|
||||||
|
### Job options
|
||||||
|
|
||||||
|
| Flag | Default | What it does |
|
||||||
|
| ------------------- | ------------------------- | ------------------------------------------------------------------------------- |
|
||||||
|
| `--job.target` | `local` | HF Jobs flavor to run on (e.g. `h200`, `h200x4`). Omitted/`local` runs here. |
|
||||||
|
| `--job.image` | `vllm/vllm-openai:latest` | Runtime image for the pod. |
|
||||||
|
| `--job.timeout` | `2h` | Wall-clock cap. Raise it for large datasets. |
|
||||||
|
| `--job.detach` | `false` | Submit and exit instead of streaming logs. |
|
||||||
|
| `--job.lerobot_ref` | `main` | Git ref of lerobot installed on the pod — point it at a branch to test changes. |
|
||||||
|
| `--job.tags` | `[]` | Extra tags on the job and on any dataset it pushes (`lerobot` is always added). |
|
||||||
|
|
||||||
|
For a bigger dataset, scale to `h200x4` and raise
|
||||||
|
`--vlm.parallel_servers` / `--vlm.num_gpus` to match, and give the job
|
||||||
|
more headroom with e.g. `--job.timeout=8h`.
|
||||||
|
|
||||||
|
Remote runs need `--repo_id` (the pod pulls the dataset from the Hub;
|
||||||
|
`--root` names a directory only your machine has). A dataset that exists
|
||||||
|
only in your local cache is pushed to a **private** repo first.
|
||||||
|
|
||||||
## Key options
|
## Key options
|
||||||
|
|
||||||
|
|||||||
@@ -165,6 +165,8 @@ Batches are flat dictionaries keyed by the constants in [`lerobot.utils.constant
|
|||||||
|
|
||||||
LeRobot uses `PolicyProcessorPipeline`s to normalize inputs and de-normalize outputs around your policy. For a concrete reference, see [`processor_act.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/act/processor_act.py) or [`processor_diffusion.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/diffusion/processor_diffusion.py).
|
LeRobot uses `PolicyProcessorPipeline`s to normalize inputs and de-normalize outputs around your policy. For a concrete reference, see [`processor_act.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/act/processor_act.py) or [`processor_diffusion.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/diffusion/processor_diffusion.py).
|
||||||
|
|
||||||
|
Pay close attention here: processors are the most common reproducibility pain point. A mismatch in normalization mode (`IDENTITY` vs `MEAN_STD` vs `MIN_MAX` vs `QUANTILES`/`QUANTILE10`) or in which features get normalized will train and eval without erroring, yet silently wreck results. Make sure the modes match how the checkpoint was trained, that the required stats exist (e.g. `QUANTILES` needs `q01`/`q99`), and that the pre- and post-processors stay consistent.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# processor_my_policy.py
|
# processor_my_policy.py
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -304,7 +306,9 @@ Mirror an existing policy that's structurally similar to yours; the diff is smal
|
|||||||
|
|
||||||
### Heavy / optional dependencies
|
### Heavy / optional dependencies
|
||||||
|
|
||||||
Most policies need a heavy backbone (transformers, diffusers, a specific VLM SDK). The convention is **two-step gating**: a `TYPE_CHECKING`-guarded import at module top, and a `require_package` runtime check in the constructor. [`modeling_diffusion.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/diffusion/modeling_diffusion.py) is the canonical reference:
|
Most policies need a heavy backbone (transformers, diffusers, a specific VLM SDK). Wherever one exists, prefer loading it e.g from `transformers` or `diffusers` rather than re-implementing the architecture in-tree.
|
||||||
|
|
||||||
|
The convention is **two-step gating**: a `TYPE_CHECKING`-guarded import at module top, and a `require_package` runtime check in the constructor. [`modeling_diffusion.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/diffusion/modeling_diffusion.py) is the canonical reference:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
@@ -374,6 +378,7 @@ The general expectations are in [`CONTRIBUTING.md`](https://github.com/huggingfa
|
|||||||
- [ ] Optional deps live behind a `[project.optional-dependencies]` extra and the `TYPE_CHECKING + require_package` guard.
|
- [ ] Optional deps live behind a `[project.optional-dependencies]` extra and the `TYPE_CHECKING + require_package` guard.
|
||||||
- [ ] `tests/policies/` updated; backward-compat artifact committed & policy-specific tests.
|
- [ ] `tests/policies/` updated; backward-compat artifact committed & policy-specific tests.
|
||||||
- [ ] `src/lerobot/policies/<name>/README.md` symlinked into `docs/source/policy_<name>_README.md`; user-facing `docs/source/<name>.mdx` written and added to `_toctree.yml`.
|
- [ ] `src/lerobot/policies/<name>/README.md` symlinked into `docs/source/policy_<name>_README.md`; user-facing `docs/source/<name>.mdx` written and added to `_toctree.yml`.
|
||||||
|
- [ ] `lerobot-train --policy.type my_policy ...` runs end-to-end for at least a few steps + save a checkpoint that can be loaded and run by `lerobot-eval` or `lerobot-rollout`.
|
||||||
- [ ] `templates/lerobot_modelcard_template.md` has a description entry and a `policy_docs` link for your policy.
|
- [ ] `templates/lerobot_modelcard_template.md` has a description entry and a `policy_docs` link for your policy.
|
||||||
- [ ] The models table in the root `README.md` lists your policy in the right category, linking to your doc page.
|
- [ ] The models table in the root `README.md` lists your policy in the right category, linking to your doc page.
|
||||||
- [ ] At least one reproducible benchmark eval in the policy MDX with a published checkpoint (sim benchmark, or real-robot dataset + checkpoint).
|
- [ ] At least one reproducible benchmark eval in the policy MDX with a published checkpoint (sim benchmark, or real-robot dataset + checkpoint).
|
||||||
|
|||||||
@@ -150,11 +150,12 @@ lerobot-rollout \
|
|||||||
Foot pedal input is also supported via `--strategy.input_device=pedal`. Configure pedal codes with `--strategy.pedal.*` flags.
|
Foot pedal input is also supported via `--strategy.input_device=pedal`. Configure pedal codes with `--strategy.pedal.*` flags.
|
||||||
|
|
||||||
| Flag | Description |
|
| Flag | Description |
|
||||||
| ------------------------------------ | ------------------------------------------------------- |
|
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `--strategy.num_episodes` | Number of correction episodes to record (default: 10) |
|
| `--strategy.num_episodes` | Number of correction episodes to record (default: 10) |
|
||||||
| `--strategy.record_autonomous` | Record autonomous frames too (default: false) |
|
| `--strategy.record_autonomous` | Record autonomous frames too (default: false) |
|
||||||
| `--strategy.upload_every_n_episodes` | Push to Hub every N episodes (default: 5) |
|
| `--strategy.upload_every_n_episodes` | Push to Hub every N episodes (default: 5) |
|
||||||
| `--strategy.input_device` | Input device: `keyboard` or `pedal` (default: keyboard) |
|
| `--strategy.input_device` | Input device: `keyboard` or `pedal` (default: keyboard) |
|
||||||
|
| `--strategy.smooth_handover` | Smoothly hand control over at pause / correction start (default: true). Disable for clutch-style teleops that re-reference at the current robot pose on engage |
|
||||||
| `--teleop.type` | **Required.** Teleoperator type |
|
| `--teleop.type` | **Required.** Teleoperator type |
|
||||||
|
|
||||||
### Episodic (`--strategy.type=episodic`)
|
### Episodic (`--strategy.type=episodic`)
|
||||||
|
|||||||
@@ -1,3 +1,11 @@
|
|||||||
|
# OMX
|
||||||
|
|
||||||
|
<img
|
||||||
|
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/omx_mainimage.png"
|
||||||
|
alt="OMX"
|
||||||
|
width=600
|
||||||
|
/>
|
||||||
|
|
||||||
## Order and Assemble the parts
|
## Order and Assemble the parts
|
||||||
|
|
||||||
First, assemble the OMX hardware following the official assembly guide.
|
First, assemble the OMX hardware following the official assembly guide.
|
||||||
|
|||||||
@@ -252,6 +252,10 @@ lerobot-dataset-viz \
|
|||||||
--episode-index 0
|
--episode-index 0
|
||||||
```
|
```
|
||||||
|
|
||||||
|
For a private or gated dataset, authenticate first with `hf auth login`, or set the
|
||||||
|
`HF_TOKEN` environment variable. The Hub client then discovers the credential
|
||||||
|
automatically; no token argument is needed.
|
||||||
|
|
||||||
**From a local folder:**
|
**From a local folder:**
|
||||||
Add the `--root` option and set `--mode local`. For example, to search in `./my_local_data_dir/lerobot/pusht`:
|
Add the `--root` option and set `--mode local`. For example, to search in `./my_local_data_dir/lerobot/pusht`:
|
||||||
|
|
||||||
|
|||||||
@@ -1,80 +0,0 @@
|
|||||||
#!/usr/bin/env python
|
|
||||||
|
|
||||||
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
"""Launch ``lerobot-annotate`` on a Hugging Face job (vllm + Qwen3.6-27B VLM).
|
|
||||||
|
|
||||||
Spawns one single-GPU ``h200`` job that:
|
|
||||||
|
|
||||||
1. installs ``lerobot`` from ``main`` plus the annotation extras,
|
|
||||||
2. boots one vllm server with Qwen3.6-27B (dense VLM),
|
|
||||||
3. runs the plan / interjections / vqa modules across the dataset
|
|
||||||
in free-form mode (each episode generates its own subtasks +
|
|
||||||
memory),
|
|
||||||
4. uploads the annotated dataset to ``--new_repo_id`` (when set)
|
|
||||||
or back to ``--repo_id``.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
|
|
||||||
HF_TOKEN=hf_... uv run python examples/annotations/run_hf_job.py
|
|
||||||
|
|
||||||
Adjust ``CMD`` (dataset, model, hub repo) and ``flavor`` below for your
|
|
||||||
run. For larger datasets, scale to ``h200x4`` and raise
|
|
||||||
``--vlm.parallel_servers`` / ``--vlm.num_gpus`` to match.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import os
|
|
||||||
|
|
||||||
from huggingface_hub import get_token, run_job
|
|
||||||
|
|
||||||
token = os.environ.get("HF_TOKEN") or get_token()
|
|
||||||
if not token:
|
|
||||||
raise RuntimeError("No HF token. Run `huggingface-cli login` or `export HF_TOKEN=hf_...`")
|
|
||||||
|
|
||||||
CMD = (
|
|
||||||
"apt-get update -qq && apt-get install -y -qq git ffmpeg && "
|
|
||||||
"pip install --no-deps "
|
|
||||||
"'lerobot @ git+https://github.com/huggingface/lerobot.git@main' && "
|
|
||||||
# Pins mirror pyproject.toml — unpinned installs pull av 18 / datasets 5 /
|
|
||||||
# draccus 0.11, which break lerobot at import time.
|
|
||||||
"pip install --upgrade-strategy only-if-needed "
|
|
||||||
"'datasets>=4.7.0,<5.0.0' 'pyarrow>=21.0.0,<30.0.0' 'av>=15.0.0,<16.0.0' 'draccus==0.10.0' "
|
|
||||||
"'pandas>=2.0.0,<3.0.0' jsonlines gymnasium torchcodec mergedeep pyyaml-include toml typing-inspect "
|
|
||||||
"openai && "
|
|
||||||
"export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 && "
|
|
||||||
"export VLLM_VIDEO_BACKEND=pyav && "
|
|
||||||
"lerobot-annotate "
|
|
||||||
"--repo_id=pepijn223/robocasa_pretrain_human300_v4 "
|
|
||||||
"--new_repo_id=pepijn223/robocasa_pretrain_human300_v4_annotated "
|
|
||||||
"--push_to_hub=true "
|
|
||||||
"--vlm.backend=openai "
|
|
||||||
"--vlm.model_id=Qwen/Qwen3.6-27B "
|
|
||||||
"--vlm.num_gpus=1 "
|
|
||||||
'--vlm.serve_command="vllm serve Qwen/Qwen3.6-27B '
|
|
||||||
"--tensor-parallel-size 1 --max-model-len 32768 "
|
|
||||||
'--gpu-memory-utilization 0.8 --uvicorn-log-level warning --port {port}" '
|
|
||||||
"--vlm.serve_ready_timeout_s=1800 "
|
|
||||||
# Qwen3.6 ships with thinking on; annotation wants plain JSON answers.
|
|
||||||
"--vlm.chat_template_kwargs='{\"enable_thinking\": false}'"
|
|
||||||
)
|
|
||||||
|
|
||||||
job = run_job(
|
|
||||||
image="vllm/vllm-openai:latest",
|
|
||||||
command=["bash", "-c", CMD],
|
|
||||||
flavor="h200",
|
|
||||||
secrets={"HF_TOKEN": token},
|
|
||||||
timeout="2h",
|
|
||||||
)
|
|
||||||
print(f"Job URL: {job.url}")
|
|
||||||
print(f"Job ID: {job.id}")
|
|
||||||
+1
-1
@@ -155,7 +155,7 @@ accelerate-dep = ["accelerate>=1.14.0,<2.0.0"]
|
|||||||
can-dep = ["python-can>=4.2.0,<5.0.0"]
|
can-dep = ["python-can>=4.2.0,<5.0.0"]
|
||||||
peft-dep = ["peft>=0.18.0,<1.0.0"]
|
peft-dep = ["peft>=0.18.0,<1.0.0"]
|
||||||
scipy-dep = ["scipy>=1.14.0,<2.0.0"]
|
scipy-dep = ["scipy>=1.14.0,<2.0.0"]
|
||||||
diffusers-dep = ["diffusers>=0.27.2,<0.36.0"]
|
diffusers-dep = ["diffusers>=0.38.0,<0.40.0"]
|
||||||
qwen-vl-utils-dep = ["qwen-vl-utils>=0.0.11,<0.1.0"]
|
qwen-vl-utils-dep = ["qwen-vl-utils>=0.0.11,<0.1.0"]
|
||||||
matplotlib-dep = ["matplotlib>=3.10.3,<4.0.0", "contourpy>=1.3.0,<2.0.0"] # NOTE: Explicitly listing contourpy helps the resolver converge faster.
|
matplotlib-dep = ["matplotlib>=3.10.3,<4.0.0", "contourpy>=1.3.0,<2.0.0"] # NOTE: Explicitly listing contourpy helps the resolver converge faster.
|
||||||
pyserial-dep = ["pyserial>=3.5,<4.0"]
|
pyserial-dep = ["pyserial>=3.5,<4.0"]
|
||||||
|
|||||||
@@ -20,6 +20,29 @@ from dataclasses import dataclass, field
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from lerobot.configs.default import JobConfig
|
||||||
|
|
||||||
|
# The annotation pipeline boots its own vLLM server, so the pod starts from the
|
||||||
|
# official vLLM runtime rather than the prebuilt `lerobot-gpu` training image;
|
||||||
|
# `lerobot` is pip-installed on top (see `lerobot.jobs.annotate`).
|
||||||
|
DEFAULT_ANNOTATE_JOB_IMAGE = "vllm/vllm-openai:latest"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class AnnotationJobConfig(JobConfig):
|
||||||
|
"""`JobConfig` with the annotation runtime's defaults.
|
||||||
|
|
||||||
|
Adds `lerobot_ref` because the vLLM image ships no lerobot: the pod installs
|
||||||
|
it from git, and the ref decides which code actually annotates. Point it at a
|
||||||
|
branch/tag/SHA to try unmerged changes remotely.
|
||||||
|
"""
|
||||||
|
|
||||||
|
image: str = DEFAULT_ANNOTATE_JOB_IMAGE
|
||||||
|
# Annotation is a bounded pass over a dataset; a tighter cap than training's
|
||||||
|
# "2d" keeps a wedged vLLM server from burning a day of GPU time.
|
||||||
|
timeout: str | None = "2h"
|
||||||
|
lerobot_ref: str = "main"
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class PlanConfig:
|
class PlanConfig:
|
||||||
@@ -207,6 +230,11 @@ class AnnotationPipelineConfig:
|
|||||||
vlm: VlmConfig = field(default_factory=VlmConfig)
|
vlm: VlmConfig = field(default_factory=VlmConfig)
|
||||||
executor: ExecutorConfig = field(default_factory=ExecutorConfig)
|
executor: ExecutorConfig = field(default_factory=ExecutorConfig)
|
||||||
|
|
||||||
|
# Where the annotation runs: omitted / "local" annotates on this machine, any
|
||||||
|
# other value is an HF Jobs flavor (e.g. "h200") and submits the run there.
|
||||||
|
# List flavors + pricing with `hf jobs hardware`.
|
||||||
|
job: AnnotationJobConfig = field(default_factory=AnnotationJobConfig)
|
||||||
|
|
||||||
skip_validation: bool = False
|
skip_validation: bool = False
|
||||||
only_episodes: tuple[int, ...] | None = None
|
only_episodes: tuple[int, ...] | None = None
|
||||||
|
|
||||||
|
|||||||
@@ -30,8 +30,8 @@ Phase 3 is why the ``plan`` module must be re-entered after the
|
|||||||
timestamps.
|
timestamps.
|
||||||
|
|
||||||
Distributed execution is provided by Hugging Face Jobs (see
|
Distributed execution is provided by Hugging Face Jobs (see
|
||||||
``examples/annotations/run_hf_job.py``); the runner inside the job
|
``lerobot.jobs.annotate``, reached via ``--job.target=<flavor>``); the pod
|
||||||
invokes ``lerobot-annotate`` which uses this in-process executor.
|
inside the job invokes ``lerobot-annotate`` which uses this in-process executor.
|
||||||
Episode-level concurrency is controlled by
|
Episode-level concurrency is controlled by
|
||||||
``ExecutorConfig.episode_parallelism``.
|
``ExecutorConfig.episode_parallelism``.
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -194,12 +194,13 @@ def make_vlm_client(config: VlmConfig) -> VlmClient:
|
|||||||
"""Build the shared VLM client.
|
"""Build the shared VLM client.
|
||||||
|
|
||||||
Only the ``openai`` backend is supported for now. The shipped workflow
|
Only the ``openai`` backend is supported for now. The shipped workflow
|
||||||
is Hugging Face Jobs (``examples/annotations/run_hf_job.py``): it boots
|
is Hugging Face Jobs (``lerobot-annotate --job.target=<flavor>``): it
|
||||||
a vLLM server inside the ``vllm/vllm-openai`` image and the pipeline
|
boots a vLLM server inside the ``vllm/vllm-openai`` image and the
|
||||||
talks to it over the OpenAI-compatible API (``--vlm.backend=openai``,
|
pipeline talks to it over the OpenAI-compatible API
|
||||||
optionally auto-spawning the server via ``auto_serve`` /
|
(``--vlm.backend=openai``, optionally auto-spawning the server via
|
||||||
``serve_command``). The former in-process ``vllm`` / ``transformers``
|
``auto_serve`` / ``serve_command``). The former in-process ``vllm`` /
|
||||||
backends were removed to keep the support surface to the HF Jobs path.
|
``transformers`` backends were removed to keep the support surface to
|
||||||
|
the HF Jobs path.
|
||||||
|
|
||||||
For ``stub``, construct :class:`StubVlmClient` directly with a responder
|
For ``stub``, construct :class:`StubVlmClient` directly with a responder
|
||||||
callable; it is rejected here to make accidental misuse obvious.
|
callable; it is rejected here to make accidental misuse obvious.
|
||||||
@@ -213,8 +214,8 @@ def make_vlm_client(config: VlmConfig) -> VlmClient:
|
|||||||
if config.backend in {"vllm", "transformers"}:
|
if config.backend in {"vllm", "transformers"}:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"backend={config.backend!r} (in-process local model) is not supported for now — "
|
f"backend={config.backend!r} (in-process local model) is not supported for now — "
|
||||||
"only backend='openai' (the Hugging Face Jobs flow) is. Run the pipeline via "
|
"only backend='openai' (the Hugging Face Jobs flow) is. Run the pipeline with "
|
||||||
"examples/annotations/run_hf_job.py, which serves the model with vLLM in the "
|
"`lerobot-annotate --job.target=<flavor>`, which serves the model with vLLM in the "
|
||||||
"vllm/vllm-openai image and talks to it over the OpenAI-compatible API."
|
"vllm/vllm-openai image and talks to it over the OpenAI-compatible API."
|
||||||
)
|
)
|
||||||
raise ValueError(f"Unknown VLM backend: {config.backend!r}")
|
raise ValueError(f"Unknown VLM backend: {config.backend!r}")
|
||||||
|
|||||||
@@ -173,7 +173,8 @@ class Reachy2Camera(Camera):
|
|||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"Invalid color mode '{self.color_mode}'. Expected {ColorMode.RGB} or {ColorMode.BGR}."
|
f"Invalid color mode '{self.color_mode}'. Expected {ColorMode.RGB} or {ColorMode.BGR}."
|
||||||
)
|
)
|
||||||
if self.color_mode == ColorMode.RGB:
|
is_depth_frame = self.config.name == "depth" and self.config.image_type == "depth"
|
||||||
|
if not is_depth_frame and self.color_mode == ColorMode.RGB:
|
||||||
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
||||||
|
|
||||||
self.latest_frame = frame
|
self.latest_frame = frame
|
||||||
|
|||||||
@@ -453,7 +453,7 @@ class RealSenseCamera(Camera):
|
|||||||
)
|
)
|
||||||
|
|
||||||
processed_image = image
|
processed_image = image
|
||||||
if self.color_mode == ColorMode.BGR:
|
if not depth_frame and self.color_mode == ColorMode.BGR:
|
||||||
processed_image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
|
processed_image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
|
||||||
|
|
||||||
if self.rotation in [cv2.ROTATE_90_CLOCKWISE, cv2.ROTATE_90_COUNTERCLOCKWISE, cv2.ROTATE_180]:
|
if self.rotation in [cv2.ROTATE_90_CLOCKWISE, cv2.ROTATE_90_COUNTERCLOCKWISE, cv2.ROTATE_180]:
|
||||||
|
|||||||
@@ -73,6 +73,8 @@ class LeRobotDatasetMetadata:
|
|||||||
revision: str | None = None,
|
revision: str | None = None,
|
||||||
force_cache_sync: bool = False,
|
force_cache_sync: bool = False,
|
||||||
metadata_buffer_size: int = 10,
|
metadata_buffer_size: int = 10,
|
||||||
|
*,
|
||||||
|
token: str | bool | None = None,
|
||||||
):
|
):
|
||||||
"""Load or download metadata for an existing LeRobot dataset.
|
"""Load or download metadata for an existing LeRobot dataset.
|
||||||
|
|
||||||
@@ -94,6 +96,10 @@ class LeRobotDatasetMetadata:
|
|||||||
even when local files exist.
|
even when local files exist.
|
||||||
metadata_buffer_size: Number of episode metadata records to buffer
|
metadata_buffer_size: Number of episode metadata records to buffer
|
||||||
in memory before flushing to parquet.
|
in memory before flushing to parquet.
|
||||||
|
token: Authentication token used for Hub requests. Pass a string
|
||||||
|
token, ``True`` to require the locally stored token, ``False``
|
||||||
|
to disable authentication, or ``None`` to use the Hugging Face
|
||||||
|
Hub default.
|
||||||
"""
|
"""
|
||||||
self.repo_id = repo_id
|
self.repo_id = repo_id
|
||||||
self.revision = revision if revision else CODEBASE_VERSION
|
self.revision = revision if revision else CODEBASE_VERSION
|
||||||
@@ -113,9 +119,12 @@ class LeRobotDatasetMetadata:
|
|||||||
self._load_metadata()
|
self._load_metadata()
|
||||||
except (FileNotFoundError, NotADirectoryError):
|
except (FileNotFoundError, NotADirectoryError):
|
||||||
if is_valid_version(self.revision):
|
if is_valid_version(self.revision):
|
||||||
|
if token is None:
|
||||||
self.revision = get_safe_version(self.repo_id, self.revision)
|
self.revision = get_safe_version(self.repo_id, self.revision)
|
||||||
|
else:
|
||||||
|
self.revision = get_safe_version(self.repo_id, self.revision, token=token)
|
||||||
|
|
||||||
self._pull_from_repo(allow_patterns="meta/")
|
self._pull_from_repo(allow_patterns="meta/", token=token)
|
||||||
self._load_metadata()
|
self._load_metadata()
|
||||||
|
|
||||||
def _flush_metadata_buffer(self) -> None:
|
def _flush_metadata_buffer(self) -> None:
|
||||||
@@ -220,7 +229,10 @@ class LeRobotDatasetMetadata:
|
|||||||
self,
|
self,
|
||||||
allow_patterns: list[str] | str | None = None,
|
allow_patterns: list[str] | str | None = None,
|
||||||
ignore_patterns: list[str] | str | None = None,
|
ignore_patterns: list[str] | str | None = None,
|
||||||
|
*,
|
||||||
|
token: str | bool | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
|
token_kwargs = {} if token is None else {"token": token}
|
||||||
if self._requested_root is None:
|
if self._requested_root is None:
|
||||||
self.root = Path(
|
self.root = Path(
|
||||||
snapshot_download(
|
snapshot_download(
|
||||||
@@ -230,6 +242,7 @@ class LeRobotDatasetMetadata:
|
|||||||
cache_dir=HF_LEROBOT_HUB_CACHE,
|
cache_dir=HF_LEROBOT_HUB_CACHE,
|
||||||
allow_patterns=allow_patterns,
|
allow_patterns=allow_patterns,
|
||||||
ignore_patterns=ignore_patterns,
|
ignore_patterns=ignore_patterns,
|
||||||
|
**token_kwargs,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
@@ -242,6 +255,7 @@ class LeRobotDatasetMetadata:
|
|||||||
local_dir=self._requested_root,
|
local_dir=self._requested_root,
|
||||||
allow_patterns=allow_patterns,
|
allow_patterns=allow_patterns,
|
||||||
ignore_patterns=ignore_patterns,
|
ignore_patterns=ignore_patterns,
|
||||||
|
**token_kwargs,
|
||||||
)
|
)
|
||||||
self.root = self._requested_root
|
self.root = self._requested_root
|
||||||
|
|
||||||
|
|||||||
@@ -65,6 +65,8 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
encoder_threads: int | None = None,
|
encoder_threads: int | None = None,
|
||||||
streaming_encoding: bool = False,
|
streaming_encoding: bool = False,
|
||||||
encoder_queue_maxsize: int = 30,
|
encoder_queue_maxsize: int = 30,
|
||||||
|
*,
|
||||||
|
token: str | bool | None = None,
|
||||||
):
|
):
|
||||||
"""
|
"""
|
||||||
2 modes are available for instantiating this class, depending on 2 different use cases:
|
2 modes are available for instantiating this class, depending on 2 different use cases:
|
||||||
@@ -197,6 +199,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
instead of writing PNG images first. This makes save_episode() near-instant. Defaults to False.
|
instead of writing PNG images first. This makes save_episode() near-instant. Defaults to False.
|
||||||
encoder_queue_maxsize (int, optional): Maximum number of frames to buffer per camera when using
|
encoder_queue_maxsize (int, optional): Maximum number of frames to buffer per camera when using
|
||||||
streaming encoding. Defaults to 30 (~1s at 30fps).
|
streaming encoding. Defaults to 30 (~1s at 30fps).
|
||||||
|
token: Authentication token used while downloading this dataset
|
||||||
|
from the Hub. Pass a string token, ``True`` to require the
|
||||||
|
locally stored token, ``False`` to disable authentication, or
|
||||||
|
``None`` to use the Hugging Face Hub default. The token is not
|
||||||
|
retained on the dataset instance after initialization.
|
||||||
|
|
||||||
Note:
|
Note:
|
||||||
Write-mode parameters (``streaming_encoding``, ``batch_encoding_size``) passed to
|
Write-mode parameters (``streaming_encoding``, ``batch_encoding_size``) passed to
|
||||||
@@ -220,7 +227,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
|
|
||||||
# Load metadata (sets self.root once from the resolved metadata root)
|
# Load metadata (sets self.root once from the resolved metadata root)
|
||||||
self.meta = LeRobotDatasetMetadata(
|
self.meta = LeRobotDatasetMetadata(
|
||||||
self.repo_id, self._requested_root, self.revision, force_cache_sync=force_cache_sync
|
self.repo_id,
|
||||||
|
self._requested_root,
|
||||||
|
self.revision,
|
||||||
|
force_cache_sync=force_cache_sync,
|
||||||
|
token=token,
|
||||||
)
|
)
|
||||||
self.root = self.meta.root
|
self.root = self.meta.root
|
||||||
self.revision = self.meta.revision
|
self.revision = self.meta.revision
|
||||||
@@ -260,8 +271,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
# Load actual data
|
# Load actual data
|
||||||
if force_cache_sync or not self.reader.try_load():
|
if force_cache_sync or not self.reader.try_load():
|
||||||
if is_valid_version(self.revision):
|
if is_valid_version(self.revision):
|
||||||
|
if token is None:
|
||||||
self.revision = get_safe_version(self.repo_id, self.revision)
|
self.revision = get_safe_version(self.repo_id, self.revision)
|
||||||
self._download(download_videos)
|
else:
|
||||||
|
self.revision = get_safe_version(self.repo_id, self.revision, token=token)
|
||||||
|
self._download(download_videos, token=token)
|
||||||
self.reader.load_and_activate()
|
self.reader.load_and_activate()
|
||||||
|
|
||||||
# Detect write-mode params for backward compatibility
|
# Detect write-mode params for backward compatibility
|
||||||
@@ -478,18 +492,19 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
"""Return the number of frames in the selected episodes."""
|
"""Return the number of frames in the selected episodes."""
|
||||||
return self.num_frames
|
return self.num_frames
|
||||||
|
|
||||||
def __getitem__(self, idx) -> dict:
|
def __getitem__(self, idx: int | slice) -> dict | list[dict]:
|
||||||
"""Return a single frame by index, with all transforms applied.
|
"""Return one frame or a slice of frames, with all transforms applied.
|
||||||
|
|
||||||
Loads the frame from the underlying HF dataset, expands delta-timestamp
|
Loads the frame from the underlying HF dataset, expands delta-timestamp
|
||||||
windows, decodes video frames, and applies image transforms. Delegates
|
windows, decodes video frames, and applies image transforms. Delegates
|
||||||
the core logic to :meth:`DatasetReader.get_item`.
|
the core logic to :class:`DatasetReader`.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
idx: Index into the (possibly episode-filtered) dataset.
|
idx: Integer index or slice into the possibly episode-filtered dataset.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Dict mapping feature names to their tensor values for this frame.
|
A frame dictionary for an integer index, or a list of frame
|
||||||
|
dictionaries for a slice.
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
RuntimeError: If the dataset is currently being recorded and
|
RuntimeError: If the dataset is currently being recorded and
|
||||||
@@ -499,6 +514,9 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
"Cannot read from a dataset that is being recorded. Call finalize() first, then access items."
|
"Cannot read from a dataset that is being recorded. Call finalize() first, then access items."
|
||||||
)
|
)
|
||||||
|
if isinstance(idx, slice):
|
||||||
|
return [self[item_idx] for item_idx in range(*idx.indices(len(self)))]
|
||||||
|
|
||||||
reader = self._ensure_reader()
|
reader = self._ensure_reader()
|
||||||
if reader.hf_dataset is None:
|
if reader.hf_dataset is None:
|
||||||
# One-shot load after finalize()
|
# One-shot load after finalize()
|
||||||
@@ -622,10 +640,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
hub_api.delete_tag(self.repo_id, tag=CODEBASE_VERSION, repo_type="dataset")
|
hub_api.delete_tag(self.repo_id, tag=CODEBASE_VERSION, repo_type="dataset")
|
||||||
hub_api.create_tag(self.repo_id, tag=CODEBASE_VERSION, revision=branch, repo_type="dataset")
|
hub_api.create_tag(self.repo_id, tag=CODEBASE_VERSION, revision=branch, repo_type="dataset")
|
||||||
|
|
||||||
def _download(self, download_videos: bool = True) -> None:
|
def _download(self, download_videos: bool = True, *, token: str | bool | None = None) -> None:
|
||||||
"""Downloads the dataset from the given 'repo_id' at the provided version."""
|
"""Downloads the dataset from the given 'repo_id' at the provided version."""
|
||||||
ignore_patterns = None if download_videos else "videos/"
|
ignore_patterns = None if download_videos else "videos/"
|
||||||
files = None
|
files = None
|
||||||
|
token_kwargs = {} if token is None else {"token": token}
|
||||||
if self.episodes is not None:
|
if self.episodes is not None:
|
||||||
# Reader is guaranteed to exist here (created in __init__ before _download)
|
# Reader is guaranteed to exist here (created in __init__ before _download)
|
||||||
files = self.reader.get_episodes_file_paths()
|
files = self.reader.get_episodes_file_paths()
|
||||||
@@ -639,6 +658,7 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
cache_dir=HF_LEROBOT_HUB_CACHE,
|
cache_dir=HF_LEROBOT_HUB_CACHE,
|
||||||
allow_patterns=files,
|
allow_patterns=files,
|
||||||
ignore_patterns=ignore_patterns,
|
ignore_patterns=ignore_patterns,
|
||||||
|
**token_kwargs,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
@@ -650,6 +670,7 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
local_dir=self._requested_root,
|
local_dir=self._requested_root,
|
||||||
allow_patterns=files,
|
allow_patterns=files,
|
||||||
ignore_patterns=ignore_patterns,
|
ignore_patterns=ignore_patterns,
|
||||||
|
**token_kwargs,
|
||||||
)
|
)
|
||||||
self.meta.root = self._requested_root
|
self.meta.root = self._requested_root
|
||||||
|
|
||||||
@@ -789,6 +810,8 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
image_writer_threads: int = 0,
|
image_writer_threads: int = 0,
|
||||||
streaming_encoding: bool = False,
|
streaming_encoding: bool = False,
|
||||||
encoder_queue_maxsize: int = 30,
|
encoder_queue_maxsize: int = 30,
|
||||||
|
*,
|
||||||
|
token: str | bool | None = None,
|
||||||
) -> "LeRobotDataset":
|
) -> "LeRobotDataset":
|
||||||
"""Resume recording on an existing dataset.
|
"""Resume recording on an existing dataset.
|
||||||
|
|
||||||
@@ -822,6 +845,8 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
streaming_encoding: If ``True``, encode video in real-time during
|
streaming_encoding: If ``True``, encode video in real-time during
|
||||||
capture.
|
capture.
|
||||||
encoder_queue_maxsize: Max buffered frames per camera for streaming.
|
encoder_queue_maxsize: Max buffered frames per camera for streaming.
|
||||||
|
token: Authentication token used if metadata must be downloaded
|
||||||
|
from the Hub. The token is not retained on the dataset instance.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
A :class:`LeRobotDataset` in write mode, ready to append episodes.
|
A :class:`LeRobotDataset` in write mode, ready to append episodes.
|
||||||
@@ -850,7 +875,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
|
|||||||
|
|
||||||
# Load metadata (revision-safe when root is not provided)
|
# Load metadata (revision-safe when root is not provided)
|
||||||
obj.meta = LeRobotDatasetMetadata(
|
obj.meta = LeRobotDatasetMetadata(
|
||||||
obj.repo_id, obj._requested_root, obj.revision, force_cache_sync=force_cache_sync
|
obj.repo_id,
|
||||||
|
obj._requested_root,
|
||||||
|
obj.revision,
|
||||||
|
force_cache_sync=force_cache_sync,
|
||||||
|
token=token,
|
||||||
)
|
)
|
||||||
|
|
||||||
obj._encoder_threads = encoder_threads
|
obj._encoder_threads = encoder_threads
|
||||||
|
|||||||
@@ -48,6 +48,8 @@ class MultiLeRobotDataset(torch.utils.data.Dataset):
|
|||||||
tolerances_s: dict | None = None,
|
tolerances_s: dict | None = None,
|
||||||
download_videos: bool = True,
|
download_videos: bool = True,
|
||||||
video_backend: str | None = None,
|
video_backend: str | None = None,
|
||||||
|
*,
|
||||||
|
token: str | bool | None = None,
|
||||||
):
|
):
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self.repo_ids = repo_ids
|
self.repo_ids = repo_ids
|
||||||
@@ -65,6 +67,7 @@ class MultiLeRobotDataset(torch.utils.data.Dataset):
|
|||||||
tolerance_s=self.tolerances_s[repo_id],
|
tolerance_s=self.tolerances_s[repo_id],
|
||||||
download_videos=download_videos,
|
download_videos=download_videos,
|
||||||
video_backend=video_backend,
|
video_backend=video_backend,
|
||||||
|
token=token,
|
||||||
)
|
)
|
||||||
for repo_id in repo_ids
|
for repo_id in repo_ids
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -256,6 +256,8 @@ class StreamingLeRobotDataset(torch.utils.data.IterableDataset):
|
|||||||
shuffle: bool = True,
|
shuffle: bool = True,
|
||||||
return_uint8: bool = False,
|
return_uint8: bool = False,
|
||||||
depth_output_unit: str = DEFAULT_DEPTH_UNIT,
|
depth_output_unit: str = DEFAULT_DEPTH_UNIT,
|
||||||
|
*,
|
||||||
|
token: str | bool | None = None,
|
||||||
):
|
):
|
||||||
"""Initialize a StreamingLeRobotDataset.
|
"""Initialize a StreamingLeRobotDataset.
|
||||||
|
|
||||||
@@ -278,6 +280,11 @@ class StreamingLeRobotDataset(torch.utils.data.IterableDataset):
|
|||||||
shuffle (bool, optional): Whether to shuffle the dataset across exhaustions. Defaults to True.
|
shuffle (bool, optional): Whether to shuffle the dataset across exhaustions. Defaults to True.
|
||||||
depth_output_unit (str, optional): Physical unit depth maps are dequantized to ("m" or "mm").
|
depth_output_unit (str, optional): Physical unit depth maps are dequantized to ("m" or "mm").
|
||||||
Defaults to "mm".
|
Defaults to "mm".
|
||||||
|
token: Authentication token used while streaming this dataset from
|
||||||
|
the Hub. Pass a string token, ``True`` to require the locally
|
||||||
|
stored token, ``False`` to disable authentication, or ``None``
|
||||||
|
to use the Hugging Face Hub default. The token is not retained
|
||||||
|
on the dataset instance after initialization.
|
||||||
"""
|
"""
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self.repo_id = repo_id
|
self.repo_id = repo_id
|
||||||
@@ -306,7 +313,11 @@ class StreamingLeRobotDataset(torch.utils.data.IterableDataset):
|
|||||||
|
|
||||||
# Load metadata
|
# Load metadata
|
||||||
self.meta = LeRobotDatasetMetadata(
|
self.meta = LeRobotDatasetMetadata(
|
||||||
self.repo_id, self._requested_root, self.revision, force_cache_sync=force_cache_sync
|
self.repo_id,
|
||||||
|
self._requested_root,
|
||||||
|
self.revision,
|
||||||
|
force_cache_sync=force_cache_sync,
|
||||||
|
token=token,
|
||||||
)
|
)
|
||||||
self.root = self.meta.root
|
self.root = self.meta.root
|
||||||
self.revision = self.meta.revision
|
self.revision = self.meta.revision
|
||||||
@@ -334,12 +345,14 @@ class StreamingLeRobotDataset(torch.utils.data.IterableDataset):
|
|||||||
self.delta_timestamps = delta_timestamps
|
self.delta_timestamps = delta_timestamps
|
||||||
self.delta_indices = get_delta_indices(self.delta_timestamps, self.fps)
|
self.delta_indices = get_delta_indices(self.delta_timestamps, self.fps)
|
||||||
|
|
||||||
|
token_kwargs = {} if token is None or self.streaming_from_local else {"token": token}
|
||||||
self.hf_dataset: datasets.IterableDataset = load_dataset(
|
self.hf_dataset: datasets.IterableDataset = load_dataset(
|
||||||
self.repo_id if not self.streaming_from_local else str(self.root),
|
self.repo_id if not self.streaming_from_local else str(self.root),
|
||||||
split="train",
|
split="train",
|
||||||
streaming=self.streaming,
|
streaming=self.streaming,
|
||||||
data_files="data/*/*.parquet",
|
data_files="data/*/*.parquet",
|
||||||
revision=self.revision,
|
revision=self.revision,
|
||||||
|
**token_kwargs,
|
||||||
)
|
)
|
||||||
|
|
||||||
self.num_shards = min(self.hf_dataset.num_shards, max_num_shards)
|
self.num_shards = min(self.hf_dataset.num_shards, max_num_shards)
|
||||||
|
|||||||
@@ -325,16 +325,19 @@ def check_version_compatibility(
|
|||||||
logging.warning(FUTURE_MESSAGE.format(repo_id=repo_id, version=v_check))
|
logging.warning(FUTURE_MESSAGE.format(repo_id=repo_id, version=v_check))
|
||||||
|
|
||||||
|
|
||||||
def get_repo_versions(repo_id: str) -> list[packaging.version.Version]:
|
def get_repo_versions(repo_id: str, *, token: str | bool | None = None) -> list[packaging.version.Version]:
|
||||||
"""Return available valid versions (branches and tags) on a given Hub repo.
|
"""Return available valid versions (branches and tags) on a given Hub repo.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
repo_id (str): The repository ID on the Hugging Face Hub.
|
repo_id (str): The repository ID on the Hugging Face Hub.
|
||||||
|
token: Authentication token used for Hub requests. Pass a string token,
|
||||||
|
``True`` to require the locally stored token, ``False`` to disable
|
||||||
|
authentication, or ``None`` to use the Hugging Face Hub default.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
list[packaging.version.Version]: A list of valid versions found.
|
list[packaging.version.Version]: A list of valid versions found.
|
||||||
"""
|
"""
|
||||||
api = HfApi()
|
api = HfApi() if token is None else HfApi(token=token)
|
||||||
repo_refs = api.list_repo_refs(repo_id, repo_type="dataset")
|
repo_refs = api.list_repo_refs(repo_id, repo_type="dataset")
|
||||||
repo_refs = [b.name for b in repo_refs.branches + repo_refs.tags]
|
repo_refs = [b.name for b in repo_refs.branches + repo_refs.tags]
|
||||||
repo_versions = []
|
repo_versions = []
|
||||||
@@ -345,7 +348,12 @@ def get_repo_versions(repo_id: str) -> list[packaging.version.Version]:
|
|||||||
return repo_versions
|
return repo_versions
|
||||||
|
|
||||||
|
|
||||||
def get_safe_version(repo_id: str, version: str | packaging.version.Version) -> str:
|
def get_safe_version(
|
||||||
|
repo_id: str,
|
||||||
|
version: str | packaging.version.Version,
|
||||||
|
*,
|
||||||
|
token: str | bool | None = None,
|
||||||
|
) -> str:
|
||||||
"""Return the specified version if available on repo, or the latest compatible one.
|
"""Return the specified version if available on repo, or the latest compatible one.
|
||||||
|
|
||||||
If the exact version is not found, it looks for the latest version with the
|
If the exact version is not found, it looks for the latest version with the
|
||||||
@@ -354,6 +362,7 @@ def get_safe_version(repo_id: str, version: str | packaging.version.Version) ->
|
|||||||
Args:
|
Args:
|
||||||
repo_id (str): The repository ID on the Hugging Face Hub.
|
repo_id (str): The repository ID on the Hugging Face Hub.
|
||||||
version (str | packaging.version.Version): The target version.
|
version (str | packaging.version.Version): The target version.
|
||||||
|
token: Authentication token forwarded to the Hub version lookup.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
str: The safe version string (e.g., "v1.2.3") to use as a revision.
|
str: The safe version string (e.g., "v1.2.3") to use as a revision.
|
||||||
@@ -366,7 +375,7 @@ def get_safe_version(repo_id: str, version: str | packaging.version.Version) ->
|
|||||||
target_version = (
|
target_version = (
|
||||||
packaging.version.parse(version) if not isinstance(version, packaging.version.Version) else version
|
packaging.version.parse(version) if not isinstance(version, packaging.version.Version) else version
|
||||||
)
|
)
|
||||||
hub_versions = get_repo_versions(repo_id)
|
hub_versions = get_repo_versions(repo_id) if token is None else get_repo_versions(repo_id, token=token)
|
||||||
|
|
||||||
if not hub_versions:
|
if not hub_versions:
|
||||||
raise RevisionNotFoundError(
|
raise RevisionNotFoundError(
|
||||||
|
|||||||
@@ -322,7 +322,7 @@ class HILSerlRobotEnvConfig(EnvConfig):
|
|||||||
class LiberoEnv(EnvConfig):
|
class LiberoEnv(EnvConfig):
|
||||||
task: str = "libero_10" # can also choose libero_spatial, libero_object, etc.
|
task: str = "libero_10" # can also choose libero_spatial, libero_object, etc.
|
||||||
task_ids: list[int] | None = None
|
task_ids: list[int] | None = None
|
||||||
fps: int = 30
|
fps: int = 20 # Must match robosuite's default control_freq (20 Hz)
|
||||||
episode_length: int | None = None
|
episode_length: int | None = None
|
||||||
obs_type: str = "pixels_agent_pos"
|
obs_type: str = "pixels_agent_pos"
|
||||||
render_mode: str = "rgb_array"
|
render_mode: str = "rgb_array"
|
||||||
@@ -354,6 +354,9 @@ class LiberoEnv(EnvConfig):
|
|||||||
control_mode: str = "relative" # or "absolute"
|
control_mode: str = "relative" # or "absolute"
|
||||||
|
|
||||||
def __post_init__(self):
|
def __post_init__(self):
|
||||||
|
if self.fps <= 0:
|
||||||
|
raise ValueError(f"fps must be positive, got {self.fps}")
|
||||||
|
|
||||||
if self.obs_type == "pixels":
|
if self.obs_type == "pixels":
|
||||||
self.features[LIBERO_KEY_PIXELS_AGENTVIEW] = PolicyFeature(
|
self.features[LIBERO_KEY_PIXELS_AGENTVIEW] = PolicyFeature(
|
||||||
type=FeatureType.VISUAL, shape=(self.observation_height, self.observation_width, 3)
|
type=FeatureType.VISUAL, shape=(self.observation_height, self.observation_width, 3)
|
||||||
@@ -412,6 +415,7 @@ class LiberoEnv(EnvConfig):
|
|||||||
"render_mode": self.render_mode,
|
"render_mode": self.render_mode,
|
||||||
"observation_height": self.observation_height,
|
"observation_height": self.observation_height,
|
||||||
"observation_width": self.observation_width,
|
"observation_width": self.observation_width,
|
||||||
|
"control_freq": self.fps,
|
||||||
}
|
}
|
||||||
if self.task_ids is not None:
|
if self.task_ids is not None:
|
||||||
kwargs["task_ids"] = self.task_ids
|
kwargs["task_ids"] = self.task_ids
|
||||||
|
|||||||
@@ -125,10 +125,13 @@ class LiberoEnv(gym.Env):
|
|||||||
n_envs: int = 1,
|
n_envs: int = 1,
|
||||||
camera_name_mapping: dict[str, str] | None = None,
|
camera_name_mapping: dict[str, str] | None = None,
|
||||||
num_steps_wait: int = 10,
|
num_steps_wait: int = 10,
|
||||||
|
control_freq: int = 20,
|
||||||
control_mode: str = "relative",
|
control_mode: str = "relative",
|
||||||
is_libero_plus: bool = False,
|
is_libero_plus: bool = False,
|
||||||
):
|
):
|
||||||
super().__init__()
|
super().__init__()
|
||||||
|
if control_freq <= 0:
|
||||||
|
raise ValueError(f"control_freq must be positive, got {control_freq}")
|
||||||
self.task_id = task_id
|
self.task_id = task_id
|
||||||
self.is_libero_plus = is_libero_plus
|
self.is_libero_plus = is_libero_plus
|
||||||
self.obs_type = obs_type
|
self.obs_type = obs_type
|
||||||
@@ -154,6 +157,7 @@ class LiberoEnv(gym.Env):
|
|||||||
}
|
}
|
||||||
self.camera_name_mapping = camera_name_mapping
|
self.camera_name_mapping = camera_name_mapping
|
||||||
self.num_steps_wait = num_steps_wait
|
self.num_steps_wait = num_steps_wait
|
||||||
|
self.control_freq = control_freq
|
||||||
self.episode_index = episode_index
|
self.episode_index = episode_index
|
||||||
self.episode_length = episode_length
|
self.episode_length = episode_length
|
||||||
# Load once and keep
|
# Load once and keep
|
||||||
@@ -260,6 +264,7 @@ class LiberoEnv(gym.Env):
|
|||||||
bddl_file_name=self._task_bddl_file,
|
bddl_file_name=self._task_bddl_file,
|
||||||
camera_heights=self.observation_height,
|
camera_heights=self.observation_height,
|
||||||
camera_widths=self.observation_width,
|
camera_widths=self.observation_width,
|
||||||
|
control_freq=self.control_freq,
|
||||||
)
|
)
|
||||||
env.reset()
|
env.reset()
|
||||||
self._env = env
|
self._env = env
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ from lerobot.utils.import_utils import require_package
|
|||||||
# guard the optional dependency here so importing this package fails loudly if it's missing.
|
# guard the optional dependency here so importing this package fails loudly if it's missing.
|
||||||
require_package("datasets", extra="dataset")
|
require_package("datasets", extra="dataset")
|
||||||
|
|
||||||
|
from .annotate import submit_annotate_to_hf
|
||||||
from .hf import submit_to_hf
|
from .hf import submit_to_hf
|
||||||
|
|
||||||
__all__ = ["submit_to_hf"]
|
__all__ = ["submit_annotate_to_hf", "submit_to_hf"]
|
||||||
|
|||||||
@@ -0,0 +1,176 @@
|
|||||||
|
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
"""Run ``lerobot-annotate`` on HF Jobs (HuggingFace GPUs).
|
||||||
|
|
||||||
|
Same shape as the training submitter in ``hf.py``, with one difference: the
|
||||||
|
annotation pipeline serves its own VLM, so the pod starts from the official
|
||||||
|
``vllm/vllm-openai`` image (which has no lerobot) instead of the prebuilt
|
||||||
|
``lerobot-gpu`` image, and installs lerobot on top before running.
|
||||||
|
|
||||||
|
Because there is no config repo to stage, the pod replays the user's own CLI
|
||||||
|
flags — everything except the client-only ``--job.*`` and the host-local
|
||||||
|
``--root``, which is replaced by ``--repo_id`` so the pod pulls the dataset
|
||||||
|
from the Hub.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import shlex
|
||||||
|
import sys
|
||||||
|
from dataclasses import is_dataclass
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from huggingface_hub import HfApi, get_token, run_job
|
||||||
|
|
||||||
|
from .dataset import ensure_dataset_available
|
||||||
|
|
||||||
|
# Package-internal reuse of the training submitter's job plumbing: following a
|
||||||
|
# submitted job and forwarding argv are identical for annotation runs.
|
||||||
|
from .hf import _pod_forwarded_args, follow_job, resolve_job_tags
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from lerobot.annotations.steerable_pipeline.config import AnnotationPipelineConfig
|
||||||
|
|
||||||
|
LEROBOT_GIT_URL = "https://github.com/huggingface/lerobot.git"
|
||||||
|
|
||||||
|
# Mirrors the pins in pyproject.toml. The vLLM image resolves dependencies on its
|
||||||
|
# own otherwise, and pulls av 18 / datasets 5 / draccus 0.11 — each of which breaks
|
||||||
|
# lerobot at import time. `--upgrade-strategy only-if-needed` keeps vLLM's own
|
||||||
|
# (torch, transformers, ...) pins intact.
|
||||||
|
_RUNTIME_REQUIREMENTS = (
|
||||||
|
"'datasets>=4.7.0,<5.0.0' 'pyarrow>=21.0.0,<30.0.0' 'av>=15.0.0,<16.0.0' 'draccus==0.10.0' "
|
||||||
|
"'pandas>=2.0.0,<3.0.0' jsonlines gymnasium torchcodec mergedeep pyyaml-include toml typing-inspect "
|
||||||
|
"openai"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Flags the submitter resolves itself instead of forwarding verbatim: `--root`
|
||||||
|
# names a directory only this machine has, `--repo_id` is re-emitted from the
|
||||||
|
# config, and the config-file args name local files (rejected up front by
|
||||||
|
# `submit_annotate_to_hf`). `--job.*` is dropped separately, by prefix; bare
|
||||||
|
# `--job` is not, hence its entry here — it is the one arg that could smuggle a
|
||||||
|
# remote `target` onto the pod and have the job recursively submit itself.
|
||||||
|
_SUBMITTER_OWNED_ARGS = ("--root", "--repo_id", "--config_path", "--job")
|
||||||
|
|
||||||
|
|
||||||
|
def _local_config_file_args(cfg: AnnotationPipelineConfig) -> list[str]:
|
||||||
|
"""The CLI args that name a config file on the client's disk.
|
||||||
|
|
||||||
|
draccus exposes ``--config_path`` for the whole config plus a ``--<field>``
|
||||||
|
for every nested dataclass (``--vlm``, ``--plan``, ``--job``, ...). The pod has
|
||||||
|
none of those files, so a remote run has to reject them rather than silently
|
||||||
|
drop the settings they carry.
|
||||||
|
"""
|
||||||
|
return ["--config_path", *(f"--{name}" for name in vars(cfg) if is_dataclass(getattr(cfg, name)))]
|
||||||
|
|
||||||
|
|
||||||
|
def build_pod_setup(lerobot_ref: str) -> str:
|
||||||
|
"""Shell prelude that turns the vLLM image into a ``lerobot-annotate`` runtime."""
|
||||||
|
spec = f"lerobot @ git+{LEROBOT_GIT_URL}@{lerobot_ref}"
|
||||||
|
return (
|
||||||
|
# git to install from the repo, ffmpeg to decode the dataset's videos.
|
||||||
|
"apt-get update -qq && apt-get install -y -qq git ffmpeg && "
|
||||||
|
f"pip install --no-deps {shlex.quote(spec)} && "
|
||||||
|
f"pip install --upgrade-strategy only-if-needed {_RUNTIME_REQUIREMENTS} && "
|
||||||
|
# vLLM's cudagraph memory estimate over-reserves and starves the KV cache;
|
||||||
|
# PyAV is the video backend the server can decode our frames with.
|
||||||
|
"export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 && "
|
||||||
|
"export VLLM_VIDEO_BACKEND=pyav"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def build_pod_command(repo_id: str, lerobot_ref: str, argv: list[str]) -> list[str]:
|
||||||
|
"""Build the ``bash -c`` command the pod runs: setup prelude, then annotation.
|
||||||
|
|
||||||
|
``argv`` is the user's CLI (``sys.argv[1:]``) minus the flags in
|
||||||
|
``_SUBMITTER_OWNED_ARGS``; ``--repo_id`` is re-added from the config so the pod
|
||||||
|
always annotates the dataset we just made sure is reachable on the Hub.
|
||||||
|
``--job.target=local`` stops the pod from re-dispatching to itself.
|
||||||
|
"""
|
||||||
|
forwarded = _pod_forwarded_args(argv, drop_names=_SUBMITTER_OWNED_ARGS, drop_prefixes=("--job.",))
|
||||||
|
annotate = shlex.join(["lerobot-annotate", f"--repo_id={repo_id}", *forwarded, "--job.target=local"])
|
||||||
|
return ["bash", "-c", f"{build_pod_setup(lerobot_ref)} && {annotate}"]
|
||||||
|
|
||||||
|
|
||||||
|
def submit_annotate_to_hf(cfg: AnnotationPipelineConfig) -> None:
|
||||||
|
"""Submit an annotation run to HF Jobs infrastructure.
|
||||||
|
|
||||||
|
Resolves credentials, makes sure the source dataset is reachable from the pod,
|
||||||
|
submits the job, then tails its logs until the job reaches a terminal stage —
|
||||||
|
or returns immediately with ``--job.detach``. Ctrl-C detaches without
|
||||||
|
cancelling the remote job.
|
||||||
|
"""
|
||||||
|
token = get_token()
|
||||||
|
if not token:
|
||||||
|
raise RuntimeError("Not logged in to Hugging Face. Run `hf auth login` first.")
|
||||||
|
|
||||||
|
if cfg.repo_id is None:
|
||||||
|
raise ValueError(
|
||||||
|
"Remote annotation requires --repo_id: the pod downloads the dataset from the Hub, "
|
||||||
|
"and --root only names a directory on this machine."
|
||||||
|
)
|
||||||
|
|
||||||
|
argv = sys.argv[1:]
|
||||||
|
passed = {tok.split("=", 1)[0] for tok in argv}
|
||||||
|
used_config_files = sorted(passed.intersection(_local_config_file_args(cfg)))
|
||||||
|
if used_config_files:
|
||||||
|
raise ValueError(
|
||||||
|
f"{', '.join(used_config_files)} cannot be used with a remote --job.target: the pod "
|
||||||
|
"cannot read config files from this machine. Pass the settings as CLI flags instead."
|
||||||
|
)
|
||||||
|
|
||||||
|
if not cfg.push_to_hub:
|
||||||
|
# The pod's filesystem is discarded when the job ends, so without a push the
|
||||||
|
# run produces nothing. Warn rather than fail: a smoke test over
|
||||||
|
# --only_episodes that only inspects the logs is a legitimate use.
|
||||||
|
print(
|
||||||
|
"WARNING: --push_to_hub is off. The annotated dataset lives only on the pod and is "
|
||||||
|
"discarded when the job ends. Pass --push_to_hub=true to keep the result."
|
||||||
|
)
|
||||||
|
|
||||||
|
api = HfApi(token=token)
|
||||||
|
tags = resolve_job_tags(cfg.job.tags)
|
||||||
|
ensure_dataset_available(cfg.repo_id, api=api, tags=tags)
|
||||||
|
|
||||||
|
command = build_pod_command(cfg.repo_id, cfg.job.lerobot_ref, argv)
|
||||||
|
|
||||||
|
print(f"Submitting job to HF Jobs (flavor={cfg.job.target}, image={cfg.job.image}) ...")
|
||||||
|
job_info = run_job(
|
||||||
|
image=cfg.job.image,
|
||||||
|
command=command,
|
||||||
|
flavor=cfg.job.target,
|
||||||
|
secrets={"HF_TOKEN": token},
|
||||||
|
timeout=cfg.job.timeout,
|
||||||
|
# HF Jobs labels are key/value; expose each tag as a queryable label.
|
||||||
|
labels=dict.fromkeys(tags, "true"),
|
||||||
|
)
|
||||||
|
job_id = job_info.id
|
||||||
|
job_url = getattr(job_info, "url", None)
|
||||||
|
print(f"Job submitted: {job_id}")
|
||||||
|
if job_url:
|
||||||
|
print(f" Job page: {job_url}")
|
||||||
|
target_repo_id = cfg.new_repo_id or cfg.repo_id
|
||||||
|
if cfg.push_to_hub:
|
||||||
|
print(f" Dataset repo: https://huggingface.co/datasets/{target_repo_id}")
|
||||||
|
print(f" Monitor: hf jobs logs {job_id}")
|
||||||
|
print(f" Cancel: hf jobs cancel {job_id}")
|
||||||
|
|
||||||
|
# No success marker: `lerobot-annotate` keeps working after the upload log line
|
||||||
|
# (dataset card, version tag), so completion has to be stage-based.
|
||||||
|
if not follow_job(job_id, detach=cfg.job.detach):
|
||||||
|
return
|
||||||
|
|
||||||
|
if cfg.push_to_hub:
|
||||||
|
print(f"\nAnnotation complete — dataset pushed to https://huggingface.co/datasets/{target_repo_id}")
|
||||||
|
else:
|
||||||
|
print("\nAnnotation complete. Note: --push_to_hub was off, so the result stayed on the pod.")
|
||||||
+69
-54
@@ -223,6 +223,74 @@ def _poll_until_done(
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def follow_job(job_id: str, *, detach: bool = False, success_marker: str | None = None) -> bool:
|
||||||
|
"""Watch a submitted job to the end, streaming its logs to stdout.
|
||||||
|
|
||||||
|
Returns True when the job finished successfully and False when we stopped watching
|
||||||
|
without a verdict — `detach`, or the user pressing Ctrl-C, which detaches rather than
|
||||||
|
cancelling the remote job. Raises RuntimeError when the job reaches a terminal stage
|
||||||
|
other than COMPLETED.
|
||||||
|
|
||||||
|
`success_marker` finishes as soon as that string appears in the logs instead of waiting
|
||||||
|
out the platform's post-run finalization (~30s). Callers that have a log line meaning
|
||||||
|
"the artifact is on the Hub" should pass it; without one, completion is stage-based.
|
||||||
|
"""
|
||||||
|
if detach:
|
||||||
|
return False
|
||||||
|
|
||||||
|
done = threading.Event()
|
||||||
|
detached = threading.Event()
|
||||||
|
marker_seen = threading.Event()
|
||||||
|
stage_holder: dict[str, str | None] = {}
|
||||||
|
|
||||||
|
def _poll() -> None:
|
||||||
|
stage_holder["stage"] = _poll_until_done(job_id, done, status_holder=stage_holder)
|
||||||
|
|
||||||
|
poll_thread = threading.Thread(target=_poll, daemon=True)
|
||||||
|
poll_thread.start()
|
||||||
|
log_thread = threading.Thread(
|
||||||
|
target=_tail_logs, args=(job_id, done, success_marker, marker_seen), daemon=True
|
||||||
|
)
|
||||||
|
log_thread.start()
|
||||||
|
|
||||||
|
def _detach(sig, frame):
|
||||||
|
detached.set()
|
||||||
|
done.set()
|
||||||
|
print("\nDetached. Job is still running.")
|
||||||
|
print(f" Monitor: hf jobs logs {job_id}")
|
||||||
|
print(f" Cancel: hf jobs cancel {job_id}")
|
||||||
|
|
||||||
|
# signal.signal only works on the main thread; when called from a worker thread
|
||||||
|
# (e.g. an orchestration framework) skip the Ctrl-C-detaches-instead-of-cancels
|
||||||
|
# handler rather than crashing with ValueError.
|
||||||
|
install_sigint = threading.current_thread() is threading.main_thread()
|
||||||
|
original_sigint = signal.getsignal(signal.SIGINT) if install_sigint else None
|
||||||
|
if install_sigint:
|
||||||
|
signal.signal(signal.SIGINT, _detach)
|
||||||
|
try:
|
||||||
|
# Timeout-based join so SIGINT is delivered to the main thread promptly.
|
||||||
|
while poll_thread.is_alive():
|
||||||
|
poll_thread.join(timeout=0.5)
|
||||||
|
log_thread.join(timeout=5)
|
||||||
|
finally:
|
||||||
|
if install_sigint:
|
||||||
|
signal.signal(signal.SIGINT, original_sigint)
|
||||||
|
|
||||||
|
if detached.is_set():
|
||||||
|
return False
|
||||||
|
if marker_seen.is_set():
|
||||||
|
return True
|
||||||
|
|
||||||
|
stage = stage_holder.get("stage")
|
||||||
|
if stage != "COMPLETED":
|
||||||
|
message = stage_holder.get("message")
|
||||||
|
detail = f" ({message})" if message else ""
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Job {job_id} ended with stage={stage}{detail}. Check logs: hf jobs logs {job_id}"
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
def _pod_forwarded_args(
|
def _pod_forwarded_args(
|
||||||
argv: list[str], drop_names: tuple[str, ...] = (), drop_prefixes: tuple[str, ...] = ()
|
argv: list[str], drop_names: tuple[str, ...] = (), drop_prefixes: tuple[str, ...] = ()
|
||||||
) -> list[str]:
|
) -> list[str]:
|
||||||
@@ -362,64 +430,11 @@ def submit_to_hf(cfg: TrainPipelineConfig) -> None:
|
|||||||
print(f" Monitor: hf jobs logs {job_id}")
|
print(f" Monitor: hf jobs logs {job_id}")
|
||||||
print(f" Cancel: hf jobs cancel {job_id}")
|
print(f" Cancel: hf jobs cancel {job_id}")
|
||||||
|
|
||||||
if cfg.job.detach:
|
|
||||||
return
|
|
||||||
|
|
||||||
done = threading.Event()
|
|
||||||
detached = threading.Event()
|
|
||||||
pushed_ok = threading.Event()
|
|
||||||
stage_holder: dict[str, str | None] = {}
|
|
||||||
|
|
||||||
def _poll() -> None:
|
|
||||||
stage_holder["stage"] = _poll_until_done(job_id, done, status_holder=stage_holder)
|
|
||||||
|
|
||||||
poll_thread = threading.Thread(target=_poll, daemon=True)
|
|
||||||
poll_thread.start()
|
|
||||||
# Finish as soon as the model is pushed, rather than waiting out the platform's
|
# Finish as soon as the model is pushed, rather than waiting out the platform's
|
||||||
# post-run finalization before the job stage flips to COMPLETED. This matches the
|
# post-run finalization before the job stage flips to COMPLETED. This matches the
|
||||||
# exact log line emitted by PreTrainedPolicy.push_model_to_hub — the two must stay
|
# exact log line emitted by PreTrainedPolicy.push_model_to_hub — the two must stay
|
||||||
# in sync. If it ever stops matching we just fall back to stage-based completion
|
# in sync. If it ever stops matching we just fall back to stage-based completion
|
||||||
# (~30s slower), so the contract is an optimization, not a correctness requirement.
|
# (~30s slower), so the contract is an optimization, not a correctness requirement.
|
||||||
success_marker = f"Model pushed to https://huggingface.co/{repo_id}"
|
success_marker = f"Model pushed to https://huggingface.co/{repo_id}"
|
||||||
log_thread = threading.Thread(
|
if follow_job(job_id, detach=cfg.job.detach, success_marker=success_marker):
|
||||||
target=_tail_logs, args=(job_id, done, success_marker, pushed_ok), daemon=True
|
|
||||||
)
|
|
||||||
log_thread.start()
|
|
||||||
|
|
||||||
def _detach(sig, frame):
|
|
||||||
detached.set()
|
|
||||||
done.set()
|
|
||||||
print("\nDetached. Job is still running.")
|
|
||||||
print(f" Monitor: hf jobs logs {job_id}")
|
|
||||||
print(f" Cancel: hf jobs cancel {job_id}")
|
|
||||||
|
|
||||||
# signal.signal only works on the main thread; when called from a worker thread
|
|
||||||
# (e.g. an orchestration framework) skip the Ctrl-C-detaches-instead-of-cancels
|
|
||||||
# handler rather than crashing with ValueError.
|
|
||||||
install_sigint = threading.current_thread() is threading.main_thread()
|
|
||||||
original_sigint = signal.getsignal(signal.SIGINT) if install_sigint else None
|
|
||||||
if install_sigint:
|
|
||||||
signal.signal(signal.SIGINT, _detach)
|
|
||||||
try:
|
|
||||||
# Timeout-based join so SIGINT is delivered to the main thread promptly.
|
|
||||||
while poll_thread.is_alive():
|
|
||||||
poll_thread.join(timeout=0.5)
|
|
||||||
log_thread.join(timeout=5)
|
|
||||||
finally:
|
|
||||||
if install_sigint:
|
|
||||||
signal.signal(signal.SIGINT, original_sigint)
|
|
||||||
|
|
||||||
if detached.is_set():
|
|
||||||
return
|
|
||||||
|
|
||||||
if pushed_ok.is_set():
|
|
||||||
print(f"\nTraining complete — model pushed to https://huggingface.co/{repo_id}")
|
print(f"\nTraining complete — model pushed to https://huggingface.co/{repo_id}")
|
||||||
return
|
|
||||||
|
|
||||||
stage = stage_holder.get("stage")
|
|
||||||
if stage != "COMPLETED":
|
|
||||||
message = stage_holder.get("message")
|
|
||||||
detail = f" ({message})" if message else ""
|
|
||||||
raise RuntimeError(
|
|
||||||
f"Job {job_id} ended with stage={stage}{detail}. Check logs: hf jobs logs {job_id}"
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -20,7 +20,6 @@ import logging
|
|||||||
import time
|
import time
|
||||||
from contextlib import contextmanager
|
from contextlib import contextmanager
|
||||||
from copy import deepcopy
|
from copy import deepcopy
|
||||||
from functools import cached_property
|
|
||||||
from typing import TYPE_CHECKING, Any, TypedDict
|
from typing import TYPE_CHECKING, Any, TypedDict
|
||||||
|
|
||||||
from lerobot.utils.decorators import check_if_already_connected, check_if_not_connected
|
from lerobot.utils.decorators import check_if_already_connected, check_if_not_connected
|
||||||
@@ -854,7 +853,7 @@ class DamiaoMotorsBus(MotorsBusBase):
|
|||||||
else:
|
else:
|
||||||
raise ValueError(f"Motor {motor_obj} doesn't have a valid recv_id (None).")
|
raise ValueError(f"Motor {motor_obj} doesn't have a valid recv_id (None).")
|
||||||
|
|
||||||
@cached_property
|
@property
|
||||||
def is_calibrated(self) -> bool:
|
def is_calibrated(self) -> bool:
|
||||||
"""Check if motors are calibrated."""
|
"""Check if motors are calibrated."""
|
||||||
return bool(self.calibration)
|
return bool(self.calibration)
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import abc
|
import abc
|
||||||
import logging
|
import logging
|
||||||
|
import time
|
||||||
from collections.abc import Sequence
|
from collections.abc import Sequence
|
||||||
from contextlib import contextmanager
|
from contextlib import contextmanager
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
@@ -818,13 +819,13 @@ class SerialMotorsBus(MotorsBusBase):
|
|||||||
"""
|
"""
|
||||||
motor_names = self._get_motors_list(motors)
|
motor_names = self._get_motors_list(motors)
|
||||||
|
|
||||||
start_positions = self.sync_read("Present_Position", motor_names, normalize=False)
|
start_positions = self.sync_read("Present_Position", motor_names, normalize=False, num_retry=5)
|
||||||
mins = start_positions.copy()
|
mins = start_positions.copy()
|
||||||
maxes = start_positions.copy()
|
maxes = start_positions.copy()
|
||||||
|
|
||||||
user_pressed_enter = False
|
user_pressed_enter = False
|
||||||
while not user_pressed_enter:
|
while not user_pressed_enter:
|
||||||
positions = self.sync_read("Present_Position", motor_names, normalize=False)
|
positions = self.sync_read("Present_Position", motor_names, normalize=False, num_retry=5)
|
||||||
mins = {motor: min(positions[motor], min_) for motor, min_ in mins.items()}
|
mins = {motor: min(positions[motor], min_) for motor, min_ in mins.items()}
|
||||||
maxes = {motor: max(positions[motor], max_) for motor, max_ in maxes.items()}
|
maxes = {motor: max(positions[motor], max_) for motor, max_ in maxes.items()}
|
||||||
|
|
||||||
@@ -837,9 +838,12 @@ class SerialMotorsBus(MotorsBusBase):
|
|||||||
if enter_pressed():
|
if enter_pressed():
|
||||||
user_pressed_enter = True
|
user_pressed_enter = True
|
||||||
|
|
||||||
if display_values and not user_pressed_enter:
|
if not user_pressed_enter:
|
||||||
|
if display_values:
|
||||||
# Move cursor up to overwrite the previous output
|
# Move cursor up to overwrite the previous output
|
||||||
move_cursor_up(len(motor_names) + 3)
|
move_cursor_up(len(motor_names) + 3)
|
||||||
|
# Throttle reads even when the live table is disabled.
|
||||||
|
time.sleep(0.02)
|
||||||
|
|
||||||
same_min_max = [motor for motor in motor_names if mins[motor] == maxes[motor]]
|
same_min_max = [motor for motor in motor_names if mins[motor] == maxes[motor]]
|
||||||
if same_min_max:
|
if same_min_max:
|
||||||
|
|||||||
@@ -79,6 +79,8 @@ class DiffusionConfig(PreTrainedConfig):
|
|||||||
use_film_scale_modulation: FiLM (https://huggingface.co/papers/1709.07871) is used for the Unet conditioning.
|
use_film_scale_modulation: FiLM (https://huggingface.co/papers/1709.07871) is used for the Unet conditioning.
|
||||||
Bias modulation is used be default, while this parameter indicates whether to also use scale
|
Bias modulation is used be default, while this parameter indicates whether to also use scale
|
||||||
modulation.
|
modulation.
|
||||||
|
gradient_checkpointing: Whether to checkpoint the Unet residual blocks during training. This reduces
|
||||||
|
activation memory at the cost of recomputing those blocks during the backward pass.
|
||||||
noise_scheduler_type: Name of the noise scheduler to use. Supported options: ["DDPM", "DDIM"].
|
noise_scheduler_type: Name of the noise scheduler to use. Supported options: ["DDPM", "DDIM"].
|
||||||
num_train_timesteps: Number of diffusion steps for the forward diffusion schedule.
|
num_train_timesteps: Number of diffusion steps for the forward diffusion schedule.
|
||||||
beta_schedule: Name of the diffusion beta schedule as per DDPMScheduler from Hugging Face diffusers.
|
beta_schedule: Name of the diffusion beta schedule as per DDPMScheduler from Hugging Face diffusers.
|
||||||
@@ -132,6 +134,7 @@ class DiffusionConfig(PreTrainedConfig):
|
|||||||
n_groups: int = 8
|
n_groups: int = 8
|
||||||
diffusion_step_embed_dim: int = 128
|
diffusion_step_embed_dim: int = 128
|
||||||
use_film_scale_modulation: bool = True
|
use_film_scale_modulation: bool = True
|
||||||
|
gradient_checkpointing: bool = False
|
||||||
# Noise scheduler.
|
# Noise scheduler.
|
||||||
noise_scheduler_type: str = "DDPM"
|
noise_scheduler_type: str = "DDPM"
|
||||||
num_train_timesteps: int = 100
|
num_train_timesteps: int = 100
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ import torch
|
|||||||
import torch.nn.functional as F # noqa: N812
|
import torch.nn.functional as F # noqa: N812
|
||||||
import torchvision
|
import torchvision
|
||||||
from torch import Tensor, nn
|
from torch import Tensor, nn
|
||||||
|
from torch.utils.checkpoint import checkpoint
|
||||||
|
|
||||||
from lerobot.utils.constants import ACTION, OBS_ENV_STATE, OBS_IMAGES, OBS_STATE
|
from lerobot.utils.constants import ACTION, OBS_ENV_STATE, OBS_IMAGES, OBS_STATE
|
||||||
from lerobot.utils.import_utils import _diffusers_available, require_package
|
from lerobot.utils.import_utils import _diffusers_available, require_package
|
||||||
@@ -727,20 +728,33 @@ class DiffusionConditionalUnet1d(nn.Module):
|
|||||||
else:
|
else:
|
||||||
global_feature = timesteps_embed
|
global_feature = timesteps_embed
|
||||||
|
|
||||||
|
use_gc = self.config.gradient_checkpointing and self.training
|
||||||
|
|
||||||
# Run encoder, keeping track of skip features to pass to the decoder.
|
# Run encoder, keeping track of skip features to pass to the decoder.
|
||||||
encoder_skip_features: list[Tensor] = []
|
encoder_skip_features: list[Tensor] = []
|
||||||
for resnet, resnet2, downsample in self.down_modules:
|
for resnet, resnet2, downsample in self.down_modules:
|
||||||
|
if use_gc:
|
||||||
|
x = checkpoint(resnet, x, global_feature, use_reentrant=False)
|
||||||
|
x = checkpoint(resnet2, x, global_feature, use_reentrant=False)
|
||||||
|
else:
|
||||||
x = resnet(x, global_feature)
|
x = resnet(x, global_feature)
|
||||||
x = resnet2(x, global_feature)
|
x = resnet2(x, global_feature)
|
||||||
encoder_skip_features.append(x)
|
encoder_skip_features.append(x)
|
||||||
x = downsample(x)
|
x = downsample(x)
|
||||||
|
|
||||||
for mid_module in self.mid_modules:
|
for mid_module in self.mid_modules:
|
||||||
|
if use_gc:
|
||||||
|
x = checkpoint(mid_module, x, global_feature, use_reentrant=False)
|
||||||
|
else:
|
||||||
x = mid_module(x, global_feature)
|
x = mid_module(x, global_feature)
|
||||||
|
|
||||||
# Run decoder, using the skip features from the encoder.
|
# Run decoder, using the skip features from the encoder.
|
||||||
for resnet, resnet2, upsample in self.up_modules:
|
for resnet, resnet2, upsample in self.up_modules:
|
||||||
x = torch.cat((x, encoder_skip_features.pop()), dim=1)
|
x = torch.cat((x, encoder_skip_features.pop()), dim=1)
|
||||||
|
if use_gc:
|
||||||
|
x = checkpoint(resnet, x, global_feature, use_reentrant=False)
|
||||||
|
x = checkpoint(resnet2, x, global_feature, use_reentrant=False)
|
||||||
|
else:
|
||||||
x = resnet(x, global_feature)
|
x = resnet(x, global_feature)
|
||||||
x = resnet2(x, global_feature)
|
x = resnet2(x, global_feature)
|
||||||
x = upsample(x)
|
x = upsample(x)
|
||||||
|
|||||||
@@ -524,8 +524,6 @@ class PI05Pytorch(nn.Module): # see openpi `PI0Pytorch`
|
|||||||
|
|
||||||
def embed_suffix(self, noisy_actions, timestep):
|
def embed_suffix(self, noisy_actions, timestep):
|
||||||
"""Embed noisy_actions, timestep to prepare for Expert Gemma processing."""
|
"""Embed noisy_actions, timestep to prepare for Expert Gemma processing."""
|
||||||
embs = []
|
|
||||||
pad_masks = []
|
|
||||||
att_masks = []
|
att_masks = []
|
||||||
|
|
||||||
# Embed timestep using sine-cosine positional encoding
|
# Embed timestep using sine-cosine positional encoding
|
||||||
@@ -551,23 +549,17 @@ class PI05Pytorch(nn.Module): # see openpi `PI0Pytorch`
|
|||||||
return F.silu(x)
|
return F.silu(x)
|
||||||
|
|
||||||
time_emb = self._apply_checkpoint(time_mlp_func, time_emb)
|
time_emb = self._apply_checkpoint(time_mlp_func, time_emb)
|
||||||
action_time_emb = action_emb
|
|
||||||
adarms_cond = time_emb
|
adarms_cond = time_emb
|
||||||
|
|
||||||
embs.append(action_time_emb)
|
bsize, action_time_dim = action_emb.shape[:2]
|
||||||
bsize, action_time_dim = action_time_emb.shape[:2]
|
pad_masks = torch.ones(bsize, action_time_dim, dtype=torch.bool, device=timestep.device)
|
||||||
action_time_mask = torch.ones(bsize, action_time_dim, dtype=torch.bool, device=timestep.device)
|
|
||||||
pad_masks.append(action_time_mask)
|
|
||||||
|
|
||||||
# Set attention masks so that image, language and state inputs do not attend to action tokens
|
# Set attention masks so that image, language and state inputs do not attend to action tokens
|
||||||
att_masks += [1] + ([0] * (self.config.chunk_size - 1))
|
att_masks += [1] + ([0] * (self.config.chunk_size - 1))
|
||||||
|
att_masks = torch.tensor(att_masks, dtype=action_emb.dtype, device=action_emb.device)
|
||||||
embs = torch.cat(embs, dim=1)
|
|
||||||
pad_masks = torch.cat(pad_masks, dim=1)
|
|
||||||
att_masks = torch.tensor(att_masks, dtype=embs.dtype, device=embs.device)
|
|
||||||
att_masks = att_masks[None, :].expand(bsize, len(att_masks))
|
att_masks = att_masks[None, :].expand(bsize, len(att_masks))
|
||||||
|
|
||||||
return embs, pad_masks, att_masks, adarms_cond
|
return action_emb, pad_masks, att_masks, adarms_cond
|
||||||
|
|
||||||
def forward(self, images, img_masks, tokens, masks, actions, noise, time) -> Tensor:
|
def forward(self, images, img_masks, tokens, masks, actions, noise, time) -> Tensor:
|
||||||
"""Do a full training forward pass and compute the loss."""
|
"""Do a full training forward pass and compute the loss."""
|
||||||
|
|||||||
@@ -61,9 +61,15 @@ import torch.nn.functional as F # noqa: N812
|
|||||||
from torch import Tensor, nn
|
from torch import Tensor, nn
|
||||||
|
|
||||||
from lerobot.utils.constants import ACTION, OBS_LANGUAGE_ATTENTION_MASK, OBS_LANGUAGE_TOKENS, OBS_STATE
|
from lerobot.utils.constants import ACTION, OBS_LANGUAGE_ATTENTION_MASK, OBS_LANGUAGE_TOKENS, OBS_STATE
|
||||||
from lerobot.utils.device_utils import get_safe_dtype
|
|
||||||
from lerobot.utils.import_utils import require_package
|
from lerobot.utils.import_utils import require_package
|
||||||
|
|
||||||
|
from ..common.flow_matching import euler_integrate, sample_noise, sample_time_beta
|
||||||
|
from ..common.vla_utils import (
|
||||||
|
create_sinusoidal_pos_embedding,
|
||||||
|
make_att_2d_masks,
|
||||||
|
pad_vector,
|
||||||
|
resize_with_pad,
|
||||||
|
)
|
||||||
from ..pretrained import PreTrainedPolicy
|
from ..pretrained import PreTrainedPolicy
|
||||||
from ..rtc.modeling_rtc import RTCProcessor
|
from ..rtc.modeling_rtc import RTCProcessor
|
||||||
from ..utils import (
|
from ..utils import (
|
||||||
@@ -79,96 +85,6 @@ class ActionSelectKwargs(TypedDict, total=False):
|
|||||||
execution_horizon: int | None
|
execution_horizon: int | None
|
||||||
|
|
||||||
|
|
||||||
def create_sinusoidal_pos_embedding(
|
|
||||||
time: torch.tensor, dimension: int, min_period: float, max_period: float, device="cpu"
|
|
||||||
) -> Tensor:
|
|
||||||
"""Computes sine-cosine positional embedding vectors for scalar positions."""
|
|
||||||
if dimension % 2 != 0:
|
|
||||||
raise ValueError(f"dimension ({dimension}) must be divisible by 2")
|
|
||||||
|
|
||||||
if time.ndim != 1:
|
|
||||||
raise ValueError("The time tensor is expected to be of shape `(batch_size, )`.")
|
|
||||||
|
|
||||||
dtype = get_safe_dtype(torch.float64, device.type)
|
|
||||||
fraction = torch.linspace(0.0, 1.0, dimension // 2, dtype=dtype, device=device)
|
|
||||||
period = min_period * (max_period / min_period) ** fraction
|
|
||||||
|
|
||||||
# Compute the outer product
|
|
||||||
scaling_factor = 1.0 / period * 2 * math.pi
|
|
||||||
sin_input = scaling_factor[None, :] * time[:, None]
|
|
||||||
pos_emb = torch.cat([torch.sin(sin_input), torch.cos(sin_input)], dim=1)
|
|
||||||
return pos_emb
|
|
||||||
|
|
||||||
|
|
||||||
def make_att_2d_masks(pad_masks, att_masks):
|
|
||||||
"""Copied from big_vision.
|
|
||||||
|
|
||||||
Tokens can attend to valid inputs tokens which have a cumulative mask_ar
|
|
||||||
smaller or equal to theirs. This way `mask_ar` int[B, N] can be used to
|
|
||||||
setup several types of attention, for example:
|
|
||||||
|
|
||||||
[[1 1 1 1 1 1]]: pure causal attention.
|
|
||||||
|
|
||||||
[[0 0 0 1 1 1]]: prefix-lm attention. The first 3 tokens can attend between
|
|
||||||
themselves and the last 3 tokens have a causal attention. The first
|
|
||||||
entry could also be a 1 without changing behaviour.
|
|
||||||
|
|
||||||
[[1 0 1 0 1 0 0 1 0 0]]: causal attention between 4 blocks. Tokens of a
|
|
||||||
block can attend all previous blocks and all tokens on the same block.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
input_mask: bool[B, N] true if its part of the input, false if padding.
|
|
||||||
mask_ar: int32[B, N] mask that's 1 where previous tokens cannot depend on
|
|
||||||
it and 0 where it shares the same attention mask as the previous token.
|
|
||||||
"""
|
|
||||||
if att_masks.ndim != 2:
|
|
||||||
raise ValueError(att_masks.ndim)
|
|
||||||
if pad_masks.ndim != 2:
|
|
||||||
raise ValueError(pad_masks.ndim)
|
|
||||||
|
|
||||||
cumsum = torch.cumsum(att_masks, dim=1)
|
|
||||||
att_2d_masks = cumsum[:, None, :] <= cumsum[:, :, None]
|
|
||||||
pad_2d_masks = pad_masks[:, None, :] * pad_masks[:, :, None]
|
|
||||||
att_2d_masks = att_2d_masks & pad_2d_masks
|
|
||||||
return att_2d_masks
|
|
||||||
|
|
||||||
|
|
||||||
def resize_with_pad(img, width, height, pad_value=-1):
|
|
||||||
# assume no-op when width height fits already
|
|
||||||
if img.ndim != 4:
|
|
||||||
raise ValueError(f"(b,c,h,w) expected, but {img.shape}")
|
|
||||||
|
|
||||||
cur_height, cur_width = img.shape[2:]
|
|
||||||
|
|
||||||
ratio = max(cur_width / width, cur_height / height)
|
|
||||||
resized_height = int(cur_height / ratio)
|
|
||||||
resized_width = int(cur_width / ratio)
|
|
||||||
resized_img = F.interpolate(
|
|
||||||
img, size=(resized_height, resized_width), mode="bilinear", align_corners=False
|
|
||||||
)
|
|
||||||
|
|
||||||
pad_height = max(0, int(height - resized_height))
|
|
||||||
pad_width = max(0, int(width - resized_width))
|
|
||||||
|
|
||||||
# pad on left and top of image
|
|
||||||
padded_img = F.pad(resized_img, (pad_width, 0, pad_height, 0), value=pad_value)
|
|
||||||
return padded_img
|
|
||||||
|
|
||||||
|
|
||||||
def pad_vector(vector, new_dim):
|
|
||||||
"""Can be (batch_size x sequence_length x features_dimension)
|
|
||||||
or (batch_size x features_dimension)
|
|
||||||
"""
|
|
||||||
if vector.shape[-1] == new_dim:
|
|
||||||
return vector
|
|
||||||
shape = list(vector.shape)
|
|
||||||
current_dim = shape[-1]
|
|
||||||
shape[-1] = new_dim
|
|
||||||
new_vector = torch.zeros(*shape, dtype=vector.dtype, device=vector.device)
|
|
||||||
new_vector[..., :current_dim] = vector
|
|
||||||
return new_vector
|
|
||||||
|
|
||||||
|
|
||||||
def normalize(x, min_val, max_val):
|
def normalize(x, min_val, max_val):
|
||||||
return (x - min_val) / (max_val - min_val)
|
return (x - min_val) / (max_val - min_val)
|
||||||
|
|
||||||
@@ -429,7 +345,13 @@ class SmolVLAPolicy(PreTrainedPolicy):
|
|||||||
for key in present_img_keys:
|
for key in present_img_keys:
|
||||||
img = batch[key][:, -1, :, :, :] if batch[key].ndim == 5 else batch[key]
|
img = batch[key][:, -1, :, :, :] if batch[key].ndim == 5 else batch[key]
|
||||||
if self.config.resize_imgs_with_padding is not None:
|
if self.config.resize_imgs_with_padding is not None:
|
||||||
img = resize_with_pad(img, *self.config.resize_imgs_with_padding, pad_value=0)
|
# SmolVLA stores the target as (width, height); the shared helper expects (height, width).
|
||||||
|
img = resize_with_pad(
|
||||||
|
img,
|
||||||
|
self.config.resize_imgs_with_padding[1],
|
||||||
|
self.config.resize_imgs_with_padding[0],
|
||||||
|
pad_value=0,
|
||||||
|
)
|
||||||
|
|
||||||
# Normalize from range [0,1] to [-1,1] as expacted by siglip
|
# Normalize from range [0,1] to [-1,1] as expacted by siglip
|
||||||
img = img * 2.0 - 1.0
|
img = img * 2.0 - 1.0
|
||||||
@@ -619,20 +541,10 @@ class VLAFlowMatching(nn.Module):
|
|||||||
params.requires_grad = self.config.train_state_proj
|
params.requires_grad = self.config.train_state_proj
|
||||||
|
|
||||||
def sample_noise(self, shape, device):
|
def sample_noise(self, shape, device):
|
||||||
noise = torch.normal(
|
return sample_noise(shape, device)
|
||||||
mean=0.0,
|
|
||||||
std=1.0,
|
|
||||||
size=shape,
|
|
||||||
dtype=torch.float32,
|
|
||||||
device=device,
|
|
||||||
)
|
|
||||||
return noise
|
|
||||||
|
|
||||||
def sample_time(self, bsize, device):
|
def sample_time(self, bsize, device):
|
||||||
beta_dist = torch.distributions.Beta(concentration1=1.5, concentration0=1.0)
|
return sample_time_beta(bsize, device, alpha=1.5, beta=1.0, scale=0.999, offset=0.001)
|
||||||
time_beta = beta_dist.sample((bsize,)).to(device=device, dtype=torch.float32)
|
|
||||||
time = time_beta * 0.999 + 0.001
|
|
||||||
return time
|
|
||||||
|
|
||||||
def embed_prefix(
|
def embed_prefix(
|
||||||
self, images, img_masks, lang_tokens, lang_masks, state: torch.Tensor = None
|
self, images, img_masks, lang_tokens, lang_masks, state: torch.Tensor = None
|
||||||
@@ -800,7 +712,6 @@ class VLAFlowMatching(nn.Module):
|
|||||||
past_key_values=None,
|
past_key_values=None,
|
||||||
inputs_embeds=[prefix_embs, suffix_embs],
|
inputs_embeds=[prefix_embs, suffix_embs],
|
||||||
use_cache=False,
|
use_cache=False,
|
||||||
fill_kv_cache=False,
|
|
||||||
)
|
)
|
||||||
suffix_out = suffix_out[:, -self.config.chunk_size :]
|
suffix_out = suffix_out[:, -self.config.chunk_size :]
|
||||||
# Original openpi code, upcast attention output
|
# Original openpi code, upcast attention output
|
||||||
@@ -839,47 +750,25 @@ class VLAFlowMatching(nn.Module):
|
|||||||
past_key_values=None,
|
past_key_values=None,
|
||||||
inputs_embeds=[prefix_embs, None],
|
inputs_embeds=[prefix_embs, None],
|
||||||
use_cache=self.config.use_cache,
|
use_cache=self.config.use_cache,
|
||||||
fill_kv_cache=True,
|
|
||||||
)
|
)
|
||||||
num_steps = self.config.num_steps
|
num_steps = self.config.num_steps
|
||||||
dt = -1.0 / num_steps
|
|
||||||
|
|
||||||
x_t = noise
|
return euler_integrate(
|
||||||
for step in range(num_steps):
|
lambda input_x_t, current_timestep: self.denoise_step(
|
||||||
time = 1.0 + step * dt
|
|
||||||
time_tensor = torch.tensor(time, dtype=torch.float32, device=device).expand(bsize)
|
|
||||||
|
|
||||||
def denoise_step_partial_call(input_x_t, current_timestep=time_tensor):
|
|
||||||
return self.denoise_step(
|
|
||||||
x_t=input_x_t,
|
x_t=input_x_t,
|
||||||
prefix_pad_masks=prefix_pad_masks,
|
prefix_pad_masks=prefix_pad_masks,
|
||||||
past_key_values=past_key_values,
|
past_key_values=past_key_values,
|
||||||
timestep=current_timestep,
|
timestep=current_timestep,
|
||||||
|
),
|
||||||
|
noise,
|
||||||
|
num_steps,
|
||||||
|
rtc_processor=self.rtc_processor,
|
||||||
|
rtc_enabled=self._rtc_enabled(),
|
||||||
|
inference_delay=kwargs.get("inference_delay"),
|
||||||
|
prev_chunk_left_over=kwargs.get("prev_chunk_left_over"),
|
||||||
|
execution_horizon=kwargs.get("execution_horizon"),
|
||||||
)
|
)
|
||||||
|
|
||||||
if self._rtc_enabled():
|
|
||||||
inference_delay = kwargs.get("inference_delay")
|
|
||||||
prev_chunk_left_over = kwargs.get("prev_chunk_left_over")
|
|
||||||
execution_horizon = kwargs.get("execution_horizon")
|
|
||||||
|
|
||||||
v_t = self.rtc_processor.denoise_step(
|
|
||||||
x_t=x_t,
|
|
||||||
prev_chunk_left_over=prev_chunk_left_over,
|
|
||||||
inference_delay=inference_delay,
|
|
||||||
time=time,
|
|
||||||
original_denoise_step_partial=denoise_step_partial_call,
|
|
||||||
execution_horizon=execution_horizon,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
v_t = denoise_step_partial_call(x_t)
|
|
||||||
|
|
||||||
x_t = x_t + dt * v_t
|
|
||||||
|
|
||||||
if self.rtc_processor is not None and self.rtc_processor.is_debug_enabled():
|
|
||||||
self.rtc_processor.track(time=time, x_t=x_t, v_t=v_t)
|
|
||||||
|
|
||||||
return x_t
|
|
||||||
|
|
||||||
def denoise_step(
|
def denoise_step(
|
||||||
self,
|
self,
|
||||||
prefix_pad_masks,
|
prefix_pad_masks,
|
||||||
@@ -907,8 +796,10 @@ class VLAFlowMatching(nn.Module):
|
|||||||
past_key_values=past_key_values,
|
past_key_values=past_key_values,
|
||||||
inputs_embeds=[None, suffix_embs],
|
inputs_embeds=[None, suffix_embs],
|
||||||
use_cache=self.config.use_cache,
|
use_cache=self.config.use_cache,
|
||||||
fill_kv_cache=False,
|
|
||||||
)
|
)
|
||||||
|
if past_key_values is not None:
|
||||||
|
# Self-attention layers append suffix K/V in place; restore the prefix for the next step.
|
||||||
|
past_key_values.crop(prefix_len)
|
||||||
suffix_out = outputs_embeds[1]
|
suffix_out = outputs_embeds[1]
|
||||||
suffix_out = suffix_out[:, -self.config.chunk_size :]
|
suffix_out = suffix_out[:, -self.config.chunk_size :]
|
||||||
suffix_out = suffix_out.to(dtype=torch.float32)
|
suffix_out = suffix_out.to(dtype=torch.float32)
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ if TYPE_CHECKING or _transformers_available:
|
|||||||
AutoModel,
|
AutoModel,
|
||||||
AutoModelForImageTextToText,
|
AutoModelForImageTextToText,
|
||||||
AutoProcessor,
|
AutoProcessor,
|
||||||
|
DynamicCache,
|
||||||
SmolVLMForConditionalGeneration,
|
SmolVLMForConditionalGeneration,
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
@@ -33,6 +34,7 @@ else:
|
|||||||
AutoModel = None
|
AutoModel = None
|
||||||
AutoModelForImageTextToText = None
|
AutoModelForImageTextToText = None
|
||||||
AutoProcessor = None
|
AutoProcessor = None
|
||||||
|
DynamicCache = None
|
||||||
SmolVLMForConditionalGeneration = None
|
SmolVLMForConditionalGeneration = None
|
||||||
|
|
||||||
|
|
||||||
@@ -216,9 +218,8 @@ class SmolVLMWithExpertModel(nn.Module):
|
|||||||
batch_size,
|
batch_size,
|
||||||
head_dim,
|
head_dim,
|
||||||
use_cache: bool = True,
|
use_cache: bool = True,
|
||||||
fill_kv_cache: bool = True,
|
past_key_values: "DynamicCache | None" = None,
|
||||||
past_key_values=None,
|
) -> "tuple[list[torch.Tensor], DynamicCache | None]":
|
||||||
) -> list[torch.Tensor]:
|
|
||||||
query_states = []
|
query_states = []
|
||||||
key_states = []
|
key_states = []
|
||||||
value_states = []
|
value_states = []
|
||||||
@@ -259,22 +260,16 @@ class SmolVLMWithExpertModel(nn.Module):
|
|||||||
query_states = apply_rope(query_states, position_ids_)
|
query_states = apply_rope(query_states, position_ids_)
|
||||||
key_states = apply_rope(key_states, position_ids_)
|
key_states = apply_rope(key_states, position_ids_)
|
||||||
|
|
||||||
if use_cache and past_key_values is None:
|
|
||||||
past_key_values = {}
|
|
||||||
|
|
||||||
if use_cache:
|
if use_cache:
|
||||||
if fill_kv_cache:
|
# `DynamicCache` stores tensors as [batch, heads, seq, head_dim]; this module works with
|
||||||
past_key_values[layer_idx] = {
|
# [batch, seq, heads, head_dim]. During prefix prefill this stores the (post-RoPE) K/V and
|
||||||
"key_states": key_states,
|
# returns them unchanged; during denoising it appends the suffix K/V and returns
|
||||||
"value_states": value_states,
|
# [prefix; suffix], exactly like the previous hand-rolled dict cache.
|
||||||
}
|
key_states, value_states = past_key_values.update(
|
||||||
else:
|
key_states.transpose(1, 2), value_states.transpose(1, 2), layer_idx
|
||||||
# TODO here, some optimization can be done - similar to a `StaticCache` we can declare the `max_len` before.
|
)
|
||||||
# so we create an empty cache, with just one cuda malloc, and if (in autoregressive case) we reach
|
key_states = key_states.transpose(1, 2)
|
||||||
# the max len, then we (for instance) double the cache size. This implementation already exists
|
value_states = value_states.transpose(1, 2)
|
||||||
# in `transformers`. (molbap)
|
|
||||||
key_states = torch.cat([past_key_values[layer_idx]["key_states"], key_states], dim=1)
|
|
||||||
value_states = torch.cat([past_key_values[layer_idx]["value_states"], value_states], dim=1)
|
|
||||||
|
|
||||||
attention_interface = self.get_attention_interface()
|
attention_interface = self.get_attention_interface()
|
||||||
|
|
||||||
@@ -293,13 +288,12 @@ class SmolVLMWithExpertModel(nn.Module):
|
|||||||
batch_size,
|
batch_size,
|
||||||
head_dim,
|
head_dim,
|
||||||
use_cache: bool = True,
|
use_cache: bool = True,
|
||||||
fill_kv_cache: bool = True,
|
past_key_values: "DynamicCache | None" = None,
|
||||||
past_key_values=None,
|
) -> "tuple[list[torch.Tensor], DynamicCache | None]":
|
||||||
) -> list[torch.Tensor]:
|
|
||||||
attention_interface = self.get_attention_interface()
|
attention_interface = self.get_attention_interface()
|
||||||
|
|
||||||
att_outputs = []
|
att_outputs = []
|
||||||
assert len(inputs_embeds) == 2 or (use_cache and past_key_values is not None and not fill_kv_cache), (
|
assert len(inputs_embeds) == 2 or (use_cache and past_key_values is not None), (
|
||||||
f"Both len(inputs_embeds) == {len(inputs_embeds)} and past_key_values is {past_key_values}"
|
f"Both len(inputs_embeds) == {len(inputs_embeds)} and past_key_values is {past_key_values}"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -332,22 +326,13 @@ class SmolVLMWithExpertModel(nn.Module):
|
|||||||
else:
|
else:
|
||||||
expert_position_id = position_ids
|
expert_position_id = position_ids
|
||||||
|
|
||||||
if use_cache and past_key_values is None:
|
if use_cache and past_key_values is not None:
|
||||||
past_key_values = {}
|
# Cross-attention layers never fill the cache themselves: during the prefix prefill every
|
||||||
|
# layer goes through `forward_attn_layer`, which stores the (post-RoPE) VLM K/V for this
|
||||||
if use_cache:
|
# layer index. Here we only read them back (no concatenation: the expert cross-attends to
|
||||||
if fill_kv_cache:
|
# the fixed prefix). `DynamicCache` stores [batch, heads, seq, head_dim]; transpose back.
|
||||||
past_key_values[layer_idx] = {
|
key_states = past_key_values.layers[layer_idx].keys.transpose(1, 2)
|
||||||
"key_states": key_states,
|
value_states = past_key_values.layers[layer_idx].values.transpose(1, 2)
|
||||||
"value_states": value_states,
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
# TODO here, some optimization can be done - similar to a `StaticCache` we can declare the `max_len` before.
|
|
||||||
# so we create an empty cache, with just one cuda malloc, and if (in autoregressive case) we reach
|
|
||||||
# the max len, then we (for instance) double the cache size. This implementation already exists
|
|
||||||
# in `transformers`. (molbap)
|
|
||||||
key_states = past_key_values[layer_idx]["key_states"]
|
|
||||||
value_states = past_key_values[layer_idx]["value_states"]
|
|
||||||
|
|
||||||
# Expert
|
# Expert
|
||||||
expert_layer = model_layers[1][layer_idx]
|
expert_layer = model_layers[1][layer_idx]
|
||||||
@@ -360,14 +345,15 @@ class SmolVLMWithExpertModel(nn.Module):
|
|||||||
expert_hidden_states = expert_hidden_states.to(dtype=expert_layer.self_attn.q_proj.weight.dtype)
|
expert_hidden_states = expert_hidden_states.to(dtype=expert_layer.self_attn.q_proj.weight.dtype)
|
||||||
expert_query_state = expert_layer.self_attn.q_proj(expert_hidden_states).view(expert_hidden_shape)
|
expert_query_state = expert_layer.self_attn.q_proj(expert_hidden_states).view(expert_hidden_shape)
|
||||||
|
|
||||||
_key_states = key_states.to(dtype=expert_layer.self_attn.k_proj.weight.dtype).view(
|
# reshape (not view): K/V read back from the cache are transposed, hence non-contiguous
|
||||||
|
_key_states = key_states.to(dtype=expert_layer.self_attn.k_proj.weight.dtype).reshape(
|
||||||
*key_states.shape[:2], -1
|
*key_states.shape[:2], -1
|
||||||
)
|
)
|
||||||
expert_key_states = expert_layer.self_attn.k_proj(_key_states).view(
|
expert_key_states = expert_layer.self_attn.k_proj(_key_states).view(
|
||||||
*_key_states.shape[:-1], -1, expert_layer.self_attn.head_dim
|
*_key_states.shape[:-1], -1, expert_layer.self_attn.head_dim
|
||||||
) # k_proj should have same dim as kv
|
) # k_proj should have same dim as kv
|
||||||
|
|
||||||
_value_states = value_states.to(dtype=expert_layer.self_attn.v_proj.weight.dtype).view(
|
_value_states = value_states.to(dtype=expert_layer.self_attn.v_proj.weight.dtype).reshape(
|
||||||
*value_states.shape[:2], -1
|
*value_states.shape[:2], -1
|
||||||
)
|
)
|
||||||
expert_value_states = expert_layer.self_attn.v_proj(_value_states).view(
|
expert_value_states = expert_layer.self_attn.v_proj(_value_states).view(
|
||||||
@@ -416,10 +402,9 @@ class SmolVLMWithExpertModel(nn.Module):
|
|||||||
self,
|
self,
|
||||||
attention_mask: torch.Tensor | None = None,
|
attention_mask: torch.Tensor | None = None,
|
||||||
position_ids: torch.LongTensor | None = None,
|
position_ids: torch.LongTensor | None = None,
|
||||||
past_key_values: list[torch.FloatTensor] | None = None,
|
past_key_values: "DynamicCache | None" = None,
|
||||||
inputs_embeds: list[torch.FloatTensor] = None,
|
inputs_embeds: list[torch.FloatTensor] = None,
|
||||||
use_cache: bool | None = None,
|
use_cache: bool | None = None,
|
||||||
fill_kv_cache: bool | None = None,
|
|
||||||
):
|
):
|
||||||
models = [self.get_vlm_model().text_model, self.lm_expert]
|
models = [self.get_vlm_model().text_model, self.lm_expert]
|
||||||
model_layers = self.get_model_layers(models)
|
model_layers = self.get_model_layers(models)
|
||||||
@@ -431,6 +416,13 @@ class SmolVLMWithExpertModel(nn.Module):
|
|||||||
continue
|
continue
|
||||||
batch_size = hidden_states.shape[0]
|
batch_size = hidden_states.shape[0]
|
||||||
|
|
||||||
|
# Prefix prefill: no cache was passed, so create one and fill it (every layer runs
|
||||||
|
# self-attention over the prefix). When a filled cache is passed (denoising), layers
|
||||||
|
# read from it instead.
|
||||||
|
fill_kv_cache = use_cache and past_key_values is None
|
||||||
|
if fill_kv_cache:
|
||||||
|
past_key_values = DynamicCache()
|
||||||
|
|
||||||
# RMSNorm
|
# RMSNorm
|
||||||
num_layers = self.num_vlm_layers
|
num_layers = self.num_vlm_layers
|
||||||
head_dim = self.vlm.config.text_config.head_dim
|
head_dim = self.vlm.config.text_config.head_dim
|
||||||
@@ -449,7 +441,6 @@ class SmolVLMWithExpertModel(nn.Module):
|
|||||||
batch_size,
|
batch_size,
|
||||||
head_dim,
|
head_dim,
|
||||||
use_cache=use_cache,
|
use_cache=use_cache,
|
||||||
fill_kv_cache=fill_kv_cache,
|
|
||||||
past_key_values=past_key_values,
|
past_key_values=past_key_values,
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
@@ -462,7 +453,6 @@ class SmolVLMWithExpertModel(nn.Module):
|
|||||||
batch_size,
|
batch_size,
|
||||||
head_dim,
|
head_dim,
|
||||||
use_cache=use_cache,
|
use_cache=use_cache,
|
||||||
fill_kv_cache=fill_kv_cache,
|
|
||||||
past_key_values=past_key_values,
|
past_key_values=past_key_values,
|
||||||
)
|
)
|
||||||
outputs_embeds = []
|
outputs_embeds = []
|
||||||
|
|||||||
@@ -58,6 +58,9 @@ class BiSOFollower(BimanualMixin, Robot):
|
|||||||
port=config.left_arm_config.port,
|
port=config.left_arm_config.port,
|
||||||
disable_torque_on_disconnect=config.left_arm_config.disable_torque_on_disconnect,
|
disable_torque_on_disconnect=config.left_arm_config.disable_torque_on_disconnect,
|
||||||
max_relative_target=config.left_arm_config.max_relative_target,
|
max_relative_target=config.left_arm_config.max_relative_target,
|
||||||
|
position_p_coefficient=config.left_arm_config.position_p_coefficient,
|
||||||
|
position_i_coefficient=config.left_arm_config.position_i_coefficient,
|
||||||
|
position_d_coefficient=config.left_arm_config.position_d_coefficient,
|
||||||
use_degrees=config.left_arm_config.use_degrees,
|
use_degrees=config.left_arm_config.use_degrees,
|
||||||
cameras=left_arm_cameras,
|
cameras=left_arm_cameras,
|
||||||
)
|
)
|
||||||
@@ -68,6 +71,9 @@ class BiSOFollower(BimanualMixin, Robot):
|
|||||||
port=config.right_arm_config.port,
|
port=config.right_arm_config.port,
|
||||||
disable_torque_on_disconnect=config.right_arm_config.disable_torque_on_disconnect,
|
disable_torque_on_disconnect=config.right_arm_config.disable_torque_on_disconnect,
|
||||||
max_relative_target=config.right_arm_config.max_relative_target,
|
max_relative_target=config.right_arm_config.max_relative_target,
|
||||||
|
position_p_coefficient=config.right_arm_config.position_p_coefficient,
|
||||||
|
position_i_coefficient=config.right_arm_config.position_i_coefficient,
|
||||||
|
position_d_coefficient=config.right_arm_config.position_d_coefficient,
|
||||||
use_degrees=config.right_arm_config.use_degrees,
|
use_degrees=config.right_arm_config.use_degrees,
|
||||||
cameras=config.right_arm_config.cameras,
|
cameras=config.right_arm_config.cameras,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -150,9 +150,6 @@ class OpenArmFollower(Robot):
|
|||||||
|
|
||||||
self.configure()
|
self.configure()
|
||||||
|
|
||||||
if self.is_calibrated:
|
|
||||||
self.bus.set_zero_position()
|
|
||||||
|
|
||||||
self.bus.enable_torque()
|
self.bus.enable_torque()
|
||||||
|
|
||||||
logger.info(f"{self} connected.")
|
logger.info(f"{self} connected.")
|
||||||
|
|||||||
@@ -41,6 +41,11 @@ class SOFollowerConfig:
|
|||||||
# Set to `True` for backward compatibility with previous policies/dataset
|
# Set to `True` for backward compatibility with previous policies/dataset
|
||||||
use_degrees: bool = True
|
use_degrees: bool = True
|
||||||
|
|
||||||
|
# Position-mode PID gains written to Feetech STS3215 motors at connect time.
|
||||||
|
position_p_coefficient: int = 16
|
||||||
|
position_i_coefficient: int = 0
|
||||||
|
position_d_coefficient: int = 32
|
||||||
|
|
||||||
|
|
||||||
@RobotConfig.register_subclass("so101_follower")
|
@RobotConfig.register_subclass("so101_follower")
|
||||||
@RobotConfig.register_subclass("so100_follower")
|
@RobotConfig.register_subclass("so100_follower")
|
||||||
|
|||||||
@@ -161,11 +161,9 @@ class SOFollower(Robot):
|
|||||||
self.bus.configure_motors()
|
self.bus.configure_motors()
|
||||||
for motor in self.bus.motors:
|
for motor in self.bus.motors:
|
||||||
self.bus.write("Operating_Mode", motor, OperatingMode.POSITION.value)
|
self.bus.write("Operating_Mode", motor, OperatingMode.POSITION.value)
|
||||||
# Set P_Coefficient to lower value to avoid shakiness (Default is 32)
|
self.bus.write("P_Coefficient", motor, self.config.position_p_coefficient)
|
||||||
self.bus.write("P_Coefficient", motor, 16)
|
self.bus.write("I_Coefficient", motor, self.config.position_i_coefficient)
|
||||||
# Set I_Coefficient and D_Coefficient to default value 0 and 32
|
self.bus.write("D_Coefficient", motor, self.config.position_d_coefficient)
|
||||||
self.bus.write("I_Coefficient", motor, 0)
|
|
||||||
self.bus.write("D_Coefficient", motor, 32)
|
|
||||||
|
|
||||||
if motor == "gripper":
|
if motor == "gripper":
|
||||||
self.bus.write("Max_Torque_Limit", motor, 500) # 50% of max torque to avoid burnout
|
self.bus.write("Max_Torque_Limit", motor, 500) # 50% of max torque to avoid burnout
|
||||||
|
|||||||
@@ -180,6 +180,14 @@ class DAggerStrategyConfig(RolloutStrategyConfig):
|
|||||||
# Target video file size in MB for episode rotation (record_autonomous
|
# Target video file size in MB for episode rotation (record_autonomous
|
||||||
# mode only). Defaults to DEFAULT_VIDEO_FILE_SIZE_IN_MB when None.
|
# mode only). Defaults to DEFAULT_VIDEO_FILE_SIZE_IN_MB when None.
|
||||||
target_video_file_size_mb: int | None = None
|
target_video_file_size_mb: int | None = None
|
||||||
|
# Whether to turn on or off the smooth handover behavior at phase transitions:
|
||||||
|
# the leader is driven to the follower position on pause (teleops with
|
||||||
|
# `send_feedback` capability), and the follower is slid to the teleop pose when
|
||||||
|
# a correction starts (non-actuated teleops). Disable for clutch-style
|
||||||
|
# teleoperators (e.g. VR controllers) that re-reference at the current robot
|
||||||
|
# pose on engage: the handover is already continuous there, and the blocking
|
||||||
|
# interpolation only delays the start of the correction.
|
||||||
|
smooth_handover: bool = True
|
||||||
input_device: str = "keyboard"
|
input_device: str = "keyboard"
|
||||||
keyboard: DAggerKeyboardConfig = field(default_factory=DAggerKeyboardConfig)
|
keyboard: DAggerKeyboardConfig = field(default_factory=DAggerKeyboardConfig)
|
||||||
pedal: DAggerPedalConfig = field(default_factory=DAggerPedalConfig)
|
pedal: DAggerPedalConfig = field(default_factory=DAggerPedalConfig)
|
||||||
|
|||||||
@@ -623,8 +623,8 @@ class DAggerStrategy(RolloutStrategy):
|
|||||||
# State-machine transition side-effects
|
# State-machine transition side-effects
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _apply_transition(
|
def _apply_transition(
|
||||||
|
self,
|
||||||
old_phase: DAggerPhase,
|
old_phase: DAggerPhase,
|
||||||
new_phase: DAggerPhase,
|
new_phase: DAggerPhase,
|
||||||
engine,
|
engine,
|
||||||
@@ -634,6 +634,10 @@ class DAggerStrategy(RolloutStrategy):
|
|||||||
) -> None:
|
) -> None:
|
||||||
"""Execute side-effects for a validated phase transition, including smooth handovers.
|
"""Execute side-effects for a validated phase transition, including smooth handovers.
|
||||||
|
|
||||||
|
The smooth handovers below can be disabled with
|
||||||
|
``--strategy.smooth_handover=false`` (useful for clutch-style teleops
|
||||||
|
that re-reference at the current robot pose on engage).
|
||||||
|
|
||||||
AUTONOMOUS -> PAUSED (actuated teleop):
|
AUTONOMOUS -> PAUSED (actuated teleop):
|
||||||
Pause the engine, then drive the leader arm to the follower's last
|
Pause the engine, then drive the leader arm to the follower's last
|
||||||
commanded position so the operator takes over without a jerk.
|
commanded position so the operator takes over without a jerk.
|
||||||
@@ -657,7 +661,7 @@ class DAggerStrategy(RolloutStrategy):
|
|||||||
logger.info("Pausing engine - robot holds position")
|
logger.info("Pausing engine - robot holds position")
|
||||||
engine.pause()
|
engine.pause()
|
||||||
|
|
||||||
if teleop_supports_feedback(teleop) and prev_action is not None:
|
if self.config.smooth_handover and teleop_supports_feedback(teleop) and prev_action is not None:
|
||||||
# TODO(Maxime): prev_action is in robot action key space (output of robot_action_processor).
|
# TODO(Maxime): prev_action is in robot action key space (output of robot_action_processor).
|
||||||
# send_feedback expects teleop feedback key space. For homogeneous setups (e.g. SO-101
|
# send_feedback expects teleop feedback key space. For homogeneous setups (e.g. SO-101
|
||||||
# leader + SO-101 follower) the keys are identical so this works. If the processor pipeline
|
# leader + SO-101 follower) the keys are identical so this works. If the processor pipeline
|
||||||
@@ -668,7 +672,11 @@ class DAggerStrategy(RolloutStrategy):
|
|||||||
|
|
||||||
elif old_phase == DAggerPhase.PAUSED and new_phase == DAggerPhase.CORRECTING:
|
elif old_phase == DAggerPhase.PAUSED and new_phase == DAggerPhase.CORRECTING:
|
||||||
logger.info("Entering correction mode - human teleop control")
|
logger.info("Entering correction mode - human teleop control")
|
||||||
if not teleop_supports_feedback(teleop) and prev_action is not None:
|
if (
|
||||||
|
self.config.smooth_handover
|
||||||
|
and not teleop_supports_feedback(teleop)
|
||||||
|
and prev_action is not None
|
||||||
|
):
|
||||||
logger.info("Smooth handover: sliding follower to teleop position")
|
logger.info("Smooth handover: sliding follower to teleop position")
|
||||||
obs = robot.get_observation()
|
obs = robot.get_observation()
|
||||||
teleop_action = teleop.get_action()
|
teleop_action = teleop.get_action()
|
||||||
|
|||||||
@@ -24,7 +24,14 @@ Example:
|
|||||||
--root=/path/to/dataset \\
|
--root=/path/to/dataset \\
|
||||||
--vlm.model_id=Qwen/Qwen2.5-VL-7B-Instruct
|
--vlm.model_id=Qwen/Qwen2.5-VL-7B-Instruct
|
||||||
|
|
||||||
For distributed runs, see ``examples/annotations/run_hf_job.py``.
|
Pass ``--job.target=<flavor>`` to run the same command on a Hugging Face
|
||||||
|
Jobs GPU instead of this machine (see ``lerobot.jobs.annotate``):
|
||||||
|
|
||||||
|
uv run lerobot-annotate \\
|
||||||
|
--repo_id=user/dataset \\
|
||||||
|
--new_repo_id=user/dataset_annotated \\
|
||||||
|
--push_to_hub=true \\
|
||||||
|
--job.target=h200
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
@@ -69,6 +76,14 @@ def _resolve_root(cfg: AnnotationPipelineConfig) -> Path:
|
|||||||
def annotate(cfg: AnnotationPipelineConfig) -> None:
|
def annotate(cfg: AnnotationPipelineConfig) -> None:
|
||||||
"""Run the steerable annotation pipeline against a dataset."""
|
"""Run the steerable annotation pipeline against a dataset."""
|
||||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||||
|
|
||||||
|
if cfg.job.is_remote:
|
||||||
|
# Imported lazily: the submitter pulls in LeRobotDataset (the `dataset`
|
||||||
|
# extra), which a local annotation run over --root doesn't need.
|
||||||
|
from lerobot.jobs.annotate import submit_annotate_to_hf
|
||||||
|
|
||||||
|
return submit_annotate_to_hf(cfg)
|
||||||
|
|
||||||
root = _resolve_root(cfg)
|
root = _resolve_root(cfg)
|
||||||
logger.info("annotate: root=%s", root)
|
logger.info("annotate: root=%s", root)
|
||||||
|
|
||||||
|
|||||||
@@ -51,19 +51,7 @@ from lerobot.teleoperators import ( # noqa: F401
|
|||||||
rebot_102_leader,
|
rebot_102_leader,
|
||||||
so_leader,
|
so_leader,
|
||||||
)
|
)
|
||||||
|
from lerobot.utils.import_utils import register_third_party_plugins
|
||||||
COMPATIBLE_DEVICES = [
|
|
||||||
"koch_follower",
|
|
||||||
"koch_leader",
|
|
||||||
"omx_follower",
|
|
||||||
"omx_leader",
|
|
||||||
"openarm_mini",
|
|
||||||
"so100_follower",
|
|
||||||
"so100_leader",
|
|
||||||
"so101_follower",
|
|
||||||
"so101_leader",
|
|
||||||
"lekiwi",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -80,18 +68,19 @@ class SetupConfig:
|
|||||||
|
|
||||||
@draccus.wrap()
|
@draccus.wrap()
|
||||||
def setup_motors(cfg: SetupConfig):
|
def setup_motors(cfg: SetupConfig):
|
||||||
if cfg.device.type not in COMPATIBLE_DEVICES:
|
|
||||||
raise NotImplementedError
|
|
||||||
|
|
||||||
if isinstance(cfg.device, RobotConfig):
|
if isinstance(cfg.device, RobotConfig):
|
||||||
device = make_robot_from_config(cfg.device)
|
device = make_robot_from_config(cfg.device)
|
||||||
else:
|
else:
|
||||||
device = make_teleoperator_from_config(cfg.device)
|
device = make_teleoperator_from_config(cfg.device)
|
||||||
|
|
||||||
device.setup_motors()
|
setup = getattr(device, "setup_motors", None)
|
||||||
|
if not callable(setup):
|
||||||
|
raise NotImplementedError(f"Device type '{cfg.device.type}' does not support motor setup.")
|
||||||
|
setup()
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
|
register_third_party_plugins()
|
||||||
setup_motors()
|
setup_motors()
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -23,3 +23,5 @@ from ..config import TeleoperatorConfig
|
|||||||
@dataclass
|
@dataclass
|
||||||
class GamepadTeleopConfig(TeleoperatorConfig):
|
class GamepadTeleopConfig(TeleoperatorConfig):
|
||||||
use_gripper: bool = True
|
use_gripper: bool = True
|
||||||
|
# Use hidapi instead of pygame for controllers that pygame cannot detect reliably.
|
||||||
|
hidapi_fallback: bool = False
|
||||||
|
|||||||
@@ -14,6 +14,7 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
|
import logging
|
||||||
import sys
|
import sys
|
||||||
from enum import IntEnum
|
from enum import IntEnum
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -27,6 +28,8 @@ from ..teleoperator import Teleoperator
|
|||||||
from ..utils import TeleopEvents
|
from ..utils import TeleopEvents
|
||||||
from .configuration_gamepad import GamepadTeleopConfig
|
from .configuration_gamepad import GamepadTeleopConfig
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
class GripperAction(IntEnum):
|
class GripperAction(IntEnum):
|
||||||
CLOSE = 0
|
CLOSE = 0
|
||||||
@@ -56,6 +59,13 @@ class GamepadTeleop(Teleoperator):
|
|||||||
|
|
||||||
self.gamepad = None
|
self.gamepad = None
|
||||||
|
|
||||||
|
self.hidapi_fallback = config.hidapi_fallback
|
||||||
|
if sys.platform == "darwin" and not self.hidapi_fallback:
|
||||||
|
logger.warning(
|
||||||
|
"On macOS, pygame may not reliably detect input from some controllers. "
|
||||||
|
"If you experience issues, set `hidapi_fallback=true`."
|
||||||
|
)
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def action_features(self) -> dict:
|
def action_features(self) -> dict:
|
||||||
if self.config.use_gripper:
|
if self.config.use_gripper:
|
||||||
@@ -76,9 +86,7 @@ class GamepadTeleop(Teleoperator):
|
|||||||
return {}
|
return {}
|
||||||
|
|
||||||
def connect(self) -> None:
|
def connect(self) -> None:
|
||||||
# use HidApi for macos
|
if self.hidapi_fallback:
|
||||||
if sys.platform == "darwin":
|
|
||||||
# NOTE: On macOS, pygame doesn’t reliably detect input from some controllers so we fall back to hidapi
|
|
||||||
from .gamepad_utils import GamepadControllerHID as Gamepad
|
from .gamepad_utils import GamepadControllerHID as Gamepad
|
||||||
else:
|
else:
|
||||||
from .gamepad_utils import GamepadController as Gamepad
|
from .gamepad_utils import GamepadController as Gamepad
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ import cv2
|
|||||||
import numpy as np
|
import numpy as np
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from lerobot.cameras.configs import Cv2Rotation
|
from lerobot.cameras.configs import ColorMode, Cv2Rotation
|
||||||
from lerobot.cameras.opencv import OpenCVCamera, OpenCVCameraConfig
|
from lerobot.cameras.opencv import OpenCVCamera, OpenCVCameraConfig
|
||||||
from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError
|
from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError
|
||||||
|
|
||||||
@@ -132,6 +132,28 @@ def test_read(index_or_path):
|
|||||||
assert isinstance(img, np.ndarray)
|
assert isinstance(img, np.ndarray)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("index_or_path", TEST_IMAGE_PATHS, ids=TEST_IMAGE_SIZES)
|
||||||
|
def test_color_mode_conversion(index_or_path):
|
||||||
|
"""RGB and BGR reads of the same frame must differ only by a channel-axis reversal."""
|
||||||
|
rgb_config = OpenCVCameraConfig(index_or_path=index_or_path, color_mode=ColorMode.RGB, warmup_s=0)
|
||||||
|
bgr_config = OpenCVCameraConfig(index_or_path=index_or_path, color_mode=ColorMode.BGR, warmup_s=0)
|
||||||
|
with OpenCVCamera(rgb_config) as rgb_cam:
|
||||||
|
rgb = rgb_cam.read()
|
||||||
|
with OpenCVCamera(bgr_config) as bgr_cam:
|
||||||
|
bgr = bgr_cam.read()
|
||||||
|
|
||||||
|
assert rgb.shape == bgr.shape
|
||||||
|
np.testing.assert_array_equal(rgb, bgr[..., ::-1])
|
||||||
|
|
||||||
|
|
||||||
|
def test_postprocess_invalid_color_mode():
|
||||||
|
config = OpenCVCameraConfig(index_or_path=DEFAULT_PNG_FILE_PATH)
|
||||||
|
camera = OpenCVCamera(config)
|
||||||
|
camera.color_mode = "invalid"
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
camera._postprocess_image(np.zeros((120, 160, 3), dtype=np.uint8))
|
||||||
|
|
||||||
|
|
||||||
def test_read_before_connect():
|
def test_read_before_connect():
|
||||||
config = OpenCVCameraConfig(index_or_path=DEFAULT_PNG_FILE_PATH)
|
config = OpenCVCameraConfig(index_or_path=DEFAULT_PNG_FILE_PATH)
|
||||||
|
|
||||||
|
|||||||
@@ -22,6 +22,7 @@ import pytest
|
|||||||
|
|
||||||
pytest.importorskip("reachy2_sdk")
|
pytest.importorskip("reachy2_sdk")
|
||||||
|
|
||||||
|
from lerobot.cameras.configs import ColorMode
|
||||||
from lerobot.cameras.reachy2_camera import Reachy2Camera, Reachy2CameraConfig
|
from lerobot.cameras.reachy2_camera import Reachy2Camera, Reachy2CameraConfig
|
||||||
from lerobot.utils.errors import DeviceNotConnectedError
|
from lerobot.utils.errors import DeviceNotConnectedError
|
||||||
|
|
||||||
@@ -33,28 +34,19 @@ PARAMS = [
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
def _make_cam_manager_mock():
|
def _make_cam_manager_mock(color_frame, depth_frame=None):
|
||||||
c = MagicMock(name="CameraManagerMock")
|
c = MagicMock(name="CameraManagerMock")
|
||||||
|
|
||||||
teleop = MagicMock(name="TeleopCam")
|
teleop = MagicMock(name="TeleopCam")
|
||||||
teleop.width = 640
|
teleop.width = 640
|
||||||
teleop.height = 480
|
teleop.height = 480
|
||||||
teleop.get_frame = MagicMock(
|
teleop.get_frame = MagicMock(side_effect=lambda *_, **__: (color_frame, time.time()))
|
||||||
side_effect=lambda *_, **__: (
|
|
||||||
np.zeros((480, 640, 3), dtype=np.uint8),
|
|
||||||
time.time(),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
depth = MagicMock(name="DepthCam")
|
depth = MagicMock(name="DepthCam")
|
||||||
depth.width = 640
|
depth.width = 640
|
||||||
depth.height = 480
|
depth.height = 480
|
||||||
depth.get_frame = MagicMock(
|
depth.get_frame = MagicMock(side_effect=lambda *_, **__: (color_frame, time.time()))
|
||||||
side_effect=lambda *_, **__: (
|
depth.get_depth_frame = MagicMock(side_effect=lambda *_, **__: (depth_frame, time.time()))
|
||||||
np.zeros((480, 640, 3), dtype=np.uint8),
|
|
||||||
time.time(),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
c.is_connected.return_value = True
|
c.is_connected.return_value = True
|
||||||
c.teleop = teleop
|
c.teleop = teleop
|
||||||
@@ -84,12 +76,14 @@ def _make_cam_manager_mock():
|
|||||||
# ids=["teleop-left", "teleop-right", "torso-rgb", "torso-depth"],
|
# ids=["teleop-left", "teleop-right", "torso-rgb", "torso-depth"],
|
||||||
ids=["teleop-left", "teleop-right", "torso-rgb"],
|
ids=["teleop-left", "teleop-right", "torso-rgb"],
|
||||||
)
|
)
|
||||||
def camera(request):
|
def camera(request, img_array_factory):
|
||||||
name, image_type = request.param
|
name, image_type = request.param
|
||||||
|
color_frame = img_array_factory(height=480, width=640)
|
||||||
|
depth_frame = img_array_factory(height=480, width=640, channels=1, dtype=np.uint16)[..., 0]
|
||||||
with (
|
with (
|
||||||
patch(
|
patch(
|
||||||
"lerobot.cameras.reachy2_camera.reachy2_camera.CameraManager",
|
"lerobot.cameras.reachy2_camera.reachy2_camera.CameraManager",
|
||||||
side_effect=lambda *a, **k: _make_cam_manager_mock(),
|
side_effect=lambda *a, **k: _make_cam_manager_mock(color_frame, depth_frame),
|
||||||
),
|
),
|
||||||
):
|
):
|
||||||
config = Reachy2CameraConfig(name=name, image_type=image_type)
|
config = Reachy2CameraConfig(name=name, image_type=image_type)
|
||||||
@@ -188,6 +182,41 @@ def test_read_latest_too_old(camera):
|
|||||||
_ = camera.read_latest(max_age_ms=0) # immediately too old
|
_ = camera.read_latest(max_age_ms=0) # immediately too old
|
||||||
|
|
||||||
|
|
||||||
|
def test_color_mode_conversion(img_array_factory):
|
||||||
|
"""teleop frames are native BGR: RGB reverses the channel axis, BGR is passed through."""
|
||||||
|
frame = img_array_factory(height=8, width=8)
|
||||||
|
|
||||||
|
outputs = {}
|
||||||
|
for color_mode in (ColorMode.RGB, ColorMode.BGR):
|
||||||
|
with patch(
|
||||||
|
"lerobot.cameras.reachy2_camera.reachy2_camera.CameraManager",
|
||||||
|
side_effect=lambda *a, **k: _make_cam_manager_mock(frame),
|
||||||
|
):
|
||||||
|
cam = Reachy2Camera(Reachy2CameraConfig(name="teleop", image_type="left", color_mode=color_mode))
|
||||||
|
cam.connect()
|
||||||
|
outputs[color_mode] = cam.read()
|
||||||
|
cam.disconnect()
|
||||||
|
|
||||||
|
np.testing.assert_array_equal(outputs[ColorMode.BGR], frame)
|
||||||
|
np.testing.assert_array_equal(outputs[ColorMode.RGB], frame[..., ::-1])
|
||||||
|
|
||||||
|
|
||||||
|
def test_depth_frame_not_color_converted(img_array_factory):
|
||||||
|
"""A depth/depth frame must be returned as-is, without BGR<->RGB conversion."""
|
||||||
|
color_frame = img_array_factory(height=8, width=8)
|
||||||
|
depth = img_array_factory(height=8, width=8, channels=1, dtype=np.uint16)[..., 0]
|
||||||
|
with patch(
|
||||||
|
"lerobot.cameras.reachy2_camera.reachy2_camera.CameraManager",
|
||||||
|
side_effect=lambda *a, **k: _make_cam_manager_mock(color_frame, depth_frame=depth),
|
||||||
|
):
|
||||||
|
cam = Reachy2Camera(Reachy2CameraConfig(name="depth", image_type="depth"))
|
||||||
|
cam.connect()
|
||||||
|
out = cam.read()
|
||||||
|
cam.disconnect()
|
||||||
|
|
||||||
|
np.testing.assert_array_equal(out, depth)
|
||||||
|
|
||||||
|
|
||||||
def test_wrong_camera_name():
|
def test_wrong_camera_name():
|
||||||
with pytest.raises(ValueError):
|
with pytest.raises(ValueError):
|
||||||
_ = Reachy2CameraConfig(name="wrong-name", image_type="left")
|
_ = Reachy2CameraConfig(name="wrong-name", image_type="left")
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ from unittest.mock import patch
|
|||||||
import numpy as np
|
import numpy as np
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from lerobot.cameras.configs import Cv2Rotation
|
from lerobot.cameras.configs import ColorMode, Cv2Rotation
|
||||||
from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError
|
from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError
|
||||||
|
|
||||||
pytest.importorskip("pyrealsense2")
|
pytest.importorskip("pyrealsense2")
|
||||||
@@ -109,6 +109,32 @@ def test_read_depth():
|
|||||||
assert isinstance(img, np.ndarray)
|
assert isinstance(img, np.ndarray)
|
||||||
|
|
||||||
|
|
||||||
|
# These exercise _postprocess_image directly rather than read(): the bag playback returns
|
||||||
|
# non-deterministic frames we can't compare against, and the depth read() path is skipped
|
||||||
|
# (see test_read_depth) with the current pyrealsense2 version.
|
||||||
|
def test_color_mode_conversion(img_array_factory):
|
||||||
|
"""RGB (native for RealSense) is passed through; BGR reverses the channel axis."""
|
||||||
|
color = img_array_factory(height=3, width=4)
|
||||||
|
|
||||||
|
outputs = {}
|
||||||
|
for color_mode in (ColorMode.RGB, ColorMode.BGR):
|
||||||
|
camera = RealSenseCamera(RealSenseCameraConfig(serial_number_or_name="042", color_mode=color_mode))
|
||||||
|
camera.capture_height, camera.capture_width = color.shape[:2]
|
||||||
|
outputs[color_mode] = camera._postprocess_image(color)
|
||||||
|
|
||||||
|
np.testing.assert_array_equal(outputs[ColorMode.RGB], color)
|
||||||
|
np.testing.assert_array_equal(outputs[ColorMode.BGR], color[..., ::-1])
|
||||||
|
|
||||||
|
|
||||||
|
def test_depth_frame_not_color_converted(img_array_factory):
|
||||||
|
"""Depth frames must bypass color conversion, even when a BGR color_mode is set."""
|
||||||
|
camera = RealSenseCamera(RealSenseCameraConfig(serial_number_or_name="042", color_mode=ColorMode.BGR))
|
||||||
|
depth = img_array_factory(height=3, width=4, channels=1, dtype=np.uint16)[..., 0]
|
||||||
|
camera.capture_height, camera.capture_width = depth.shape
|
||||||
|
|
||||||
|
np.testing.assert_array_equal(camera._postprocess_image(depth, depth_frame=True), depth)
|
||||||
|
|
||||||
|
|
||||||
def test_read_before_connect():
|
def test_read_before_connect():
|
||||||
config = RealSenseCameraConfig(serial_number_or_name="042")
|
config = RealSenseCameraConfig(serial_number_or_name="042")
|
||||||
camera = RealSenseCamera(config)
|
camera = RealSenseCamera(config)
|
||||||
|
|||||||
@@ -14,16 +14,21 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import Mock
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
import torch
|
import torch
|
||||||
|
from packaging.version import Version
|
||||||
|
|
||||||
pytest.importorskip("datasets", reason="datasets is required (install lerobot[dataset])")
|
pytest.importorskip("datasets", reason="datasets is required (install lerobot[dataset])")
|
||||||
|
|
||||||
from datasets import Dataset # noqa: E402
|
from datasets import Dataset # noqa: E402
|
||||||
from huggingface_hub import DatasetCard
|
from huggingface_hub import DatasetCard
|
||||||
|
|
||||||
|
import lerobot.datasets.utils as dataset_utils
|
||||||
from lerobot.datasets.io_utils import hf_transform_to_torch
|
from lerobot.datasets.io_utils import hf_transform_to_torch
|
||||||
from lerobot.datasets.utils import create_lerobot_dataset_card
|
from lerobot.datasets.utils import create_lerobot_dataset_card, get_repo_versions, get_safe_version
|
||||||
from lerobot.utils.constants import ACTION, OBS_IMAGES
|
from lerobot.utils.constants import ACTION, OBS_IMAGES
|
||||||
from lerobot.utils.feature_utils import combine_feature_dicts
|
from lerobot.utils.feature_utils import combine_feature_dicts
|
||||||
|
|
||||||
@@ -57,6 +62,30 @@ def test_default_parameters():
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||||
|
def test_get_repo_versions_forwards_token(monkeypatch, token):
|
||||||
|
api = Mock()
|
||||||
|
api.list_repo_refs.return_value = SimpleNamespace(
|
||||||
|
branches=[SimpleNamespace(name="v3.0")],
|
||||||
|
tags=[],
|
||||||
|
)
|
||||||
|
hf_api = Mock(return_value=api)
|
||||||
|
monkeypatch.setattr(dataset_utils, "HfApi", hf_api)
|
||||||
|
|
||||||
|
assert get_repo_versions("private/repo", token=token) == [Version("3.0")]
|
||||||
|
hf_api.assert_called_once_with(token=token)
|
||||||
|
api.list_repo_refs.assert_called_once_with("private/repo", repo_type="dataset")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||||
|
def test_get_safe_version_forwards_token(monkeypatch, token):
|
||||||
|
get_versions = Mock(return_value=[Version("3.0")])
|
||||||
|
monkeypatch.setattr(dataset_utils, "get_repo_versions", get_versions)
|
||||||
|
|
||||||
|
assert get_safe_version("private/repo", "v3.0", token=token) == "v3.0"
|
||||||
|
get_versions.assert_called_once_with("private/repo", token=token)
|
||||||
|
|
||||||
|
|
||||||
def test_with_tags():
|
def test_with_tags():
|
||||||
tags = ["tag1", "tag2"]
|
tags = ["tag1", "tag2"]
|
||||||
card = create_lerobot_dataset_card(tags=tags)
|
card = create_lerobot_dataset_card(tags=tags)
|
||||||
|
|||||||
@@ -114,6 +114,20 @@ def test_dataset_initialization(tmp_path, lerobot_dataset_factory):
|
|||||||
assert dataset.num_frames == len(dataset)
|
assert dataset.num_frames == len(dataset)
|
||||||
|
|
||||||
|
|
||||||
|
def test_dataset_slice(tmp_path, lerobot_dataset_factory):
|
||||||
|
dataset = lerobot_dataset_factory(
|
||||||
|
root=tmp_path / "test", total_episodes=3, total_frames=30, use_videos=False
|
||||||
|
)
|
||||||
|
|
||||||
|
assert len(dataset[:5]) == 5
|
||||||
|
assert len(dataset[::2]) == (len(dataset) + 1) // 2
|
||||||
|
assert [item["index"].item() for item in dataset[4::-1]] == [4, 3, 2, 1, 0]
|
||||||
|
assert [item["index"].item() for item in dataset[-3:]] == list(range(len(dataset) - 3, len(dataset)))
|
||||||
|
assert dataset[len(dataset) :] == []
|
||||||
|
assert isinstance(dataset[0], dict)
|
||||||
|
assert dataset[:1][0].keys() == dataset[0].keys()
|
||||||
|
|
||||||
|
|
||||||
# TODO(rcadene, aliberts): do not run LeRobotDataset.create, instead refactor LeRobotDatasetMetadata.create
|
# TODO(rcadene, aliberts): do not run LeRobotDataset.create, instead refactor LeRobotDatasetMetadata.create
|
||||||
# and test the small resulting function that validates the features
|
# and test the small resulting function that validates the features
|
||||||
def test_dataset_feature_with_forward_slash_raises_error():
|
def test_dataset_feature_with_forward_slash_raises_error():
|
||||||
@@ -1741,6 +1755,38 @@ def test_delta_timestamps_query_returns_correct_values(tmp_path, empty_lerobot_d
|
|||||||
assert is_pad == [True, False], f"Expected [True, False], got {is_pad}"
|
assert is_pad == [True, False], f"Expected [True, False], got {is_pad}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_dataset_slice_with_delta_timestamps(tmp_path, empty_lerobot_dataset_factory):
|
||||||
|
features = {
|
||||||
|
"observation.state": {"dtype": "float32", "shape": (1,), "names": ["x"]},
|
||||||
|
}
|
||||||
|
dataset = empty_lerobot_dataset_factory(
|
||||||
|
root=tmp_path / "test_slice_delta", features=features, use_videos=False, fps=10
|
||||||
|
)
|
||||||
|
|
||||||
|
for frame_idx in range(5):
|
||||||
|
dataset.add_frame(
|
||||||
|
{
|
||||||
|
"observation.state": torch.tensor([frame_idx], dtype=torch.float32),
|
||||||
|
"task": "task_0",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
dataset.save_episode()
|
||||||
|
dataset.finalize()
|
||||||
|
|
||||||
|
sliced_dataset = LeRobotDataset(
|
||||||
|
dataset.repo_id,
|
||||||
|
root=dataset.root,
|
||||||
|
delta_timestamps={"observation.state": [-0.1, 0.0]},
|
||||||
|
tolerance_s=0.04,
|
||||||
|
)
|
||||||
|
|
||||||
|
items = sliced_dataset[:2]
|
||||||
|
|
||||||
|
assert items[0]["observation.state"].tolist() == [0.0, 0.0]
|
||||||
|
assert items[0]["observation.state_is_pad"].tolist() == [True, False]
|
||||||
|
assert items[1]["observation.state"].tolist() == [0.0, 1.0]
|
||||||
|
|
||||||
|
|
||||||
def test_episode_filter_filters_dataset(tmp_path, lerobot_dataset_factory):
|
def test_episode_filter_filters_dataset(tmp_path, lerobot_dataset_factory):
|
||||||
"""episode_filter on LeRobotDataset narrows the loaded dataset to matching episodes."""
|
"""episode_filter on LeRobotDataset narrows the loaded dataset to matching episodes."""
|
||||||
dataset = lerobot_dataset_factory(root=tmp_path / "test", total_episodes=8, total_frames=200)
|
dataset = lerobot_dataset_factory(root=tmp_path / "test", total_episodes=8, total_frames=200)
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ property delegation, and the full create-record-finalize-read lifecycle.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from types import SimpleNamespace
|
||||||
from unittest.mock import Mock
|
from unittest.mock import Mock
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
@@ -191,6 +192,48 @@ def test_metadata_without_root_uses_hub_cache_snapshot_download(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||||
|
def test_metadata_download_forwards_token(tmp_path, monkeypatch, token):
|
||||||
|
snapshot_root = tmp_path / "snapshot"
|
||||||
|
snapshot_download = Mock(return_value=str(snapshot_root))
|
||||||
|
get_safe_version = Mock(return_value="v3.0")
|
||||||
|
load_metadata = Mock(side_effect=[FileNotFoundError, None])
|
||||||
|
monkeypatch.setattr(dataset_metadata_module, "snapshot_download", snapshot_download)
|
||||||
|
monkeypatch.setattr(dataset_metadata_module, "get_safe_version", get_safe_version)
|
||||||
|
monkeypatch.setattr(LeRobotDatasetMetadata, "_load_metadata", load_metadata)
|
||||||
|
|
||||||
|
meta = LeRobotDatasetMetadata(
|
||||||
|
repo_id=DUMMY_REPO_ID,
|
||||||
|
revision="v3.0",
|
||||||
|
token=token,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert meta.root == snapshot_root
|
||||||
|
assert not hasattr(meta, "_token")
|
||||||
|
get_safe_version.assert_called_once_with(DUMMY_REPO_ID, "v3.0", token=token)
|
||||||
|
assert snapshot_download.call_args.kwargs["token"] is token
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||||
|
def test_data_download_forwards_token(tmp_path, monkeypatch, token):
|
||||||
|
snapshot_root = tmp_path / "snapshot"
|
||||||
|
snapshot_download = Mock(return_value=str(snapshot_root))
|
||||||
|
monkeypatch.setattr(lerobot_dataset_module, "snapshot_download", snapshot_download)
|
||||||
|
|
||||||
|
dataset = LeRobotDataset.__new__(LeRobotDataset)
|
||||||
|
dataset.repo_id = DUMMY_REPO_ID
|
||||||
|
dataset.revision = "main"
|
||||||
|
dataset.episodes = None
|
||||||
|
dataset._requested_root = None
|
||||||
|
dataset.meta = SimpleNamespace(root=None)
|
||||||
|
dataset.reader = SimpleNamespace(root=None)
|
||||||
|
|
||||||
|
dataset._download(token=token)
|
||||||
|
|
||||||
|
assert dataset.root == snapshot_root
|
||||||
|
assert snapshot_download.call_args.kwargs["token"] is token
|
||||||
|
|
||||||
|
|
||||||
def test_without_root_reads_different_revisions_from_distinct_snapshot_roots(
|
def test_without_root_reads_different_revisions_from_distinct_snapshot_roots(
|
||||||
tmp_path,
|
tmp_path,
|
||||||
info_factory,
|
info_factory,
|
||||||
|
|||||||
@@ -13,12 +13,16 @@
|
|||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import Mock
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import pytest
|
import pytest
|
||||||
import torch
|
import torch
|
||||||
|
|
||||||
pytest.importorskip("datasets", reason="datasets is required (install lerobot[dataset])")
|
pytest.importorskip("datasets", reason="datasets is required (install lerobot[dataset])")
|
||||||
|
|
||||||
|
import lerobot.datasets.streaming_dataset as streaming_dataset_module
|
||||||
from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset
|
from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset
|
||||||
from lerobot.datasets.utils import safe_shard
|
from lerobot.datasets.utils import safe_shard
|
||||||
from lerobot.utils.constants import ACTION
|
from lerobot.utils.constants import ACTION
|
||||||
@@ -71,6 +75,40 @@ def get_frames_expected_order(streaming_ds: StreamingLeRobotDataset) -> list[int
|
|||||||
return expected_indices
|
return expected_indices
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("token", ["hf_test_token", True, False])
|
||||||
|
@pytest.mark.parametrize("from_local", [False, True])
|
||||||
|
def test_streaming_dataset_forwards_hub_token_only_for_remote_data(tmp_path, monkeypatch, token, from_local):
|
||||||
|
requested_root = tmp_path / "local" if from_local else None
|
||||||
|
metadata = SimpleNamespace(
|
||||||
|
root=requested_root or tmp_path / "snapshot",
|
||||||
|
revision=streaming_dataset_module.CODEBASE_VERSION,
|
||||||
|
_version=streaming_dataset_module.CODEBASE_VERSION,
|
||||||
|
features={},
|
||||||
|
depth_keys=[],
|
||||||
|
image_keys=[],
|
||||||
|
rescale_depth_stats=Mock(),
|
||||||
|
)
|
||||||
|
metadata_cls = Mock(return_value=metadata)
|
||||||
|
load_dataset = Mock(return_value=SimpleNamespace(num_shards=1))
|
||||||
|
monkeypatch.setattr(streaming_dataset_module, "LeRobotDatasetMetadata", metadata_cls)
|
||||||
|
monkeypatch.setattr(streaming_dataset_module, "load_dataset", load_dataset)
|
||||||
|
|
||||||
|
dataset = StreamingLeRobotDataset(DUMMY_REPO_ID, root=requested_root, token=token)
|
||||||
|
|
||||||
|
metadata_cls.assert_called_once_with(
|
||||||
|
DUMMY_REPO_ID,
|
||||||
|
requested_root,
|
||||||
|
streaming_dataset_module.CODEBASE_VERSION,
|
||||||
|
force_cache_sync=False,
|
||||||
|
token=token,
|
||||||
|
)
|
||||||
|
if from_local:
|
||||||
|
assert "token" not in load_dataset.call_args.kwargs
|
||||||
|
else:
|
||||||
|
assert load_dataset.call_args.kwargs["token"] is token
|
||||||
|
assert not hasattr(dataset, "_token")
|
||||||
|
|
||||||
|
|
||||||
def test_single_frame_consistency(tmp_path, lerobot_dataset_factory):
|
def test_single_frame_consistency(tmp_path, lerobot_dataset_factory):
|
||||||
"""Test if are correctly accessed"""
|
"""Test if are correctly accessed"""
|
||||||
ds_num_frames = 400
|
ds_num_frames = 400
|
||||||
|
|||||||
@@ -35,6 +35,17 @@ def test_unknown_type():
|
|||||||
make_env_config("nonexistent")
|
make_env_config("nonexistent")
|
||||||
|
|
||||||
|
|
||||||
|
def test_libero_fps_controls_simulator_frequency():
|
||||||
|
cfg = LiberoEnv(fps=17)
|
||||||
|
|
||||||
|
assert cfg.gym_kwargs["control_freq"] == 17
|
||||||
|
|
||||||
|
|
||||||
|
def test_libero_rejects_nonpositive_fps():
|
||||||
|
with pytest.raises(ValueError, match="fps must be positive"):
|
||||||
|
LiberoEnv(fps=0)
|
||||||
|
|
||||||
|
|
||||||
def test_identity_processors():
|
def test_identity_processors():
|
||||||
"""Base class get_env_processors() returns identity pipelines."""
|
"""Base class get_env_processors() returns identity pipelines."""
|
||||||
cfg = make_env_config("aloha")
|
cfg = make_env_config("aloha")
|
||||||
|
|||||||
@@ -0,0 +1,245 @@
|
|||||||
|
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
import shlex
|
||||||
|
import sys
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import draccus
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
pytest.importorskip("datasets", reason="datasets is required (install lerobot[dataset])")
|
||||||
|
|
||||||
|
from lerobot.annotations.steerable_pipeline.config import (
|
||||||
|
DEFAULT_ANNOTATE_JOB_IMAGE,
|
||||||
|
AnnotationJobConfig,
|
||||||
|
AnnotationPipelineConfig,
|
||||||
|
)
|
||||||
|
from lerobot.jobs.annotate import build_pod_command, build_pod_setup, submit_annotate_to_hf
|
||||||
|
|
||||||
|
|
||||||
|
def _parse(*args):
|
||||||
|
return draccus.parse(AnnotationPipelineConfig, args=list(args))
|
||||||
|
|
||||||
|
|
||||||
|
def _set_argv(monkeypatch, *args):
|
||||||
|
monkeypatch.setattr(sys, "argv", ["lerobot-annotate", *args])
|
||||||
|
|
||||||
|
|
||||||
|
# --- config ----------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_annotation_job_defaults_are_local_with_vllm_image():
|
||||||
|
cfg = AnnotationJobConfig()
|
||||||
|
assert cfg.target is None
|
||||||
|
assert cfg.is_remote is False
|
||||||
|
assert cfg.image == DEFAULT_ANNOTATE_JOB_IMAGE
|
||||||
|
assert cfg.timeout == "2h"
|
||||||
|
assert cfg.lerobot_ref == "main"
|
||||||
|
|
||||||
|
|
||||||
|
def test_annotation_config_parses_job_target():
|
||||||
|
cfg = _parse("--repo_id", "u/d", "--job.target", "h200")
|
||||||
|
assert cfg.job.target == "h200"
|
||||||
|
assert cfg.job.is_remote is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_annotation_config_defaults_to_local():
|
||||||
|
assert _parse("--repo_id", "u/d").job.is_remote is False
|
||||||
|
|
||||||
|
|
||||||
|
# --- pod command -----------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_pod_setup_installs_requested_ref():
|
||||||
|
setup = build_pod_setup("my-branch")
|
||||||
|
assert "git+https://github.com/huggingface/lerobot.git@my-branch" in setup
|
||||||
|
# The vLLM image has neither ffmpeg (video decode) nor lerobot's pinned deps.
|
||||||
|
assert "ffmpeg" in setup
|
||||||
|
assert "'draccus==0.10.0'" in setup
|
||||||
|
|
||||||
|
|
||||||
|
def _annotate_argv(command):
|
||||||
|
"""Extract the `lerobot-annotate ...` argv from a `bash -c` pod command."""
|
||||||
|
assert command[:2] == ["bash", "-c"]
|
||||||
|
_setup, _, annotate = command[2].rpartition(" && ")
|
||||||
|
return shlex.split(annotate)
|
||||||
|
|
||||||
|
|
||||||
|
def test_pod_command_forwards_user_flags_and_pins_local_target():
|
||||||
|
command = build_pod_command(
|
||||||
|
"u/d",
|
||||||
|
"main",
|
||||||
|
["--repo_id=u/d", "--new_repo_id=u/d_annotated", "--push_to_hub=true", "--job.target=h200"],
|
||||||
|
)
|
||||||
|
argv = _annotate_argv(command)
|
||||||
|
assert argv[0] == "lerobot-annotate"
|
||||||
|
# --job.* is client-side orchestration; the pod must not re-dispatch itself.
|
||||||
|
assert not any(a.startswith("--job.") for a in argv[1:-1])
|
||||||
|
assert argv[-1] == "--job.target=local"
|
||||||
|
assert "--new_repo_id=u/d_annotated" in argv
|
||||||
|
assert "--push_to_hub=true" in argv
|
||||||
|
|
||||||
|
|
||||||
|
def test_pod_command_replaces_host_local_root_with_repo_id():
|
||||||
|
"""--root points at a directory only the client has; the pod resolves by repo_id."""
|
||||||
|
command = build_pod_command("u/d", "main", ["--root", "/home/me/datasets/d", "--seed=7"])
|
||||||
|
argv = _annotate_argv(command)
|
||||||
|
assert "--root" not in argv
|
||||||
|
assert "/home/me/datasets/d" not in argv
|
||||||
|
assert argv.count("--repo_id=u/d") == 1
|
||||||
|
assert "--seed=7" in argv
|
||||||
|
|
||||||
|
|
||||||
|
def test_pod_command_does_not_duplicate_repo_id():
|
||||||
|
command = build_pod_command("u/d", "main", ["--repo_id", "u/d"])
|
||||||
|
assert _annotate_argv(command).count("--repo_id=u/d") == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_pod_command_quotes_flags_containing_spaces_and_json():
|
||||||
|
"""serve_command and chat_template_kwargs must survive the trip through `bash -c`."""
|
||||||
|
serve = "--vlm.serve_command=vllm serve Qwen/Qwen3.6-27B --max-model-len 32768 --port {port}"
|
||||||
|
kwargs = '--vlm.chat_template_kwargs={"enable_thinking": false}'
|
||||||
|
command = build_pod_command("u/d", "main", [serve, kwargs])
|
||||||
|
argv = _annotate_argv(command)
|
||||||
|
assert serve in argv
|
||||||
|
assert kwargs in argv
|
||||||
|
|
||||||
|
|
||||||
|
# --- submission ------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_submit_requires_login(monkeypatch):
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: None)
|
||||||
|
with pytest.raises(RuntimeError, match="hf auth login"):
|
||||||
|
submit_annotate_to_hf(_parse("--repo_id", "u/d", "--job.target", "h200"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_submit_requires_repo_id(monkeypatch):
|
||||||
|
"""A remote run over --root alone can't work: the pod can't see the client's disk."""
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||||
|
cfg = _parse("--root", "/tmp/d", "--job.target", "h200")
|
||||||
|
with pytest.raises(ValueError, match="--repo_id"):
|
||||||
|
submit_annotate_to_hf(cfg)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("arg", ["--config_path=annotate.yaml", "--vlm=vlm.yaml", "--job=job.yaml"])
|
||||||
|
def test_submit_rejects_local_config_files(monkeypatch, arg):
|
||||||
|
"""draccus takes a config file for the whole config and for each nested one; the
|
||||||
|
pod can read none of them, so a remote run must refuse rather than drop them."""
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||||
|
_set_argv(monkeypatch, arg, "--job.target=h200")
|
||||||
|
cfg = _parse("--repo_id", "u/d", "--job.target", "h200")
|
||||||
|
with pytest.raises(ValueError, match="cannot read config files"):
|
||||||
|
submit_annotate_to_hf(cfg)
|
||||||
|
|
||||||
|
|
||||||
|
def test_pod_command_drops_bare_job_config_file_arg():
|
||||||
|
"""`--job` isn't caught by the `--job.` prefix, and could carry a remote target
|
||||||
|
that would make the pod submit a job of its own — recursively."""
|
||||||
|
argv = _annotate_argv(build_pod_command("u/d", "main", ["--job", "job.yaml", "--seed=7"]))
|
||||||
|
assert "--job" not in argv
|
||||||
|
assert "job.yaml" not in argv
|
||||||
|
assert argv[-1] == "--job.target=local"
|
||||||
|
|
||||||
|
|
||||||
|
def test_submit_dispatches_job(monkeypatch):
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.HfApi", lambda token=None: MagicMock())
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.ensure_dataset_available", lambda *a, **kw: None)
|
||||||
|
|
||||||
|
run_job_calls = []
|
||||||
|
|
||||||
|
def fake_run_job(**kwargs):
|
||||||
|
run_job_calls.append(kwargs)
|
||||||
|
return MagicMock(id="job-123")
|
||||||
|
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.run_job", fake_run_job)
|
||||||
|
_set_argv(monkeypatch, "--repo_id=u/d", "--push_to_hub=true", "--job.target=h200", "--job.detach=true")
|
||||||
|
|
||||||
|
cfg = _parse("--repo_id", "u/d", "--push_to_hub", "true", "--job.target", "h200", "--job.detach", "true")
|
||||||
|
submit_annotate_to_hf(cfg)
|
||||||
|
|
||||||
|
assert len(run_job_calls) == 1
|
||||||
|
call = run_job_calls[0]
|
||||||
|
assert call["flavor"] == "h200"
|
||||||
|
assert call["image"] == DEFAULT_ANNOTATE_JOB_IMAGE
|
||||||
|
assert call["timeout"] == "2h"
|
||||||
|
# The Hub token is forwarded so the pod can pull a private dataset and push the result.
|
||||||
|
assert call["secrets"]["HF_TOKEN"] == "tok"
|
||||||
|
assert call["labels"].get("lerobot") == "true"
|
||||||
|
argv = _annotate_argv(call["command"])
|
||||||
|
assert argv[0] == "lerobot-annotate"
|
||||||
|
assert "--push_to_hub=true" in argv
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.timeout(15)
|
||||||
|
def test_submit_follows_job_to_completion(monkeypatch, capsys):
|
||||||
|
"""Non-detach path must stream logs and RETURN (not hang) once the job is terminal.
|
||||||
|
|
||||||
|
Exercises the `follow_job` helper shared with the training submitter from the
|
||||||
|
annotation side, which is why the job-state patches target `lerobot.jobs.hf`.
|
||||||
|
Asserting on the completion message and not merely on "didn't hang" is what makes
|
||||||
|
this fail if `follow_job` ever reports detached-without-a-verdict instead.
|
||||||
|
"""
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.HfApi", lambda token=None: MagicMock())
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.ensure_dataset_available", lambda *a, **kw: None)
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.run_job", lambda **kw: MagicMock(id="job-1", url="http://x"))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"lerobot.jobs.hf.inspect_job",
|
||||||
|
lambda job_id: MagicMock(status=MagicMock(stage=MagicMock(value="COMPLETED"), message=None)),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr("lerobot.jobs.hf.fetch_job_logs", lambda job_id, follow=True: iter(()))
|
||||||
|
_set_argv(monkeypatch, "--repo_id=u/d", "--job.target=h200")
|
||||||
|
|
||||||
|
submit_annotate_to_hf(_parse("--repo_id", "u/d", "--push_to_hub", "true", "--job.target", "h200"))
|
||||||
|
assert "Annotation complete" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.timeout(15)
|
||||||
|
def test_submit_raises_when_job_fails(monkeypatch):
|
||||||
|
"""A job that ends in a non-COMPLETED stage must surface as an error, not a silent return."""
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.HfApi", lambda token=None: MagicMock())
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.ensure_dataset_available", lambda *a, **kw: None)
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.run_job", lambda **kw: MagicMock(id="job-1", url=None))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"lerobot.jobs.hf.inspect_job",
|
||||||
|
lambda job_id: MagicMock(status=MagicMock(stage=MagicMock(value="ERROR"), message="Job timeout")),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr("lerobot.jobs.hf.fetch_job_logs", lambda job_id, follow=True: iter(()))
|
||||||
|
_set_argv(monkeypatch, "--repo_id=u/d", "--job.target=h200")
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="stage=ERROR .Job timeout."):
|
||||||
|
submit_annotate_to_hf(_parse("--repo_id", "u/d", "--job.target", "h200"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_submit_ensures_dataset_is_on_the_hub(monkeypatch):
|
||||||
|
"""A local-only dataset is pushed (privately) before the job can reach it by repo_id."""
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.get_token", lambda: "tok")
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.HfApi", lambda token=None: MagicMock())
|
||||||
|
monkeypatch.setattr("lerobot.jobs.annotate.run_job", lambda **kw: MagicMock(id="job-1"))
|
||||||
|
|
||||||
|
seen = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"lerobot.jobs.annotate.ensure_dataset_available",
|
||||||
|
lambda repo_id, *, api, tags=None: seen.append((repo_id, tags)),
|
||||||
|
)
|
||||||
|
_set_argv(monkeypatch, "--repo_id=u/d", "--job.target=h200", "--job.detach=true")
|
||||||
|
|
||||||
|
submit_annotate_to_hf(
|
||||||
|
_parse("--repo_id", "u/d", "--job.target", "h200", "--job.detach", "true", "--job.tags", '["lelab"]')
|
||||||
|
)
|
||||||
|
assert seen == [("u/d", ["lerobot", "lelab"])]
|
||||||
@@ -29,12 +29,26 @@ from lerobot.jobs.hf import (
|
|||||||
_poll_until_done,
|
_poll_until_done,
|
||||||
build_remote_config_file,
|
build_remote_config_file,
|
||||||
build_repo_id,
|
build_repo_id,
|
||||||
|
follow_job,
|
||||||
resolve_job_tags,
|
resolve_job_tags,
|
||||||
resolve_wandb_api_key,
|
resolve_wandb_api_key,
|
||||||
submit_to_hf,
|
submit_to_hf,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_follow_job_detach_returns_without_watching(monkeypatch):
|
||||||
|
"""`detach` must short-circuit before any polling or log streaming starts."""
|
||||||
|
|
||||||
|
def _boom(*a, **kw):
|
||||||
|
raise AssertionError("detach must not touch the job")
|
||||||
|
|
||||||
|
monkeypatch.setattr("lerobot.jobs.hf.inspect_job", _boom)
|
||||||
|
monkeypatch.setattr("lerobot.jobs.hf.fetch_job_logs", _boom)
|
||||||
|
# False = "stopped watching without a verdict", so callers stay quiet rather than
|
||||||
|
# claiming success for a job that is still running.
|
||||||
|
assert follow_job("job-1", detach=True) is False
|
||||||
|
|
||||||
|
|
||||||
def test_resolve_job_tags_always_includes_lerobot_and_dedups():
|
def test_resolve_job_tags_always_includes_lerobot_and_dedups():
|
||||||
assert resolve_job_tags(None) == ["lerobot"]
|
assert resolve_job_tags(None) == ["lerobot"]
|
||||||
assert resolve_job_tags([]) == ["lerobot"]
|
assert resolve_job_tags([]) == ["lerobot"]
|
||||||
|
|||||||
@@ -405,12 +405,18 @@ def test_record_ranges_of_motion(mock_motors, dummy_motors):
|
|||||||
read_pos_stub = mock_motors.build_sequential_sync_read_stub(
|
read_pos_stub = mock_motors.build_sequential_sync_read_stub(
|
||||||
*X_SERIES_CONTROL_TABLE["Present_Position"], positions
|
*X_SERIES_CONTROL_TABLE["Present_Position"], positions
|
||||||
)
|
)
|
||||||
with patch("lerobot.motors.motors_bus.enter_pressed", side_effect=[False, True]):
|
|
||||||
bus = DynamixelMotorsBus(port=mock_motors.port, motors=dummy_motors)
|
bus = DynamixelMotorsBus(port=mock_motors.port, motors=dummy_motors)
|
||||||
bus.connect(handshake=False)
|
bus.connect(handshake=False)
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch("lerobot.motors.motors_bus.enter_pressed", side_effect=[False, True]),
|
||||||
|
patch("lerobot.motors.motors_bus.time.sleep") as mock_sleep,
|
||||||
|
patch.object(bus, "sync_read", wraps=bus.sync_read) as mock_sync_read,
|
||||||
|
):
|
||||||
mins, maxes = bus.record_ranges_of_motion(display_values=False)
|
mins, maxes = bus.record_ranges_of_motion(display_values=False)
|
||||||
|
|
||||||
assert mock_motors.stubs[read_pos_stub].calls == 3
|
assert mock_motors.stubs[read_pos_stub].calls == 3
|
||||||
|
assert all(call.kwargs["num_retry"] == 5 for call in mock_sync_read.call_args_list)
|
||||||
|
mock_sleep.assert_called_once_with(0.02)
|
||||||
assert mins == expected_mins
|
assert mins == expected_mins
|
||||||
assert maxes == expected_maxes
|
assert maxes == expected_maxes
|
||||||
|
|||||||
@@ -509,12 +509,18 @@ def test_record_ranges_of_motion(mock_motors, dummy_motors):
|
|||||||
stub = mock_motors.build_sequential_sync_read_stub(
|
stub = mock_motors.build_sequential_sync_read_stub(
|
||||||
*STS_SMS_SERIES_CONTROL_TABLE["Present_Position"], positions
|
*STS_SMS_SERIES_CONTROL_TABLE["Present_Position"], positions
|
||||||
)
|
)
|
||||||
with patch("lerobot.motors.motors_bus.enter_pressed", side_effect=[False, True]):
|
|
||||||
bus = FeetechMotorsBus(port=mock_motors.port, motors=dummy_motors)
|
bus = FeetechMotorsBus(port=mock_motors.port, motors=dummy_motors)
|
||||||
bus.connect(handshake=False)
|
bus.connect(handshake=False)
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch("lerobot.motors.motors_bus.enter_pressed", side_effect=[False, True]),
|
||||||
|
patch("lerobot.motors.motors_bus.time.sleep") as mock_sleep,
|
||||||
|
patch.object(bus, "sync_read", wraps=bus.sync_read) as mock_sync_read,
|
||||||
|
):
|
||||||
mins, maxes = bus.record_ranges_of_motion(display_values=False)
|
mins, maxes = bus.record_ranges_of_motion(display_values=False)
|
||||||
|
|
||||||
assert mock_motors.stubs[stub].calls == 3
|
assert mock_motors.stubs[stub].calls == 3
|
||||||
|
assert all(call.kwargs["num_retry"] == 5 for call in mock_sync_read.call_args_list)
|
||||||
|
mock_sleep.assert_called_once_with(0.02)
|
||||||
assert mins == expected_mins
|
assert mins == expected_mins
|
||||||
assert maxes == expected_maxes
|
assert maxes == expected_maxes
|
||||||
|
|||||||
@@ -109,3 +109,22 @@ def test_send_action(follower):
|
|||||||
|
|
||||||
goal_pos = {m: (i + 1) * 10 for i, m in enumerate(follower.bus.motors)}
|
goal_pos = {m: (i + 1) * 10 for i, m in enumerate(follower.bus.motors)}
|
||||||
follower.bus.sync_write.assert_called_once_with("Goal_Position", goal_pos)
|
follower.bus.sync_write.assert_called_once_with("Goal_Position", goal_pos)
|
||||||
|
|
||||||
|
|
||||||
|
def test_configure_writes_position_pid_coefficients():
|
||||||
|
bus_mock = _make_bus_mock()
|
||||||
|
bus_mock.motors = ["shoulder_pan"]
|
||||||
|
robot = MagicMock()
|
||||||
|
robot.bus = bus_mock
|
||||||
|
robot.config = SO100FollowerConfig(
|
||||||
|
port="/dev/null",
|
||||||
|
position_p_coefficient=32,
|
||||||
|
position_i_coefficient=1,
|
||||||
|
position_d_coefficient=16,
|
||||||
|
)
|
||||||
|
|
||||||
|
SO100Follower.configure(robot)
|
||||||
|
|
||||||
|
bus_mock.write.assert_any_call("P_Coefficient", "shoulder_pan", 32)
|
||||||
|
bus_mock.write.assert_any_call("I_Coefficient", "shoulder_pan", 1)
|
||||||
|
bus_mock.write.assert_any_call("D_Coefficient", "shoulder_pan", 16)
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# Copyright 2026 The HuggingFace Inc. team. All rights reserved.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import lerobot.scripts.lerobot_setup_motors as motors_module
|
||||||
|
|
||||||
|
|
||||||
|
def test_main_registers_plugins_before_parsing(monkeypatch):
|
||||||
|
calls = []
|
||||||
|
monkeypatch.setattr(motors_module, "register_third_party_plugins", lambda: calls.append("register"))
|
||||||
|
monkeypatch.setattr(motors_module, "setup_motors", lambda: calls.append("setup"))
|
||||||
|
|
||||||
|
motors_module.main()
|
||||||
|
|
||||||
|
assert calls == ["register", "setup"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_setup_motors_accepts_third_party_device(monkeypatch):
|
||||||
|
device = MagicMock()
|
||||||
|
monkeypatch.setattr(motors_module, "make_teleoperator_from_config", lambda _: device)
|
||||||
|
cfg = SimpleNamespace(device=SimpleNamespace(type="third_party"))
|
||||||
|
|
||||||
|
motors_module.setup_motors.__wrapped__(cfg)
|
||||||
|
|
||||||
|
device.setup_motors.assert_called_once_with()
|
||||||
|
|
||||||
|
|
||||||
|
def test_setup_motors_reports_unsupported_device(monkeypatch):
|
||||||
|
device = object()
|
||||||
|
monkeypatch.setattr(motors_module, "make_teleoperator_from_config", lambda _: device)
|
||||||
|
cfg = SimpleNamespace(device=SimpleNamespace(type="third_party"))
|
||||||
|
|
||||||
|
with pytest.raises(NotImplementedError, match="third_party"):
|
||||||
|
motors_module.setup_motors.__wrapped__(cfg)
|
||||||
Reference in New Issue
Block a user