mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-30 13:09:40 +00:00
feat(dataset filtering): adding support for VLM based dataset filtering following lerobot annotation pipeline style
This commit is contained in:
@@ -239,6 +239,56 @@ Every module is on by default and can be toggled independently (set to
|
||||
| `--vqa.restrict_to_default_camera` | `false` | Ground VQA only on `--vlm.camera_key` (else every camera). |
|
||||
| `--executor.episode_parallelism` | `16` | Episodes processed concurrently within each phase. |
|
||||
|
||||
## Camera-view curation
|
||||
|
||||
`lerobot-curate-cameras` is a separate, lightweight command that uses the same
|
||||
VLM backend for a **dataset-filtering / curation** pass. It downloads only the
|
||||
**first episode**, then for each camera view asks the VLM to:
|
||||
|
||||
1. **flag** whether the view is blurry / unusable, and
|
||||
2. **label** the view with a canonical name from a closed vocabulary
|
||||
(`top`, `wrist`, `front`, `bottom`, `left`, `right`, plus two-word combos
|
||||
like `left_wrist`).
|
||||
|
||||
It runs in one of two modes:
|
||||
|
||||
- `--mode=report` (default) — write the labels + verdicts into `meta/`
|
||||
(`meta/camera_curation.json` and a `curation` block on each camera in
|
||||
`meta/info.json`). Nothing is moved; this is the cheap triage pass and works
|
||||
for any dataset.
|
||||
- `--mode=rename` — apply the labels by renaming each camera key to
|
||||
`observation.images.<label>`. For **video** datasets this is a
|
||||
**download-free, server-side Hub commit**: the `videos/<key>/` files are moved
|
||||
with the Hub's LFS copy/delete (no video is downloaded or re-encoded), and only
|
||||
the small `meta/` files are edited.
|
||||
|
||||
```bash
|
||||
# Cheap, mutation-free triage (writes meta/camera_curation.json):
|
||||
uv run lerobot-curate-cameras --repo_id=user/dataset --mode=report
|
||||
|
||||
# Apply the labels by renaming camera keys on a new branch (keeps `main` intact):
|
||||
uv run lerobot-curate-cameras --repo_id=user/dataset --mode=rename --branch=curated
|
||||
|
||||
# Run the VLM decision on a GPU via HF Jobs (same --job.* flags as above):
|
||||
uv run lerobot-curate-cameras --repo_id=user/dataset --mode=rename --job.target=h200
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- The Hub rename is **in place** on the source repo — the Hub does not support
|
||||
cross-repo LFS copies. Use `--branch` to commit to a branch so `main` is
|
||||
preserved.
|
||||
- **Image** datasets store frames inside the data parquet, so their rename can't
|
||||
avoid touching the data; the rename falls back to a local rewrite (via
|
||||
[`rename_features`](./using_dataset_tools#rename-features)). Prefer `--mode=report`
|
||||
for image datasets.
|
||||
- Views judged unusable are only flagged by default (still renamed). Pass
|
||||
`--drop_unusable=true` (local path) to remove them.
|
||||
|
||||
Key options: `--mode`, `--branch`, `--n_frames`, `--view_vocabulary`,
|
||||
`--allow_combos`, `--on_collision`, `--drop_unusable`, and the shared
|
||||
`--vlm.*` / `--job.*` flags documented above.
|
||||
|
||||
## Contributing new modules
|
||||
|
||||
The pipeline is built to grow, and **contributions are very welcome** —
|
||||
|
||||
@@ -89,6 +89,28 @@ lerobot-edit-dataset \
|
||||
--operation.feature_names "['observation.images.top']"
|
||||
```
|
||||
|
||||
#### Rename Features
|
||||
|
||||
Rename feature keys — typically to canonicalize camera views (e.g.
|
||||
`observation.images.cam_0` → `observation.images.left_wrist`). A rename changes
|
||||
no pixel data, so it is a cheap key-remap: it rewrites `meta/` (info features,
|
||||
episode `videos/*` and `stats/*` columns, `stats.json`), moves the
|
||||
`videos/<key>/` directory, and — for image datasets — renames the embedded
|
||||
image column. Videos are **not** re-encoded.
|
||||
|
||||
```bash
|
||||
# Rename one or more camera keys
|
||||
lerobot-edit-dataset \
|
||||
--repo_id lerobot/pusht \
|
||||
--operation.type rename_features \
|
||||
--operation.name_mapping '{"observation.images.cam_0": "observation.images.left_wrist"}'
|
||||
```
|
||||
|
||||
If two targets collide (e.g. two cameras both labeled `top`), the operation
|
||||
raises by default; pass `--operation.on_collision suffix` to disambiguate
|
||||
deterministically (`top`, `top_2`, …). To label camera views automatically with
|
||||
a VLM, see the [Annotation Pipeline](./annotation_pipeline#camera-view-curation).
|
||||
|
||||
#### Convert to Video
|
||||
|
||||
Convert an image-based dataset to video format, creating a new LeRobotDataset where images are stored as videos. This is useful for reducing storage requirements and improving data loading performance. The new dataset will have the exact same structure as the original, but with images encoded as MP4 videos in the proper LeRobot format.
|
||||
|
||||
Reference in New Issue
Block a user