admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-28 12:15:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Harden production dataset streaming pipeline

Browse Source
This commit is contained in:
Pepijn
2026-07-27 16:07:48 +02:00
parent c0e9b0bbff
commit 79cfb52a71
14 changed files with 397 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/lerobot-dataset-v3.mdx
+3 -1
View File
@@ -178,7 +178,9 @@ To capture comparable data-pipeline results on a training host:
uv run python scripts/bench_streaming_dataset.py \
--repo-id=yaak-ai/L2D-v3 \
--batch-size=16 \
--num-workers=4 \
--num-workers=1 \
--fetch-workers=4 \
--decode-threads=2 \
--summary-json=streaming-benchmark.json
```
docs/source/training_dataset_streaming.mdx
+16 -6
View File
@@ -58,12 +58,17 @@ Subset sidecars cannot be published.
The production defaults are:
| Option | Default | Meaning |
| ----------------------------- | ------: | --------------------------------------------------- |
| `streaming_episode_pool_size` | 32 | Maximum complete episodes mixed by each rank |
| `streaming_prefetch_episodes` | 8 | Episodes fetched ahead of the active pool |
| `streaming_byte_budget_gb` | 8 | Maximum synthesized MP4 bytes per rank |
| `streaming_data_root` | unset | Optional local, Hub, bucket, or fsspec payload root |
| Option | Default | Meaning |
| ----------------------------------- | ------: | --------------------------------------------------- |
| `streaming_episode_pool_size` | 32 | Maximum complete episodes mixed by each rank |
| `streaming_prefetch_episodes` | 8 | Episodes fetched ahead of the active pool |
| `streaming_byte_budget_gb` | 8 | Maximum synthesized MP4 bytes per rank |
| `streaming_decode_threads` | 2 | Parallel sample assembly and video decode workers |
| `streaming_decoded_queue_size` | 8 | Decoded samples buffered ahead, in planner order |
| `streaming_max_open_decoders` | 64 | Independent open-decoder LRU cap per rank |
| `streaming_native_http_connections` | unset | Native HTTP connection cap per rank |
| `streaming_native_http_subranges` | 1 | Concurrent subranges per native HTTP range read |
| `streaming_data_root` | unset | Optional local, Hub, bucket, or fsspec payload root |
The active episode set is capped by both episode count and the exact synthesized mini-MP4 sizes
computed from the sidecar. An episode larger than the complete rank budget fails before training
@@ -77,6 +82,9 @@ lerobot-train \
--dataset.streaming_episode_pool_size=16 \
--dataset.streaming_prefetch_episodes=4 \
--dataset.streaming_byte_budget_gb=4 \
--dataset.streaming_decode_threads=2 \
--dataset.streaming_decoded_queue_size=8 \
--dataset.streaming_max_open_decoders=64 \
--num_workers=4 \
--policy.type=act \
--output_dir=outputs/train/act_streaming
@@ -109,6 +117,8 @@ uv run python scripts/bench_streaming_dataset.py \
--batch-size=16 \
--num-workers=1 \
--fetch-workers=4 \
--decode-threads=2 \
--decoded-queue-size=8 \
--summary-json=streaming-benchmark.json
```