admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-28 12:15:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Derive streaming decoder cap from episode pool

Browse Source
This commit is contained in:
Pepijn
2026-07-27 20:49:19 +02:00
parent 79cfb52a71
commit 4db03ed535
6 changed files with 59 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/training_dataset_streaming.mdx
+11 -12
View File
@@ -58,17 +58,17 @@ Subset sidecars cannot be published.
The production defaults are:
| Option | Default | Meaning |
| ----------------------------------- | ------: | --------------------------------------------------- |
| `streaming_episode_pool_size` | 32 | Maximum complete episodes mixed by each rank |
| `streaming_prefetch_episodes` | 8 | Episodes fetched ahead of the active pool |
| `streaming_byte_budget_gb` | 8 | Maximum synthesized MP4 bytes per rank |
| `streaming_decode_threads` | 2 | Parallel sample assembly and video decode workers |
| `streaming_decoded_queue_size` | 8 | Decoded samples buffered ahead, in planner order |
| `streaming_max_open_decoders` | 64 | Independent open-decoder LRU cap per rank |
| `streaming_native_http_connections` | unset | Native HTTP connection cap per rank |
| `streaming_native_http_subranges` | 1 | Concurrent subranges per native HTTP range read |
| `streaming_data_root` | unset | Optional local, Hub, bucket, or fsspec payload root |
| Option | Default | Meaning |
| ----------------------------------- | -------------: | --------------------------------------------------- |
| `streaming_episode_pool_size` | 32 | Maximum complete episodes mixed by each rank |
| `streaming_prefetch_episodes` | 8 | Episodes fetched ahead of the active pool |
| `streaming_byte_budget_gb` | 8 | Maximum synthesized MP4 bytes per rank |
| `streaming_decode_threads` | 2 | Parallel sample assembly and video decode workers |
| `streaming_decoded_queue_size` | 8 | Decoded samples buffered ahead, in planner order |
| `streaming_max_open_decoders` | pool × cameras | Independent open-decoder LRU cap per rank |
| `streaming_native_http_connections` | unset | Native HTTP connection cap per rank |
| `streaming_native_http_subranges` | 1 | Concurrent subranges per native HTTP range read |
| `streaming_data_root` | unset | Optional local, Hub, bucket, or fsspec payload root |
The active episode set is capped by both episode count and the exact synthesized mini-MP4 sizes
computed from the sidecar. An episode larger than the complete rank budget fails before training
@@ -84,7 +84,6 @@ lerobot-train \
--dataset.streaming_byte_budget_gb=4 \
--dataset.streaming_decode_threads=2 \
--dataset.streaming_decoded_queue_size=8 \
--dataset.streaming_max_open_decoders=64 \
--num_workers=4 \
--policy.type=act \
--output_dir=outputs/train/act_streaming