mirror of https://github.com/huggingface/lerobot.git synced 2026-07-07 18:11:50 +00:00

Files

T

Pepijn f7c8a526e8 feat(streaming): wallclock benchmark throughput, cross-worker cache stats, bucket source

- benchmark: frames_per_s_node now measures sustained wall-clock throughput over the
  post-warmup window. The previous metric summed inter-batch gaps, which collapse to ~0
  under async prefetch (consumer drains a pre-filled queue) and overstated throughput ~100x.
- VideoDecoderCache gains an optional shared [hits, misses, evictions] counter tensor;
  StreamingLeRobotDataset.video_decoder_cache_stats() aggregates it across DataLoader
  workers (lock-free, approximate; hit_rate preserved). Fixes empty cache stats with workers.
- StreamingLeRobotDataset.data_files_root: read bulk data/ + videos/ from an fsspec root
  (e.g. hf://buckets/<owner>/<name>) while metadata still loads from repo_id. Enables
  bucket / prewarmed-bucket benchmark sources without copying metadata. Exposed as
  benchmark --data_files_root.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-09 15:25:44 +02:00

benchmark_streaming.py

feat(streaming): wallclock benchmark throughput, cross-worker cache stats, bucket source

2026-06-09 15:25:44 +02:00

README.md

feat(streaming): multinode example, dataloading benchmark, distributed smoke test

2026-06-09 13:48:23 +02:00

README.md

Streaming dataloading benchmark

Measures dataloading only (no model) for StreamingLeRobotDataset: parquet read + video decode + delta windowing + shuffle. A dummy consumer pulls batches and moves them to the device, so the numbers isolate the data pipeline. Use it to compare sources (Hub vs. storage bucket vs. prewarmed bucket), frame modes, and node counts, and to catch p95/p99 video-decode regressions.

Run

python benchmarks/streaming/benchmark_streaming.py \
    --repo_id pepijn223/robocasa_pretrain_human300_v4 \
    --mode sarm --batch_size 64 --num_workers 12 --num_batches 200 \
    --source hub --out_dir benchmarks/streaming/results

Multinode (per-node throughput) goes through Accelerate under SLURM:

sbatch slurm/benchmark_streaming_robocasa.sh

Matrix

Axis	Values
Source	`hub` (verify now), `bucket`, `warmed_bucket` (bucket + prewarming; with user's help later)
Baseline	current `main` `StreamingLeRobotDataset` on Hub streaming
Nodes	1 and 2 (per-node throughput should be independent)
Frame mode	`single` (1 frame, all cameras; target ≥ 120 frames/s/node) · `sarm` (8 steps spaced 1s; target ≥ 320 frames/s/node)

--source is a label only; the actual source is whatever --repo_id / --root point at.

Metrics emitted (JSON + CSV)

frames_per_s_node, samples_per_s, first_batch_latency_s, p50/p95/p99_sample_latency_ms, wallclock_s, and video_decoder_cache (hits, misses, evictions, hit_rate, size). A low cache hit_rate with high p99 is the decoder-thrash signature — raise --video_decoder_cache_size or --buffer_size, or reduce num_workers.

Bucket sources & prewarming (manual)

Prewarming is a server-side Hugging Face storage-bucket feature — there is no client script. To benchmark the warmed_bucket source:

Attach a storage bucket to the dataset and enable it (see https://huggingface.co/docs/hub/storage-buckets). Buckets resolve through fsspec, the same as hf://, so no code change is needed — point --repo_id/--revision (or --root) at the bucket.
Enable prewarming in the bucket settings and wait for warm-up to complete.
Run the benchmark with --source warmed_bucket. Compare against the cold --source bucket and the --source hub baseline.

Manual only — not run in CI.