admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-28 20:26:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove streaming benchmark prototypes

Browse Source
This commit is contained in:
Pepijn
2026-07-28 11:14:10 +02:00
parent 4db03ed535
commit fdf6014214
7 changed files with 0 additions and 1742 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/lerobot-dataset-v3.mdx
-16
View File
@@ -172,22 +172,6 @@ lerobot-train \
Use `--dataset.streaming_data_root=hf://buckets/OWNER/BUCKET/PREFIX` when metadata lives in a
dataset repository but Parquet and MP4 data are mirrored in an HF Bucket.
To capture comparable data-pipeline results on a training host:
```bash
uv run python scripts/bench_streaming_dataset.py \
--repo-id=yaak-ai/L2D-v3 \
--batch-size=16 \
--num-workers=1 \
--fetch-workers=4 \
--decode-threads=2 \
--summary-json=streaming-benchmark.json
```
The JSON records the code and dataset revisions, initialization/first-batch time, steady-state
samples per second, batch-wait p50/p95, duplicate indices, and the exact cache/worker settings.
Run end-to-end `lerobot-train` separately to measure GPU utilization and full training step time.
The Python API uses the same implementation:
```python
docs/source/training_dataset_streaming.mdx
-26
View File
@@ -105,32 +105,6 @@ Streaming checkpoints created by the earlier multi-worker sampler record a diffe
topology and are rejected for sample-exact resume. Start a new run from the saved policy weights
rather than claiming that its dataset stream resumes exactly.
## Benchmarking
Measure the integrated path on the same hosts used for training:
```bash
uv run python scripts/bench_streaming_dataset.py \
--repo-id=OWNER/DATASET \
--revision=COMMIT_SHA \
--batch-size=16 \
--num-workers=1 \
--fetch-workers=4 \
--decode-threads=2 \
--decoded-queue-size=8 \
--summary-json=streaming-benchmark.json
```
The output records source revisions, startup and first-batch latency, steady-state throughput,
batch-wait percentiles, duplicate indices, memory high-water marks, and the exact settings. Run a
separate end-to-end training A/B to include policy compute, device transfer, and optimizer time.
Do not compare results unless the code revision, dataset revision, hardware, and settings match.
For lower-level byte-fetch, retry, decoder-open, and refill diagnostics, build the sidecar explicitly
and run `scripts/bench_episode_byte_cache.py` with the same `--repo-id`, `--revision`,
`--data-root`, and `--sidecar-path`. Treat this as a stage profiler; throughput claims should come
from the integrated dataset benchmark and end-to-end training A/B.
## Troubleshooting
- **The first batch takes a long time:** check logs for a sidecar build. Reuse a shared cache or