Harden production dataset streaming pipeline

2026-07-28 20:26:05 +00:00 · 2026-07-27 16:07:48 +02:00
parent c0e9b0bbff
commit 79cfb52a71
14 changed files with 397 additions and 56 deletions
@@ -178,7 +178,9 @@ To capture comparable data-pipeline results on a training host:
 uv run python scripts/bench_streaming_dataset.py \
  --repo-id=yaak-ai/L2D-v3 \
  --batch-size=16 \
-  --num-workers=4 \
+  --num-workers=1 \
+  --fetch-workers=4 \
+  --decode-threads=2 \
  --summary-json=streaming-benchmark.json
 ```