One-flag A/B for the fetch-concurrency fix on the cluster benchmark.
Component-level A/B against the Hub (old EpisodeByteCache @7b6f4f2b1 vs
new, decoder stubbed, 16 episodes x 3 cameras x 6 MB slices of a 518 MB
file, both run orders to bound CDN warmth):
OLD (4 workers, sequential cams): 22.2 / 24.3 MiB/s
NEW (16 workers, parallel cams): 47.3 / 45.1 MiB/s (~2.0x)
on a residential link; the cluster needed 1.44x (465 -> 670 MiB/s
aggregate) to hit its 1000 samples/s target.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reshard parquet per row group (1 shard == 1 row group == 1 episode) and feed the
episode-pool shuffle with max_buffer_input_shards so the pool is a uniform random
sample of the corpus, independent of episodes-per-file. Add validate_row_groups
guardrails (collapsed-row-group + distributed divisibility), require datasets>=5.0.0,
make the test fixture write one row group per episode, and plumb max_buffer_input_shards
through the dataloading benchmark.
Co-authored-by: Cursor <cursoragent@cursor.com>
Single-file SLURM-oriented benchmark comparing the map-style and native
streaming loaders on single-image samples: a self-submitting serial chain
that measures peak RSS, samples/s (and decoded frames/s), fetch-vs-decode
split, shuffle randomness, and p50/p95/p99 sample latency over a fixed
wall-clock window, including a 2-node split_dataset_by_node leg.
Co-authored-by: Cursor <cursoragent@cursor.com>
Pepijn