mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-06 09:37:06 +00:00
Pepijn
be64ded80f
The 64-vs-128-worker benchmark pair proved a per-host throughput ceiling (~270 MiB/s) on the HF bucket path: doubling connections exactly halved per-connection speed (4.8 -> 2.2 MiB/s) and left the aggregate flat, while per-episode latency doubled (5.7s -> 12s) and keep-up worsened. Steady-state demand (148 MiB/s) is well below the ceiling; the keep-up misses come entirely from consumer stalls (refill_wait 14-19s of ~84s): the sim blocks the training hot path on ensure_ready() for the FIFO-head replacement while episodes take 5.7-12s to arrive. Two fixes: - Non-blocking replacements: EpisodeByteCache.is_ready() (all cameras cached or futures done, no blocking) and the stream sim now swaps a replacement only when it is already resident, deferring otherwise; fetch capacity (~2x demand) repays the debt on later batches. A deferred_swaps metric is reported. - Sub-range parallel fetch (native-http): --range-subranges N splits one camera GET into N concurrent sub-range GETs. Under a per-host ceiling this adds no bandwidth but divides per-episode latency by ~N. Keep workers x subranges near the ~64-connection saturation point (e.g. --workers 16 --range-subranges 4). Verified: sub-range span math + order-preserving concat and is_ready semantics (unit-level, network stubbed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>