feat(streaming): exact-once epoch coverage for the byte-cache episode pool

The pool path sampled frames with replacement and never guaranteed a full
epoch (episodes rotated on a fixed cadence; frames drawn randomly, none
tracked). Add ExactCoveragePool: a deterministic planner that enumerates
every frame of every episode exactly once per epoch while keeping at most
pool_size episodes resident, so batch mixing stays high (uniform draw over
all remaining frames in the pool) but coverage is complete and reproducible.

Mechanics (the "evict only when all frames sampled" model): episodes are
admitted in a seeded global permutation; each resident episode carries a
seeded frame-index shuffle; each draw picks a resident episode with
probability proportional to its remaining frames and pops one; an episode
is evicted only when its last frame is emitted, then a new one is admitted;
the epoch ends when admission is exhausted and every resident episode drains.
Order is a pure function of (seed, epoch) -> resumable by deterministic
fast-forward. The planner does no I/O and exposes admission_order so callers
can prefetch episodes ahead of the sampling frontier.

Wired into the benchmark as --coverage {sampled,exact}: run_exact_coverage_stream
prefetches stream_prefetch_episodes beyond the frontier so a freshly admitted
episode's bytes are resident before it is drawn, then decodes each frame once,
paced to target.

Tests: 7 planner unit tests (exact-once coverage incl. a 45k-frame epoch,
pool-size bound, per-(seed,epoch) determinism with coverage preserved,
admission/eviction events, coupon-collector mixing, zero-length episodes) and
a mocked-cache structural test of run_exact_coverage_stream asserting
every-frame-once-per-camera plus the prefetch-before-decode invariant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Pepijn
2026-07-03 15:01:35 +02:00
parent be64ded80f
commit 06aa6a0425
3 changed files with 371 additions and 16 deletions
@@ -981,6 +981,108 @@ class EpisodeVideoManifest:
)
class ExactCoveragePool:
"""Deterministic, exactly-once frame coverage over a byte-cache episode pool.
The sampled/with-replacement pool (``run_pool_stream_simulation``) never guarantees a full
epoch: frames are drawn randomly and episodes rotate on a fixed cadence. This planner instead
enumerates *every frame of every episode exactly once per epoch* while keeping at most
``pool_size`` episodes resident, so batch mixing stays high but coverage is complete and
reproducible.
Mechanics (this is the "evict only when all frames sampled" model):
- Episodes are admitted in a seeded global permutation; the first ``pool_size`` fill the pool.
- Each resident episode carries a seeded shuffle of its own frame indices.
- Each draw picks a resident episode with probability proportional to its *remaining* frames
(i.e. a uniform draw over all remaining frames in the pool, the map-style ideal) and pops
one frame.
- An episode is evicted only when its last frame is emitted; a new episode is then admitted.
- The epoch ends when the admission order is exhausted and every resident episode is drained.
Newly admitted episodes are surfaced via :attr:`newly_admitted` (drain it to drive prefetch)
and evictions via :attr:`evicted` (drain to release cache bytes). The planner does no I/O and
is fully unit-testable. It yields ``(episode_index, frame_index)``; map to a decode timestamp
with ``frame_index / max(frame_count - 1, 1)``.
Determinism: the order is a pure function of ``(seed, epoch)`` and the episode->frame_count
map. Resume is a deterministic fast-forward: re-instantiate with the same seed/epoch and skip
``n`` samples (tabular only, no decode).
"""
def __init__(
self,
episode_frame_counts: Sequence[tuple[int, int]],
pool_size: int,
*,
seed: int,
epoch: int = 0,
):
self._counts = {int(ep): int(n) for ep, n in episode_frame_counts if int(n) > 0}
self._rng = np.random.default_rng([seed, epoch])
order = np.array(sorted(self._counts), dtype=np.int64)
self._rng.shuffle(order)
# Full admission order is exposed so a caller can prefetch episodes ahead of when they
# enter the sampling pool (a freshly admitted episode is immediately eligible to be drawn,
# so its bytes must already be resident).
self.admission_order: list[int] = order.tolist()
self._admit_cursor = 0
self._remaining: dict[int, list[int]] = {}
self._remaining_total = 0
self.newly_admitted: list[int] = []
self.evicted: list[int] = []
for _ in range(max(1, pool_size)):
self._admit_one()
def _admit_one(self) -> None:
if self._admit_cursor >= len(self.admission_order):
return
ep = self.admission_order[self._admit_cursor]
self._admit_cursor += 1
n = self._counts[ep]
frames = np.arange(n, dtype=np.int64)
self._rng.shuffle(frames)
self._remaining[ep] = frames.tolist()
self._remaining_total += n
self.newly_admitted.append(ep)
@property
def remaining_total(self) -> int:
return self._remaining_total
@property
def admitted_count(self) -> int:
"""Number of episodes pulled from the admission order so far (pool fills + rotations)."""
return self._admit_cursor
@property
def resident(self) -> list[int]:
return list(self._remaining)
def __iter__(self) -> "ExactCoveragePool":
return self
def __next__(self) -> tuple[int, int]:
if self._remaining_total == 0:
raise StopIteration
# Uniform draw over all remaining frames in the pool: walk the residents by cumulative
# remaining count. O(pool_size) per draw (~1024) -> negligible next to decode.
target = int(self._rng.integers(self._remaining_total))
chosen = None
for ep, frames in self._remaining.items():
if target < len(frames):
chosen = ep
break
target -= len(frames)
frames = self._remaining[chosen]
frame_index = frames.pop()
self._remaining_total -= 1
if not frames:
del self._remaining[chosen]
self.evicted.append(chosen)
self._admit_one()
return chosen, frame_index
class EpisodeByteCache:
def __init__(
self,