mirror of
https://github.com/huggingface/lerobot.git
synced 2026-06-18 00:37:10 +00:00
test(sampler): drain resumed trillion-frame sampler via iter() to avoid list() prealloc
list(sampler) calls PyObject_LengthHint -> __len__ (the full 10**12 epoch length) and preallocates that many slots before iterating, OOMing even though the resumed epoch only yields 3 frames. Collect through the iterator (no length hint) so the test exercises the real O(1) seek/drain instead of CPython's list growth heuristic.
This commit is contained in:
@@ -246,7 +246,10 @@ def test_deterministic_sampler_constant_memory():
|
||||
sampler = deterministic_sampler([0], [num_frames], shuffle=True, seed=0)
|
||||
assert len(sampler) == num_frames
|
||||
sampler.load_state_dict({"epoch": 3, "start_index": num_frames - 3})
|
||||
tail = list(sampler)
|
||||
# Collect via the iterator: list(sampler) would call PyObject_LengthHint -> sampler.__len__
|
||||
# (the full epoch length, here 10**12) and pre-allocate that many slots before iterating. The
|
||||
# iterator itself exposes no length hint, so this stays O(1) like the resumed epoch it drains.
|
||||
tail = list(iter(sampler))
|
||||
assert len(tail) == 3
|
||||
assert all(0 <= idx < num_frames for idx in tail)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user