Predrag Cvetkovic 35339d31e5 fix(datasets): bound memory of augment_dataset_quantile_stats by sampling frames (#3749) ...

* fix(datasets): bound memory of augment_dataset_quantile_stats by sampling frames

Per-episode stats previously materialized every frame (and decoded up to 16
episodes in parallel), so peak memory scaled with episode length and OOM'd on
large datasets (#2889). Numeric features are now read in full from the table
(exact), while only image/video frames are sub-sampled per episode using the
existing sample_indices heuristic. Worker count is configurable via
LEROBOT_STATS_MAX_WORKERS; --no-sampling restores exact behavior.

* Update tests/datasets/test_augment_quantile_stats.py

Co-authored-by: Haoming Song <1847575517@qq.com>
Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>

---------

Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
Co-authored-by: Haoming Song <1847575517@qq.com>

2026-07-29 12:32:03 +02:00

..

test_aggregate.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_augment_quantile_stats.py

fix(datasets): bound memory of augment_dataset_quantile_stats by sampling frames (#3749)

2026-07-29 12:32:03 +02:00

test_compute_stats.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_dataset_metadata.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_dataset_reader.py

feat(encoding parameters): adding support for user provided video encoding parameters (#3455)

2026-05-14 23:46:42 +02:00

test_dataset_tools.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_dataset_utils.py

feat(dataset): accept token argument for private HF Hub datasets (#4136)

2026-07-24 18:51:35 +02:00

test_dataset_writer.py

fix(datasets): clear video frame staging on episode reset (#3683)

2026-07-27 13:44:32 +02:00

test_datasets.py

feat(dataset): add slice support to LeRobotDataset.__getitem__ (#4129)

2026-07-23 22:05:29 +02:00

test_delta_timestamps.py

feat(dependencies): minimal default tag install (#3362)

2026-04-12 20:03:04 +02:00

test_depth.py

fix(depth unit): adding input depth unit storage in the dataset metadata (#3899)

2026-07-02 11:53:13 +02:00

test_image_transforms.py

feat(dependencies): minimal default tag install (#3362)

2026-04-12 20:03:04 +02:00

test_image_writer.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_language_render.py

Add extensive language support (#3467)

2026-05-19 14:46:11 +02:00

test_language.py

Add extensive language support (#3467)

2026-05-19 14:46:11 +02:00

test_lerobot_dataset.py

Handle resuming empty local datasets (#3859)

2026-07-28 16:25:42 +02:00

test_quantiles_dataset_integration.py

feat(dependencies): minimal default tag install (#3362)

2026-04-12 20:03:04 +02:00

test_sampler.py

feat(datasets): deterministic, resumable shuffling for EpisodeAwareSampler (#3769)

2026-06-12 11:47:16 +02:00

test_streaming_video_encoder.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_streaming.py

feat(dataset): accept token argument for private HF Hub datasets (#4136)

2026-07-24 18:51:35 +02:00

test_video_decoder_cache.py

fix(datasets): bound VideoDecoderCache to prevent OOM on large datasets (#3614)

2026-05-19 16:54:25 +02:00

test_video_encoding.py

fix(save codec options): making sure codec options are always set via set_if (#3910)

2026-07-02 15:29:14 +02:00

test_visualize_dataset.py

feat(dependencies): minimal default tag install (#3362)

2026-04-12 20:03:04 +02:00