Commit Graph

10 Commits

Author SHA1 Message Date
CarolinePascal 96d5c3258a chore(format): formatting code 2026-08-06 13:20:22 +02:00
CarolinePascal d7a460d80c tests(simplifications): further simplifying tests, reducing docstrings size and making them clearer. 2026-08-06 13:20:22 +02:00
CarolinePascal f84495d206 feat(tests): simplifying streaming dataset tests with assert_frame_matches() 2026-08-06 13:20:22 +02:00
dongmao.zhang e2804c9fbd test(datasets): support the multi-file video layout in dataset fixtures
Add `episodes_per_video_file` to `episodes_factory`: episode `i` goes to
`file_index = i // episodes_per_video_file` and `from_timestamp` restarts at 0 on
each rollover. `create_videos` encodes one .mp4 per file, and
`mock_snapshot_download` lists and creates every video file rather than only
file-000. Without the option the fixtures emit the same single-file layout as
before.

Add streaming tests over that layout, on the plain and the delta path.
2026-08-06 13:20:22 +02:00
Steven Palma 0d0737ab57 feat(dataset): Support streaming from HF Storage Buckets + Bump HF hub & datasets (#4236) 2026-07-31 01:18:51 +02:00
Steven Palma fbe8f5c9da fix(datasets): stop frame errors being treated as shard exhaustion in StreamingLeRobotDataset (#4237)
* fix(datasets): stop frame errors being treated as shard exhaustion in StreamingLeRobotDataset

StreamingLeRobotDataset.__iter__ caught every RuntimeError and treated all as exhausted shard. Real errors like video decode failure made each shard get dropped on the first frame, so iteration ended while yeilding zero frames with no errors.

Shard exahustion is StopIteration raised from make_frame generator, which python converts to RuntimeError with StopIteration as __cause__. Added check to tell StopIteration from everything else, consuming real shard exhaustion while re-raising everything else.

Added a test that injects a decode failure and asserts iteration raises instead of returning on empty stream.

Fixes #4066

* refactor(datasets): exception streaming

---------

Co-authored-by: Mohit Yadav <mohitydv09@gmail.com>
2026-07-30 16:19:37 +02:00
Steven Palma 0d383d09f2 feat(dataset): accept token argument for private HF Hub datasets (#4136) 2026-07-24 18:51:35 +02:00
Steven Palma df0763a2bc feat(dependencies): minimal default tag install (#3362) 2026-04-12 20:03:04 +02:00
Steven Palma d2782cf66b chore: replace hard-coded action values with constants throughout all the source code (#2055)
* chore: replace hard-coded 'action' values with constants throughout all the source code

* chore(tests): replace hard-coded action values with constants throughout all the test code
2025-09-26 13:33:18 +02:00
Francesco Capuano 33cad37054 Add Streaming Dataset (#1613)
Co-authored-by: Michel Aractingi <michel.aractingi@huggingface.co>
2025-09-15 14:08:01 +02:00