4940281120
feat(streaming): random-episode admission via reshard() + multi-input-shard shuffle
...
Reshard parquet per row group (1 shard == 1 row group == 1 episode) and feed the
episode-pool shuffle with max_buffer_input_shards so the pool is a uniform random
sample of the corpus, independent of episodes-per-file. Add validate_row_groups
guardrails (collapsed-row-group + distributed divisibility), require datasets>=5.0.0,
make the test fixture write one row group per episode, and plumb max_buffer_input_shards
through the dataloading benchmark.
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-06-15 13:33:27 +00:00
d90e4bcfd3
refactor(dataset): modular files ( #3171 )
...
* refactor(dataset): modular files
* refactor(dataset): update imports across the codebase
2026-03-15 23:58:09 -07:00
f55c6e89f0
Dataset v3 ( #1412 )
...
Co-authored-by: Simon Alibert <75076266+aliberts@users.noreply.github.com >
Co-authored-by: Remi Cadene <re.cadene@gmail.com >
Co-authored-by: Tavish <tavish9.chen@gmail.com >
Co-authored-by: fracapuano <francesco.capuano@huggingface.co >
Co-authored-by: CarolinePascal <caroline8.pascal@gmail.com >
2025-09-15 09:53:30 +02:00
d4ee470b00
Package folder structure ( #1417 )
...
* Move files
* Replace imports & paths
* Update relative paths
* Update doc symlinks
* Update instructions paths
* Fix imports
* Update grpc files
* Update more instructions
* Downgrade grpc-tools
* Update manifest
* Update more paths
* Update config paths
* Update CI paths
* Update bandit exclusions
* Remove walkthrough section
2025-07-01 16:34:46 +02:00
5d24ce3160
chore(doc): add license header to all files ( #818 )
2025-03-05 17:56:51 +01:00
3354d919fc
LeRobotDataset v2.1 ( #711 )
...
Co-authored-by: Remi <remi.cadene@huggingface.co >
Co-authored-by: Remi Cadene <re.cadene@gmail.com >
2025-02-25 15:27:29 +01:00
32eb0cec8f
Dataset v2.0 ( #461 )
...
Co-authored-by: Remi <remi.cadene@huggingface.co >
2024-11-29 19:04:00 +01:00
pepijn