Commit Graph

220 Commits

Author SHA1 Message Date
Remi Cadene 8d360927af WIP aggregate 2025-05-16 17:41:47 +00:00
Remi Cadene 1ecaeabad0 Uploaded droid 1.0.1 2025-05-08 15:14:15 +00:00
Remi Cadene 0309a9fcbc Speedup data loading 2025-05-06 15:13:50 +00:00
Remi Cadene 588bf96559 Fix aggregate (num_frames, dataset_from_index, index) 2025-05-06 15:13:35 +00:00
Remi Cadene e11d2e4197 Aggregate: Add concatenation 2025-05-02 13:33:57 +02:00
Remi Cadene 253c649507 Fix convert v30 with image datasets 2025-04-24 18:51:53 +02:00
Remi Cadene 71715c3914 fix hf_dataset.set_transform(hf_transform_to_torch) 2025-04-23 11:42:21 +02:00
Remi Cadene 7c005c2aa1 Merge remote-tracking branch 'origin/user/rcadene/2025_04_11_dataset_v3' into user/rcadene/2025_04_11_dataset_v3 2025-04-23 09:16:37 +00:00
Remi Cadene d518b036d0 Faster self.meta.episodes[...]
switch back to set_transform instead of set_format

Add video_files_size_in_mb

pre-commit run --all-files
2025-04-23 09:14:02 +00:00
Remi Cadene 367d9bda7d Fix unit tests 2025-04-22 10:35:20 +02:00
Remi Cadene 601b5fdbfe Merge remote-tracking branch 'origin/user/rcadene/2025_04_11_dataset_v3' into user/rcadene/2025_04_11_dataset_v3 2025-04-22 08:19:30 +00:00
Remi Cadene 20b74ae1eb fix 2025-04-21 13:38:29 +00:00
Remi Cadene b9b880bd8b fix get_parquet_file_size_in_mb + DEFAULT_FILE_SIZE_IN_MB=100 2025-04-21 12:59:35 +00:00
Remi Cadene 5bd9cb1e72 Merge remote-tracking branch 'origin/main' into user/rcadene/2025_04_11_dataset_v3 2025-04-21 11:03:12 +02:00
Remi Cadene 2866d0770f small fix ffmpeg encoding 2025-04-21 10:59:06 +02:00
Remi Cadene 4375a05a9f Add push to hub for convert_dataset_v21_to_v30 2025-04-21 10:08:25 +02:00
Remi Cadene 4acf99f622 pre-commit run --all-files 2025-04-21 09:34:19 +02:00
Remi Cadene 9c0836c8d0 Remove legacy from datasets/utils.py 2025-04-19 19:27:14 +05:30
Remi Cadene b0cca75e5e Progress on aggregate_datasets 2025-04-19 19:11:53 +05:30
Remi Cadene 54b5c805bf Revert mistake convert_dataset_v20_to_v21.py 2025-04-17 04:47:00 +02:00
Remi Cadene eab5543750 Merge (No verify) 2025-04-17 04:46:09 +02:00
Remi Cadene 6b6a990f4c most unit tests passing (TODO: convert datasets) 2025-04-16 21:30:58 +02:00
Remi Cadene c2a05a1fde Fix (Now loading all frames is possible) 2025-04-14 14:47:18 +00:00
Remi Cadene 34c5d4ce07 Most unit tests are passing 2025-04-11 14:04:22 +02:00
Remi Cadene c1b28f0b58 Commit before episodes episodes_stats merging 2025-04-09 15:20:15 +02:00
pre-commit-ci[bot] 437fc29e12 [pre-commit.ci] pre-commit autoupdate (#871)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-04-08 06:58:46 +02:00
Remi Cadene 53ecec5fb2 WIP v21 to v30 2025-03-31 07:38:01 +00:00
Steven Palma b568de35ad fix(datasets): cast imgs_dir as Path (#915) 2025-03-28 18:08:12 +01:00
Steven Palma a6015a55f9 chore(scripts): remove deprecated script (#887) 2025-03-23 01:16:50 +01:00
Remi Cadene 1a5c1ef9c7 Rename openx to droid + Improve all (not tested) 2025-03-18 16:28:09 +00:00
Steven Palma 1c15bab70f fix(codec): hot-fix for default codec in linux arm platforms (#868) 2025-03-17 13:23:11 +01:00
Jade Choghari 0e98c6ee96 Add torchcodec cpu (#798)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Remi <re.cadene@gmail.com>
Co-authored-by: Remi <remi.cadene@huggingface.co>
Co-authored-by: Simon Alibert <simon.alibert@huggingface.co>
Co-authored-by: Simon Alibert <75076266+aliberts@users.noreply.github.com>
2025-03-14 16:53:42 +01:00
Steven Palma 5d24ce3160 chore(doc): add license header to all files (#818) 2025-03-05 17:56:51 +01:00
Remi Cadene 7866c1f7d1 Merge remote-tracking branch 'origin/main' into user/rcadene/2025_02_19_port_openx 2025-03-01 19:17:18 +00:00
Remi Cadene 3666ac9346 WIP UploadDataset 2025-03-01 19:07:22 +00:00
Simon Alibert e81c36cf74 Fix dataset version tags (#790) 2025-02-28 14:36:20 +01:00
Simon Alibert a1809ad3de Add typos checks (#770) 2025-02-25 23:51:15 +01:00
Haskely eecf32e77a feat: Add root directory option for dataset configuration (#765)
Co-authored-by: Simon Alibert <75076266+aliberts@users.noreply.github.com>
2025-02-25 17:27:36 +01:00
Simon Alibert 3354d919fc LeRobotDataset v2.1 (#711)
Co-authored-by: Remi <remi.cadene@huggingface.co>
Co-authored-by: Remi Cadene <re.cadene@gmail.com>
2025-02-25 15:27:29 +01:00
Remi Cadene 3daab2acbb Add upload_large_folder 2025-02-23 18:19:12 +00:00
Remi Cadene c36d2253d0 Aggregate works 2025-02-23 18:18:46 +00:00
Remi Cadene e2e6f6e666 Add auto_downsample_height_width 2025-02-23 18:15:39 +00:00
Remi Cadene ff0029f84b aggregate works 2025-02-22 15:33:47 +00:00
Remi Cadene 39ad2d16d4 let's go 2025-02-22 11:12:39 +00:00
Remi Cadene 5fbbaa1bc0 fix No such file or directory error 2025-02-20 23:04:58 +00:00
Simon Alibert 64ed5258e6 Fix batch convert 2025-02-20 09:00:14 +01:00
Simon Alibert 392a8c32a7 Improve doc 2025-02-20 08:24:41 +01:00
Simon Alibert 969ef745a2 Remove dataset consolidate (#752) 2025-02-19 16:02:54 +01:00
Simon Alibert 2487228ea7 Use HF_HOME env variable (#753) 2025-02-19 14:49:46 +01:00
Simon Alibert fbf2f2222a Remove local_files_only and use codebase_version instead of branches (#734) 2025-02-19 08:36:32 +01:00