Sundar Raghavan 9782bfd64b Support streaming from HF Storage Buckets in StreamingLeRobotDataset ...

Add an opt-in repo_type="bucket" parameter to StreamingLeRobotDataset
and LeRobotDatasetMetadata so a dataset can be streamed directly from an
HF Storage Bucket (hf://buckets/...) with no local download.

When repo_type="bucket":
- skip git-version resolution (buckets have no refs/tags),
- pull the meta/ directory via HfFileSystem.get,
- point url_root at hf://buckets/{repo_id},
- read parquet shards via load_dataset("parquet",
  data_files="hf://buckets/{repo_id}/data/*/*.parquet", ...).

The default repo_type="dataset" preserves all existing behavior.
LeRobotDataset (non-streaming) and create() are unchanged.

Closes #3969

Signed-off-by: Sundar Raghavan <sdraghav@amazon.com>

2026-07-30 15:45:42 +02:00

..

test_aggregate.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_augment_quantile_stats.py

fix(datasets): bound memory of augment_dataset_quantile_stats by sampling frames (#3749)

2026-07-29 12:32:03 +02:00

test_compute_stats.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_dataset_metadata.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_dataset_reader.py

feat(encoding parameters): adding support for user provided video encoding parameters (#3455)

2026-05-14 23:46:42 +02:00

test_dataset_tools.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_dataset_utils.py

feat(dataset): accept token argument for private HF Hub datasets (#4136)

2026-07-24 18:51:35 +02:00

test_dataset_writer.py

fix(datasets): clear video frame staging on episode reset (#3683)

2026-07-27 13:44:32 +02:00

test_datasets.py

feat(dataset): add slice support to LeRobotDataset.__getitem__ (#4129)

2026-07-23 22:05:29 +02:00

test_delta_timestamps.py

feat(dependencies): minimal default tag install (#3362)

2026-04-12 20:03:04 +02:00

test_depth.py

fix(depth unit): adding input depth unit storage in the dataset metadata (#3899)

2026-07-02 11:53:13 +02:00

test_image_transforms.py

feat(dependencies): minimal default tag install (#3362)

2026-04-12 20:03:04 +02:00

test_image_writer.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_language_render.py

Add extensive language support (#3467)

2026-05-19 14:46:11 +02:00

test_language.py

Add extensive language support (#3467)

2026-05-19 14:46:11 +02:00

test_lerobot_dataset.py

Handle resuming empty local datasets (#3859)

2026-07-28 16:25:42 +02:00

test_quantiles_dataset_integration.py

feat(dependencies): minimal default tag install (#3362)

2026-04-12 20:03:04 +02:00

test_sampler.py

feat(datasets): deterministic, resumable shuffling for EpisodeAwareSampler (#3769)

2026-06-12 11:47:16 +02:00

test_streaming_video_encoder.py

feat(depth maps): adding support for depth in LeRobot (#3644)

2026-06-27 14:21:21 +02:00

test_streaming.py

Support streaming from HF Storage Buckets in StreamingLeRobotDataset

2026-07-30 15:45:42 +02:00

test_video_decoder_cache.py

fix(datasets): bound VideoDecoderCache to prevent OOM on large datasets (#3614)

2026-05-19 16:54:25 +02:00

test_video_encoding.py

fix(save codec options): making sure codec options are always set via set_if (#3910)

2026-07-02 15:29:14 +02:00

test_visualize_dataset.py

feat(dependencies): minimal default tag install (#3362)

2026-04-12 20:03:04 +02:00