lerobot

mirror of https://github.com/huggingface/lerobot.git synced 2026-05-26 14:09:47 +00:00

Files

T

Pepijn a088c10c80 examples(port_datasets): SLURM+datatrove RoboCasa composite_seen build

Parallel variant of build_robocasa_composite_seen.py modeled after the
existing slurm_port_shards.py / slurm_aggregate_shards.py pattern.

Two-phase datatrove pipeline:
  * Phase 1 DOWNLOAD: tasks=16 (one per RoboCasa composite_seen task),
    each worker downloads its assigned tar via RoboCasa's own
    download_datasets helper. Network-bound, idempotent.
  * Phase 2 AGGREGATE: tasks=1, single worker calls aggregate_datasets
    over the 16 extracted directories. Submitted with depends=phase1 so
    SLURM only releases it once all 16 downloads succeed.

Reuses the COMPOSITE_SEEN_TASKS list and per-task download/resolve
helpers from the single-machine script via aliased imports — single
source of truth for 'what does it mean to download a composite_seen
task'.

Local (--slurm 0) mode runs the two phases sequentially in-process for
debugging on a workstation.

Usage on SLURM:
    uv run python examples/port_datasets/slurm_build_robocasa_composite_seen.py \
        --output-dir=/scratch/${USER}/robocasa_composite_seen \
        --hub-repo-id=${HF_USER}/robocasa_composite_seen \
        --logs-dir=/scratch/${USER}/logs/robocasa \
        --partition=cpu --push-to-hub

Prereq: uv sync --extra annotations  (pulls datatrove)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 14:10:05 +02:00

display_error_files.py

Dataset v3 (#1412 )

2025-09-15 09:53:30 +02:00

port_droid.py

feat(dependencies): minimal default tag install (#3362 )

2026-04-12 20:03:04 +02:00

slurm_aggregate_shards.py

feat(dependencies): minimal default tag install (#3362 )

2026-04-12 20:03:04 +02:00

slurm_build_robocasa_composite_seen.py

examples(port_datasets): SLURM+datatrove RoboCasa composite_seen build