mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-08 02:22:02 +00:00
fix(streaming): do not prepare the dataloader with accelerate
The dataset is already rank-disjoint via split_dataset_by_node; accelerate's IterableDatasetShard wrapper kept only every Nth batch of each rank's stream, silently training on 1/N of the data per pass while decoding all of it. The --dummy benchmark path never prepared the loader, so benchmarks were unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -115,7 +115,11 @@ def main() -> None:
|
||||
cfg = ACTConfig(input_features=input_features, output_features=output_features)
|
||||
model = ACTPolicy(cfg)
|
||||
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
|
||||
model, optimizer, loader = accelerator.prepare(model, optimizer, loader)
|
||||
# Do NOT prepare the dataloader: the dataset is already rank-disjoint via
|
||||
# split_dataset_by_node, and accelerate's IterableDatasetShard would keep only every
|
||||
# world_size-th batch of it (silently training on 1/N of the data while decoding all
|
||||
# of it). Batches are moved to the device manually in the loop.
|
||||
model, optimizer = accelerator.prepare(model, optimizer)
|
||||
|
||||
# Resume: restore the dataset's stream position so we don't replay already-seen data. The state holds
|
||||
# plain HF stream dicts + RNG state (not tensors), so weights_only=False is required; the file is a
|
||||
|
||||
Reference in New Issue
Block a user