admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-23 17:56:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add inline offline validation with train/eval split (#3824)

Browse Source 
* refactor(training): rename eval_freq to env_eval_freq

- Rename eval_freq to env_eval_freq to distinguish sim environment evaluation from offline loss evaluation.

* feat(training): add inline offline validation with train/eval split

- Add eval_split config for balanced per-task holdout
- Add eval_steps for periodic inline eval loss computation
- Add max_eval_samples to cap eval cost

* fix(datasets): remap absolute indices in __getitem__ for filtered datasets

* fix(train): vectorize eval subset selection for max_eval_samples

* fix(datasets): Move the remapping into EpisodeAwareSampler via absolute_to_relative_idx

* fix(validation): add eval_split range check and eval_steps warning

Validate eval_split is in [0.0, 1.0) to prevent garbage splits from
out-of-range values. Raise when eval_steps > 0 but eval_split is 0.0
since no offline eval will run.

* fix(train): prepare eval dataloader with accelerator for multi-GPU

Prepare eval_dataloader through accelerator.prepare() so eval data is
sharded across ranks instead of duplicated. Reduce eval_loss across
ranks with mean reduction for consistent logging.

* fix(test): rename eval_freq to env_eval_freq for multi-GPU training
This commit is contained in:
Khalil Meftah
2026-06-25 15:31:24 +02:00
committed by GitHub
parent c3f180e115
commit 6a788fbdb0
18 changed files with 199 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/metaworld.mdx
+2 -2
View File
@@ -120,11 +120,11 @@ lerobot-train \
--batch_size=4 \
--eval.batch_size=1 \
--eval.n_episodes=1 \
--eval_freq=1000
--env_eval_freq=1000
```
## Practical tips
- Use the one-hot task conditioning for multi-task training (MT10/MT50 conventions) so policies have explicit task context.
- Inspect the dataset task descriptions and the `info["is_success"]` keys when writing post-processing or logging so your success metrics line up with the benchmark.
- Adjust `batch_size`, `steps`, and `eval_freq` to match your compute budget.
- Adjust `batch_size`, `steps`, and `env_eval_freq` to match your compute budget.