admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-27 11:46:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(robomme): enumerate distinct test episodes

Browse Source
This commit is contained in:
Pepijn
2026-07-15 19:55:56 +02:00
parent d10efb9cb5
commit 5c453c3786
2 changed files with 13 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/robomme.mdx
+12 -10
View File
@@ -61,7 +61,7 @@ lerobot-eval \
### Multi-task evaluation ### Multi-task evaluation
Evaluate multiple tasks in one run by comma-separating task names. Use `task_ids` to control which episodes are evaluated per task. Recommended: 50 episodes per task for the test split. Evaluate multiple tasks in one run by comma-separating task names. Use `task_ids` to select the dataset-backed episodes evaluated for every task. For the standard 50-episode test protocol, select IDs 0 through 49 and run each selected episode once.
```bash ```bash
lerobot-eval \ lerobot-eval \
@@ -69,20 +69,22 @@ lerobot-eval \
--env.type=robomme \ --env.type=robomme \
--env.task=PickXtimes,BinFill,StopCube,MoveCube,InsertPeg \ --env.task=PickXtimes,BinFill,StopCube,MoveCube,InsertPeg \
--env.dataset_split=test \ --env.dataset_split=test \
--env.task_ids=[0,1,2,3,4,5,6,7,8,9] \ --env.task_ids=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49] \
--eval.batch_size=1 \ --eval.batch_size=1 \
--eval.n_episodes=50 --eval.n_episodes=1
``` ```
### Key CLI options for `env.type=robomme` ### Key CLI options for `env.type=robomme`
| Option | Default | Description | | Option | Default | Description |
| -------------------- | ------------- | -------------------------------------------------- | | -------------------- | ------------- | ------------------------------------------------ |
| `env.task` | `PickXtimes` | Any of the 16 task names above (comma-separated) | | `env.task` | `PickXtimes` | Any of the 16 task names above (comma-separated) |
| `env.dataset_split` | `test` | `train`, `val`, or `test` | | `env.dataset_split` | `test` | `train`, `val`, or `test` |
| `env.action_space` | `joint_angle` | `joint_angle` (8-D) or `ee_pose` (7-D) | | `env.action_space` | `joint_angle` | `joint_angle` (8-D) or `ee_pose` (7-D) |
| `env.episode_length` | `300` | Max steps per episode | | `env.episode_length` | `300` | Max steps per episode |
| `env.task_ids` | `null` | List of episode indices to evaluate (null = `[0]`) | | `env.task_ids` | `null` | Dataset episode indices (null = `[0]`) |
`eval.n_episodes` repeats each selected `task_id`; it does not advance to the next dataset episode. Keep it at `1` when enumerating the 50 distinct test episodes with `env.task_ids`.
## Dataset ## Dataset
examples/robomme/smolvla_visual_memory_ablation.sh
+1 -1
View File
@@ -58,7 +58,7 @@ if [[ "${RUN_EVAL}" == "true" ]]; then
--env.task_ids="${TASK_IDS}" \ --env.task_ids="${TASK_IDS}" \
'--rename_map={"observation.images.image":"observation.images.camera1","observation.images.wrist_image":"observation.images.camera2"}' \ '--rename_map={"observation.images.image":"observation.images.camera1","observation.images.wrist_image":"observation.images.camera2"}' \
--eval.batch_size=1 \ --eval.batch_size=1 \
--eval.n_episodes=50 \ --eval.n_episodes=1 \
--seed="${SEED}" \ --seed="${SEED}" \
--output_dir="${OUTPUT_ROOT}/eval-${variant}" --output_dir="${OUTPUT_ROOT}/eval-${variant}"
done done