admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-26 19:26:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci(robomme): smoke-eval 10 tasks instead of 5

Browse Source 
Broader coverage on the RoboMME benchmark CI job: bump the smoke eval
from 5 tasks to 10 (one episode each), all drawn from ROBOMME_TASKS.

Tasks now run: PickXtimes, BinFill, StopCube, MoveCube, InsertPeg,
SwingXtimes, VideoUnmask, ButtonUnmask, PickHighlight, PatternLock.

Updated the parse_eval_metrics.py `--task` label from the single
`PickXtimes` stub to the full comma list so the metrics artifact
reflects what was actually run. `parse_eval_metrics.py` already reads
`overall` for multi-task runs, so no parser change is needed.

Made-with: Cursor
This commit is contained in:
Pepijn
2026-04-17 13:40:31 +01:00
parent 8b4f345a04
commit 0f03129536
1 changed files with 4 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/benchmark_tests.yml
+4 -4
View File
@@ -347,7 +347,7 @@ jobs:
load: true load: true
tags: lerobot-benchmark-robomme:ci tags: lerobot-benchmark-robomme:ci
- name: Run RoboMME smoke eval (1 episode) - name: Run RoboMME smoke eval (10 tasks, 1 episode each)
if: env.HF_USER_TOKEN != '' if: env.HF_USER_TOKEN != ''
run: | run: |
docker run --name robomme-eval --gpus all \ docker run --name robomme-eval --gpus all \
@@ -361,7 +361,7 @@ jobs:
lerobot-eval \ lerobot-eval \
--policy.path=lerobot/smolvla_robomme \ --policy.path=lerobot/smolvla_robomme \
--env.type=robomme \ --env.type=robomme \
--env.task=PickXtimes,BinFill,StopCube,MoveCube,InsertPeg \ --env.task=PickXtimes,BinFill,StopCube,MoveCube,InsertPeg,SwingXtimes,VideoUnmask,ButtonUnmask,PickHighlight,PatternLock \
--env.dataset_split=test \ --env.dataset_split=test \
--env.task_ids=[0] \ --env.task_ids=[0] \
--eval.batch_size=1 \ --eval.batch_size=1 \
@@ -372,7 +372,7 @@ jobs:
--policy.empty_cameras=3 \ --policy.empty_cameras=3 \
--output_dir=/tmp/eval-artifacts --output_dir=/tmp/eval-artifacts
python scripts/ci/extract_task_descriptions.py \ python scripts/ci/extract_task_descriptions.py \
--env robomme --task PickXtimes,BinFill,StopCube,MoveCube,InsertPeg \ --env robomme --task PickXtimes,BinFill,StopCube,MoveCube,InsertPeg,SwingXtimes,VideoUnmask,ButtonUnmask,PickHighlight,PatternLock \
--output /tmp/eval-artifacts/task_descriptions.json --output /tmp/eval-artifacts/task_descriptions.json
" "
@@ -389,7 +389,7 @@ jobs:
python3 scripts/ci/parse_eval_metrics.py \ python3 scripts/ci/parse_eval_metrics.py \
--artifacts-dir /tmp/robomme-artifacts \ --artifacts-dir /tmp/robomme-artifacts \
--env robomme \ --env robomme \
--task PickXtimes \ --task PickXtimes,BinFill,StopCube,MoveCube,InsertPeg,SwingXtimes,VideoUnmask,ButtonUnmask,PickHighlight,PatternLock \
--policy lerobot/smolvla_robomme --policy lerobot/smolvla_robomme
- name: Upload RoboMME rollout video - name: Upload RoboMME rollout video