admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-26 11:16:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci(vlabench): smoke-eval 10 tasks instead of 1

Browse Source 
Broader coverage on the VLABench benchmark CI job: bump the smoke eval
from 1 task to 10 (one episode each), all drawn from PRIMITIVE_TASKS.

Tasks now run: select_fruit, select_toy, select_book, select_painting,
select_drink, select_ingredient, select_mahjong, select_poker,
add_condiment, insert_flower.

Bumped the parse_eval_metrics.py `--task` label to the full comma
list so the metrics artifact reflects what was actually run.
`parse_eval_metrics.py` already reads `overall` for multi-task runs,
so no parser change is needed.

Made-with: Cursor
This commit is contained in:
Pepijn
2026-04-17 13:40:42 +01:00
parent 0ac6118bc5
commit 893b10b825
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/benchmark_tests.yml
+3 -3
View File
@@ -357,7 +357,7 @@ jobs:
build-args: | build-args: |
VLABENCH_ASSETS_REPO=lerobot/vlabench-assets VLABENCH_ASSETS_REPO=lerobot/vlabench-assets
- name: Run VLABench smoke eval (1 episode) - name: Run VLABench smoke eval (10 tasks, 1 episode each)
if: env.HF_USER_TOKEN != '' if: env.HF_USER_TOKEN != ''
run: | run: |
docker run --name vlabench-eval --gpus all \ docker run --name vlabench-eval --gpus all \
@@ -372,7 +372,7 @@ jobs:
lerobot-eval \ lerobot-eval \
--policy.path=lerobot/smolvla_vlabench \ --policy.path=lerobot/smolvla_vlabench \
--env.type=vlabench \ --env.type=vlabench \
--env.task=select_fruit \ --env.task=select_fruit,select_toy,select_book,select_painting,select_drink,select_ingredient,select_mahjong,select_poker,add_condiment,insert_flower \
--eval.batch_size=1 \ --eval.batch_size=1 \
--eval.n_episodes=1 \ --eval.n_episodes=1 \
--eval.use_async_envs=false \ --eval.use_async_envs=false \
@@ -394,7 +394,7 @@ jobs:
python3 scripts/ci/parse_eval_metrics.py \ python3 scripts/ci/parse_eval_metrics.py \
--artifacts-dir /tmp/vlabench-artifacts \ --artifacts-dir /tmp/vlabench-artifacts \
--env vlabench \ --env vlabench \
--task select_fruit \ --task select_fruit,select_toy,select_book,select_painting,select_drink,select_ingredient,select_mahjong,select_poker,add_condiment,insert_flower \
--policy lerobot/smolvla_vlabench --policy lerobot/smolvla_vlabench
- name: Upload VLABench rollout video - name: Upload VLABench rollout video