From e9e7cbbbdaaa6e8e0be7abdbc6e3bcad2b1730a5 Mon Sep 17 00:00:00 2001
From: Pepijn <pepijn@huggingface.co>
Date: Fri, 17 Apr 2026 13:34:01 +0100
Subject: [PATCH] ci(robocasa): smoke-eval 10 atomic tasks instead of 1

Broader coverage in the benchmark CI job: evaluate SmolVLA on ten
fixture-centric atomic RoboCasa tasks (one episode each) instead of
just CloseFridge. The tasks are all drawn from TARGET_TASKS.atomic_seen
and selected to avoid object-manipulation categories that would require
the objaverse/aigen asset packs (we only ship objs_lw in the Docker
image, paired with obj_registries=["lightwheel"] on the runtime side).

Tasks: CloseFridge, OpenCabinet, OpenDrawer, TurnOnMicrowave,
TurnOffStove, CloseToasterOvenDoor, SlideDishwasherRack,
TurnOnSinkFaucet, NavigateKitchen, TurnOnElectricKettle.

`scripts/ci/parse_eval_metrics.py` already handles multi-task output
via the `overall` key, so no parser changes needed. Bumped the metrics
artifact's task label to `atomic_smoke_10` to reflect the grouping.

Made-with: Cursor
---
 .github/workflows/benchmark_tests.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/benchmark_tests.yml b/.github/workflows/benchmark_tests.yml
index 83da814e3..8bf788c20 100644
--- a/.github/workflows/benchmark_tests.yml
+++ b/.github/workflows/benchmark_tests.yml
@@ -346,7 +346,7 @@ jobs:
           load: true
           tags: lerobot-benchmark-robocasa:ci
 
-      - name: Run RoboCasa365 smoke eval (1 episode)
+      - name: Run RoboCasa365 smoke eval (10 atomic tasks, 1 episode each)
         if: env.HF_USER_TOKEN != ''
         run: |
           docker run --name robocasa-eval --gpus all \
@@ -361,7 +361,7 @@ jobs:
               lerobot-eval \
                 --policy.path=lerobot/smolvla_robocasa \
                 --env.type=robocasa \
-                --env.task=CloseFridge \
+                --env.task=CloseFridge,OpenCabinet,OpenDrawer,TurnOnMicrowave,TurnOffStove,CloseToasterOvenDoor,SlideDishwasherRack,TurnOnSinkFaucet,NavigateKitchen,TurnOnElectricKettle \
                 --eval.batch_size=1 \
                 --eval.n_episodes=1 \
                 --eval.use_async_envs=false \
@@ -383,7 +383,7 @@ jobs:
           python3 scripts/ci/parse_eval_metrics.py \
             --artifacts-dir /tmp/robocasa-artifacts \
             --env robocasa \
-            --task CloseFridge \
+            --task atomic_smoke_10 \
             --policy lerobot/smolvla_robocasa
 
       - name: Upload RoboCasa365 rollout video