fix(smolvla2): high-level steps must run before LowLevelForward refills

Both HighLevelSubtaskFwd and LowLevelForward are gated on
'action queue is empty'. With LowLevelForward listed first, it refilled
the queue on the empty-queue tick before HighLevelSubtaskFwd got to
check — so the gate I added in the previous commit made the high-level
step a permanent no-op after the initial bootstrap. Visible symptom:
subtask string never advances past whatever bootstrap seeded, no
subtask_change events, memory stays unset, and the new overfit
diagnostics never appear on the panel because last_subtask_raw is
never written.

Move all high-level steps (subtask, memory, interjection, vqa) ahead
of LowLevelForward. On an empty-queue tick the subtask refreshes
first, the new string flows into the next chunk's prompt, then
LowLevelForward generates the chunk, then DispatchAction drains it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pepijn
2026-05-12 17:38:06 +02:00
parent aecb80a9d2
commit b9db4d21a2
@@ -69,16 +69,18 @@ class SmolVLA2Runtime:
_stop: bool = field(default=False, init=False)
def __post_init__(self) -> None:
# Pipeline order matters. Both ``HighLevelSubtaskFwd`` and
# ``LowLevelForward`` are gated on "action queue is empty" so
# the slow LLM call (select_message) doesn't starve dispatch.
# If LowLevelForward runs first, it refills the queue and the
# high-level step never sees ``queue == 0`` afterwards.
#
# Order is therefore: high-level steps that read state (subtask,
# memory, interjection, vqa) → low-level chunk refresh → action
# dispatch → tool dispatch. So on an empty-queue tick the
# subtask refreshes first, the new subtask string flows into
# the next chunk's prompt, and DispatchAction drains.
self.pipeline = [
LowLevelForward(
trigger=HzTrigger(self.chunk_hz),
policy=self.policy,
observation_provider=self.observation_provider,
),
DispatchAction(
trigger=HzTrigger(self.ctrl_hz),
robot_executor=self.robot_executor,
),
HighLevelSubtaskFwd(
trigger=HzTrigger(self.high_level_hz),
policy=self.policy,
@@ -96,6 +98,15 @@ class SmolVLA2Runtime:
policy=self.policy,
observation_provider=self.observation_provider,
),
LowLevelForward(
trigger=HzTrigger(self.chunk_hz),
policy=self.policy,
observation_provider=self.observation_provider,
),
DispatchAction(
trigger=HzTrigger(self.ctrl_hz),
robot_executor=self.robot_executor,
),
DispatchToolCalls(tools=self.tools),
]
self.state = initial_runtime_state()