admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-07-30 13:09:40 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(smolvla): freeze the intended VLM layers when train_expert_only=False (#4019)

Browse Source 
* fix(smolvla): freeze the intended VLM layers when train_expert_only=False

The partial-freeze patterns in set_requires_grad() used a
'text_model.model.' prefix that does not exist in SmolVLM parameter
names ('SmolVLMModel.text_model' is a bare LlamaModel, with no nested
'.model'). As a result the last VLM layer and the final norm were
silently left trainable, defeating the freeze that was added to avoid
unused-parameter errors with DDP; only lm_head was frozen by substring
luck.

Use the real flat names, and raise if any freeze pattern stops matching
so a future transformers renaming cannot silently reintroduce the bug.
Add a CPU regression test covering both last_layers branches.

Fixes #4018

* test(smolvla): drop regression test per review

---------

Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
This commit is contained in:
Xingdong Zuo
2026-07-30 20:17:54 +09:00
committed by GitHub
parent 49d5ea49bc
commit ede1fc2978
1 changed files with 12 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lerobot/policies/smolvla/smolvlm_with_expert.py
+12 -3
View File
@@ -168,14 +168,23 @@ class SmolVLMWithExpertModel(nn.Module):
last_layers.append(self.num_vlm_layers - 2) last_layers.append(self.num_vlm_layers - 2)
frozen_layers = [ frozen_layers = [
"lm_head", "lm_head",
"text_model.model.norm.weight", "text_model.norm.weight",
] ]
for layer in last_layers: for layer in last_layers:
frozen_layers.append(f"text_model.model.layers.{layer}.") frozen_layers.append(f"text_model.layers.{layer}.")
unmatched_patterns = set(frozen_layers)
for name, params in self.vlm.named_parameters(): for name, params in self.vlm.named_parameters():
if any(k in name for k in frozen_layers): matched_patterns = [k for k in frozen_layers if k in name]
if matched_patterns:
params.requires_grad = False params.requires_grad = False
unmatched_patterns.difference_update(matched_patterns)
if unmatched_patterns:
raise RuntimeError(
"Some frozen layer patterns matched no VLM parameters, so the corresponding layers "
"would silently remain trainable (parameter naming may have changed in transformers): "
f"{sorted(unmatched_patterns)}"
)
# To avoid unused params issue with distributed training # To avoid unused params issue with distributed training
for name, params in self.lm_expert.named_parameters(): for name, params in self.lm_expert.named_parameters():
if "lm_head" in name: if "lm_head" in name: