Switching embodiment on a packaged checkpoint keeps the previous
embodiment's camera_sizes entries because dict config values merge
instead of replacing, which tripped the exact-key validation. Rebuild
sizes from the named profile whenever the keys diverge from
camera_order, and add regression tests for both fixes.
Co-authored-by: Cursor <cursoragent@cursor.com>
LeRobot dataset stats include a scalar per-feature sample count that is
not a per-dimension statistic, so projecting it to the embodiment index
map raised a width-mismatch ValueError for every real dataset.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add dataloader_multiprocessing_context, default to spawn
Make the DataLoader multiprocessing start method configurable on
TrainPipelineConfig and default it to 'spawn'.
The previous default (fork on Linux) is unsafe with libraries that hold
non-fork-safe state in the parent process — common ones in this codebase
are PyAV, torchcodec, and the ffmpeg shared libs they wrap. Symptoms
reported in #2488, #2209, and observed locally include:
- multiprocessing.context.AuthenticationError: digest received was wrong
- RuntimeError: Pin memory thread exited unexpectedly
- RuntimeError: DataLoader worker exited unexpectedly
- Random SIGSEGV inside worker processes during video decode
Switching to spawn re-imports modules cleanly in each worker and
eliminates these failure modes. Added the setting as a config field
rather than hard-coding so users on platforms where fork is preferred
can opt back in via --dataloader-multiprocessing-context=fork.
* Address review: shorten config comment, note spawn startup tradeoff
Per @jashshah999, mention that spawn workers re-import modules and so
add some startup time vs fork. Also trim the failure-mode dump from
the inline comment — the linked issue covers the symptoms in detail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(scripts): add multiprocessing_context safeguards
* chore(config): add libs note
---------
Co-authored-by: 0o8o0-blip <0o8o0-blip@users.noreply.github.com>
Nikodem Bartnik