* feat(train): parallel training engine with FSDP2, HSDP, and DCP checkpoints
Replace the FSDP1 training path with a config-owned parallel-training
engine:
- Topology and runtime configs (--parallelism.*, --accelerator.*):
dp_replicate x dp_shard degrees select single-process, DDP (unchanged
default), FSDP2, or HSDP; mixed precision, first-class gradient
accumulation, and FSDP/DDP tuning knobs are mirrored as plain
dataclasses that build the accelerate objects at runtime, so every
run is reproducible from its train_config.json alone. Accelerate env
vars are guarded against configuring the engine behind the config
system's back.
- Declarative policy surface: policies declare FSDP2 wrap units
(_fsdp_wrap_modules) and non-forward entry points
(_fsdp_forward_methods); a shared engine resolves them around
accelerator.prepare(). Context-parallel fields are reserved and
validated to 1.
- Checkpoints: selectable --checkpoint_format (safetensors | dcp |
safetensors_dcp); the sharded optimizer channel is always DCP;
two-phase resume (step+RNG before prepare, DCP model/optimizer after)
reshards across GPU-topology changes; lerobot-convert-dcp merges DCP
shards into a distributable model.safetensors offline.
- Publishing: PreTrainedPolicy.push_model_to_hub is replaced by the
free publish_trained_model (model + processors + card + train config,
all-ranks gather with main-rank writes);
PreTrainedPolicy._save_pretrained gathers state dicts internally,
removing the state_dict= threading from save_pretrained.
- lerobot_train is restructured around the engine: optimizer built
before the single prepare() call, deferred weight load on DCP
resumes, collective save_checkpoint with no call-site rank branches,
dp-world-size-based sample accounting.
Breaking changes: FSDP checkpoints from lerobot <= 0.6.x are not
resumable (weights stay loadable via from_pretrained; pin
lerobot==0.6.x to finish old runs); the `accelerate launch
--config_file` yaml flow is superseded by the config flags; training
autocast is owned exclusively by --accelerator.mixed_precision
(policy.dtype only casts parameters).
Also fixes: reward-model hub publishing crash (TypeError on extra
kwargs).
Verified by ~200 new CPU tests (config round-trips, checkpoint
round-trips per format, two-phase resume, publisher contracts,
converter equivalence, accelerate canaries), a 5-test 4-GPU suite
(FSDP2 save/resume bit-exactness, HSDP/DDP loss parity,
changed-topology resume, all-ranks save_pretrained, grad-accum
equivalence), and end-to-end ACT (1/4/8 GPUs) + FastWAM 6B
(FSDP2 + HSDP) training runs.
* feat(rewards): add RewardModelConfig and PreTrainedRewardModel base classes
* refactor(rewards): migrate Classifier from policies/sac/reward_model/ to rewards/classifier/
* refactor(rewards): migrate SARM from policies/sarm/ to rewards/sarm/
* refactor(rewards): add rewards/factory.py and remove reward model code from policies/factory.py
* refactor(rewards): update imports and delete old reward model locations
* test(rewards): add reward model tests and update existing test imports
* fix(rewards): restore full Classifier and SARM implementations
* test(rewards): restore missing CUDA and mixed precision classifier processor tests
* refactor(lerobot_train.py): remove rabc specific configuration and replace it with a generic samplerweight class in lerobot_train
* refactor(lerobot_train.py): add missing sampling weight script
* linter + missing files
* add testing for sampl weighter
* revert some useless changes, improve typing
* update docs
* add automatic detection of the progress path
* remove type exp
* improve comment
* fix: move rabc.py to rewards/sarm/ and update import paths
* refactor(imports): update reward model imports to new module structure
* refactor(imports): update reward model imports to reflect new module structure
* refactor(imports): conditionally import pandas based on availability
* feat(configs): add reward_model field to TrainPipelineConfig and Hub fields to RewardModelConfig
* refactor(policies): remove reward model branches from policy factory and __init__
* refactor(rewards): expand __init__ facade and fix SARMConfig __post_init__ crash
* feat(train): route reward model training through rewards/factory instead of policies/factory
* refactor(train): streamline reward model training logic
* fix(rewards): ensure FileNotFoundError is raised for missing config_file
* refactor(train): update __get_path_fields__ to include reward_model for config loading
* refactor(classifier): remove redundant input normalization in predict_reward method
* fix(train): raise ValueError for non-trainable reward models in train function
* refactor(pretrained_rm): add model card template
* refactor(tests): reward models
* refactor(sarm): update reset method and remove unused action prediction methods
* refactor(wandb): differentiate tags for reward model and policy training in cfg_to_group function
* fix(train): raise ValueError for PEFT usage in reward model training
* refactor(rewards): enhance RewardModelConfig with device handling and delta indices properties
---------
Co-authored-by: Michel Aractingi <michel.aractingi@huggingface.co>
Haoming Song