mirror of
https://github.com/huggingface/lerobot.git
synced 2026-08-01 22:19:48 +00:00
Maxime Ellerbach
c15f6acb7e
LingBot-VA's action stream doubles as its memory: _compute_kv_cache feeds the previous chunk's predicted actions back as clean context. With use_relative_actions each chunk is anchored on the state at its own start, so the same physical motion encodes differently depending on which chunk it fell in, and the cached stream sawtooths back to zero displacement at every boundary (mean 0.22, p95 0.91 normalized units against a total output range of 2.0). Training never contains such a reset -- one sample is exactly one chunk, hence one anchor -- and the model's only cross-frame supervision is "continue from the previous clean action frame", so it continues rather than restarting and the postprocessor double-counts the displacement: ~7.3 deg/chunk on the folding data, matching the observed monotonic drift after 4-5 chunks. First-difference the conditioning instead. An increment carries no anchor, so the cached history stays consistent across chunks; measured boundary inconsistency drops 7.30 -> 1.78 deg, the latter being the controller's tracking error, which is generic to chunked control rather than an artifact of the action encoding. The conversion touches ONLY the conditioning -- at inference in _preprocess_action_state, and on the matching clean copy in _add_noise_stream so training agrees. Predicted actions keep the cumulative-offset encoding, so the processors, the checkpoint pipeline and the executed commands are untouched. That is also what makes it safe to do here: normalization is affine with a non-zero constant, so cumsum does not commute with unnormalization -- inverting an accumulated stream would drift by (n-1)*(q99+q01)/2, which is 0 on a symmetric channel but 118 deg over one chunk on right_joint_6. Nothing inverts the conditioning, so that cannot arise. Gated on use_relative_actions only: an absolute action stream is already globally consistent, so those checkpoints keep their exact behaviour. A relative checkpoint is only valid for the encoding it was trained with. Also fixes observation_delta_indices: AutoencoderKLWan._encode consumes only the first 4 * (iter_ - 1) + 1 frames of an n-frame clip, so asking for frame_chunk_size * 4 decoded 3 frames per sample that never reached the encoder (verified by ablation -- scrambling them left the latents bit-identical). Same latent frame count, less video decode. The observation window stays shorter than the action window, so sample validity is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>