Files
lerobot/docs/source/backwardcomp.mdx
T
Adil Zouitine a877c596ba chore(docs): Processor doc (#1685)
* chore(docs): initialize doc

* Added script for the second part of the processor doc

* precommit style nit

* improved part 2 of processor guide

* Add comprehensive documentation for processors in robotics

- Introduced a detailed guide on processors, covering their role in transforming raw robot data into model-ready inputs and vice versa.
- Explained core concepts such as EnvTransition, ProcessorStep, and RobotProcessor, along with their functionalities.
- Included examples of common processor steps like normalization, device management, batch processing, and text tokenization.
- Provided insights on building complete pipelines, integrating processors into training loops, and saving/loading configurations.
- Emphasized best practices and advanced features for effective usage of processors in robotics applications.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat(docs): Enhance introduction to processors with additional converter functions

- Updated the introduction to processors documentation to include default batch-to-transition and transition-to-batch converters.
- Added detailed descriptions and examples for new specialized converter functions: `to_transition_teleop_action`, `to_transition_robot_observation`, `to_output_robot_action`, and `to_dataset_frame`.
- Improved clarity on how these converters facilitate integration with existing robotics applications.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Improved doc implement_your_own_pipeline
- Use normalization processor as default example
- Add section on transform features
- Add section on overrides.

* Add phone docs and use pipeline for robots/teleop docs

* Fix typo in documentation for adapters in robots/teleop section

* Enhance documentation for processors with detailed explanations and examples

- Updated the introduction to processors, clarifying the role of `EnvTransition` and `ProcessorStep`.
- Introduced `DataProcessorPipeline` as a generic orchestrator for chaining processor steps.
- Added comprehensive descriptions of new converter functions and their applications.
- Improved clarity on type safety and the differences between `RobotProcessorPipeline` and `PolicyProcessorPipeline`.
- Included examples for various processing scenarios, emphasizing best practices for data handling in robotics.

* Enhance documentation for processor migration and debugging

- Added detailed sections on the migration of models to the new `PolicyProcessorPipeline` system, including breaking changes and migration scripts.
- Introduced a comprehensive guide for debugging processor pipelines, covering common issues, step-by-step inspection, and runtime monitoring techniques.
- Updated examples to reflect new usage patterns and best practices for processor implementation and error handling.
- Clarified the role of various processor steps and their configurations in the context of robotics applications.

---------

Co-authored-by: Michel Aractingi <michel.aractingi@huggingface.co>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Pepijn <pepijn@huggingface.co>
2025-09-12 18:00:37 +02:00

152 lines
6.2 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Backward compatibility
## Policy Normalization Migration (PR #1452)
**Breaking Change**: LeRobot policies no longer have built-in normalization layers embedded in their weights. Normalization is now handled by external `PolicyProcessorPipeline` components.
### What changed?
| | Before PR #1452 | After PR #1452 |
| -------------------------- | ------------------------------------------------ | ------------------------------------------------------------ |
| **Normalization Location** | Embedded in model weights (`normalize_inputs.*`) | External `PolicyProcessorPipeline` components |
| **Model State Dict** | Contains normalization statistics | **Clean weights only** - no normalization parameters |
| **Usage** | `policy(batch)` handles everything | `preprocessor(batch)` → `policy(...)` → `postprocessor(...)` |
### Impact on existing models
- Models trained **before** PR #1452 have normalization embedded in their weights
- These models need migration to work with the new `PolicyProcessorPipeline` system
- The migration extracts normalization statistics and creates separate processor pipelines
### Migrating old models
Use the migration script to convert models with embedded normalization:
```shell
python src/lerobot/processor/migrate_policy_normalization.py \
--pretrained-path lerobot/act_aloha_sim_transfer_cube_human \
--push-to-hub \
--branch migrated
```
The script:
1. **Extracts** normalization statistics from model weights
2. **Creates** external preprocessor and postprocessor pipelines
3. **Removes** normalization layers from model weights
4. **Saves** clean model + processor pipelines
5. **Pushes** to Hub with automatic PR creation
### Using migrated models
```python
# New usage pattern (after migration)
from lerobot.policies.factory import make_policy, make_pre_post_processors
# Load model and processors separately
policy = make_policy(config, ds_meta=dataset.meta)
preprocessor, postprocessor = make_pre_post_processors(
policy_cfg=config,
dataset_stats=dataset.meta.stats
)
# Process data through pipeline
processed_batch = preprocessor(raw_batch)
action = policy.select_action(processed_batch)
final_action = postprocessor(action)
```
## Hardware API redesign
PR [#777](https://github.com/huggingface/lerobot/pull/777) improves the LeRobot calibration but is **not backward-compatible**. Below is a overview of what changed and how you can continue to work with datasets created before this pull request.
### What changed?
| | Before PR #777 | After PR #777 |
| --------------------------------- | ------------------------------------------------- | ------------------------------------------------------------ |
| **Joint range** | Degrees `-180...180°` | **Normalised range** Joints: `100...100` Gripper: `0...100` |
| **Zero position (SO100 / SO101)** | Arm fully extended horizontally | **In middle of the range for each joint** |
| **Boundary handling** | Software safeguards to detect ±180 ° wrap-arounds | No wrap-around logic needed due to mid-range zero |
---
### Impact on existing datasets
- Recorded trajectories created **before** PR #777 will replay incorrectly if loaded directly:
- Joint angles are offset and incorrectly normalized.
- Any models directly finetuned or trained on the old data will need their inputs and outputs converted.
### Using datasets made with the previous calibration system
We provide a migration example script for replaying an episode recorded with the previous calibration here: `examples/backward_compatibility/replay.py`.
Below we take you through the modifications that are done in the example script to make the previous calibration datasets work.
```diff
+ key = f"{name.removeprefix('main_')}.pos"
action[key] = action_array[i].item()
+ action["shoulder_lift.pos"] = -(action["shoulder_lift.pos"] - 90)
+ action["elbow_flex.pos"] -= 90
```
Let's break this down.
New codebase uses `.pos` suffix for the position observations and we have removed `main_` prefix:
<!-- prettier-ignore-start -->
```python
key = f"{name.removeprefix('main_')}.pos"
```
<!-- prettier-ignore-end -->
For `"shoulder_lift"` (id = 2), the 0 position is changed by -90 degrees and the direction is reversed compared to old calibration/code.
<!-- prettier-ignore-start -->
```python
action["shoulder_lift.pos"] = -(action["shoulder_lift.pos"] - 90)
```
<!-- prettier-ignore-end -->
For `"elbow_flex"` (id = 3), the 0 position is changed by -90 degrees compared to old calibration/code.
<!-- prettier-ignore-start -->
```python
action["elbow_flex.pos"] -= 90
```
<!-- prettier-ignore-end -->
To use degrees normalization we then set the `--robot.use_degrees` option to `true`.
```diff
python examples/backward_compatibility/replay.py \
--robot.type=so101_follower \
--robot.port=/dev/tty.usbmodem5A460814411 \
--robot.id=blue \
+ --robot.use_degrees=true \
--dataset.repo_id=my_dataset_id \
--dataset.episode=0
```
### Using policies trained with the previous calibration system
Policies output actions in the same format as the datasets (`torch.Tensors`). Therefore, the same transformations should be applied.
To find these transformations, we recommend to first try and and replay an episode of the dataset your policy was trained on using the section above.
Then, add these same transformations on your inference script (shown here in the `record.py` script):
```diff
action_values = predict_action(
observation_frame,
policy,
get_safe_torch_device(policy.config.device),
policy.config.use_amp,
task=single_task,
robot_type=robot.robot_type,
)
action = {key: action_values[i].item() for i, key in enumerate(robot.action_features)}
+ action["shoulder_lift.pos"] = -(action["shoulder_lift.pos"] - 90)
+ action["elbow_flex.pos"] -= 90
robot.send_action(action)
```
If you have questions or run into migration issues, feel free to ask them on [Discord](https://discord.gg/s3KuuzsPFb)