Compare commits

...

4 Commits

Author SHA1 Message Date
Maxime Ellerbach c961595644 adding relative joint exclude test 2026-07-23 17:04:51 +00:00
Maxime Ellerbach 295c612711 adding relative actions to sync engine 2026-07-17 16:09:12 +00:00
Steven Palma c5371d0691 refactor(processors): share policy pipeline builders (#4016)
* refactor(processors): share policy pipeline builders

* Apply suggestions from code review

Co-authored-by: Martino Russi <77496684+nepyope@users.noreply.github.com>
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>

* fix(processor): solve style after commit suggestions

---------

Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Martino Russi <77496684+nepyope@users.noreply.github.com>
2026-07-17 14:10:32 +02:00
Steven Palma b2c062c0f4 refactor(policies): resolve policy components by convention (#4015)
* refactor(policies): resolve policy components by convention

* remove fron None no-op

* extend processor resolver error handling logic to policy class resolver as well

---------

Co-authored-by: Martino Russi <nopyeps@gmail.com>
2026-07-17 13:59:38 +02:00
27 changed files with 662 additions and 926 deletions
+16 -14
View File
@@ -150,14 +150,14 @@ class MyPolicy(PreTrainedPolicy):
The methods called by the train/eval loops:
| Method | Used by | What it does |
| ----------------------------------------------------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `reset() -> None` | `lerobot-eval` | Clear per-episode state at the start of each episode. |
| `select_action(batch, **kwargs) -> Tensor` | `lerobot-eval` | Return the next action `(B, action_dim)`. Called every step. |
| `predict_action_chunk(batch, **kwargs) -> Tensor` | the policy itself | Return an action chunk `(B, chunk_size, action_dim)`. Currently abstract on the base class — raise `NotImplementedError` if your policy doesn't chunk. |
| `forward(batch, reduction="mean") -> tuple[Tensor, dict \| None]` | `lerobot-train` | Return `(loss, output_dict)`. Accept `reduction="none"` if you want to support per-sample weighting. |
| `get_optim_params() -> dict` | the optimizer | Return `self.parameters()` for simple policies; return a named parameter dict for [multi-optimizer policies](https://github.com/huggingface/lerobot/blob/ecd38c50d7d15b4184cf42649ff1185ee2e11eeb/src/lerobot/policies/sac/modeling_sac.py#L61-L73). |
| `update() -> None` _(optional)_ | `lerobot-train` | Called after each optimizer step _if defined_. Use for EMA, target nets, replay buffers (TDMPC uses this). |
| Method | Used by | What it does |
| ----------------------------------------------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `reset() -> None` | `lerobot-eval` | Clear per-episode state at the start of each episode. |
| `select_action(batch, **kwargs) -> Tensor` | `lerobot-eval` | Return the next action `(B, action_dim)`. Called every step. |
| `predict_action_chunk(batch, **kwargs) -> Tensor` | the policy itself | Return an action chunk `(B, chunk_size, action_dim)`. Currently abstract on the base class — raise `NotImplementedError` if your policy doesn't chunk. |
| `forward(batch, reduction="mean") -> tuple[Tensor, dict \| None]` | `lerobot-train` | Return `(loss, output_dict)`. Accept `reduction="none"` if you want to support per-sample weighting. |
| `get_optim_params() -> dict` | the optimizer | Return `self.parameters()` for simple policies; return a named parameter dict for multi-optimizer policies (see `get_optim_params` in [`modeling_act.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/act/modeling_act.py) for a per-group learning-rate example). |
| `update() -> None` _(optional)_ | `lerobot-train` | Called after each optimizer step _if defined_. Use for EMA, target nets, replay buffers (TDMPC uses this). |
Batches are flat dictionaries keyed by the constants in [`lerobot.utils.constants`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/utils/constants.py): `OBS_STATE` (`observation.state.<motor>`), `OBS_IMAGES` (`observation.images.<camera>`), `OBS_LANGUAGE`, `ACTION`, etc. Reuse the constants — don't invent new prefixes.
@@ -295,12 +295,10 @@ The file names are load-bearing: the factory does lazy imports by name, and the
### Wiring
Four places need to know about your policy. All by name.
Two places need to know about your policy. All by name.
1. **`policies/__init__.py`** — re-export `MyPolicyConfig` and add it to `__all__`. **Don't** re-export the modeling class; it loads lazily through the factory (so `import lerobot` stays fast).
2. **`factory.py:get_policy_class`** — add a branch returning `MyPolicy` from a lazy import.
3. **`factory.py:make_policy_config`** and **`factory.py:make_pre_post_processors`** — same idea, two more branches.
4. **`templates/lerobot_modelcard_template.md` and the root `README.md`** — the template is what `push_model_to_hub` renders into the model card of every checkpoint trained with your policy: add a one-line description of your policy in the `model_name` branches, map it in `policy_docs` so cards link to your MDX guide, and optionally add an architecture image to `diagrams`. Then add your policy to the models table in the root `README.md`, under the right category, linking to your doc page.
1. **`policies/__init__.py`** — re-export `MyPolicyConfig` and add it to `__all__`. This import is what registers your policy: `@PreTrainedConfig.register_subclass("my_policy")` runs, and from then on the factory resolves everything by convention. **Don't** re-export the modeling class; it loads lazily through the factory (so `import lerobot` stays fast).
2. **`templates/lerobot_modelcard_template.md` and the root `README.md`** — the template is what `push_model_to_hub` renders into the model card of every checkpoint trained with your policy: add a one-line description of your policy in the `model_name` branches, map it in `policy_docs` so cards link to your MDX guide, and optionally add an architecture image to `diagrams`. Then add your policy to the models table in the root `README.md`, under the right category, linking to your doc page.
Mirror an existing policy that's structurally similar to yours; the diff is small.
@@ -332,6 +330,10 @@ This way:
Add a matching extra to [`pyproject.toml`](https://github.com/huggingface/lerobot/blob/main/pyproject.toml) `[project.optional-dependencies]` and include it in the `all` extra so `pip install 'lerobot[all]'` keeps installing everything.
### Avoid copying a modeling file — subclass it
If your policy needs to modify a backbone that already exists in `transformers` (custom conditioning, extra inputs, a swapped sub-module), **do not vendor a copy of its `modeling_*.py`**. Instead, subclass the smallest upstream unit and override only what changes. [`pi_gemma.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/policies/pi_gemma.py) is the canonical reference: it injects AdaRMS conditioning into PaliGemma/Gemma in ~370 lines by subclassing `GemmaModel`/`PaliGemmaModel` and overriding the decoder-layer forward, instead of forking the ~2,000-line modeling file. Model surgery on a _loaded_ native model is also fine (layer truncation, tokenizer expansion, hidden-state capture — see `evo1/internvl3_embedder.py`, `eo1/modeling_eo1.py`, `groot/groot_n1_7.py` for working examples). Reviewers will ask for this pattern when a PR arrives with a copied modeling file; the only accepted exception is a model that does not exist in `transformers` at all.
### Benchmarks and a published checkpoint
A new policy is much easier to review — and far more useful — when it ships with a working checkpoint and at least one number you can reproduce.
@@ -367,7 +369,7 @@ If your policy is real-robot-only and no sim benchmark applies, swap the sim eva
The general expectations are in [`CONTRIBUTING.md`](https://github.com/huggingface/lerobot/blob/main/CONTRIBUTING.md) and the [PR template](https://github.com/huggingface/lerobot/blob/main/.github/PULL_REQUEST_TEMPLATE.md). On top of those, reviewers will look for:
- [ ] `MyPolicy` and `MyPolicyConfig` cover the surface above; `__init_subclass__` accepts the class.
- [ ] `factory.py` and `policies/__init__.py` are wired (lazy imports for modeling).
- [ ] `policies/__init__.py` re-exports the config (this registers the policy; the factory resolves modeling/processor by naming convention).
- [ ] `make_my_policy_pre_post_processors` follows the naming convention.
- [ ] Optional deps live behind a `[project.optional-dependencies]` extra and the `TYPE_CHECKING + require_package` guard.
- [ ] `tests/policies/` updated; backward-compat artifact committed & policy-specific tests.
+15 -9
View File
@@ -205,24 +205,30 @@ class PreTrainedConfig(draccus.ChoiceRegistry, HubMixin, abc.ABC): # type: igno
f"{CONFIG_NAME} not found on the HuggingFace Hub in {model_id}"
) from e
# HACK: Parse the original config to get the config subclass, so that we can
# apply cli overrides.
# This is very ugly, ideally we'd like to be able to do that natively with draccus
# something like --policy.path (in addition to --policy.type)
with draccus.config_type("json"):
orig_config = draccus.parse(cls, config_file, args=[])
if config_file is None:
raise FileNotFoundError(f"{CONFIG_NAME} not found in {model_id}")
with open(config_file) as f:
config = json.load(f)
config.pop("type")
# Resolve the concrete config subclass from the serialized "type" tag, then parse
# the config (with CLI overrides) directly for that class. The "type" key is
# stripped because draccus only consumes it when parsing the registry base class.
policy_type = config.pop("type", None)
if policy_type is None:
raise ValueError(f"Missing 'type' field in {CONFIG_NAME} of {model_id}")
try:
config_cls = cls.get_choice_class(policy_type)
except Exception as e:
raise ValueError(
f"Policy type '{policy_type}' (from {CONFIG_NAME} of {model_id}) is not registered. "
f"Available policy types: {cls.get_known_choices()}"
) from e
with tempfile.NamedTemporaryFile("w+", delete=False, suffix=".json") as f:
json.dump(config, f)
config_file = f.name
cli_overrides = policy_kwargs.pop("cli_overrides", [])
with draccus.config_type("json"):
return draccus.parse(orig_config.__class__, config_file, args=cli_overrides)
return draccus.parse(config_cls, config_file, args=cli_overrides)
+2
View File
@@ -32,6 +32,7 @@ from .pretrained import PreTrainedPolicy as PreTrainedPolicy
from .smolvla.configuration_smolvla import SmolVLAConfig as SmolVLAConfig
from .tdmpc.configuration_tdmpc import TDMPCConfig as TDMPCConfig
from .utils import make_robot_action, prepare_observation_for_inference
from .vla_jepa.configuration_vla_jepa import VLAJEPAConfig as VLAJEPAConfig
from .vqbet.configuration_vqbet import VQBeTConfig as VQBeTConfig
from .wall_x.configuration_wall_x import WallXConfig as WallXConfig
from .xvla.configuration_xvla import XVLAConfig as XVLAConfig
@@ -57,6 +58,7 @@ __all__ = [
"PI05Config",
"SmolVLAConfig",
"TDMPCConfig",
"VLAJEPAConfig",
"VQBeTConfig",
"WallXConfig",
"XVLAConfig",
+2 -39
View File
@@ -18,17 +18,10 @@ from typing import Any
import torch
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
RenameObservationsProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_pre_post_processors,
)
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .configuration_act import ACTConfig
@@ -54,34 +47,4 @@ def make_act_pre_post_processors(
tuple[PolicyProcessorPipeline[dict[str, Any], dict[str, Any]], PolicyProcessorPipeline[PolicyAction, PolicyAction]]: A tuple containing the
pre-processor pipeline and the post-processor pipeline.
"""
input_steps = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
device=config.device,
),
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
DeviceProcessorStep(device="cpu"),
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_default_pre_post_processors(config, dataset_stats, normalizer_device=config.device)
@@ -19,17 +19,10 @@ from typing import Any
import torch
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
RenameObservationsProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_pre_post_processors,
)
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .configuration_diffusion import DiffusionConfig
@@ -63,32 +56,4 @@ def make_diffusion_pre_post_processors(
Returns:
A tuple containing the configured pre-processor and post-processor pipelines.
"""
input_steps = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
DeviceProcessorStep(device="cpu"),
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_default_pre_post_processors(config, dataset_stats)
+12 -37
View File
@@ -23,24 +23,16 @@ import torch
from lerobot.configs.types import FeatureType, PipelineFeatureType, PolicyFeature
from lerobot.processor import (
AddBatchDimensionProcessorStep,
ComplementaryDataProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
ProcessorStep,
ProcessorStepRegistry,
RenameObservationsProcessorStep,
UnnormalizerProcessorStep,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from lerobot.processor.converters import policy_action_to_transition, transition_to_policy_action
from lerobot.types import TransitionKey
from lerobot.utils.constants import (
OBS_STATE,
POLICY_POSTPROCESSOR_DEFAULT_NAME,
POLICY_PREPROCESSOR_DEFAULT_NAME,
)
from lerobot.utils.constants import OBS_STATE
from lerobot.utils.import_utils import _transformers_available, require_package
from .configuration_eo1 import EO1Config
@@ -242,14 +234,12 @@ def make_eo1_pre_post_processors(
]:
"""Build pre/post processor pipelines for EO1."""
steps = make_default_policy_processor_steps(config, dataset_stats)
input_steps: list[ProcessorStep] = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
steps.rename_observations,
steps.add_batch_dim,
steps.normalize,
EO1ConversationTemplateStep(input_features=config.input_features, chunk_size=config.chunk_size),
EO1QwenProcessorStep(
processor_name=config.vlm_base,
@@ -257,27 +247,12 @@ def make_eo1_pre_post_processors(
image_max_pixels=config.image_max_pixels,
use_fast_processor=config.use_fast_processor,
),
DeviceProcessorStep(device=config.device),
steps.to_device,
]
output_steps: list[ProcessorStep] = [
UnnormalizerProcessorStep(
features=config.output_features,
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
DeviceProcessorStep(device="cpu"),
steps.unnormalize,
steps.to_cpu,
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
+66 -318
View File
@@ -17,6 +17,7 @@
from __future__ import annotations
import importlib
import inspect
import logging
from typing import TYPE_CHECKING, Any, TypedDict, Unpack
@@ -44,26 +45,10 @@ from lerobot.utils.constants import (
)
from lerobot.utils.feature_utils import dataset_to_policy_features
from .act.configuration_act import ACTConfig
from .diffusion.configuration_diffusion import DiffusionConfig
from .eo1.configuration_eo1 import EO1Config
from .evo1.configuration_evo1 import Evo1Config
from .fastwam.configuration_fastwam import FastWAMConfig
from .gaussian_actor.configuration_gaussian_actor import GaussianActorConfig
from .groot.configuration_groot import GrootConfig
from .lingbot_va.configuration_lingbot_va import LingBotVAConfig
from .molmoact2.configuration_molmoact2 import MolmoAct2Config
from .multi_task_dit.configuration_multi_task_dit import MultiTaskDiTConfig
from .pi0.configuration_pi0 import PI0Config
from .pi05.configuration_pi05 import PI05Config
from .pretrained import PreTrainedPolicy
from .smolvla.configuration_smolvla import SmolVLAConfig
from .tdmpc.configuration_tdmpc import TDMPCConfig
from .utils import validate_visual_features_consistency
from .vla_jepa.configuration_vla_jepa import VLAJEPAConfig
from .vqbet.configuration_vqbet import VQBeTConfig
from .wall_x.configuration_wall_x import WallXConfig
from .xvla.configuration_xvla import XVLAConfig
def _reconnect_relative_absolute_steps(
@@ -88,100 +73,23 @@ def get_policy_class(name: str) -> type[PreTrainedPolicy]:
"""
Retrieves a policy class by its registered name.
This function uses dynamic imports to avoid loading all policy classes into memory
at once, improving startup time and reducing dependencies.
Resolution is convention-based: the draccus-registered config class of ``name`` is
looked up, its ``configuration_*`` module path is rewritten to ``modeling_*``, and
the ``<X>Policy`` class is imported from there. The modeling module is only imported
at call time, keeping heavy optional dependencies lazy. This works for both built-in
policies and third-party lerobot plugins (anything registered via
``@PreTrainedConfig.register_subclass``).
Args:
name: The name of the policy. Supported names are "tdmpc", "diffusion", "act",
"multi_task_dit", "vqbet", "pi0", "pi05", "gaussian_actor", "smolvla", "wall_x",
"molmoact2", "eo1", "evo1".
name: The registered name of the policy (e.g. "act", "diffusion", "pi0").
Returns:
The policy class corresponding to the given name.
Raises:
NotImplementedError: If the policy name is not recognized.
ValueError: If the policy name is not registered.
ImportError: If the policy's optional dependencies are not installed.
"""
if name == "tdmpc":
from .tdmpc.modeling_tdmpc import TDMPCPolicy
return TDMPCPolicy
elif name == "diffusion":
from .diffusion.modeling_diffusion import DiffusionPolicy
return DiffusionPolicy
elif name == "act":
from .act.modeling_act import ACTPolicy
return ACTPolicy
elif name == "multi_task_dit":
from .multi_task_dit.modeling_multi_task_dit import MultiTaskDiTPolicy
return MultiTaskDiTPolicy
elif name == "vqbet":
from .vqbet.modeling_vqbet import VQBeTPolicy
return VQBeTPolicy
elif name == "pi0":
from .pi0.modeling_pi0 import PI0Policy
return PI0Policy
elif name == "pi0_fast":
from .pi0_fast.modeling_pi0_fast import PI0FastPolicy
return PI0FastPolicy
elif name == "pi05":
from .pi05.modeling_pi05 import PI05Policy
return PI05Policy
elif name == "gaussian_actor":
from .gaussian_actor.modeling_gaussian_actor import GaussianActorPolicy
return GaussianActorPolicy
elif name == "smolvla":
from .smolvla.modeling_smolvla import SmolVLAPolicy
return SmolVLAPolicy
elif name == "groot":
from .groot.modeling_groot import GrootPolicy
return GrootPolicy
elif name == "xvla":
from .xvla.modeling_xvla import XVLAPolicy
return XVLAPolicy
elif name == "wall_x":
from .wall_x.modeling_wall_x import WallXPolicy
return WallXPolicy
elif name == "eo1":
from .eo1.modeling_eo1 import EO1Policy
return EO1Policy
elif name == "molmoact2":
from .molmoact2.modeling_molmoact2 import MolmoAct2Policy
return MolmoAct2Policy
elif name == "vla_jepa":
from .vla_jepa.modeling_vla_jepa import VLAJEPAPolicy
return VLAJEPAPolicy
elif name == "lingbot_va":
from .lingbot_va.modeling_lingbot_va import LingBotVAPolicy
return LingBotVAPolicy
elif name == "fastwam":
from .fastwam.modeling_fastwam import FastWAMPolicy
return FastWAMPolicy
elif name == "evo1":
from .evo1.modeling_evo1 import Evo1Policy
return Evo1Policy
else:
try:
return _get_policy_cls_from_policy_name(name=name)
except Exception as e:
raise ValueError(f"Policy type '{name}' is not available.") from e
return _get_policy_cls_from_policy_name(name=name)
def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
@@ -192,9 +100,8 @@ def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
mapping a string identifier to the corresponding config class.
Args:
policy_type: The type of the policy. Supported types include "tdmpc",
"multi_task_dit", "diffusion", "act", "vqbet", "pi0", "pi05", "gaussian_actor",
"smolvla", "wall_x", "molmoact2", "eo1", "evo1".
policy_type: The registered type of the policy (any name registered via
``@PreTrainedConfig.register_subclass``, e.g. "act", "diffusion", "pi0").
**kwargs: Keyword arguments to be passed to the configuration class constructor.
Returns:
@@ -203,48 +110,11 @@ def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
Raises:
ValueError: If the `policy_type` is not recognized.
"""
if policy_type == "tdmpc":
return TDMPCConfig(**kwargs)
elif policy_type == "diffusion":
return DiffusionConfig(**kwargs)
elif policy_type == "act":
return ACTConfig(**kwargs)
elif policy_type == "multi_task_dit":
return MultiTaskDiTConfig(**kwargs)
elif policy_type == "vqbet":
return VQBeTConfig(**kwargs)
elif policy_type == "pi0":
return PI0Config(**kwargs)
elif policy_type == "pi05":
return PI05Config(**kwargs)
elif policy_type == "gaussian_actor":
return GaussianActorConfig(**kwargs)
elif policy_type == "smolvla":
return SmolVLAConfig(**kwargs)
elif policy_type == "groot":
return GrootConfig(**kwargs)
elif policy_type == "xvla":
return XVLAConfig(**kwargs)
elif policy_type == "wall_x":
return WallXConfig(**kwargs)
elif policy_type == "eo1":
return EO1Config(**kwargs)
elif policy_type == "molmoact2":
return MolmoAct2Config(**kwargs)
elif policy_type == "vla_jepa":
return VLAJEPAConfig(**kwargs)
elif policy_type == "lingbot_va":
return LingBotVAConfig(**kwargs)
elif policy_type == "fastwam":
return FastWAMConfig(**kwargs)
elif policy_type == "evo1":
return Evo1Config(**kwargs)
else:
try:
config_cls = PreTrainedConfig.get_choice_class(policy_type)
return config_cls(**kwargs)
except Exception as e:
raise ValueError(f"Policy type '{policy_type}' is not available.") from e
try:
config_cls = PreTrainedConfig.get_choice_class(policy_type)
except Exception as e:
raise ValueError(f"Policy type '{policy_type}' is not available.") from e
return config_cls(**kwargs)
class ProcessorConfigKwargs(TypedDict, total=False):
@@ -298,8 +168,7 @@ def make_pre_post_processors(
A tuple containing the input (pre-processor) and output (post-processor) pipelines.
Raises:
NotImplementedError: If a processor factory is not implemented for the given
policy configuration type.
ValueError: If no processor factory exists for the given policy configuration type.
"""
if pretrained_path:
if isinstance(policy_cfg, GrootConfig):
@@ -351,166 +220,13 @@ def make_pre_post_processors(
)
return preprocessor, postprocessor
# Create a new processor based on policy type
if isinstance(policy_cfg, TDMPCConfig):
from .tdmpc.processor_tdmpc import make_tdmpc_pre_post_processors
processors = make_tdmpc_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, DiffusionConfig):
from .diffusion.processor_diffusion import make_diffusion_pre_post_processors
processors = make_diffusion_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, ACTConfig):
from .act.processor_act import make_act_pre_post_processors
processors = make_act_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, MultiTaskDiTConfig):
from .multi_task_dit.processor_multi_task_dit import (
make_multi_task_dit_pre_post_processors,
)
processors = make_multi_task_dit_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, VQBeTConfig):
from .vqbet.processor_vqbet import make_vqbet_pre_post_processors
processors = make_vqbet_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, PI0Config):
from .pi0.processor_pi0 import make_pi0_pre_post_processors
processors = make_pi0_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, PI05Config):
from .pi05.processor_pi05 import make_pi05_pre_post_processors
processors = make_pi05_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, GaussianActorConfig):
from .gaussian_actor.processor_gaussian_actor import make_gaussian_actor_pre_post_processors
processors = make_gaussian_actor_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, SmolVLAConfig):
from .smolvla.processor_smolvla import make_smolvla_pre_post_processors
processors = make_smolvla_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, GrootConfig):
from .groot.processor_groot import make_groot_pre_post_processors
processors = make_groot_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
dataset_meta=kwargs.get("dataset_meta"),
)
elif isinstance(policy_cfg, XVLAConfig):
from .xvla.processor_xvla import (
make_xvla_pre_post_processors,
)
processors = make_xvla_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, WallXConfig):
from .wall_x.processor_wall_x import make_wall_x_pre_post_processors
processors = make_wall_x_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, EO1Config):
from .eo1.processor_eo1 import make_eo1_pre_post_processors
processors = make_eo1_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, Evo1Config):
from .evo1.processor_evo1 import make_evo1_pre_post_processors
processors = make_evo1_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, MolmoAct2Config):
from .molmoact2.processor_molmoact2 import make_molmoact2_pre_post_processors
processors = make_molmoact2_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
dataset_meta=kwargs.get("dataset_meta"),
)
elif isinstance(policy_cfg, VLAJEPAConfig):
from .vla_jepa.processor_vla_jepa import make_vla_jepa_pre_post_processors
processors = make_vla_jepa_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, LingBotVAConfig):
from .lingbot_va.processor_lingbot_va import make_lingbot_va_pre_post_processors
processors = make_lingbot_va_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
elif isinstance(policy_cfg, FastWAMConfig):
from .fastwam.processor_fastwam import make_fastwam_pre_post_processors
processors = make_fastwam_pre_post_processors(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
else:
try:
processors = _make_processors_from_policy_config(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
)
except Exception as e:
raise ValueError(f"Processor for policy type '{policy_cfg.type}' is not implemented.") from e
return processors
# Create new processors from the policy config, resolving the per-policy factory
# function by naming convention (lazy import keeps optional dependencies optional).
return _make_processors_from_policy_config(
config=policy_cfg,
dataset_stats=kwargs.get("dataset_stats"),
dataset_meta=kwargs.get("dataset_meta"),
)
def make_policy(
@@ -654,10 +370,12 @@ def make_policy(
return policy
def _get_policy_cls_from_policy_name(name: str) -> type[PreTrainedConfig]:
def _get_policy_cls_from_policy_name(name: str) -> type[PreTrainedPolicy]:
"""Get policy class from its registered name using dynamic imports.
This is used as a helper function to import policies from 3rd party lerobot plugins.
Works for built-in policies and 3rd party lerobot plugins alike: the config class
registered under ``name`` is resolved via the draccus ChoiceRegistry, and the policy
class is imported from the sibling ``modeling_*`` module by naming convention.
Args:
name: The name of the policy.
@@ -683,22 +401,39 @@ def _get_policy_cls_from_policy_name(name: str) -> type[PreTrainedConfig]:
"configuration_", "modeling_"
) # e.g., configuration_diffusion -> modeling_diffusion
module = importlib.import_module(module_path)
policy_cls = getattr(module, cls_name)
try:
module = importlib.import_module(module_path)
except ModuleNotFoundError as e:
if e.name == module_path:
# The modeling_* module itself does not exist for this policy type. A missing
# optional dependency inside an existing module propagates unchanged instead,
# so its actionable install hint stays visible.
raise ValueError(f"Policy class for '{name}' is not implemented.") from e
raise
policy_cls = getattr(module, cls_name, None)
if policy_cls is None:
raise ValueError(
f"Policy class '{cls_name}' not found in '{module_path}'. "
f"Policies must expose '<Name>Policy' in the sibling 'modeling_*' module by naming convention."
)
return policy_cls
def _make_processors_from_policy_config(
config: PreTrainedConfig,
dataset_stats: dict[str, dict[str, torch.Tensor]] | None = None,
dataset_meta: Any | None = None,
) -> tuple[Any, Any]:
"""Create pre- and post-processors from a policy configuration using dynamic imports.
This is used as a helper function to import processor factories from 3rd party lerobot plugins.
Resolves ``make_{type}_pre_post_processors`` from the policy's ``processor_*`` module
by naming convention. Works for built-in policies and 3rd party lerobot plugins.
Args:
config: The policy configuration object.
dataset_stats: Dataset statistics for normalization.
dataset_meta: Dataset metadata, forwarded only to factories that declare a
``dataset_meta`` parameter (e.g. groot, molmoact2).
Returns:
A tuple containing the input (pre-processor) and output (post-processor) pipelines.
"""
@@ -711,6 +446,19 @@ def _make_processors_from_policy_config(
logging.debug(
f"Instantiating pre/post processors using function '{function_name}' from module '{module_path}'"
)
module = importlib.import_module(module_path)
function = getattr(module, function_name)
return function(config, dataset_stats=dataset_stats)
try:
module = importlib.import_module(module_path)
except ModuleNotFoundError as e:
if e.name == module_path:
# The processor_* module itself does not exist for this policy type. A missing
# optional dependency inside an existing module propagates unchanged instead,
# so its actionable install hint stays visible.
raise ValueError(f"Processor for policy type '{policy_type}' is not implemented.") from e
raise
function = getattr(module, function_name, None)
if function is None:
raise ValueError(f"Processor for policy type '{policy_type}' is not implemented.")
call_kwargs: dict[str, Any] = {"dataset_stats": dataset_stats}
if "dataset_meta" in inspect.signature(function).parameters:
call_kwargs["dataset_meta"] = dataset_meta
return function(config, **call_kwargs)
@@ -22,20 +22,11 @@ import torch
from lerobot.configs import PipelineFeatureType, PolicyFeature
from lerobot.processor import (
ActionProcessorStep,
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
ProcessorStepRegistry,
RenameObservationsProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
)
from lerobot.utils.constants import (
POLICY_POSTPROCESSOR_DEFAULT_NAME,
POLICY_PREPROCESSOR_DEFAULT_NAME,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from .configuration_fastwam import FastWAMConfig
@@ -105,38 +96,20 @@ def make_fastwam_pre_post_processors(
# anyway) and unsafe across fine-tuning: its `resize_size` would be inherited from the base
# checkpoint's camera geometry, not this dataset's, making the concatenation N_cameras x too wide.
steps = make_default_policy_processor_steps(config, normalization_stats, normalizer_device=config.device)
input_steps = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=normalization_stats,
device=config.device,
),
steps.rename_observations,
steps.add_batch_dim,
steps.to_device,
steps.normalize,
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features,
norm_map=config.normalization_mapping,
stats=normalization_stats,
),
steps.unnormalize,
]
if config.toggle_action_dimensions:
output_steps.append(
FastWAMActionToggleProcessorStep(toggle_dimensions=config.toggle_action_dimensions)
)
output_steps.append(DeviceProcessorStep(device="cpu"))
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
output_steps.append(steps.to_cpu)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
@@ -20,17 +20,10 @@ from typing import Any
import torch
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
RenameObservationsProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_pre_post_processors,
)
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .configuration_gaussian_actor import GaussianActorConfig
@@ -62,33 +55,4 @@ def make_gaussian_actor_pre_post_processors(
Returns:
A tuple containing the configured pre-processor and post-processor pipelines.
"""
# Add remaining processors
input_steps = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
DeviceProcessorStep(device="cpu"),
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_default_pre_post_processors(config, dataset_stats)
@@ -25,19 +25,12 @@ import torch
from lerobot.configs.types import FeatureType, NormalizationMode
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
ProcessorStep,
RenameObservationsProcessorStep,
UnnormalizerProcessorStep,
)
from lerobot.processor.converters import policy_action_to_transition, transition_to_policy_action
from lerobot.utils.constants import (
POLICY_POSTPROCESSOR_DEFAULT_NAME,
POLICY_PREPROCESSOR_DEFAULT_NAME,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from .configuration_lingbot_va import LingBotVAConfig
@@ -52,15 +45,13 @@ def make_lingbot_va_pre_post_processors(
]:
"""Build the pre/post processor pipelines for LingBot-VA."""
steps = make_default_policy_processor_steps(config, dataset_stats)
input_steps: list[ProcessorStep] = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
DeviceProcessorStep(device=config.device),
steps.rename_observations,
steps.add_batch_dim,
steps.normalize,
steps.to_device,
]
# Unnormalize actions from [-1, 1] to physical units (QUANTILES) using q01/q99 restored from the checkpoint.
@@ -70,18 +61,7 @@ def make_lingbot_va_pre_post_processors(
norm_map={FeatureType.ACTION: NormalizationMode.QUANTILES},
stats=dataset_stats,
),
DeviceProcessorStep(device="cpu"),
steps.to_cpu,
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
@@ -19,18 +19,12 @@ from typing import Any
import torch
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
RenameObservationsProcessorStep,
TokenizerProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .configuration_multi_task_dit import MultiTaskDiTConfig
@@ -66,9 +60,11 @@ def make_multi_task_dit_pre_post_processors(
A tuple containing the configured pre-processor and post-processor pipelines.
"""
steps = make_default_policy_processor_steps(config, dataset_stats, normalizer_device=config.device)
input_steps = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
steps.rename_observations,
steps.add_batch_dim,
TokenizerProcessorStep(
tokenizer_name=config.text_encoder_name,
padding=config.tokenizer_padding,
@@ -76,32 +72,12 @@ def make_multi_task_dit_pre_post_processors(
max_length=config.tokenizer_max_length,
truncation=config.tokenizer_truncation,
),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
device=config.device,
),
steps.to_device,
steps.normalize,
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features,
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
DeviceProcessorStep(device="cpu"),
steps.unnormalize,
steps.to_cpu,
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
+11 -32
View File
@@ -21,22 +21,16 @@ import torch
from lerobot.configs import PipelineFeatureType, PolicyFeature
from lerobot.processor import (
AbsoluteActionsProcessorStep,
AddBatchDimensionProcessorStep,
ComplementaryDataProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
ProcessorStep,
ProcessorStepRegistry,
RelativeActionsProcessorStep,
RenameObservationsProcessorStep,
TokenizerProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .configuration_pi0 import PI0Config
@@ -136,10 +130,12 @@ def make_pi0_pre_post_processors(
action_names=getattr(config, "action_feature_names", None),
)
steps = make_default_policy_processor_steps(config, dataset_stats)
# OpenPI order: raw → relative → normalize → model → unnormalize → absolute
input_steps: list[ProcessorStep] = [
RenameObservationsProcessorStep(rename_map={}), # To mimic the same processor as pretrained one
AddBatchDimensionProcessorStep(),
steps.rename_observations, # To mimic the same processor as pretrained one
steps.add_batch_dim,
Pi0NewLineProcessor(), # Add newlines before tokenization for PaliGemma
TokenizerProcessorStep(
tokenizer_name="google/paligemma-3b-pt-224",
@@ -147,32 +143,15 @@ def make_pi0_pre_post_processors(
padding_side="right",
padding="max_length",
),
DeviceProcessorStep(device=config.device),
steps.to_device,
relative_step,
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
steps.normalize,
]
output_steps: list[ProcessorStep] = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
steps.unnormalize,
AbsoluteActionsProcessorStep(enabled=config.use_relative_actions, relative_step=relative_step),
DeviceProcessorStep(device="cpu"),
steps.to_cpu,
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
+12 -36
View File
@@ -24,26 +24,17 @@ import torch
from lerobot.configs import PipelineFeatureType, PolicyFeature
from lerobot.processor import (
AbsoluteActionsProcessorStep,
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
ProcessorStep,
ProcessorStepRegistry,
RelativeActionsProcessorStep,
RenameObservationsProcessorStep,
TokenizerProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from lerobot.types import EnvTransition, TransitionKey
from lerobot.utils.constants import (
OBS_STATE,
POLICY_POSTPROCESSOR_DEFAULT_NAME,
POLICY_PREPROCESSOR_DEFAULT_NAME,
)
from lerobot.utils.constants import OBS_STATE
from .configuration_pi05 import PI05Config
@@ -135,18 +126,16 @@ def make_pi05_pre_post_processors(
action_names=getattr(config, "action_feature_names", None),
)
steps = make_default_policy_processor_steps(config, dataset_stats)
# OpenPI order: raw → relative → normalize → model → unnormalize → absolute
input_steps: list[ProcessorStep] = [
RenameObservationsProcessorStep(rename_map={}), # To mimic the same processor as pretrained one
AddBatchDimensionProcessorStep(),
steps.rename_observations, # To mimic the same processor as pretrained one
steps.add_batch_dim,
relative_step,
# NOTE: NormalizerProcessorStep MUST come before Pi05PrepareStateTokenizerProcessorStep
# because the tokenizer step expects normalized state in [-1, 1] range for discretization
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
steps.normalize,
Pi05PrepareStateTokenizerProcessorStep(max_state_dim=config.max_state_dim),
TokenizerProcessorStep(
tokenizer_name="google/paligemma-3b-pt-224",
@@ -154,26 +143,13 @@ def make_pi05_pre_post_processors(
padding_side="right",
padding="max_length",
),
DeviceProcessorStep(device=config.device),
steps.to_device,
]
output_steps: list[ProcessorStep] = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
steps.unnormalize,
AbsoluteActionsProcessorStep(enabled=config.use_relative_actions, relative_step=relative_step),
DeviceProcessorStep(device="cpu"),
steps.to_cpu,
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
@@ -25,26 +25,17 @@ from lerobot.configs import PipelineFeatureType, PolicyFeature
from lerobot.processor import (
AbsoluteActionsProcessorStep,
ActionTokenizerProcessorStep,
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
ProcessorStep,
ProcessorStepRegistry,
RelativeActionsProcessorStep,
RenameObservationsProcessorStep,
TokenizerProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from lerobot.types import EnvTransition, TransitionKey
from lerobot.utils.constants import (
OBS_STATE,
POLICY_POSTPROCESSOR_DEFAULT_NAME,
POLICY_PREPROCESSOR_DEFAULT_NAME,
)
from lerobot.utils.constants import OBS_STATE
from .configuration_pi0_fast import PI0FastConfig
@@ -135,6 +126,8 @@ def make_pi0_fast_pre_post_processors(
action_names=getattr(config, "action_feature_names", None),
)
steps = make_default_policy_processor_steps(config, dataset_stats)
# Pi0Fast order: relative → normalize → tokenize → model → unnormalize → absolute
# This matches pi0/pi0.5: RelativeActionsProcessorStep runs first on raw absolute actions,
# caching the raw state. NormalizerProcessorStep then normalizes the raw relative actions,
@@ -144,14 +137,10 @@ def make_pi0_fast_pre_post_processors(
# before Pi0FastPrepareStateAndLanguageTokenizerProcessorStep, so the state tokenizer
# continues to receive normalized state in [-1, 1] as expected.
input_steps: list[ProcessorStep] = [
RenameObservationsProcessorStep(rename_map={}), # To mimic the same processor as pretrained one
AddBatchDimensionProcessorStep(),
steps.rename_observations, # To mimic the same processor as pretrained one
steps.add_batch_dim,
relative_step,
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
steps.normalize,
Pi0FastPrepareStateAndLanguageTokenizerProcessorStep(max_state_dim=config.max_state_dim),
TokenizerProcessorStep(
tokenizer_name=config.text_tokenizer_name,
@@ -165,26 +154,13 @@ def make_pi0_fast_pre_post_processors(
fast_skip_tokens=config.fast_skip_tokens,
paligemma_tokenizer_name=config.text_tokenizer_name,
),
DeviceProcessorStep(device=config.device),
steps.to_device,
]
output_steps: list[ProcessorStep] = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
steps.unnormalize,
AbsoluteActionsProcessorStep(enabled=config.use_relative_actions, relative_step=relative_step),
DeviceProcessorStep(device="cpu"),
steps.to_cpu,
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
@@ -19,19 +19,13 @@ from typing import Any
import torch
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NewLineTaskProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
RenameObservationsProcessorStep,
TokenizerProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .configuration_smolvla import SmolVLAConfig
@@ -66,9 +60,11 @@ def make_smolvla_pre_post_processors(
A tuple containing the configured pre-processor and post-processor pipelines.
"""
steps = make_default_policy_processor_steps(config, dataset_stats)
input_steps = [
RenameObservationsProcessorStep(rename_map={}), # To mimic the same processor as pretrained one
AddBatchDimensionProcessorStep(),
steps.rename_observations, # To mimic the same processor as pretrained one
steps.add_batch_dim,
NewLineTaskProcessorStep(),
TokenizerProcessorStep(
tokenizer_name=config.vlm_model_name,
@@ -76,28 +72,11 @@ def make_smolvla_pre_post_processors(
padding_side="right",
max_length=config.tokenizer_max_length,
),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
steps.to_device,
steps.normalize,
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
DeviceProcessorStep(device="cpu"),
steps.unnormalize,
steps.to_cpu,
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
+2 -37
View File
@@ -19,17 +19,10 @@ from typing import Any
import torch
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
RenameObservationsProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_pre_post_processors,
)
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .configuration_tdmpc import TDMPCConfig
@@ -61,32 +54,4 @@ def make_tdmpc_pre_post_processors(
Returns:
A tuple containing the configured pre-processor and post-processor pipelines.
"""
input_steps = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
DeviceProcessorStep(device="cpu"),
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_default_pre_post_processors(config, dataset_stats)
@@ -20,20 +20,16 @@ import torch
from lerobot.policies.vla_jepa.configuration_vla_jepa import VLAJEPAConfig
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
EnvTransition,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
ProcessorStep,
ProcessorStepRegistry,
RenameObservationsProcessorStep,
TransitionKey,
UnnormalizerProcessorStep,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from lerobot.processor.converters import policy_action_to_transition, transition_to_policy_action
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
@ProcessorStepRegistry.register(name="vla_jepa_clip_actions")
@@ -112,15 +108,12 @@ def make_vla_jepa_pre_post_processors(
PolicyProcessorPipeline[PolicyAction, PolicyAction],
]:
features = {**config.input_features, **config.output_features}
steps = make_default_policy_processor_steps(config, dataset_stats)
input_steps = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features=features,
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
steps.rename_observations,
steps.add_batch_dim,
steps.to_device,
steps.normalize,
]
output_steps: list[ProcessorStep] = []
if config.clip_normalized_actions:
@@ -129,6 +122,8 @@ def make_vla_jepa_pre_post_processors(
output_steps.append(
PreSnapGripperProcessorStep(gripper_dim=config.gripper_dim, threshold=config.gripper_threshold)
)
# NOTE: unlike the default policy unnormalizer (output features only), VLA-JEPA
# unnormalizes over BOTH input and output features.
output_steps.append(
UnnormalizerProcessorStep(
features=features,
@@ -140,16 +135,5 @@ def make_vla_jepa_pre_post_processors(
output_steps.append(
BinarizeGripperProcessorStep(gripper_dim=config.gripper_dim, threshold=config.gripper_threshold)
)
output_steps.append(DeviceProcessorStep(device="cpu"))
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
output_steps.append(steps.to_cpu)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
+2 -37
View File
@@ -20,17 +20,10 @@ from typing import Any
import torch
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
RenameObservationsProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_pre_post_processors,
)
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .configuration_vqbet import VQBeTConfig
@@ -62,32 +55,4 @@ def make_vqbet_pre_post_processors(
Returns:
A tuple containing the configured pre-processor and post-processor pipelines.
"""
input_steps = [
RenameObservationsProcessorStep(rename_map={}), # Let the possibility to the user to rename the keys
AddBatchDimensionProcessorStep(),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
DeviceProcessorStep(device="cpu"),
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_default_pre_post_processors(config, dataset_stats)
+11 -32
View File
@@ -20,19 +20,13 @@ import torch
from lerobot.configs import PipelineFeatureType, PolicyFeature
from lerobot.processor import (
AddBatchDimensionProcessorStep,
ComplementaryDataProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
ProcessorStepRegistry,
RenameObservationsProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .configuration_wall_x import WallXConfig
@@ -65,37 +59,22 @@ def make_wall_x_pre_post_processors(
A tuple containing the configured pre-processor and post-processor pipelines
"""
steps = make_default_policy_processor_steps(config, dataset_stats)
input_steps = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
steps.rename_observations,
steps.add_batch_dim,
WallXTaskProcessor(), # Process task description
NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
DeviceProcessorStep(device=config.device),
steps.normalize,
steps.to_device,
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
DeviceProcessorStep(device="cpu"),
steps.unnormalize,
steps.to_cpu,
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
@ProcessorStepRegistry.register(name="wall_x_task_processor")
+11 -34
View File
@@ -22,19 +22,14 @@ import torch
from lerobot.configs import PipelineFeatureType, PolicyFeature
from lerobot.processor import (
AddBatchDimensionProcessorStep,
DeviceProcessorStep,
NormalizerProcessorStep,
ObservationProcessorStep,
PolicyAction,
PolicyProcessorPipeline,
ProcessorStep,
ProcessorStepRegistry,
RenameObservationsProcessorStep,
TokenizerProcessorStep,
UnnormalizerProcessorStep,
policy_action_to_transition,
transition_to_policy_action,
make_default_policy_processor_steps,
make_policy_processor_pipelines,
)
from lerobot.types import EnvTransition, TransitionKey
from lerobot.utils.constants import (
@@ -42,8 +37,6 @@ from lerobot.utils.constants import (
OBS_IMAGES,
OBS_PREFIX,
OBS_STATE,
POLICY_POSTPROCESSOR_DEFAULT_NAME,
POLICY_PREPROCESSOR_DEFAULT_NAME,
)
from .configuration_xvla import XVLAConfig
@@ -61,10 +54,11 @@ def make_xvla_pre_post_processors(
Build the LeRobot processor pipelines for XVLA.
"""
features = {**config.input_features, **config.output_features}
steps = make_default_policy_processor_steps(config, dataset_stats)
input_steps = [
RenameObservationsProcessorStep(rename_map={}),
AddBatchDimensionProcessorStep(),
steps.rename_observations,
steps.add_batch_dim,
TokenizerProcessorStep(
tokenizer_name=config.tokenizer_name,
max_length=config.tokenizer_max_length,
@@ -74,32 +68,15 @@ def make_xvla_pre_post_processors(
XVLAImageToFloatProcessorStep(),
XVLAImageNetNormalizeProcessorStep(),
XVLAAddDomainIdProcessorStep(),
DeviceProcessorStep(device=config.device),
NormalizerProcessorStep(
features=features, norm_map=config.normalization_mapping, stats=dataset_stats
),
steps.to_device,
steps.normalize,
]
output_steps = [
UnnormalizerProcessorStep(
features=config.output_features,
norm_map=config.normalization_mapping,
stats=dataset_stats,
),
DeviceProcessorStep(device="cpu"),
steps.unnormalize,
steps.to_cpu,
]
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
return make_policy_processor_pipelines(input_steps=input_steps, output_steps=output_steps)
# Custom XVLA processor steps
+8
View File
@@ -42,10 +42,14 @@ from .delta_action_processor import MapDeltaActionToRobotActionStep, MapTensorTo
from .device_processor import DeviceProcessorStep
from .env_processor import IsaaclabArenaProcessorStep, LiberoProcessorStep
from .factory import (
DefaultPolicyProcessorSteps,
make_default_policy_processor_steps,
make_default_pre_post_processors,
make_default_processors,
make_default_robot_action_processor,
make_default_robot_observation_processor,
make_default_teleop_action_processor,
make_policy_processor_pipelines,
)
from .gym_action_processor import (
Numpy2TorchActionProcessorStep,
@@ -129,10 +133,14 @@ __all__ = [
"ImageCropResizeProcessorStep",
"InfoProcessorStep",
"InterventionActionProcessorStep",
"DefaultPolicyProcessorSteps",
"make_default_policy_processor_steps",
"make_default_pre_post_processors",
"make_default_processors",
"make_default_teleop_action_processor",
"make_default_robot_action_processor",
"make_default_robot_observation_processor",
"make_policy_processor_pipelines",
"AbsoluteActionsProcessorStep",
"RelativeActionsProcessorStep",
"MapDeltaActionToRobotActionStep",
+114 -2
View File
@@ -14,15 +14,33 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from lerobot.types import RobotAction, RobotObservation
from dataclasses import dataclass
from typing import Any
import torch
from lerobot.configs.policies import PreTrainedConfig
from lerobot.types import PolicyAction, RobotAction, RobotObservation
from lerobot.utils.constants import POLICY_POSTPROCESSOR_DEFAULT_NAME, POLICY_PREPROCESSOR_DEFAULT_NAME
from .batch_processor import AddBatchDimensionProcessorStep
from .converters import (
observation_to_transition,
policy_action_to_transition,
robot_action_observation_to_transition,
transition_to_observation,
transition_to_policy_action,
transition_to_robot_action,
)
from .pipeline import IdentityProcessorStep, RobotProcessorPipeline
from .device_processor import DeviceProcessorStep
from .normalize_processor import NormalizerProcessorStep, UnnormalizerProcessorStep
from .pipeline import (
IdentityProcessorStep,
PolicyProcessorPipeline,
ProcessorStep,
RobotProcessorPipeline,
)
from .rename_processor import RenameObservationsProcessorStep
def make_default_teleop_action_processor() -> RobotProcessorPipeline[
@@ -61,3 +79,97 @@ def make_default_processors():
robot_action_processor = make_default_robot_action_processor()
robot_observation_processor = make_default_robot_observation_processor()
return (teleop_action_processor, robot_action_processor, robot_observation_processor)
@dataclass
class DefaultPolicyProcessorSteps:
"""The canonical processor steps shared by most policies' pre/post pipelines.
Policies compose these in their own order (step ORDER is a Hub-serialized contract
and intentionally stays explicit per policy) and interleave their custom steps.
"""
rename_observations: RenameObservationsProcessorStep
add_batch_dim: AddBatchDimensionProcessorStep
to_device: DeviceProcessorStep
normalize: NormalizerProcessorStep
unnormalize: UnnormalizerProcessorStep
to_cpu: DeviceProcessorStep
def make_default_policy_processor_steps(
config: PreTrainedConfig,
dataset_stats: dict[str, dict[str, torch.Tensor]] | None = None,
*,
normalizer_device: torch.device | str | None = None,
) -> DefaultPolicyProcessorSteps:
"""Construct the canonical policy processor steps from a policy config.
Args:
config: A `PreTrainedConfig` providing `device`, `input_features`,
`output_features` and `normalization_mapping`.
dataset_stats: Dataset statistics used for (un)normalization.
normalizer_device: Device passed to `NormalizerProcessorStep` (some policies pin
their normalization stats to the policy device; most leave it unset).
"""
return DefaultPolicyProcessorSteps(
rename_observations=RenameObservationsProcessorStep(rename_map={}),
add_batch_dim=AddBatchDimensionProcessorStep(),
to_device=DeviceProcessorStep(device=config.device),
normalize=NormalizerProcessorStep(
features={**config.input_features, **config.output_features},
norm_map=config.normalization_mapping,
stats=dataset_stats,
device=normalizer_device,
),
unnormalize=UnnormalizerProcessorStep(
features=config.output_features, norm_map=config.normalization_mapping, stats=dataset_stats
),
to_cpu=DeviceProcessorStep(device="cpu"),
)
def make_policy_processor_pipelines(
input_steps: list[ProcessorStep],
output_steps: list[ProcessorStep],
) -> tuple[
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
PolicyProcessorPipeline[PolicyAction, PolicyAction],
]:
"""Wrap pre/post step lists into the canonical policy pipeline pair.
Uses the standard pipeline names (which determine the serialized JSON filenames on
the Hub) and the standard policy-action converters on the postprocessor.
"""
return (
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
steps=input_steps,
name=POLICY_PREPROCESSOR_DEFAULT_NAME,
),
PolicyProcessorPipeline[PolicyAction, PolicyAction](
steps=output_steps,
name=POLICY_POSTPROCESSOR_DEFAULT_NAME,
to_transition=policy_action_to_transition,
to_output=transition_to_policy_action,
),
)
def make_default_pre_post_processors(
config: PreTrainedConfig,
dataset_stats: dict[str, dict[str, torch.Tensor]] | None = None,
*,
normalizer_device: torch.device | str | None = None,
) -> tuple[
PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
PolicyProcessorPipeline[PolicyAction, PolicyAction],
]:
"""The pure-scaffold policy pipeline pair: Rename -> Batch -> Device -> Normalize,
and Unnormalize -> Device(cpu). Policies with custom steps or a different step order
compose `make_default_policy_processor_steps` themselves instead.
"""
s = make_default_policy_processor_steps(config, dataset_stats, normalizer_device=normalizer_device)
return make_policy_processor_pipelines(
input_steps=[s.rename_observations, s.add_batch_dim, s.to_device, s.normalize],
output_steps=[s.unnormalize, s.to_cpu],
)
@@ -126,7 +126,7 @@ class RelativeActionsProcessorStep(ProcessorStep):
observation = transition.get(TransitionKey.OBSERVATION, {})
state = observation.get(OBS_STATE) if observation else None
# Always cache state for the paired AbsoluteActionsProcessorStep
# Always cache state for the paired AbsoluteActionsProcessorStep.
if state is not None:
self._last_state = state
@@ -146,6 +146,11 @@ class RelativeActionsProcessorStep(ProcessorStep):
"""Return the cached ``observation.state`` used as the reference point for relative/absolute action conversions."""
return self._last_state
def set_cached_state(self, state: torch.Tensor | None) -> None:
"""Override the cached anchor state, e.g. to re-pin a chunk's anchor after the
per-tick pipeline overwrote it (see ``SyncInferenceEngine``)."""
self._last_state = state
def get_config(self) -> dict[str, Any]:
return {
"enabled": self.enabled,
-11
View File
@@ -43,7 +43,6 @@ from lerobot.processor import (
make_default_processors,
rename_stats,
)
from lerobot.processor.relative_action_processor import RelativeActionsProcessorStep
from lerobot.robots import make_robot_from_config
from lerobot.teleoperators import Teleoperator, make_teleoperator_from_config
from lerobot.utils.feature_utils import combine_feature_dicts, hw_to_dataset_features
@@ -52,7 +51,6 @@ from .configs import BaseStrategyConfig, DAggerStrategyConfig, RolloutConfig
from .inference import (
InferenceEngine,
RTCInferenceConfig,
SyncInferenceConfig,
create_inference_engine,
)
from .robot_wrapper import ThreadSafeRobot
@@ -399,15 +397,6 @@ def build_rollout_context(
},
)
if isinstance(cfg.inference, SyncInferenceConfig) and any(
isinstance(step, RelativeActionsProcessorStep) and step.enabled
for step in getattr(preprocessor, "steps", ())
):
raise NotImplementedError(
"SyncInferenceEngine does not support policies with relative actions for now."
"Use --inference.type=rtc or remove relative action processor steps from the policy pipeline."
)
# --- 7. Inference strategy (needs policy + pre/post + hardware) --
logger.info(
"Creating inference engine (type=%s)...",
+73 -14
View File
@@ -24,26 +24,21 @@ import torch
from lerobot.policies.pretrained import PreTrainedPolicy
from lerobot.policies.utils import make_robot_action, prepare_observation_for_inference
from lerobot.processor import PolicyProcessorPipeline
from lerobot.processor import PolicyProcessorPipeline, RelativeActionsProcessorStep
from .base import InferenceEngine
logger = logging.getLogger(__name__)
# TODO(Steven): support relative-action policies. The per-tick flow refreshes
# ``RelativeActionsProcessorStep._last_state`` every call, so cached chunk
# actions popped on later ticks get reanchored to the *current* robot state and
# absolute targets drift through the chunk. Relative-action policies are
# rejected at context-build time today; RTC postprocesses the whole chunk and
# is unaffected.
#
# Candidate fix: drive the policy via ``predict_action_chunk`` and serve a
# local FIFO of postprocessed actions. Eliminates drift by construction and
# saves per-tick pre/post work, but bypasses ``select_action`` — needs
# fallbacks for SAC (raises), ACT temporal ensembling (ensembler lives in
# ``select_action``), and Diffusion-family (obs-history queues populated as a
# side effect of ``select_action``).
# Relative-action support: a predicted chunk of offsets is anchored to the robot
# state at prediction time, but the sync engine reruns the pre/post pipeline every
# tick, so ``RelativeActionsProcessorStep`` would re-anchor cached actions to the
# current (moved) state and drift through the chunk. We pin the anchor per chunk:
# a probe on the policy's public ``predict_action_chunk`` flags the ticks that
# predict a fresh chunk; on the others the engine restores the anchor the relative
# step overwrote. ``select_action`` stays on the hot path, so per-tick side effects
# (e.g. LingBot-VA keyframe feedback) are preserved.
class SyncInferenceEngine(InferenceEngine):
@@ -73,6 +68,31 @@ class SyncInferenceEngine(InferenceEngine):
self._task = task
self._device = torch.device(device or "cpu")
self._robot_type = robot_type
# Find an enabled RelativeActionsProcessorStep to pin its anchor per chunk
# (see module comment), mirroring the RTC engine.
self._relative_step = next(
(
s
for s in getattr(preprocessor, "steps", ())
if isinstance(s, RelativeActionsProcessorStep) and s.enabled
),
None,
)
# Set by the probe for the current tick / ever, respectively.
self._chunk_predicted = False
self._ever_predicted_chunk = False
self._original_predict_action_chunk = None # set while the probe is installed
if self._relative_step is not None:
# ``action_names`` is optional on the step; fill it lazily from the
# policy/dataset so the relative<->absolute mask is built correctly. This is
# a deliberate engine->step side effect (the step is configured by its consumer).
if self._relative_step.action_names is None:
cfg_names = getattr(policy.config, "action_feature_names", None)
self._relative_step.action_names = list(cfg_names) if cfg_names else list(ordered_action_keys)
self._install_chunk_probe()
logger.info("Relative actions enabled: chunk anchor pinned per predicted chunk")
logger.info(
"SyncInferenceEngine initialized (device=%s, action_keys=%d)",
self._device,
@@ -85,6 +105,11 @@ class SyncInferenceEngine(InferenceEngine):
def stop(self) -> None:
"""No background resources to stop."""
# Undo the probe so the policy object isn't left permanently patched
# (it may outlive this engine or be reused by another).
if self._original_predict_action_chunk is not None:
self._policy.predict_action_chunk = self._original_predict_action_chunk
self._original_predict_action_chunk = None
logger.info("SyncInferenceEngine stopped")
def reset(self) -> None:
@@ -93,6 +118,27 @@ class SyncInferenceEngine(InferenceEngine):
self._policy.reset()
self._preprocessor.reset()
self._postprocessor.reset()
# New episode: the next tick predicts a fresh chunk and re-anchors.
self._chunk_predicted = False
self._ever_predicted_chunk = False
def _install_chunk_probe(self) -> None:
"""Wrap the policy's public ``predict_action_chunk`` so we learn which ticks
predict a fresh chunk (when the anchor must advance) without introspecting any
private action queue. Chunking policies call it from ``select_action``.
Wraps whatever callable is currently bound (e.g. an already-``torch.compile``d
one, since ``build_rollout_context`` compiles before building the engine); undone
in ``stop()``."""
self._original_predict_action_chunk = self._policy.predict_action_chunk
inner = self._original_predict_action_chunk
def probe(*args, **kwargs):
self._chunk_predicted = True
self._ever_predicted_chunk = True
return inner(*args, **kwargs)
self._policy.predict_action_chunk = probe
def get_action(self, obs_frame: dict | None) -> torch.Tensor | None:
"""Run the full inference pipeline on ``obs_frame`` and return an action tensor."""
@@ -107,12 +153,25 @@ class SyncInferenceEngine(InferenceEngine):
if self._device.type == "cuda" and self._policy.config.use_amp
else nullcontext()
)
# Snapshot the chunk anchor before the preprocessor overwrites it with this
# tick's state; restore it below if this tick only served a cached action.
# ``clone`` so the snapshot survives even if the cached tensor is ever mutated
# in place (today it is only rebound, but the copy is cheap for a state vector).
anchor_before = None
if self._relative_step is not None:
cached = self._relative_step.get_cached_state()
anchor_before = cached.clone() if cached is not None else None
self._chunk_predicted = False
with torch.inference_mode(), autocast_ctx:
observation = prepare_observation_for_inference(
observation, self._device, self._task, self._robot_type
)
observation = self._preprocessor(observation)
action = self._policy.select_action(observation)
# Hold the anchor only for a chunking policy serving a cached action this
# tick; policies that never chunk or that recomputed keep refreshing.
if self._relative_step is not None and self._ever_predicted_chunk and not self._chunk_predicted:
self._relative_step.set_cached_state(anchor_before)
action = self._postprocessor(action)
action_tensor = action.squeeze(0).cpu()
+7
View File
@@ -346,3 +346,10 @@ def test_state_not_modified_by_relative_processor(dataset, action_dim):
result_state = result[TransitionKey.OBSERVATION][OBS_STATE]
torch.testing.assert_close(result_state, original_state)
def test_cached_anchor_not_in_config():
"""The cached anchor is ephemeral runtime state and must not leak into the config."""
step = RelativeActionsProcessorStep(enabled=True)
step.set_cached_state(torch.tensor([[1.0, 2.0, 3.0, 4.0]]))
assert set(step.get_config()) == {"enabled", "exclude_joints", "action_names"}
+222
View File
@@ -348,3 +348,225 @@ def test_rollout_context_fields():
field_names = {f.name for f in dataclasses.fields(RolloutContext)}
assert field_names == {"runtime", "hardware", "policy", "processors", "data"}
# ---------------------------------------------------------------------------
# Sync engine: relative-action anchoring (drift-free chunk execution)
# ---------------------------------------------------------------------------
_REL_ACTION_NAMES = ["j0.pos", "j1.pos", "j2.pos", "gripper.pos"]
_REL_ACTION_DIM = len(_REL_ACTION_NAMES)
def _relative_pre_post(exclude_joints=None):
"""Pre/post processors wrapping the real relative (caches anchor) and absolute
(relative + cached state) steps, mirroring what the sync engine feeds them."""
from lerobot.processor import (
AbsoluteActionsProcessorStep,
RelativeActionsProcessorStep,
TransitionKey,
create_transition,
)
from lerobot.utils.constants import OBS_STATE
relative_step = RelativeActionsProcessorStep(
enabled=True, exclude_joints=exclude_joints or [], action_names=list(_REL_ACTION_NAMES)
)
absolute_step = AbsoluteActionsProcessorStep(enabled=True, relative_step=relative_step)
class _Pre:
steps = [relative_step]
def __call__(self, observation):
# Run the relative step so it caches the anchor, then pass the batch through.
transition = create_transition(observation={OBS_STATE: observation[OBS_STATE]})
relative_step(transition)
return observation
def reset(self):
pass
class _Post:
def __call__(self, action):
transition = create_transition(action=action)
return absolute_step(transition)[TransitionKey.ACTION]
def reset(self):
pass
return _Pre(), _Post(), relative_step
def _fake_relative_policy(chunk_rel, n_action_steps, chunking=True):
"""Fake relative-action policy for the sync engine.
``chunking=True`` buffers a chunk and serves it one action per tick, calling the
public ``predict_action_chunk`` only on refill (pi0/fastwam/lingbot). ``False``
returns an action directly and never calls it. The engine's anchor probe keys off
that public call, so the fake routes through it rather than any private queue.
"""
from collections import deque
policy = MagicMock()
policy.config.use_amp = False
policy.config.action_feature_names = list(_REL_ACTION_NAMES)
state = {"predict_calls": 0}
queue = deque(maxlen=n_action_steps)
def predict_action_chunk(_batch=None, **_kwargs):
state["predict_calls"] += 1
return chunk_rel.unsqueeze(0) # [B=1, n, dim]
def select_action(_observation):
if not chunking:
return chunk_rel[0].unsqueeze(0)
if len(queue) == 0:
actions = policy.predict_action_chunk(_observation)
queue.extend(actions.transpose(0, 1)) # [n, 1, dim]
return queue.popleft()
policy.predict_action_chunk.side_effect = predict_action_chunk
policy.select_action.side_effect = select_action
policy.reset.side_effect = queue.clear
policy._predict_state = state
return policy
def _build_sync_engine(policy, pre, post):
from lerobot.rollout import SyncInferenceEngine
return SyncInferenceEngine(
policy=policy,
preprocessor=pre,
postprocessor=post,
dataset_features={"action": {"names": list(_REL_ACTION_NAMES)}},
ordered_action_keys=list(_REL_ACTION_NAMES),
task="test",
device="cpu",
robot_type="mock",
)
def _obs_frame(state_values):
import numpy as np
return {"observation.state": np.asarray(state_values, dtype=np.float32)}
def test_sync_relative_holds_anchor_across_chunk():
"""Every action popped within a chunk must anchor to the tick-0 state (no drift)."""
n = 4
# A distinct relative offset per chunk step so a wrong anchor would be visible.
chunk_rel = torch.stack([torch.full((_REL_ACTION_DIM,), 0.1 * (i + 1)) for i in range(n)])
pre, post, relative_step = _relative_pre_post()
policy = _fake_relative_policy(chunk_rel, n_action_steps=n)
engine = _build_sync_engine(policy, pre, post)
assert engine._relative_step is relative_step # introspection wired the step
s0 = [1.0, 2.0, 3.0, 4.0]
outputs = []
for tick in range(n):
# Feed a *different* state each tick; a drifting anchor would use it.
state = [v + tick for v in s0]
outputs.append(engine.get_action(_obs_frame(state)))
# Exactly one chunk was predicted across the n ticks.
assert policy._predict_state["predict_calls"] == 1
for tick in range(n):
expected = torch.tensor(s0) + chunk_rel[tick]
torch.testing.assert_close(outputs[tick], expected)
# Next tick empties the queue -> fresh chunk -> anchor advances to the new state.
s_next = [10.0, 20.0, 30.0, 40.0]
out = engine.get_action(_obs_frame(s_next))
assert policy._predict_state["predict_calls"] == 2
torch.testing.assert_close(out, torch.tensor(s_next) + chunk_rel[0])
# The anchor now reflects the fresh-chunk state, not the held one.
torch.testing.assert_close(relative_step.get_cached_state(), torch.tensor([s_next]))
def test_sync_relative_reset_reanchors_new_episode():
"""After ``reset()`` the first tick of the next episode anchors to the new state."""
n = 3
chunk_rel = torch.stack([torch.full((_REL_ACTION_DIM,), 0.2) for _ in range(n)])
pre, post, relative_step = _relative_pre_post()
policy = _fake_relative_policy(chunk_rel, n_action_steps=n)
engine = _build_sync_engine(policy, pre, post)
# Episode 1: one tick anchors to s0 and leaves cached actions in the queue.
engine.get_action(_obs_frame([1.0, 1.0, 1.0, 1.0]))
assert policy._predict_state["predict_calls"] == 1
engine.reset() # clears the queue and the per-episode chunk flags
# Episode 2: a fresh state must produce a fresh chunk anchored to that state,
# not carry over the previous episode's anchor.
s_new = [7.0, 8.0, 9.0, 10.0]
out = engine.get_action(_obs_frame(s_new))
assert policy._predict_state["predict_calls"] == 2
torch.testing.assert_close(out, torch.tensor(s_new) + chunk_rel[0])
torch.testing.assert_close(relative_step.get_cached_state(), torch.tensor([s_new]))
def test_sync_relative_non_chunking_policy_refreshes_every_tick():
"""A policy that never calls ``predict_action_chunk`` must not freeze the anchor."""
n = 3
chunk_rel = torch.stack([torch.full((_REL_ACTION_DIM,), 0.5) for _ in range(n)])
pre, post, _ = _relative_pre_post()
policy = _fake_relative_policy(chunk_rel, n_action_steps=n, chunking=False)
engine = _build_sync_engine(policy, pre, post)
s0 = [1.0, 1.0, 1.0, 1.0]
for tick in range(3):
state = [v + tick for v in s0]
out = engine.get_action(_obs_frame(state))
# Anchor must track the current state every tick (no chunk => no hold).
torch.testing.assert_close(out, torch.tensor(state) + chunk_rel[0])
assert policy._predict_state["predict_calls"] == 0
def test_sync_engine_no_relative_step_is_none():
"""Without an enabled relative step, the engine takes the plain select_action path."""
policy = MagicMock()
policy.config.use_amp = False
engine = _build_sync_engine(policy, MagicMock(steps=[]), MagicMock())
assert engine._relative_step is None
def test_sync_relative_exclude_joints_stay_absolute():
"""With ``exclude_joints``, excluded dims pass through absolute while the relative
dims still hold the tick-0 anchor across the chunk."""
n = 4
# Distinct offset per step *and* per dim so a wrong anchor or a wrong mask shows up.
chunk_rel = torch.stack([torch.full((_REL_ACTION_DIM,), 0.1 * (i + 1)) for i in range(n)])
pre, post, relative_step = _relative_pre_post(exclude_joints=["gripper"])
policy = _fake_relative_policy(chunk_rel, n_action_steps=n)
engine = _build_sync_engine(policy, pre, post)
# gripper (last dim) is kept absolute; j0..j2 are relative.
mask = torch.tensor([1.0, 1.0, 1.0, 0.0])
s0 = [1.0, 2.0, 3.0, 4.0]
outputs = []
for tick in range(n):
state = [v + tick for v in s0] # moving state; a drifting anchor would use it
outputs.append(engine.get_action(_obs_frame(state)))
assert policy._predict_state["predict_calls"] == 1 # one chunk held across n ticks
for tick in range(n):
# relative dims: anchor held at s0; excluded gripper dim: raw predicted value.
expected = chunk_rel[tick] + torch.tensor(s0) * mask
torch.testing.assert_close(outputs[tick], expected)
def test_sync_relative_stop_restores_policy_method():
"""``stop()`` un-patches the probe so the policy object isn't permanently modified."""
n = 3
chunk_rel = torch.stack([torch.full((_REL_ACTION_DIM,), 0.2) for _ in range(n)])
pre, post, _ = _relative_pre_post()
policy = _fake_relative_policy(chunk_rel, n_action_steps=n)
original = policy.predict_action_chunk
engine = _build_sync_engine(policy, pre, post)
assert policy.predict_action_chunk is not original # probe installed
engine.stop()
assert policy.predict_action_chunk is original # restored