Changes:
- Create separate figure for correction data instead of overlaying on v_t
- Add _rescale_axes helper method to properly scale all axes
- Add 10% margin to y-axis for better visualization
- Fix v_t chart vertical compression issue
Benefits:
- Clearer v_t plot without correction overlay
- Better axis scaling with proper margins
- Separate correction figure for focused analysis
- Improved readability of all denoising visualizations
Output files:
- denoising_xt_comparison.png (x_t trajectories)
- denoising_vt_comparison.png (v_t velocity - now cleaner)
- denoising_correction_comparison.png (NEW - separate corrections)
- denoising_x1t_comparison.png (x1_t state with error)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Alexander Soare <alexander.soare159@gmail.com>
Refactor to improve separation of concerns:
modeling_smolvla.py changes:
- Remove all plotting logic from sample_actions method
- Remove viz_xt_axs, viz_vt_axs, viz_x1t_axs parameters
- Remove matplotlib and RTCDebugVisualizer imports
- Remove viz_fig, viz_axs, denoise_step_counter instance variables
- Simplify denoising loop to only track data in rtc_processor
eval_dataset.py changes:
- Add _plot_denoising_steps_from_tracker helper method
- Retrieve debug steps from tracker after inference
- Plot x_t, v_t, x1_t, correction, and error from tracker data
- Enable debug tracking (cfg.rtc.debug = True) for visualization
- Remove viz axes parameters from predict_action_chunk calls
modeling_rtc.py changes:
- Remove v_t from track() call (handled by user change)
Benefits:
- Cleaner modeling code focused on inference
- Evaluation script owns all visualization logic
- Better separation of concerns
- Tracker is single source of truth for debug data
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Alexander Soare <alexander.soare159@gmail.com>
- Add force=True to logging.basicConfig to override existing configuration
- Enable line buffering for stdout/stderr for real-time log output
- Modify init_rtc_processor to create processor when rtc_config exists
even if RTC is disabled, allowing tracking of denoising data
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Alexander Soare <alexander.soare159@gmail.com>
Update eval_dataset.py to save all comparison images to the
configured output_dir instead of the current directory. This provides
better organization and allows users to specify where outputs should be
saved.
Changes:
- Add os import at top level
- Create output_dir at start of run_evaluation()
- Save all comparison images to output_dir
- Remove duplicate os imports
- Update init_rtc_processor() docstring to be more concise
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Alexander Soare <alexander.soare159@gmail.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Implement Real-Time Chunking (RTC) for action chunking policies using flow
matching denoising. RTC enables smooth action transitions between consecutive
chunks by using prefix guidance during denoising.
Key features:
- RTCProcessor class with denoise_step method for RTC guidance
- Tracker system for debug tracking using time-based dictionary storage
- RTCDebugVisualizer with comprehensive visualization utilities
- Integration with SmolVLA policy for flow matching models
- Support for multiple prefix attention schedules (ZEROS, ONES, LINEAR, EXP)
- Configurable execution horizon and max guidance weight
- Example scripts for dataset evaluation and real-time control
Technical details:
- Uses autograd-based gradient computation for RTC corrections
- Time-based tracking eliminates duplicate step issues
- Proxy methods in RTCProcessor for cleaner API
- Full integration with LeRobot's policy and dataset systems
Files added/modified:
- src/lerobot/configs/types.py: Add RTCAttentionSchedule enum
- src/lerobot/policies/rtc/: Core RTC implementation
- configuration_rtc.py: RTC configuration
- modeling_rtc.py: RTCProcessor with denoise_step
- debug_handler.py: Tracker for debug information
- debug_visualizer.py: Visualization utilities
- src/lerobot/policies/smolvla/modeling_smolvla.py: RTC integration
- examples/rtc/: Example scripts and evaluation tools
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Alexander Soare <alexander.soare159@gmail.com>
Co-Authored-By: Claude <noreply@anthropic.com>
* (unscrewing things up) (#2288)
* fix: expose a function explicitly building a frame for inference
* fix: first make dataset frame, then make ready for inference
* fix: reducing reliance on lerobot record for policy's ouptuts too
* fix: encapsulating squeezing out + device handling from predict action
* fix: remove duplicated call to build_inference_frame and add a function to only perform data type handling (whole conversion is: keys matching + data type conversion)
* refactor(envs): add custom-observation-size (#2167)
* fix: add MockMotorBus to MockRobot
* rl: first drafts
* add: all components of HIL SERL
* fix: actor block works
* fix: less friction, less friction
* add: hil-serl complete example
* fix: dataset names
* fix: restructuring example folder
* fix: act works but found bug in how ACT works
* fix: same path for both pre and postprocessors
* fix: paths
* add: example usage for act
* add: using ACT example
* fix: training examples
* fix: using examples
* fix: camera index
* fix: rename workflows into tutorial so that the path of the files is lerobot/examples/tutorial/...
* fix: upload everything in one repo
* fix: model name
* fix: simplify model path
* add: VLAs example
---------
Signed-off-by: Francesco Capuano <74058581+fracapuano@users.noreply.github.com>
* fix: minor fix using named attributes
* fix: change model to act
* fix: named attributes for inference frame building
* fix: minor fixes to smolvla
* fix: small changes to pi0
* remove: old file that should have never been committed (ups sorry sorry)
---------
Signed-off-by: Francesco Capuano <74058581+fracapuano@users.noreply.github.com>
* make add_feature take multiple features at a time and rename to add_features
* - New function: modify_features that was a combination of remove features and add features.
- This function is important for when we want to add a feature and remove another so we can do it in one time to avoid copying and creating the dataset multiple times
* feat(dataset-tools): add dataset utilities and example script
- Introduced dataset tools for LeRobotDataset, including functions for deleting episodes, splitting datasets, adding/removing features, and merging datasets.
- Added an example script demonstrating the usage of these utilities.
- Implemented comprehensive tests for all new functionalities to ensure reliability and correctness.
* style fixes
* move example to dataset dir
* missing lisence
* fixes mostly path
* clean comments
* move tests to functions instead of class based
* - fix video editting, decode, delete frames and rencode video
- copy unchanged video and parquet files to avoid recreating the entire dataset
* Fortify tooling tests
* Fix type issue resulting from saving numpy arrays with shape 3,1,1
* added lerobot_edit_dataset
* - revert changes in examples
- remove hardcoded split names
* update comment
* fix comment
add lerobot-edit-dataset shortcut
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Michel Aractingi <michel.aractingi@huggingface.co>
* style nit after copilot review
* fix: bug in dataset root when editing the dataset in place (without setting new_repo_id
* Fix bug in aggregate.py when accumelating video timestamps; add tests to fortify aggregate videos
* Added missing output repo id
* migrate delete episode to using pyav instead of decoding, writing frames to disk and encoding again.
Co-authored-by: Caroline Pascal <caroline8.pascal@gmail.com>
* added modified suffix in case repo_id is not set in delete_episode
* adding docs for dataset tools
* bump av version and add back time_base assignment
* linter
* modified push_to_hub logic in lerobot_edit_dataset
* fix(progress bar): fixing the progress bar issue in dataset tools
* chore(concatenate): removing no longer needed concatenate_datasets usage
* fix(file sizes forwarding): forwarding files and chunk sizes in metadata info when splitting and aggregating datasets
* style fix
* refactor(aggregate): Fix video indexing and timestamp bugs in dataset merging
There were three critical bugs in aggregate.py that prevented correct dataset merging:
1. Video file indices: Changed from += to = assignment to correctly reference
merged video files
2. Video timestamps: Implemented per-source-file offset tracking to maintain
continuous timestamps when merging split datasets (was causing non-monotonic
timestamp warnings)
3. File rotation offsets: Store timestamp offsets after rotation decision to
prevent out-of-bounds frame access (was causing "Invalid frame index" errors
with small file size limits)
Changes:
- Updated update_meta_data() to apply per-source-file timestamp offsets
- Updated aggregate_videos() to track offsets correctly during file rotation
- Added get_video_duration_in_s import for duration calculation
* Improved docs for split dataset and added a check for the possible case that the split size results in zero episodes
* chore(docs): update merge documentation details
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
---------
Co-authored-by: CarolinePascal <caroline8.pascal@gmail.com>
Co-authored-by: Jack Vial <vialjack@gmail.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* chore: replace hard-coded 'action' values with constants throughout all the source code
* chore(tests): replace hard-coded action values with constants throughout all the test code
* chore: replace hard-coded OBS values with constants throughout all the source code
* chore(tests): replace hard-coded OBS values with constants throughout all the test code
* Remove unused scripts, add docs for image transforms and add example
* fix(examples): move train_policy.py under examples, remove outdated readme parts
* remove script thats copied to train folder
* remove outdated links to examples and example tests
Michel Aractingi