mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-06 09:37:06 +00:00
7fa2dfefa23a92a00a36519283ccc1ee597f7f2e
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Caroline Pascal
|
3dd19d043e |
feat(depth maps): adding support for depth in LeRobot (#3644)
* feat(depth): add depth quantization helpers and tests
* feat(video): add ffv1 to supported codecs
* feat(depth): persist depth metadata
* feat(depth): extend quantization tools to better fit the encoding/decoding pipeline
* feat(depth): plumb DepthEncoderConfig through LeRobotDataset and DatasetWriter
* feat(depth): wire StreamingVideoEncoder + writer to depth encoder
* feat(depth): wire DatasetReader to decode_depth_frames
* feat(cameras/realsense): expose async depth in metric meters
* feat(features): route 2D camera shapes to observation.depth.<key>
* feat(robots/so_follower): emit + populate depth keys when use_depth
* feat(record): plumb DepthEncoderConfig through lerobot-record
* feat(viz): render depth observations as rr.DepthImage in Viridis
* feat(depth maps writer): adding support for raw depth maps recording with image writer
* chore(format): format code
* feat(depth shape): ensuring depth maps shape is always including the channel
* feat(is_depth): simplifying is_depth nested name + legacy support
* fix(stop_event): fixing stop_event race condition in camera classes
* fix(plumbing): fixing missing parts in the depth maps pipeline
* chore(typos): fixing typos
* test(fix): fixing exisiting tests to still work with latest features
* tests(depth): adding new tests for depth integration validation
* feat(pix_fmt channels): use PyAv to check get pixel formats number of channels
* feat(refactor): refactor DepthEncoderConfig quantization pipeline, so that the methods do not live in the config class. Add pixel format - channels validation.Move the default pixel format for depth in the config file.
* fix(pre-commit): fixing mutable defautl value
* fix(info): fixing info metadata update when is_depth_map was set
* tests(typos): fixing typos in tests
* fix(realsense): fixing typo in realsense serial number
* fix(normalization): restricting 255 normalization to non depth/uint8 images only
* fix(typo): fixing typo
* fix(TIFF): add missing quantization and cleanup for TIFF files
* feat(batched dequantization): optimizing dequantize_depth for torch based batched dequantization
* feat(tools): adding depth support in LeRobotDataset edition tools
* test(aggregate): extending aggregation tests to depth frames
* test(cleaning): cleaning up tests
* fix(from_video_info): fixing early validation issue in from_video_info
* fix(typo): fixing typo
* fix(is_depth): adding missing doctrings and is_depth arguments in video decoding functions
Co-authored-by: Wensi (Vince) Ai <59036629+wensi-ai@users.noreply.github.com>
* fix(depth units): fixing depth units output for the realsense cameras
* feat(output unit): adding support for output unit specification at dataset reading/training time
Co-authored-by: Wensi (Vince) Ai <59036629+wensi-ai@users.noreply.github.com>
* test(depth): cleaning up depth tests
* test(depth encoding): updating and cleaning video/depth encoding tests
* chore(format): formatting code
* docs(depth): improving depth maps docs
* test(fix): fixing depth tests
* test(dataset tools): adding missing tests for new dataset edition tools features
* chore(format): formatting code
* fix(pyav check): fixing PyAV option validation for integer codec options by normalizing
numeric values before calling `is_integer()`
Co-authored-by: Wensi (Vince) Ai <59036629+wensi-ai@users.noreply.github.com>
* docs(mermaid): fixing mermaid diagram
* fix(rebase): rebase follow up corrections
* feat(dataset tools): adding missing docstrings and features for depth fill support in dataset edition tools
* docs(docstring): updating docstrings
* docs(dataset tools): updating docs
* fix(save images): fixing image saving in dataset tools
* fix(update video info): fixing update video info logic to match the recording and editing use cases
* test(reencode): fixing reencoding monkeypatch
* fix(review): add Claude review
* chore(format): format code
* fix(update video info): ditching the differentiated approahces for video info update - video info are always updated unless for preserved keys.
* chore(rebase): fixing rebase merge conflicts
* test(visualization): fixing visualization tests
* feat(docstrings): adding explicit docstring for encoding parameters. Docstrigns will now show up as description in the CLI --help.
* feat(mm as default): adding a global DEFAULT_DEPTH_UNIT variable setting mm as default depth unit
* fix(RGB <-> camera): renaming camera_encoder to rgb_encoder for clarity
* chore(TODO): removing deprecated TODO
* doc(write_u16_plane): improving docstrings for write_u16_plane
* feat(units): adding constants for depth frames units (m and mm)
* fix(spam): replacing spamming warning but a debug log
* feat(leagcy metadata): adding automatic metadata update for legacy 'video.is_depth_map' feature
* fix(copy&reindex): fixing metadat reshaping for single channel frames
* fix(ImageNet): excluding dpeth frames from ImageNet stats
* fix(PyAV container seek): fixing initial PyAV container seek to be robust againsy codec choice
* feat(lerobot-dataset-viz): adding support for depth in lerobot-dataset-viz
* fix(compress): removing rerun compression for DepthImages
* fix(signle channel squeeze): fixing single channel squeezing
* chore(format): format code
* fix(streaming): adding support for dequantization in streaming_dataset.py
* refactor(read depth): factorizing depth reading methods for realsense camera and adding support for depth-only usage
* chore(renaming): fixing missed RGBEncoderConfig renamings
* docs(renaming): reflecting renamings in a clearer way in the docs
* chore(annotation): excluding depth from the annotation pipeline
* feat(robots): adding depth support in compatible follower robots
* feat(LeSadKiwi): excluding LeKiwi from depth support (for now)
* chore(fail): removing misplaced file
* chore(fail): removing misplaced file
* fix(remove ffv1): removing ffv1 as it does not support MP4
* docs(cheat sheet): adding depth and video encoding to the cheat sheet
* fix(lossless): tuning depth encoding parameters for lossless depth storage
* test(fix): fixing failing tests
* depth(ZMQ): excluding ZMQ from depth support
* Revert "depth(ZMQ): excluding ZMQ from depth support"
This reverts commit
|
||
|
Caroline Pascal
|
bd9619dfc3 |
feat(encoding parameters): adding support for user provided video encoding parameters (#3455)
* chore(video backend): renaming codec into video_backend in get_safe_default_video_backend() * feat(pyav utils): adding suport for PyAV encoding parameters validation * feat(VideoEncoderConfig): creating a VideoEncoderConfig to encapsulate encoding parameters * feat(VideoEncoderConfig): propagating the VideoEncoderConfig in the codebase * chore(docs): updating the docs * feat(metadata): adding encoding parameters in dataset metadata * fix(concatenation compatibility): adding compatibility check when concatenating video files * feat(VideoEncoderConfig init): making VideoEncoderConfig more robust and adaptable to multiple backends * feat(pyav checks): making pyav parameters checks more robust * chore(duplicate): removing duplicate get_codec_options definition * test(existing): adapting existing tests * test(new): adding new tests for encoding related features * chore(format): fixing formatting issues * chore(PyAV): cleaning up PyAV utils and encoding parameters checks to stick to the minimun required tooling. * chore(format): formatting code * chore(doctrings): updating docstrings * fix(camera_encoder_config): Removing camera_encoder_config from LeRobotDataset, as it's only required in LeRobotDatasetWriter. * feat(default values): applying a consistent naming convention for default RGB cameras video encoder parameters * fix(rollout): propagating VideoEncoderConfig to the latest recording modes * chore(format): formatting code, fixing error messages and variable names * fix(arguments order): reverting changes in arguments order in StreamingVideoEncoder * chore(relative imports): switching to relative local imports within lerobot.datasets * test(artifacts): cleaning up artifacts for the video encoding tests * chore(docs): updating docs * chore(fromat): formatting code * fix(imports): refactoring the file architecture to avoid circular imports. VideoEncoderConfig is now defined in lerobot.configs and lazily imports av at runtime. * fix(typos): fixing typos and small mistakes * test(factories): updating factories * feat(aggregate): updating dataset aggregation procedure. Encoding tuning paramters (crf, g,...) are ignored for validation and changed to None in the aggregated dataset if incompatible. * docs(typos): fixing typos * fix(deletion): reverting unwanted deletion * fix(typos): fixing multiple typos * feat(codec options): passing codec options to lerobot_edit_dataset episode deletion tool * typo(typo): typo * fix(typos): fixing remaining typos * chore(rename): renaming camera_encoder_config to camera_encoder * docs(clean): cleaning and formating docs * docs(dataset): addind details about datasets * chore(format): formatting code * docs(warning): adding warning regarding encoding parameters modification * fix(re-encoding): removing inconsistent re-encoding option in lerobot_edit_dataset * typos(typos): typos * chore(format): resolving prettier issues * fix(h264_nvenc): fixing crf handling for h264_nvenc * docs(clean): removing too technical parts of the docs * fix(imports): fixing imports at the __init__ level * fix(imports): fixing not very pretty imports in video config file |
||
|
Steven Palma
|
e96339a3b4 |
feat(dataset): add streaming video encoding + HW encoder support (#2974)
* feat(dataset): init stream encoding * feat(dataset): use threads to fix frame pickle latency * refactor(dataset): remove HW encoded related changes * add lp (#2977) * feat(dataset): add Hw encoding + log drop frames (#2978) * chore(docs): add streaming video encoding guide * fix(dataset): style docs + testing * chore(docs): simplify sttreaming video encoding guide * chore(dataset): add commands + streaming encoding default false + print note if false + queue default is now 30 * chore(docs): add verification note advice * chore(dataset): adjusting defaults & docs for streaming encoding * docs(scripts): improve docstrings * test(dataset): polish streaming encoding tests * chore(dataset): move FYI log related to streaming * chore(dataset): add arg vcodec to suggestions * refactor(dataset): better handling for auto and available vcodec * chore(dataset): change log level * docs(dataset): add note related to training performance vcodec * docs(dataset): add more notes to streaming encoding --------- Co-authored-by: Caroline Pascal <caroline8.pascal@gmail.com> Co-authored-by: Pepijn <pepijn@huggingface.co> |
||
|
Steven Palma
|
5f15232271 | chore: remove usernames + use entrypoints in docs, comments & sample commands (#2988) | ||
|
Steven Palma
|
ce3b9f627e |
chore(docs): prioritize use of entry points in docs + fix nightly badge (#1692)
* chore(docs): fix typo in nightly badge * chore(docs): prioritize the use of entrypoints for consistency |
||
|
Steven Palma
|
e88b30e6cc |
fix(ci): multiple fixes (#1549)
* fix(ci): tag of image when pushing to main * fix(docs): remove symlink in docs folder * chore(docs): move .mdx files to docs/ folder * chore(docs): create symlink to docs files * chore(ci): de-couple fast and full test pipeline * fix(ci): skip GPU Tests for community PRs |
||
|
Simon Alibert
|
039de254ea |
Add Hope Jr (#935)
* Fix imports * Add feetech write tests * Nit * Add autoclosing fixture * Assert ping stub called * Add CalibrationMode * Add Motor in dxl robots * Simplify split_int_bytes * Rename read/write -> sync_read/write, refactor, add write * Rename tests * Refactor dxl tests by functionality * Add dxl write test * Refactor _is_comm_success * Refactor feetech tests by functionality * Add feetech write test * Simplify _is_comm_success & _is_error * Move mock_serial patch to dedicated file * Remove test skips & fix docstrings * Nit * Add dxl operating modes * Add is_connected in robots and teleops * Update Koch * Add feetech operating modes * Caps dxl OperatingMode * Update ensure_safe_goal_position * Update so100 * Privatize methods & renames * Fix dict * Add _configure_motors & move ping methods * Return models (str) with pings * Implement feetech broadcast ping * Add raw_values option * Rename idx -> id_ * Improve errors * Fix feetech ping tests * Ensure motors exist at connection time * Update tests * Add test_motors_bus * Move DriveMode & TorqueMode * Update Koch imports * Update so100 imports * Fix visualize_motors_bus * Fix imports * Add calibration * Rename idx -> id_ * Rename idx -> id_ * (WIP) _async_read * Add new calibration method for robot refactor (#896) Co-authored-by: Simon Alibert <simon.alibert@huggingface.co> * Remove deprecated scripts * Rename CalibrationMode -> MotorNormMode * Fix calibration functions * Remove todo * Add scan_port utility * Add calibration utilities * Move encoding functions to encoding_utils * Add test_encoding_utils * Rename test * Add more calibration utilities * Format baudrate tables * Implement SO-100 leader calibration * Implement SO-100 follower calibration * Implement Koch calibration * Add test_scan_port (TODO) * Fix calibration * Hack feetech firmware bug * Update tests * Update Koch & SO-100 * Improve format * Rename SO-100 classes * Rename Koch classes * Add calibration tests * Remove old calibration tests * Revert feetech hack and monkeypatch instead * Simplify motors mocks * Add is_calibrated test * Update viperx & widowx * Rename viperx & widowx * Remove old calibration * feat(teleop): thread-safe keyboard teleop implementation (#869) Co-authored-by: Simon Alibert <75076266+aliberts@users.noreply.github.com> * Add support for feetech scs series + various fixes * Update dynamixel with motors bus & tables changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * (WIP) Add Hope Jr * Rename arm -> hand * (WIP) Add homonculus arm & glove * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Feetech protocol version * Implement read * Use constants from sdks * (nit) move write * Fix broadcast ping type hint * Add protocol 1 broadcast ping * Refactor & add _serialize_data * Add feetech sm8512bl * Make feetech broadcast ping faster in protocol 1 * Cleanup * Add support for feetech protocol 1 to _split_into_byte_chunks * Fix unormalize * Remove test_motors_bus fixtures * Add more segmented tests (base motor bus & feetech), add feetech protocol 1 support * Add more segmented tests (dynamixel) * Refactor tests * Add handshake, fix feetech _read_firmware_version * Fix tests * Motors config & disconnect fixes * Add torque_disabled context * Update branch & fix pre-commit errors * Fix hand & glove readings * Update feetech tables * Move read/write_calibration implementations * Add setup_motor * Fix calibration msg display * Fix setup_motor & add it to robots * Fix _find_single_motor * Remove deprecated configure_motor * Remove deprecated dynamixel_calibration * Remove names * Remove deprecated import * refactor/lekiwi robot (#863) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Simon Alibert <simon.alibert@huggingface.co> Co-authored-by: Simon Alibert <75076266+aliberts@users.noreply.github.com> * fix(teleoperators): use property is_connected (#1075) * Remove deprecated manipulator * Update robot features & naming * Update teleop features & naming * Add make_teleoperator_from_config * Rename find_port * Fix config parsing * Remove app script * Add setup_motors * Add teleoperate * Add record * Add replay * Fix test_datasets * Add mock robot & teleop * Add new test_control_robot * Add test_record_and_resume * Remove deprecated scripts & tests * Add calibrate * Add docstrings * Fix tests (no-extras install) * Add SO101 * Remove pynput from optional deps * Rename example 7 * Remove unecessary id * Add MotorsBus docstrings * Rename arm -> bus * Remove Moss arm * Fix setup_motors & calibrate configs * Fix test_calibrate * Add copyrights * Update hand & arm * Update homonculus hand & arm * Fix dxl _find_single_motor * Update glove * Add setup_motors for lekiwi * Fix glove calibration * Complete docstring * Add check for same min and max during calibration * Move MockMotorsBus * Add so100_follower tests * (WIP) add calibration gui * Fix test * Add setup_motors * Update calibration gui * Remove old .cache folder * Replace deprecated abc.abstractproperty * Fix feetech protocol 1 configure * Cleanup gui & add copyrights * Anatomically precise joint names * (WIP) Add glove to hand joints translation * Move make_robot_config * Add drive_mode & norm_mode in glove calibration * Fix joints translation * Fix normalization drive_mode * nit * Fix glove to hand conversion * Adapt feetech calibration * Remove pygame prompt * Implement arm calibration (hacks) * Better MotorsBus error messages * Update feetech read_calibration * Fix feetech test_is_calibrated * Cleanup glove * (WIP) Update arm * Add changes from #1117 * refactor(cameras): cameras implementations + tests improvements (#1108) Co-authored-by: Simon Alibert <simon.alibert@huggingface.co> Co-authored-by: Simon Alibert <75076266+aliberts@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix arm joints order * Add timeout/event logic * Fix arm & glove * Fix predict_action from record * fix(cameras): update docstring + handle sn when starts with 0 + update timeouts to more reasonable value (#1154) * fix(scripts): parser instead of draccus in record + add __get_path_fields__() to RecordConfig (#1155) * Left/Right sides + other fixes * Arm fixes and add config * More hacks * Add control scripts * Fix merge errors * push changes to calibration, teleop and docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Move readme to docs * update readme Signed-off-by: Martino Russi <77496684+nepyope@users.noreply.github.com> * Add files via upload Signed-off-by: Martino Russi <77496684+nepyope@users.noreply.github.com> * Update image sources * Symlink doc * Compress image * Move image * Update docs link * fix docs * simplify teleop scripts * fix variable names * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address code review * add EMA to glove * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * integrate teleoperation for hand * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * import hopejr/homunculus in teleoperate * update docs for teleoperate, record, replay, train and inference * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * chore(hopejr): address comments * chore(hopejr): address coments 2 * chore(docs): update teleoperation instructions for the hand/glove * fix(hopejr): calibration int + update docs --------- Signed-off-by: Martino Russi <77496684+nepyope@users.noreply.github.com> Signed-off-by: Simon Alibert <75076266+aliberts@users.noreply.github.com> Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com> Co-authored-by: Steven Palma <imstevenpmwork@ieee.org> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: nepyope <nopyeps@gmail.com> Co-authored-by: Martino Russi <77496684+nepyope@users.noreply.github.com> Co-authored-by: Steven Palma <steven.palma@huggingface.co> |