admin/lerobot
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-05-21 19:49:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(format): formatting code

Browse Source
This commit is contained in:
CarolinePascal
2026-05-13 15:06:57 +02:00
parent f89da582ae
commit d1917a4871
7 changed files with 44 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/streaming_video_encoding.mdx
+13 -13
View File
@@ -14,12 +14,12 @@ This makes `save_episode()` near-instant (the video is already encoded by the ti
## 2. Tuning Parameters ## 2. Tuning Parameters
| Parameter | CLI Flag | Type | Default | Description | | Parameter | CLI Flag | Type | Default | Description |
| ----------------------- | ---------------------------------------- | ------------- | ------------- | ----------------------------------------------------------------- | | ----------------------- | --------------------------------- | ------------- | ------------- | ----------------------------------------------------------------- |
| `streaming_encoding` | `--dataset.streaming_encoding` | `bool` | `True` | Enable real-time encoding during capture | | `streaming_encoding` | `--dataset.streaming_encoding` | `bool` | `True` | Enable real-time encoding during capture |
| `vcodec` | `--dataset.camera_encoder.vcodec` | `str` | `"libsvtav1"` | Video codec. `"auto"` detects best HW encoder | | `vcodec` | `--dataset.camera_encoder.vcodec` | `str` | `"libsvtav1"` | Video codec. `"auto"` detects best HW encoder |
| `encoder_threads` | `--dataset.encoder_threads` | `int \| None` | `None` (auto) | Threads per encoder instance. `None` will leave the vcoded decide | | `encoder_threads` | `--dataset.encoder_threads` | `int \| None` | `None` (auto) | Threads per encoder instance. `None` will leave the vcoded decide |
| `encoder_queue_maxsize` | `--dataset.encoder_queue_maxsize` | `int` | `30` | Max buffered frames per camera (~1s at 30fps). Consumes RAM | | `encoder_queue_maxsize` | `--dataset.encoder_queue_maxsize` | `int` | `30` | Max buffered frames per camera (~1s at 30fps). Consumes RAM |
## 3. Performance Considerations ## 3. Performance Considerations
@@ -82,8 +82,8 @@ Use HW encoding when:
### Available HW Encoders ### Available HW Encoders
| Encoder | Platform | Hardware | CLI Value | | Encoder | Platform | Hardware | CLI Value |
| ------------------- | ------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- | | ------------------- | ------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------- |
| `h264_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.camera_encoder.vcodec=h264_videotoolbox` | | `h264_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.camera_encoder.vcodec=h264_videotoolbox` |
| `hevc_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.camera_encoder.vcodec=hevc_videotoolbox` | | `hevc_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.camera_encoder.vcodec=hevc_videotoolbox` |
| `h264_nvenc` | Linux/Windows | NVIDIA GPU | `--dataset.camera_encoder.vcodec=h264_nvenc` | | `h264_nvenc` | Linux/Windows | NVIDIA GPU | `--dataset.camera_encoder.vcodec=h264_nvenc` |
@@ -100,15 +100,15 @@ Use HW encoding when:
## 5. Troubleshooting ## 5. Troubleshooting
| Symptom | Likely Cause | Fix | | Symptom | Likely Cause | Fix |
| ------------------------------------------------------------------ | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------------------------------------------------------------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| System freezes or choppy robot movement or Rerun visualization lag | CPU starved (100% load usage) | Close other apps, reduce encoding throughput, lower `encoder_threads`, use `h264`, use `display_data=False`. If the CPU continues to be at 100% then it might be insufficient for your setup, consider `--dataset.streaming_encoding=false` or HW encoding (`--dataset.camera_encoder.vcodec=auto`) | | System freezes or choppy robot movement or Rerun visualization lag | CPU starved (100% load usage) | Close other apps, reduce encoding throughput, lower `encoder_threads`, use `h264`, use `display_data=False`. If the CPU continues to be at 100% then it might be insufficient for your setup, consider `--dataset.streaming_encoding=false` or HW encoding (`--dataset.camera_encoder.vcodec=auto`) |
| "Encoder queue full" warnings or dropped frames in dataset | Encoder can't keep up (Queue overflow) | If CPU is not at 100%: Increase `encoder_threads`, increase `encoder_queue_maxsize` or use HW encoding (`--dataset.camera_encoder.vcodec=auto`). | | "Encoder queue full" warnings or dropped frames in dataset | Encoder can't keep up (Queue overflow) | If CPU is not at 100%: Increase `encoder_threads`, increase `encoder_queue_maxsize` or use HW encoding (`--dataset.camera_encoder.vcodec=auto`). |
| High RAM usage | Queue filling faster than encoding | `encoder_threads` too low or CPU insufficient. Reduce `encoder_queue_maxsize` or use HW encoding | | High RAM usage | Queue filling faster than encoding | `encoder_threads` too low or CPU insufficient. Reduce `encoder_queue_maxsize` or use HW encoding |
| Large video files | Using HW encoder or H.264 | Expected trade-off. Switch to `libsvtav1` if CPU allows | | Large video files | Using HW encoder or H.264 | Expected trade-off. Switch to `libsvtav1` if CPU allows |
| `save_episode()` still slow | `streaming_encoding` is `False` | Set `--dataset.streaming_encoding=true` | | `save_episode()` still slow | `streaming_encoding` is `False` | Set `--dataset.streaming_encoding=true` |
| Encoder thread crash | Codec not available or invalid settings | Check `vcodec` is installed, try `--dataset.camera_encoder.vcodec=auto` | | Encoder thread crash | Codec not available or invalid settings | Check `vcodec` is installed, try `--dataset.camera_encoder.vcodec=auto` |
| Recorded dataset is missing frames | CPU/GPU starvation or occasional load spikes | If ~5% of frames are missing, your system is likely overloaded — follow the recommendations above. If fewer frames are missing (~2%), they are probably due to occasional transient load spikes (often at startup) and can be considered expected. | | Recorded dataset is missing frames | CPU/GPU starvation or occasional load spikes | If ~5% of frames are missing, your system is likely overloaded — follow the recommendations above. If fewer frames are missing (~2%), they are probably due to occasional transient load spikes (often at startup) and can be considered expected. |
## 6. Recommended Configurations ## 6. Recommended Configurations
docs/source/video_encoding_parameters.mdx
+23 -18
View File
@@ -1,13 +1,16 @@
# Video encoding parameters # Video encoding parameters
When video storage is enabled, LeRobot stores each camera stream as an **MP4** file instead of saving one image file per timestep. Video encoding compresses across time, which usually cuts dataset size and I/O compared to a pile of PNGs, while keeping MP4 — a format every player and loader understands. When video storage is enabled, LeRobot stores each camera stream as an **MP4** file instead of saving one image file per timestep. Video encoding compresses across time, which usually cuts dataset size and I/O compared to a pile of PNG, while keeping MP4 — a format every player and loader understands.
Encoding frames into an MP4 is a full FFmpeg pipeline: choice of encoder, pixel format, GOP/keyframes, quality vs. speed, and optional extra encoder flags. Most of these knobs are user-tunable through `camera_encoder`, a nested `VideoEncoderConfig` (`lerobot.configs.video.VideoEncoderConfig`) passed through PyAV. Encoding frames into an MP4 is a full FFmpeg pipeline: choice of encoder, pixel format, GOP/keyframes, quality vs. speed, and optional extra encoder flags. Most of these knobs are user-tunable through `camera_encoder`, a nested `VideoEncoderConfig` (`lerobot.configs.video.VideoEncoderConfig`) passed through PyAV.
You can set these parameters from the CLI with `--dataset.camera_encoder.<field>` (e.g. with `lerobot-record` or `lerobot-rollout`). The same block applies to every camera video stream in that run. You can set these parameters from the CLI with `--dataset.camera_encoder.<field>` (e.g. with `lerobot-record` or `lerobot-rollout`). The same block applies to every camera video stream in that run.
<Tip> <Tip>
Video storage must be on for `camera_encoder` to have any effect — `use_videos=True` in Python APIs, or `--dataset.video=true` on the CLI (the recording default). With video off, inputs stay as images and `camera_encoder` is ignored. Video storage must be on for `camera_encoder` to have any effect —
`use_videos=True` in Python APIs, or `--dataset.video=true` on the CLI (the
recording default). With video off, inputs stay as images and `camera_encoder`
is ignored.
</Tip> </Tip>
For details on **when** frames are written vs. encoded (streaming vs. post-episode), queues, and other top-level `--dataset.*` switches, see [Streaming Video Encoding](./streaming_video_encoding). For an encoding-parameter comparison and experiments, see the [video-benchmark Space](https://huggingface.co/spaces/lerobot/video-benchmark). For details on **when** frames are written vs. encoded (streaming vs. post-episode), queues, and other top-level `--dataset.*` switches, see [Streaming Video Encoding](./streaming_video_encoding). For an encoding-parameter comparison and experiments, see the [video-benchmark Space](https://huggingface.co/spaces/lerobot/video-benchmark).
@@ -42,25 +45,25 @@ lerobot-record \
All flags below are prefixed with `--dataset.camera_encoder.` on the CLI. All flags below are prefixed with `--dataset.camera_encoder.` on the CLI.
| Parameter | Type | Default | Description | | Parameter | Type | Default | Description |
| --------------- | -------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | --------------- | -------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `vcodec` | `str` | `"libsvtav1"` | Video codec name. `"auto"` picks the first available hardware encoder from a fixed preference list, falling back to `libsvtav1`. | | `vcodec` | `str` | `"libsvtav1"` | Video codec name. `"auto"` picks the first available hardware encoder from a fixed preference list, falling back to `libsvtav1`. |
| `pix_fmt` | `str` | `"yuv420p"` | Output pixel format. Must be supported by the chosen codec in your FFmpeg build. | | `pix_fmt` | `str` | `"yuv420p"` | Output pixel format. Must be supported by the chosen codec in your FFmpeg build. |
| `g` | `int` | `2` | GOP size — a keyframe every `g` frames. Emitted as FFmpeg option `g`. | | `g` | `int` | `2` | GOP size — a keyframe every `g` frames. Emitted as FFmpeg option `g`. |
| `crf` | `int` | `30` | Abstract quality value, mapped per codec (see the [mapping](#mapping-videoencoderconfig--ffmpeg-options) below). Lower → higher quality / larger output where the mapping is monotone. | | `crf` | `int` | `30` | Abstract quality value, mapped per codec (see the [mapping](#mapping-videoencoderconfig--ffmpeg-options) below). Lower → higher quality / larger output where the mapping is monotone. |
| `preset` | `int` or `str` | `12` \* | Encoder speed preset; meaning depends on the codec. <br/>\* When unset and `vcodec=libsvtav1`, LeRobot defaults to `12`. | | `preset` | `int` or `str` | `12` \* | Encoder speed preset; meaning depends on the codec. <br/>\* When unset and `vcodec=libsvtav1`, LeRobot defaults to `12`. |
| `fast_decode` | `int` | `0` | `libsvtav1`: `0–2`, passed via `svtav1-params`. <br/>`h264` / `hevc` (software): if `>0`, sets `tune=fastdecode`. <br/>Other codecs: usually unused. | | `fast_decode` | `int` | `0` | `libsvtav1`: `0–2`, passed via `svtav1-params`. <br/>`h264` / `hevc` (software): if `>0`, sets `tune=fastdecode`. <br/>Other codecs: usually unused. |
| `video_backend` | `str` | `"pyav"` | Only `"pyav"` is currently implemented for video encoding. | | `video_backend` | `str` | `"pyav"` | Only `"pyav"` is currently implemented for video encoding. |
| `extra_options` | `dict` | `{}` | Extra FFmpeg options merged after the structured fields above. Cannot override keys already set by those fields. | | `extra_options` | `dict` | `{}` | Extra FFmpeg options merged after the structured fields above. Cannot override keys already set by those fields. |
--- ---
## Validation ## Validation
| Check | Behavior | | Check | Behavior |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Video codec presence | `vcodec` must exist as a video encoder in the local FFmpeg build (after resolving `"auto"`). | | Video codec presence | `vcodec` must exist as a video encoder in the local FFmpeg build (after resolving `"auto"`). |
| Pixel format | `pix_fmt` is checked against the encoder's reported pixel formats when available. | | Pixel format | `pix_fmt` is checked against the encoder's reported pixel formats when available. |
| Options | The output of `get_codec_options()` (including values from `extra_options`) is checked against PyAV/FFmpeg option metadata — ranges, integer constraints, string choices — where applicable. | | Options | The output of `get_codec_options()` (including values from `extra_options`) is checked against PyAV/FFmpeg option metadata — ranges, integer constraints, string choices — where applicable. |
--- ---
@@ -78,7 +81,6 @@ How each field is forwarded to FFmpeg after `vcodec` resolution, via `get_codec_
| `h264_vaapi` | `qp` ← `crf` | — | — | | `h264_vaapi` | `qp` ← `crf` | — | — |
| `h264_qsv` | `global_quality` ← `crf` | `preset` | — | | `h264_qsv` | `global_quality` ← `crf` | `preset` | — |
--- ---
## Persistence in dataset metadata ## Persistence in dataset metadata
@@ -104,7 +106,7 @@ After the first episode of a video stream is encoded, the encoder configuration
"video.preset": "fast", "video.preset": "fast",
"video.fast_decode": 0, "video.fast_decode": 0,
"video.video_backend": "pyav", "video.video_backend": "pyav",
"video.extra_options": {"tune": "film", "profile:v": "high", "bf": 2} "video.extra_options": { "tune": "film", "profile:v": "high", "bf": 2 }
} }
} }
} }
@@ -117,7 +119,10 @@ Two sources contribute to the `info` block:
- **Encoder-derived** (taken from `VideoEncoderConfig`): `video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.video_backend`, `video.extra_options`. - **Encoder-derived** (taken from `VideoEncoderConfig`): `video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.video_backend`, `video.extra_options`.
<Tip> <Tip>
This block is populated **once**, from the **first** episode. It assumes every episode in the dataset was encoded with the same `camera_encoder`. Changing encoder settings partway through a recording is not supported — the `info.json` will only reflect the parameters used for the first episode. This block is populated **once**, from the **first** episode. It assumes every
episode in the dataset was encoded with the same `camera_encoder`. Changing
encoder settings partway through a recording is not supported — the
`info.json` will only reflect the parameters used for the first episode.
</Tip> </Tip>
--- ---
src/lerobot/datasets/dataset_metadata.py
+1 -3
View File
@@ -536,9 +536,7 @@ class LeRobotDatasetMetadata:
for key in video_keys: for key in video_keys:
if not self.features[key].get("info", None): if not self.features[key].get("info", None):
video_path = self.root / self.video_path.format(video_key=key, chunk_index=0, file_index=0) video_path = self.root / self.video_path.format(video_key=key, chunk_index=0, file_index=0)
self.info.features[key]["info"] = get_video_info( self.info.features[key]["info"] = get_video_info(video_path, camera_encoder=camera_encoder)
video_path, camera_encoder=camera_encoder
)
def update_chunk_settings( def update_chunk_settings(
self, self,
src/lerobot/datasets/dataset_tools.py
+1 -3
View File
@@ -232,9 +232,7 @@ def split_dataset(
video_metadata = None video_metadata = None
if dataset.meta.video_keys: if dataset.meta.video_keys:
video_metadata = _copy_and_reindex_videos( video_metadata = _copy_and_reindex_videos(dataset, new_meta, episode_mapping, camera_encoder)
dataset, new_meta, episode_mapping, camera_encoder
)
data_metadata = _copy_and_reindex_data(dataset, new_meta, episode_mapping) data_metadata = _copy_and_reindex_data(dataset, new_meta, episode_mapping)
src/lerobot/datasets/video_utils.py
+1 -3
View File
@@ -687,9 +687,7 @@ class StreamingVideoEncoder:
video_path = temp_video_dir / f"{video_key.replace('/', '_')}_streaming.mp4" video_path = temp_video_dir / f"{video_key.replace('/', '_')}_streaming.mp4"
vcodec = self._camera_encoder.vcodec vcodec = self._camera_encoder.vcodec
codec_options = self._camera_encoder.get_codec_options( codec_options = self._camera_encoder.get_codec_options(self._encoder_threads, as_strings=True)
self._encoder_threads, as_strings=True
)
encoder_thread = _CameraEncoderThread( encoder_thread = _CameraEncoderThread(
video_path=video_path, video_path=video_path,
fps=self.fps, fps=self.fps,
src/lerobot/scripts/lerobot_edit_dataset.py
+1 -2
View File
@@ -573,8 +573,7 @@ def handle_convert_image_to_video(cfg: EditDatasetConfig) -> None:
dataset=dataset, dataset=dataset,
output_dir=output_dir, output_dir=output_dir,
repo_id=output_repo_id, repo_id=output_repo_id,
camera_encoder=getattr(cfg.operation, "camera_encoder", None) camera_encoder=getattr(cfg.operation, "camera_encoder", None) or camera_encoder_defaults(),
or camera_encoder_defaults(),
episode_indices=getattr(cfg.operation, "episode_indices", None), episode_indices=getattr(cfg.operation, "episode_indices", None),
num_workers=getattr(cfg.operation, "num_workers", 4), num_workers=getattr(cfg.operation, "num_workers", 4),
max_episodes_per_batch=getattr(cfg.operation, "max_episodes_per_batch", None), max_episodes_per_batch=getattr(cfg.operation, "max_episodes_per_batch", None),
tests/datasets/test_streaming_video_encoder.py
+4 -12
View File
@@ -211,9 +211,7 @@ class TestStreamingVideoEncoder:
video_keys = [f"{OBS_IMAGES}.laptop", f"{OBS_IMAGES}.phone"] video_keys = [f"{OBS_IMAGES}.laptop", f"{OBS_IMAGES}.phone"]
encoder = StreamingVideoEncoder( encoder = StreamingVideoEncoder(
fps=30, fps=30,
camera_encoder=self._make_encoder_config( camera_encoder=self._make_encoder_config(vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30),
vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30
),
) )
encoder.start_episode(video_keys, tmp_path) encoder.start_episode(video_keys, tmp_path)
@@ -239,9 +237,7 @@ class TestStreamingVideoEncoder:
video_keys = [f"{OBS_IMAGES}.cam"] video_keys = [f"{OBS_IMAGES}.cam"]
encoder = StreamingVideoEncoder( encoder = StreamingVideoEncoder(
fps=30, fps=30,
camera_encoder=self._make_encoder_config( camera_encoder=self._make_encoder_config(vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30),
vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30
),
) )
for ep in range(3): for ep in range(3):
@@ -267,9 +263,7 @@ class TestStreamingVideoEncoder:
video_keys = [f"{OBS_IMAGES}.cam"] video_keys = [f"{OBS_IMAGES}.cam"]
encoder = StreamingVideoEncoder( encoder = StreamingVideoEncoder(
fps=30, fps=30,
camera_encoder=self._make_encoder_config( camera_encoder=self._make_encoder_config(vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30),
vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30
),
) )
encoder.start_episode(video_keys, tmp_path) encoder.start_episode(video_keys, tmp_path)
@@ -352,9 +346,7 @@ class TestStreamingVideoEncoder:
video_keys = [f"{OBS_IMAGES}.cam1", f"{OBS_IMAGES}.cam2"] video_keys = [f"{OBS_IMAGES}.cam1", f"{OBS_IMAGES}.cam2"]
encoder = StreamingVideoEncoder( encoder = StreamingVideoEncoder(
fps=30, fps=30,
camera_encoder=self._make_encoder_config( camera_encoder=self._make_encoder_config(vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30),
vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30
),
) )
encoder.start_episode(video_keys, tmp_path) encoder.start_episode(video_keys, tmp_path)