diff --git a/docs/source/streaming_video_encoding.mdx b/docs/source/streaming_video_encoding.mdx index 36c599bad..96e049eb3 100644 --- a/docs/source/streaming_video_encoding.mdx +++ b/docs/source/streaming_video_encoding.mdx @@ -14,12 +14,12 @@ This makes `save_episode()` near-instant (the video is already encoded by the ti ## 2. Tuning Parameters -| Parameter | CLI Flag | Type | Default | Description | -| ----------------------- | ---------------------------------------- | ------------- | ------------- | ----------------------------------------------------------------- | -| `streaming_encoding` | `--dataset.streaming_encoding` | `bool` | `True` | Enable real-time encoding during capture | +| Parameter | CLI Flag | Type | Default | Description | +| ----------------------- | --------------------------------- | ------------- | ------------- | ----------------------------------------------------------------- | +| `streaming_encoding` | `--dataset.streaming_encoding` | `bool` | `True` | Enable real-time encoding during capture | | `vcodec` | `--dataset.camera_encoder.vcodec` | `str` | `"libsvtav1"` | Video codec. `"auto"` detects best HW encoder | -| `encoder_threads` | `--dataset.encoder_threads` | `int \| None` | `None` (auto) | Threads per encoder instance. `None` will leave the vcoded decide | -| `encoder_queue_maxsize` | `--dataset.encoder_queue_maxsize` | `int` | `30` | Max buffered frames per camera (~1s at 30fps). Consumes RAM | +| `encoder_threads` | `--dataset.encoder_threads` | `int \| None` | `None` (auto) | Threads per encoder instance. `None` will leave the vcoded decide | +| `encoder_queue_maxsize` | `--dataset.encoder_queue_maxsize` | `int` | `30` | Max buffered frames per camera (~1s at 30fps). Consumes RAM | ## 3. Performance Considerations @@ -82,8 +82,8 @@ Use HW encoding when: ### Available HW Encoders -| Encoder | Platform | Hardware | CLI Value | -| ------------------- | ------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- | +| Encoder | Platform | Hardware | CLI Value | +| ------------------- | ------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------- | | `h264_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.camera_encoder.vcodec=h264_videotoolbox` | | `hevc_videotoolbox` | macOS | Apple Silicon / Intel | `--dataset.camera_encoder.vcodec=hevc_videotoolbox` | | `h264_nvenc` | Linux/Windows | NVIDIA GPU | `--dataset.camera_encoder.vcodec=h264_nvenc` | @@ -100,15 +100,15 @@ Use HW encoding when: ## 5. Troubleshooting -| Symptom | Likely Cause | Fix | -| ------------------------------------------------------------------ | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Symptom | Likely Cause | Fix | +| ------------------------------------------------------------------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | System freezes or choppy robot movement or Rerun visualization lag | CPU starved (100% load usage) | Close other apps, reduce encoding throughput, lower `encoder_threads`, use `h264`, use `display_data=False`. If the CPU continues to be at 100% then it might be insufficient for your setup, consider `--dataset.streaming_encoding=false` or HW encoding (`--dataset.camera_encoder.vcodec=auto`) | | "Encoder queue full" warnings or dropped frames in dataset | Encoder can't keep up (Queue overflow) | If CPU is not at 100%: Increase `encoder_threads`, increase `encoder_queue_maxsize` or use HW encoding (`--dataset.camera_encoder.vcodec=auto`). | -| High RAM usage | Queue filling faster than encoding | `encoder_threads` too low or CPU insufficient. Reduce `encoder_queue_maxsize` or use HW encoding | -| Large video files | Using HW encoder or H.264 | Expected trade-off. Switch to `libsvtav1` if CPU allows | -| `save_episode()` still slow | `streaming_encoding` is `False` | Set `--dataset.streaming_encoding=true` | +| High RAM usage | Queue filling faster than encoding | `encoder_threads` too low or CPU insufficient. Reduce `encoder_queue_maxsize` or use HW encoding | +| Large video files | Using HW encoder or H.264 | Expected trade-off. Switch to `libsvtav1` if CPU allows | +| `save_episode()` still slow | `streaming_encoding` is `False` | Set `--dataset.streaming_encoding=true` | | Encoder thread crash | Codec not available or invalid settings | Check `vcodec` is installed, try `--dataset.camera_encoder.vcodec=auto` | -| Recorded dataset is missing frames | CPU/GPU starvation or occasional load spikes | If ~5% of frames are missing, your system is likely overloaded — follow the recommendations above. If fewer frames are missing (~2%), they are probably due to occasional transient load spikes (often at startup) and can be considered expected. | +| Recorded dataset is missing frames | CPU/GPU starvation or occasional load spikes | If ~5% of frames are missing, your system is likely overloaded — follow the recommendations above. If fewer frames are missing (~2%), they are probably due to occasional transient load spikes (often at startup) and can be considered expected. | ## 6. Recommended Configurations diff --git a/docs/source/video_encoding_parameters.mdx b/docs/source/video_encoding_parameters.mdx index a257b2d9f..c07f40c5b 100644 --- a/docs/source/video_encoding_parameters.mdx +++ b/docs/source/video_encoding_parameters.mdx @@ -1,13 +1,16 @@ # Video encoding parameters -When video storage is enabled, LeRobot stores each camera stream as an **MP4** file instead of saving one image file per timestep. Video encoding compresses across time, which usually cuts dataset size and I/O compared to a pile of PNGs, while keeping MP4 — a format every player and loader understands. +When video storage is enabled, LeRobot stores each camera stream as an **MP4** file instead of saving one image file per timestep. Video encoding compresses across time, which usually cuts dataset size and I/O compared to a pile of PNG, while keeping MP4 — a format every player and loader understands. Encoding frames into an MP4 is a full FFmpeg pipeline: choice of encoder, pixel format, GOP/keyframes, quality vs. speed, and optional extra encoder flags. Most of these knobs are user-tunable through `camera_encoder`, a nested `VideoEncoderConfig` (`lerobot.configs.video.VideoEncoderConfig`) passed through PyAV. You can set these parameters from the CLI with `--dataset.camera_encoder.` (e.g. with `lerobot-record` or `lerobot-rollout`). The same block applies to every camera video stream in that run. -Video storage must be on for `camera_encoder` to have any effect — `use_videos=True` in Python APIs, or `--dataset.video=true` on the CLI (the recording default). With video off, inputs stay as images and `camera_encoder` is ignored. + Video storage must be on for `camera_encoder` to have any effect — + `use_videos=True` in Python APIs, or `--dataset.video=true` on the CLI (the + recording default). With video off, inputs stay as images and `camera_encoder` + is ignored. For details on **when** frames are written vs. encoded (streaming vs. post-episode), queues, and other top-level `--dataset.*` switches, see [Streaming Video Encoding](./streaming_video_encoding). For an encoding-parameter comparison and experiments, see the [video-benchmark Space](https://huggingface.co/spaces/lerobot/video-benchmark). @@ -42,25 +45,25 @@ lerobot-record \ All flags below are prefixed with `--dataset.camera_encoder.` on the CLI. -| Parameter | Type | Default | Description | -| --------------- | -------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `vcodec` | `str` | `"libsvtav1"` | Video codec name. `"auto"` picks the first available hardware encoder from a fixed preference list, falling back to `libsvtav1`. | -| `pix_fmt` | `str` | `"yuv420p"` | Output pixel format. Must be supported by the chosen codec in your FFmpeg build. | -| `g` | `int` | `2` | GOP size — a keyframe every `g` frames. Emitted as FFmpeg option `g`. | +| Parameter | Type | Default | Description | +| --------------- | -------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `vcodec` | `str` | `"libsvtav1"` | Video codec name. `"auto"` picks the first available hardware encoder from a fixed preference list, falling back to `libsvtav1`. | +| `pix_fmt` | `str` | `"yuv420p"` | Output pixel format. Must be supported by the chosen codec in your FFmpeg build. | +| `g` | `int` | `2` | GOP size — a keyframe every `g` frames. Emitted as FFmpeg option `g`. | | `crf` | `int` | `30` | Abstract quality value, mapped per codec (see the [mapping](#mapping-videoencoderconfig--ffmpeg-options) below). Lower → higher quality / larger output where the mapping is monotone. | -| `preset` | `int` or `str` | `12` \* | Encoder speed preset; meaning depends on the codec.
\* When unset and `vcodec=libsvtav1`, LeRobot defaults to `12`. | -| `fast_decode` | `int` | `0` | `libsvtav1`: `0–2`, passed via `svtav1-params`.
`h264` / `hevc` (software): if `>0`, sets `tune=fastdecode`.
Other codecs: usually unused. | -| `video_backend` | `str` | `"pyav"` | Only `"pyav"` is currently implemented for video encoding. | -| `extra_options` | `dict` | `{}` | Extra FFmpeg options merged after the structured fields above. Cannot override keys already set by those fields. | +| `preset` | `int` or `str` | `12` \* | Encoder speed preset; meaning depends on the codec.
\* When unset and `vcodec=libsvtav1`, LeRobot defaults to `12`. | +| `fast_decode` | `int` | `0` | `libsvtav1`: `0–2`, passed via `svtav1-params`.
`h264` / `hevc` (software): if `>0`, sets `tune=fastdecode`.
Other codecs: usually unused. | +| `video_backend` | `str` | `"pyav"` | Only `"pyav"` is currently implemented for video encoding. | +| `extra_options` | `dict` | `{}` | Extra FFmpeg options merged after the structured fields above. Cannot override keys already set by those fields. | --- ## Validation -| Check | Behavior | -| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Video codec presence | `vcodec` must exist as a video encoder in the local FFmpeg build (after resolving `"auto"`). | -| Pixel format | `pix_fmt` is checked against the encoder's reported pixel formats when available. | +| Check | Behavior | +| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Video codec presence | `vcodec` must exist as a video encoder in the local FFmpeg build (after resolving `"auto"`). | +| Pixel format | `pix_fmt` is checked against the encoder's reported pixel formats when available. | | Options | The output of `get_codec_options()` (including values from `extra_options`) is checked against PyAV/FFmpeg option metadata — ranges, integer constraints, string choices — where applicable. | --- @@ -78,7 +81,6 @@ How each field is forwarded to FFmpeg after `vcodec` resolution, via `get_codec_ | `h264_vaapi` | `qp` ← `crf` | — | — | | `h264_qsv` | `global_quality` ← `crf` | `preset` | — | - --- ## Persistence in dataset metadata @@ -104,7 +106,7 @@ After the first episode of a video stream is encoded, the encoder configuration "video.preset": "fast", "video.fast_decode": 0, "video.video_backend": "pyav", - "video.extra_options": {"tune": "film", "profile:v": "high", "bf": 2} + "video.extra_options": { "tune": "film", "profile:v": "high", "bf": 2 } } } } @@ -117,7 +119,10 @@ Two sources contribute to the `info` block: - **Encoder-derived** (taken from `VideoEncoderConfig`): `video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.video_backend`, `video.extra_options`. -This block is populated **once**, from the **first** episode. It assumes every episode in the dataset was encoded with the same `camera_encoder`. Changing encoder settings partway through a recording is not supported — the `info.json` will only reflect the parameters used for the first episode. + This block is populated **once**, from the **first** episode. It assumes every + episode in the dataset was encoded with the same `camera_encoder`. Changing + encoder settings partway through a recording is not supported — the + `info.json` will only reflect the parameters used for the first episode. --- diff --git a/src/lerobot/datasets/dataset_metadata.py b/src/lerobot/datasets/dataset_metadata.py index 289160650..51e60724a 100644 --- a/src/lerobot/datasets/dataset_metadata.py +++ b/src/lerobot/datasets/dataset_metadata.py @@ -536,9 +536,7 @@ class LeRobotDatasetMetadata: for key in video_keys: if not self.features[key].get("info", None): video_path = self.root / self.video_path.format(video_key=key, chunk_index=0, file_index=0) - self.info.features[key]["info"] = get_video_info( - video_path, camera_encoder=camera_encoder - ) + self.info.features[key]["info"] = get_video_info(video_path, camera_encoder=camera_encoder) def update_chunk_settings( self, diff --git a/src/lerobot/datasets/dataset_tools.py b/src/lerobot/datasets/dataset_tools.py index ace9f7339..438f3c0b4 100644 --- a/src/lerobot/datasets/dataset_tools.py +++ b/src/lerobot/datasets/dataset_tools.py @@ -232,9 +232,7 @@ def split_dataset( video_metadata = None if dataset.meta.video_keys: - video_metadata = _copy_and_reindex_videos( - dataset, new_meta, episode_mapping, camera_encoder - ) + video_metadata = _copy_and_reindex_videos(dataset, new_meta, episode_mapping, camera_encoder) data_metadata = _copy_and_reindex_data(dataset, new_meta, episode_mapping) diff --git a/src/lerobot/datasets/video_utils.py b/src/lerobot/datasets/video_utils.py index 5d705ca72..e823a406c 100644 --- a/src/lerobot/datasets/video_utils.py +++ b/src/lerobot/datasets/video_utils.py @@ -687,9 +687,7 @@ class StreamingVideoEncoder: video_path = temp_video_dir / f"{video_key.replace('/', '_')}_streaming.mp4" vcodec = self._camera_encoder.vcodec - codec_options = self._camera_encoder.get_codec_options( - self._encoder_threads, as_strings=True - ) + codec_options = self._camera_encoder.get_codec_options(self._encoder_threads, as_strings=True) encoder_thread = _CameraEncoderThread( video_path=video_path, fps=self.fps, diff --git a/src/lerobot/scripts/lerobot_edit_dataset.py b/src/lerobot/scripts/lerobot_edit_dataset.py index ca5a5d893..9107a29c7 100644 --- a/src/lerobot/scripts/lerobot_edit_dataset.py +++ b/src/lerobot/scripts/lerobot_edit_dataset.py @@ -573,8 +573,7 @@ def handle_convert_image_to_video(cfg: EditDatasetConfig) -> None: dataset=dataset, output_dir=output_dir, repo_id=output_repo_id, - camera_encoder=getattr(cfg.operation, "camera_encoder", None) - or camera_encoder_defaults(), + camera_encoder=getattr(cfg.operation, "camera_encoder", None) or camera_encoder_defaults(), episode_indices=getattr(cfg.operation, "episode_indices", None), num_workers=getattr(cfg.operation, "num_workers", 4), max_episodes_per_batch=getattr(cfg.operation, "max_episodes_per_batch", None), diff --git a/tests/datasets/test_streaming_video_encoder.py b/tests/datasets/test_streaming_video_encoder.py index a2a494e8b..b69f24254 100644 --- a/tests/datasets/test_streaming_video_encoder.py +++ b/tests/datasets/test_streaming_video_encoder.py @@ -211,9 +211,7 @@ class TestStreamingVideoEncoder: video_keys = [f"{OBS_IMAGES}.laptop", f"{OBS_IMAGES}.phone"] encoder = StreamingVideoEncoder( fps=30, - camera_encoder=self._make_encoder_config( - vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30 - ), + camera_encoder=self._make_encoder_config(vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30), ) encoder.start_episode(video_keys, tmp_path) @@ -239,9 +237,7 @@ class TestStreamingVideoEncoder: video_keys = [f"{OBS_IMAGES}.cam"] encoder = StreamingVideoEncoder( fps=30, - camera_encoder=self._make_encoder_config( - vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30 - ), + camera_encoder=self._make_encoder_config(vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30), ) for ep in range(3): @@ -267,9 +263,7 @@ class TestStreamingVideoEncoder: video_keys = [f"{OBS_IMAGES}.cam"] encoder = StreamingVideoEncoder( fps=30, - camera_encoder=self._make_encoder_config( - vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30 - ), + camera_encoder=self._make_encoder_config(vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30), ) encoder.start_episode(video_keys, tmp_path) @@ -352,9 +346,7 @@ class TestStreamingVideoEncoder: video_keys = [f"{OBS_IMAGES}.cam1", f"{OBS_IMAGES}.cam2"] encoder = StreamingVideoEncoder( fps=30, - camera_encoder=self._make_encoder_config( - vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30 - ), + camera_encoder=self._make_encoder_config(vcodec="libsvtav1", pix_fmt="yuv420p", g=2, crf=30), ) encoder.start_episode(video_keys, tmp_path)