From 6af4eb6da4d291f0907b5bd3820d84e5a6f5da22 Mon Sep 17 00:00:00 2001 From: CarolinePascal Date: Thu, 2 Jul 2026 14:56:37 +0200 Subject: [PATCH] docs(prettier): making video encoding parameters docs prettier --- docs/source/video_encoding_parameters.mdx | 120 ++++++++++++++-------- 1 file changed, 79 insertions(+), 41 deletions(-) diff --git a/docs/source/video_encoding_parameters.mdx b/docs/source/video_encoding_parameters.mdx index 132d25056..b3ffcf86c 100644 --- a/docs/source/video_encoding_parameters.mdx +++ b/docs/source/video_encoding_parameters.mdx @@ -6,12 +6,11 @@ Encoding frames into an MP4 is a full FFmpeg pipeline: choice of encoder, pixel You can set these parameters from the CLI with `--dataset.rgb_encoder.` (e.g. with `lerobot-record` or `lerobot-rollout`). The same block applies to every camera video stream in that run. - - Video storage must be on for `rgb_encoder` to have any effect — - `use_videos=True` in Python APIs, or `--dataset.video=true` on the CLI (the - recording default). With video off, inputs stay as images and `rgb_encoder` is - ignored. - +> [!TIP] +> Video storage must be on for `rgb_encoder` to have any effect — +> `use_videos=True` in Python APIs, or `--dataset.video=true` on the CLI (the +> recording default). With video off, inputs stay as images and `rgb_encoder` is +> ignored. For details on **when** frames are written vs. encoded (streaming vs. post-episode), queues, and other top-level `--dataset.*` switches, see [Streaming Video Encoding](./streaming_video_encoding). For an encoding-parameter comparison and experiments, see the [video-benchmark Space](https://huggingface.co/spaces/lerobot/video-benchmark). @@ -43,12 +42,10 @@ lerobot-record \ ## Tuning parameters - -The defaults are tuned to balance **compression ratio**, **visual quality**, and **decoding/seek speed** for typical robotics datasets. Changing them can affect both recording (CPU load, frame drops) and training (decoding throughput, image quality). - -Only override these parameters if you have a specific reason to, and measure the impact on your pipeline before relying on the new settings. - - +> [!WARNING] +> The defaults are tuned to balance **compression ratio**, **visual quality**, and **decoding/seek speed** for typical robotics datasets. Changing them can affect both recording (CPU load, frame drops) and training (decoding throughput, image quality). +> +> Only override these parameters if you have a specific reason to, and measure the impact on your pipeline before relying on the new settings. All flags below are prefixed with `--dataset.rgb_encoder.` on the CLI. @@ -69,25 +66,29 @@ All flags below are prefixed with `--dataset.rgb_encoder.` on the CLI. Depth maps (Intel RealSense, Reachy 2) are stored as their **own video streams** alongside the RGB streams. Raw depth (`uint16` millimetres or `float32` metres) can't survive an 8-bit codec, so LeRobot **quantizes** each map to a 12-bit code (`[0, 4095]`) — logarithmically by default, to match the `1/depth` error profile of depth sensors — then packs it into a high-bit-depth pixel format (`gray12le`) and encodes it with a 12-bit codec. -```mermaid -flowchart LR - A["Raw depth (uint16 mm / float32 m)"] --> B["Clip to depth_min, depth_max"] - B --> C["Quantize to 12-bit code 0–4095 (log or linear)"] - C --> D["Pack into gray12le"] - D --> E["Encode video (hevc Main 12)"] - E --> F[("MP4 + metadata: depth_min/max, shift, use_log")] - F -. "load time (depth_output_unit)" .-> G["Dequantize to mm or m"] - - classDef input fill:#e3f2fd,stroke:#1565c0,color:#0d47a1; - classDef encode fill:#ede7f6,stroke:#5e35b1,color:#311b92; - classDef store fill:#fff8e1,stroke:#f9a825,color:#e65100; - classDef load fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20; - - class A input; - class B,C,D,E encode; - class F store; - class G load; -``` +
+
+ Raw depthuint16 mm
float32 m + → +
+ Record time + Clipto [depth_min,
depth_max] + → + Quantize12-bit codes 0–4095
log (default) or linear + → + Packinto gray12le
plane + → + EncodeHEVC
Main 12 +
+ → + MP4stored
stream + → +
+ Load time + Dequantizeto mm / m +
+
+
Configure the depth pipeline through a parallel **`depth_encoder`** block (`DepthEncoderConfig`). It shares every `RGBEncoderConfig` field (`vcodec`, `pix_fmt`, `crf`, …) and adds four quantizer knobs, set via `--dataset.depth_encoder.`: @@ -168,15 +169,44 @@ After the first episode of a video stream is encoded, the encoder configuration Two sources contribute to the `info` block: -- **Stream-derived** (read back from the encoded MP4 with PyAV): `video.height`, `video.width`, `video.codec`, `video.pix_fmt`, `video.fps`, `video.channels`, `is_depth_map`, plus `audio.*` if an audio stream is present. -- **Encoder-derived** (taken from `RGBEncoderConfig` or `DepthEncoderConfig`): `video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.video_backend`, `video.extra_options`. +
+
+
Stream-derived
+
+
Read back from the encoded MP4 with PyAV.
+
+ video.height + video.width + video.codec + video.pix_fmt + video.fps + video.channels + is_depth_map + audio.* +
+
+
+
+
Encoder-derived
+
+
Taken from RGBEncoderConfig / DepthEncoderConfig.
+
+ video.g + video.crf + video.preset + video.fast_decode + video.video_backend + video.extra_options +
+
+
+
- - This block is populated **once**, from the **first** episode. It assumes every - episode in the dataset was encoded with the same `rgb_encoder`. Changing - encoder settings partway through a recording is not supported — the - `info.json` will only reflect the parameters used for the first episode. - +> [!IMPORTANT] +> This block is populated **once**, from the **first** episode. It assumes every +> episode in the dataset was encoded with the same `rgb_encoder`. Changing +> encoder settings partway through a recording is not supported — the +> `info.json` will only reflect the parameters used for the first episode. --- @@ -184,5 +214,13 @@ Two sources contribute to the `info` block: When aggregating datasets with `merge_datasets`, video files are concatenated as-is (no re-encoding), and encoder fields in `info.json` are merged per-key: -- **Stream-derived fields must match** across sources: `video.codec`, `video.pix_fmt`, `video.height`, `video.width`, `video.fps`. Otherwise FFmpeg's concat demuxer fails. -- **Encoder-tuning fields are merged loosely**: `video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.extra_options`. If every source agrees, the value is kept; if not, it's set to `null` (or `{}` for `video.extra_options`) and a warning is logged. +
+
+ Must match + Stream-derived fields — video.codec, video.pix_fmt, video.height, video.width, video.fps — must match across sources, otherwise FFmpeg's concat demuxer fails. +
+
+ Merged loosely + Encoder-tuning fields — video.g, video.crf, video.preset, video.fast_decode, video.extra_options. If every source agrees, the value is kept; if not, it's set to null (or {} for video.extra_options) and a warning is logged. +
+