mirror of
https://github.com/huggingface/lerobot.git
synced 2026-07-05 00:57:06 +00:00
chore(format): formatting code
This commit is contained in:
CarolinePascal
@@ -67,27 +67,90 @@ All flags below are prefixed with `--dataset.rgb_encoder.` on the CLI.
|
||||
Depth maps (Intel RealSense, Reachy 2) are stored as their **own video streams** alongside the RGB streams. Raw depth (`uint16` millimetres or `float32` metres) can't survive an 8-bit codec, so LeRobot **quantizes** each map to a 12-bit code (`[0, 4095]`) — logarithmically by default, to match the `1/depth` error profile of depth sensors — then packs it into a high-bit-depth pixel format (`gray12le`) and encodes it with a 12-bit codec.
|
||||
|
||||
<div style="margin:28px 0;padding:14px 0;">
|
||||
<div style="margin:0 auto;display:flex;flex-wrap:wrap;justify-content:center;align-items:stretch;gap:6px;font-family:'Source Sans 3',ui-sans-serif,system-ui,sans-serif;font-size:14px;font-weight:600;color:#1B1B1D;">
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#DBEAFE;color:#1D4ED8;border-radius:9px;padding:8px 12px;"><span>Raw depth</span><span style="font-size:11px;font-weight:400;color:#3B6FD4;white-space:nowrap;">uint16 mm<br/>float32 m</span></span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">→</span>
|
||||
<div style="border:2px dashed #C4B5FD;border-radius:13px;padding:18px 12px 12px;position:relative;display:flex;align-items:stretch;gap:6px;">
|
||||
<span style="position:absolute;top:-10px;left:12px;background:#fff;padding:0 6px;font-size:11px;font-weight:700;color:#7E22CE;text-transform:uppercase;letter-spacing:0.5px;white-space:nowrap;">Record time</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;"><span>Clip</span><span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">to [depth_min,<br/>depth_max]</span></span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">→</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;"><span>Quantize</span><span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">12-bit codes 0–4095<br/>log (default) or linear</span></span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">→</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;"><span>Pack</span><span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">into gray12le<br/>plane</span></span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">→</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;"><span>Encode</span><span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">HEVC<br/>Main 12</span></span>
|
||||
<div style="margin:0 auto;display:flex;flex-wrap:wrap;justify-content:center;align-items:stretch;gap:6px;font-family:'Source Sans 3',ui-sans-serif,system-ui,sans-serif;font-size:14px;font-weight:600;color:#1B1B1D;">
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#DBEAFE;color:#1D4ED8;border-radius:9px;padding:8px 12px;">
|
||||
<span>Raw depth</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#3B6FD4;white-space:nowrap;">
|
||||
uint16 mm
|
||||
<br />
|
||||
float32 m
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<div style="border:2px dashed #C4B5FD;border-radius:13px;padding:18px 12px 12px;position:relative;display:flex;align-items:stretch;gap:6px;">
|
||||
<span style="position:absolute;top:-10px;left:12px;background:#fff;padding:0 6px;font-size:11px;font-weight:700;color:#7E22CE;text-transform:uppercase;letter-spacing:0.5px;white-space:nowrap;">
|
||||
Record time
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;">
|
||||
<span>Clip</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">
|
||||
to [depth_min,
|
||||
<br />
|
||||
depth_max]
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;">
|
||||
<span>Quantize</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">
|
||||
12-bit codes 0–4095
|
||||
<br />
|
||||
log (default) or linear
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;">
|
||||
<span>Pack</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">
|
||||
into gray12le
|
||||
<br />
|
||||
plane
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#F3E8FF;color:#7E22CE;border-radius:9px;padding:8px 12px;">
|
||||
<span>Encode</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#9061C2;white-space:nowrap;">
|
||||
HEVC
|
||||
<br />
|
||||
Main 12
|
||||
</span>
|
||||
</span>
|
||||
</div>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">
|
||||
→
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#FEF3C7;color:#B45309;border-radius:9px;padding:8px 12px;">
|
||||
<span>MP4</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#C77D18;white-space:nowrap;">
|
||||
stored
|
||||
<br />
|
||||
stream
|
||||
</span>
|
||||
</span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#34A06B;">
|
||||
→
|
||||
</span>
|
||||
<div style="border:2px dashed #6EE7B7;border-radius:13px;padding:18px 12px 12px;position:relative;display:flex;align-items:center;gap:6px;">
|
||||
<span style="position:absolute;top:-10px;left:12px;background:#fff;padding:0 6px;font-size:11px;font-weight:700;color:#047857;text-transform:uppercase;letter-spacing:0.5px;white-space:nowrap;">
|
||||
Load time
|
||||
</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#D1FAE5;color:#047857;border-radius:9px;padding:8px 12px;">
|
||||
<span>Dequantize</span>
|
||||
<span style="font-size:11px;font-weight:400;color:#059669;white-space:nowrap;">
|
||||
to mm / m
|
||||
</span>
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#C3CBD9;">→</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#FEF3C7;color:#B45309;border-radius:9px;padding:8px 12px;"><span>MP4</span><span style="font-size:11px;font-weight:400;color:#C77D18;white-space:nowrap;">stored<br/>stream</span></span>
|
||||
<span style="display:flex;align-items:center;font-size:16px;color:#34A06B;">→</span>
|
||||
<div style="border:2px dashed #6EE7B7;border-radius:13px;padding:18px 12px 12px;position:relative;display:flex;align-items:center;gap:6px;">
|
||||
<span style="position:absolute;top:-10px;left:12px;background:#fff;padding:0 6px;font-size:11px;font-weight:700;color:#047857;text-transform:uppercase;letter-spacing:0.5px;white-space:nowrap;">Load time</span>
|
||||
<span style="display:flex;flex-direction:column;justify-content:center;align-items:center;text-align:center;gap:2px;background:#D1FAE5;color:#047857;border-radius:9px;padding:8px 12px;"><span>Dequantize</span><span style="font-size:11px;font-weight:400;color:#059669;white-space:nowrap;">to mm / m</span></span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Configure the depth pipeline through a parallel **`depth_encoder`** block (`DepthEncoderConfig`). It shares every `RGBEncoderConfig` field (`vcodec`, `pix_fmt`, `crf`, …) and adds four quantizer knobs, set via `--dataset.depth_encoder.<field>`:
|
||||
@@ -171,32 +234,69 @@ Two sources contribute to the `info` block:
|
||||
|
||||
<div style="display:flex;flex-wrap:wrap;gap:14px;margin:20px 0;font-family:'Source Sans 3',ui-sans-serif,system-ui,sans-serif;">
|
||||
<div style="flex:1 1 280px;border:1px solid #BFDBFE;border-radius:12px;overflow:hidden;">
|
||||
<div style="background:#DBEAFE;color:#1D4ED8;font-weight:700;font-size:14px;padding:8px 14px;">Stream-derived</div>
|
||||
<div style="background:#DBEAFE;color:#1D4ED8;font-weight:700;font-size:14px;padding:8px 14px;">
|
||||
Stream-derived
|
||||
</div>
|
||||
<div style="padding:12px 14px;">
|
||||
<div style="font-size:13px;color:#4B5563;margin-bottom:10px;">Read back from the encoded MP4 with PyAV.</div>
|
||||
<div style="font-size:13px;color:#4B5563;margin-bottom:10px;">
|
||||
Read back from the encoded MP4 with PyAV.
|
||||
</div>
|
||||
<div style="display:flex;flex-wrap:wrap;gap:6px;">
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">video.height</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">video.width</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">video.codec</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">video.pix_fmt</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">video.fps</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">video.channels</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">is_depth_map</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">audio.*</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.height
|
||||
</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.width
|
||||
</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.codec
|
||||
</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.pix_fmt
|
||||
</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.fps
|
||||
</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.channels
|
||||
</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
is_depth_map
|
||||
</code>
|
||||
<code style="background:#EFF6FF;color:#1D4ED8;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
audio.*
|
||||
</code>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div style="flex:1 1 280px;border:1px solid #DDD6FE;border-radius:12px;overflow:hidden;">
|
||||
<div style="background:#F3E8FF;color:#7E22CE;font-weight:700;font-size:14px;padding:8px 14px;">Encoder-derived</div>
|
||||
<div style="background:#F3E8FF;color:#7E22CE;font-weight:700;font-size:14px;padding:8px 14px;">
|
||||
Encoder-derived
|
||||
</div>
|
||||
<div style="padding:12px 14px;">
|
||||
<div style="font-size:13px;color:#4B5563;margin-bottom:10px;">Taken from <code style="font-size:12px;">RGBEncoderConfig</code> / <code style="font-size:12px;">DepthEncoderConfig</code>.</div>
|
||||
<div style="font-size:13px;color:#4B5563;margin-bottom:10px;">
|
||||
Taken from <code style="font-size:12px;">RGBEncoderConfig</code> /{" "}
|
||||
<code style="font-size:12px;">DepthEncoderConfig</code>.
|
||||
</div>
|
||||
<div style="display:flex;flex-wrap:wrap;gap:6px;">
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">video.g</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">video.crf</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">video.preset</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">video.fast_decode</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">video.video_backend</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">video.extra_options</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.g
|
||||
</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.crf
|
||||
</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.preset
|
||||
</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.fast_decode
|
||||
</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.video_backend
|
||||
</code>
|
||||
<code style="background:#FAF5FF;color:#7E22CE;border-radius:6px;padding:2px 8px;font-size:12px;">
|
||||
video.extra_options
|
||||
</code>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -216,11 +316,33 @@ When aggregating datasets with `merge_datasets`, video files are concatenated as
|
||||
|
||||
<div style="display:flex;flex-direction:column;gap:12px;margin:20px 0;font-family:'Source Sans 3',ui-sans-serif,system-ui,sans-serif;">
|
||||
<div style="display:flex;gap:12px;align-items:flex-start;border-left:3px solid #F87171;background:#FEF2F2;border-radius:0 10px 10px 0;padding:12px 14px;">
|
||||
<span style="flex:none;background:#FEE2E2;color:#B91C1C;font-weight:700;font-size:11px;text-transform:uppercase;letter-spacing:0.4px;border-radius:6px;padding:3px 8px;margin-top:1px;white-space:nowrap;">Must match</span>
|
||||
<span style="font-size:14px;color:#1B1B1D;">Stream-derived fields — <code style="font-size:12px;">video.codec</code>, <code style="font-size:12px;">video.pix_fmt</code>, <code style="font-size:12px;">video.height</code>, <code style="font-size:12px;">video.width</code>, <code style="font-size:12px;">video.fps</code> — must match across sources, otherwise FFmpeg's concat demuxer fails.</span>
|
||||
<span style="flex:none;background:#FEE2E2;color:#B91C1C;font-weight:700;font-size:11px;text-transform:uppercase;letter-spacing:0.4px;border-radius:6px;padding:3px 8px;margin-top:1px;white-space:nowrap;">
|
||||
Must match
|
||||
</span>
|
||||
<span style="font-size:14px;color:#1B1B1D;">
|
||||
Stream-derived fields — <code style="font-size:12px;">video.codec</code>,{" "}
|
||||
<code style="font-size:12px;">video.pix_fmt</code>,{" "}
|
||||
<code style="font-size:12px;">video.height</code>,{" "}
|
||||
<code style="font-size:12px;">video.width</code>,{" "}
|
||||
<code style="font-size:12px;">video.fps</code> — must match across
|
||||
sources, otherwise FFmpeg's concat demuxer fails.
|
||||
</span>
|
||||
</div>
|
||||
<div style="display:flex;gap:12px;align-items:flex-start;border-left:3px solid #34D399;background:#ECFDF5;border-radius:0 10px 10px 0;padding:12px 14px;">
|
||||
<span style="flex:none;background:#D1FAE5;color:#047857;font-weight:700;font-size:11px;text-transform:uppercase;letter-spacing:0.4px;border-radius:6px;padding:3px 8px;margin-top:1px;white-space:nowrap;">Merged loosely</span>
|
||||
<span style="font-size:14px;color:#1B1B1D;">Encoder-tuning fields — <code style="font-size:12px;">video.g</code>, <code style="font-size:12px;">video.crf</code>, <code style="font-size:12px;">video.preset</code>, <code style="font-size:12px;">video.fast_decode</code>, <code style="font-size:12px;">video.extra_options</code>. If every source agrees, the value is kept; if not, it's set to <code style="font-size:12px;">null</code> (or <code style="font-size:12px;">{}</code> for <code style="font-size:12px;">video.extra_options</code>) and a warning is logged.</span>
|
||||
<span style="flex:none;background:#D1FAE5;color:#047857;font-weight:700;font-size:11px;text-transform:uppercase;letter-spacing:0.4px;border-radius:6px;padding:3px 8px;margin-top:1px;white-space:nowrap;">
|
||||
Merged loosely
|
||||
</span>
|
||||
<span style="font-size:14px;color:#1B1B1D;">
|
||||
Encoder-tuning fields — <code style="font-size:12px;">video.g</code>,{" "}
|
||||
<code style="font-size:12px;">video.crf</code>,{" "}
|
||||
<code style="font-size:12px;">video.preset</code>,{" "}
|
||||
<code style="font-size:12px;">video.fast_decode</code>,{" "}
|
||||
<code style="font-size:12px;">video.extra_options</code>. If every source
|
||||
agrees, the value is kept; if not, it's set to{" "}
|
||||
<code style="font-size:12px;">null</code> (or{" "}
|
||||
<code style="font-size:12px;">{}</code> for{" "}
|
||||
<code style="font-size:12px;">video.extra_options</code>) and a warning is
|
||||
logged.
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user