mirror of
https://github.com/huggingface/lerobot.git
synced 2026-05-11 14:49:43 +00:00
Caroline Pascal
55c0471db9
* docs(cameras): revising and improving docs on cameras * resolving copilot comments
221 lines
9.5 KiB
Plaintext
221 lines
9.5 KiB
Plaintext
# Cameras
|
||
|
||
LeRobot offers multiple options for video capture:
|
||
|
||
| Class | Supported Cameras |
|
||
| ----------------- | ----------------------------------- |
|
||
| `OpenCVCamera` | Phone, built-in laptop, USB webcams |
|
||
| `ZMQCamera` | Network-connected cameras |
|
||
| `RealSenseCamera` | Intel RealSense (with depth) |
|
||
| `Reachy2Camera` | Reachy 2 robot cameras |
|
||
|
||
> [!TIP]
|
||
> For `OpenCVCamera` compatibility details, see the [Video I/O with OpenCV Overview](https://docs.opencv.org/4.x/d0/da7/videoio_overview.html).
|
||
|
||
### Find your camera
|
||
|
||
Every camera requires a unique identifier to be instantiated, allowing you to distinguish between multiple connected devices.
|
||
|
||
`OpenCVCamera` and `RealSenseCamera` support auto-discovery. Run the command below to list available devices and their identifiers. Note that these identifiers may change after rebooting your computer or re-plugging the camera, depending on your operating system.
|
||
|
||
```bash
|
||
lerobot-find-cameras opencv # or realsense for Intel Realsense cameras
|
||
```
|
||
|
||
The output will look something like this if you have two cameras connected:
|
||
|
||
```bash
|
||
--- Detected Cameras ---
|
||
Camera #0:
|
||
Name: OpenCV Camera @ 0
|
||
Type: OpenCV
|
||
Id: 0
|
||
Backend api: AVFOUNDATION
|
||
Default stream profile:
|
||
Format: 16.0
|
||
Width: 1920
|
||
Height: 1080
|
||
Fps: 15.0
|
||
--------------------
|
||
(more cameras ...)
|
||
```
|
||
|
||
> [!WARNING]
|
||
> When using Intel RealSense cameras in `macOS`, you could get this [error](https://github.com/IntelRealSense/librealsense/issues/12307): `Error finding RealSense cameras: failed to set power state`, this can be solved by running the same command with `sudo` permissions. Note that using RealSense cameras in `macOS` is unstable.
|
||
|
||
`ZMQCamera` and `Reachy2Camera` do not support auto-discovery. They must be configured manually by providing their network address and port or robot SDK settings.
|
||
|
||
## Use cameras
|
||
|
||
### Frame access modes
|
||
|
||
All camera classes implement three access modes for capturing frames:
|
||
|
||
| Method | Behavior | Blocks? | Best For |
|
||
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | ---------------------------------------- |
|
||
| `read()` | Waits for the camera hardware to return a frame. May block for a long time depending on the camera and SDK. | Yes | Simple scripts, sequential capture |
|
||
| `async_read(timeout_ms)` | Returns the latest unconsumed frame from background thread. Blocks only if buffer is empty, up to `timeout_ms`. Raises `TimeoutError` if no frame arrives. | With a timeout | Control loops synchronized to camera FPS |
|
||
| `read_latest(max_age_ms)` | Peeks at the most recent frame in buffer (may be stale). Raises `TimeoutError` if frame is older than `max_age_ms`. | No | UI visualization, logging, monitoring |
|
||
|
||
### Usage examples
|
||
|
||
The following examples show how to use the camera API to configure and capture frames from different camera types.
|
||
|
||
- **Blocking and non-blocking frame capture** using an OpenCV-based camera
|
||
- **Color and depth capture** using an Intel RealSense camera
|
||
|
||
> [!WARNING]
|
||
> Failing to cleanly disconnect cameras can cause resource leaks. Use the context manager protocol to ensure automatic cleanup:
|
||
>
|
||
> ```python
|
||
> with OpenCVCamera(config) as camera:
|
||
> ...
|
||
> ```
|
||
>
|
||
> You can also call `connect()` and `disconnect()` manually, but always use a `finally` block for the latter.
|
||
|
||
<hfoptions id="shell_restart">
|
||
<hfoption id="Open CV Camera">
|
||
|
||
<!-- prettier-ignore-start -->
|
||
```python
|
||
from lerobot.cameras.opencv.configuration_opencv import OpenCVCameraConfig
|
||
from lerobot.cameras.opencv.camera_opencv import OpenCVCamera
|
||
from lerobot.cameras.configs import ColorMode, Cv2Rotation
|
||
|
||
# Construct an `OpenCVCameraConfig` with your desired FPS, resolution, color mode, and rotation.
|
||
config = OpenCVCameraConfig(
|
||
index_or_path=0,
|
||
fps=15,
|
||
width=1920,
|
||
height=1080,
|
||
color_mode=ColorMode.RGB,
|
||
rotation=Cv2Rotation.NO_ROTATION
|
||
)
|
||
|
||
# Instantiate and connect an `OpenCVCamera`, performing a warm-up read (default).
|
||
with OpenCVCamera(config) as camera:
|
||
|
||
# Read a frame synchronously — blocks until hardware delivers a new frame
|
||
frame = camera.read()
|
||
print(f"read() call returned frame with shape:", frame.shape)
|
||
|
||
# Read a frame asynchronously with a timeout — returns the latest unconsumed frame or waits up to timeout_ms for a new one
|
||
try:
|
||
for i in range(10):
|
||
frame = camera.async_read(timeout_ms=200)
|
||
print(f"async_read call returned frame {i} with shape:", frame.shape)
|
||
except TimeoutError as e:
|
||
print(f"No frame received within timeout: {e}")
|
||
|
||
# Instantly return a frame - returns the most recent frame captured by the camera
|
||
try:
|
||
initial_frame = camera.read_latest(max_age_ms=1000)
|
||
for i in range(10):
|
||
frame = camera.read_latest(max_age_ms=1000)
|
||
print(f"read_latest call returned frame {i} with shape:", frame.shape)
|
||
print(f"Was a new frame received by the camera? {not (initial_frame == frame).any()}")
|
||
except TimeoutError as e:
|
||
print(f"Frame too old: {e}")
|
||
|
||
```
|
||
<!-- prettier-ignore-end -->
|
||
|
||
</hfoption>
|
||
<hfoption id="Intel Realsense Camera">
|
||
|
||
<!-- prettier-ignore-start -->
|
||
```python
|
||
from lerobot.cameras.realsense.configuration_realsense import RealSenseCameraConfig
|
||
from lerobot.cameras.realsense.camera_realsense import RealSenseCamera
|
||
from lerobot.cameras.configs import ColorMode, Cv2Rotation
|
||
|
||
# Create a `RealSenseCameraConfig` specifying your camera’s serial number and enabling depth.
|
||
config = RealSenseCameraConfig(
|
||
serial_number_or_name="233522074606",
|
||
fps=15,
|
||
width=640,
|
||
height=480,
|
||
color_mode=ColorMode.RGB,
|
||
use_depth=True,
|
||
rotation=Cv2Rotation.NO_ROTATION
|
||
)
|
||
|
||
# Instantiate and connect a `RealSenseCamera` with warm-up read (default).
|
||
camera = RealSenseCamera(config)
|
||
camera.connect()
|
||
|
||
# Capture a color frame via `read()` and a depth map via `read_depth()`.
|
||
try:
|
||
color_frame = camera.read()
|
||
depth_map = camera.read_depth()
|
||
print("Color frame shape:", color_frame.shape)
|
||
print("Depth map shape:", depth_map.shape)
|
||
finally:
|
||
camera.disconnect()
|
||
```
|
||
<!-- prettier-ignore-end -->
|
||
|
||
</hfoption>
|
||
</hfoptions>
|
||
|
||
## Use your phone's camera
|
||
|
||
<hfoptions id="use phone">
|
||
<hfoption id="iPhone & macOS">
|
||
|
||
To use your iPhone as a camera on macOS, enable the Continuity Camera feature:
|
||
|
||
- Ensure your Mac is running macOS 13 or later, and your iPhone is on iOS 16 or later.
|
||
- Sign in both devices with the same Apple ID.
|
||
- Connect your devices with a USB cable or turn on Wi-Fi and Bluetooth for a wireless connection.
|
||
|
||
For more details, visit [Apple support](https://support.apple.com/en-gb/guide/mac-help/mchl77879b8a/mac).
|
||
|
||
</hfoption>
|
||
<hfoption id="OBS virtual camera">
|
||
|
||
If you want to use your phone as a camera using OBS, follow these steps to set up a virtual camera.
|
||
|
||
1. _(Linux only) Install `v4l2loopback-dkms` and `v4l-utils`_. These packages create virtual camera devices and verify their settings. Install with:
|
||
|
||
```bash
|
||
sudo apt install v4l2loopback-dkms v4l-utils
|
||
```
|
||
|
||
2. _Install the [DroidCam app](https://droidcam.app) on your phone_. This app is available for both iOS and Android.
|
||
3. _Download and install [OBS Studio](https://obsproject.com)_.
|
||
4. _Download and install the [DroidCam OBS plugin](https://droidcam.app/obs)_.
|
||
5. _Start OBS Studio_.
|
||
|
||
6. _Add your phone as a source_. Follow the instructions [here](https://droidcam.app/obs/usage). Be sure to set the resolution to `640x480` to avoid the watermarks.
|
||
7. _Adjust resolution settings_. In OBS Studio, go to `File > Settings > Video` or `OBS > Preferences... > Video`. Change the `Base(Canvas) Resolution` and the `Output(Scaled) Resolution` to `640x480` by manually typing it.
|
||
8. _Start virtual camera_. In OBS Studio, follow the instructions [here](https://obsproject.com/kb/virtual-camera-guide).
|
||
9. _Verify the virtual camera setup and resolution_.
|
||
- **Linux**: Use `v4l2-ctl` to list devices and check resolution:
|
||
```bash
|
||
v4l2-ctl --list-devices # find VirtualCam and note its /dev/videoX path
|
||
v4l2-ctl -d /dev/videoX --get-fmt-video # replace with your VirtualCam path
|
||
```
|
||
You should see `VirtualCam` listed and resolution `640x480`.
|
||
- **macOS**: Open Photo Booth or FaceTime and select "OBS Virtual Camera" as the input.
|
||
- **Windows**: The native Camera app doesn't support virtual cameras. Use a video conferencing app (Zoom, Teams) or run `lerobot-find-cameras opencv` directly to verify.
|
||
|
||
<details>
|
||
<summary><strong>Troubleshooting</strong></summary>
|
||
|
||
> The virtual camera resolution is incorrect.
|
||
|
||
Delete the virtual camera source and recreate it. The resolution cannot be changed after creation.
|
||
|
||
> Error reading frame in background thread for OpenCVCamera(X): OpenCVCamera(X) frame width=640 or height=480 do not match configured width=1920 or height=1080.
|
||
|
||
This error is caused by OBS Virtual Camera advertising a `1920x1080` resolution despite rescaling. The only fix for now is to comment out the width and height check in `_postprocess_image()`.
|
||
|
||
</details>
|
||
|
||
</hfoption>
|
||
</hfoptions>
|
||
|
||
If everything is set up correctly, your phone will appear as a standard OpenCV camera and can be used with `OpenCVCamera`.
|