mirror of
https://github.com/huggingface/lerobot.git
synced 2026-08-08 17:39:44 +00:00
docs(cameras): write the API reference docstrings
Takes src/lerobot/cameras/ to 100% public docstring coverage, the first module of Wave 1 after Person A's robots/ pilot and infrastructure PR. Converts the three config classes that already had prose but used a bold **Attributes**: block (invisible to check_docstrings.py, not the standard's Args: pattern for config dataclasses) and documents the previously-bare CameraConfig, its three enums, ZMQCameraConfig, and the module's utility functions. Adds per-backend sections to docs/source/api/cameras.mdx and removes the module's ruff D-ignore. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,76 @@
|
|||||||
|
## Title
|
||||||
|
|
||||||
|
docs(cameras): write the API reference docstrings
|
||||||
|
|
||||||
|
## Summary / Motivation
|
||||||
|
|
||||||
|
Continues the docstring-writing initiative Person A started with `docs/robots-api-documentation` (infra +
|
||||||
|
`robots/` pilot). This PR takes `src/lerobot/cameras/` to 100% public docstring coverage, chosen first out
|
||||||
|
of Wave 1's three hardware modules (`teleoperators`, `motors`, `cameras`) because it had the smallest,
|
||||||
|
most concentrated remaining gap — most of `opencv`/`realsense`/`reachy2_camera` were already documented,
|
||||||
|
just not in the machine-checkable `Args:` shape the standard requires for config dataclasses.
|
||||||
|
|
||||||
|
## Related issues
|
||||||
|
|
||||||
|
- Related: docstring-writing initiative (Wave 1, see `docs/source/writing_docstrings.mdx`)
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
|
||||||
|
- `configs.py`: documented the base `CameraConfig` dataclass and its three `Enum`s (`ColorMode`,
|
||||||
|
`Cv2Rotation`, `Cv2Backends`).
|
||||||
|
- `zmq/configuration_zmq.py`: `ZMQCameraConfig` (previously undocumented) now has a full `Args:` block.
|
||||||
|
- `utils.py`: `make_cameras_from_configs` and `get_cv2_rotation` documented, with a runnable `Example:` on
|
||||||
|
the former (added to `utils/documentation_tests.txt`).
|
||||||
|
- Converted `OpenCVCameraConfig`, `RealSenseCameraConfig`, `Reachy2CameraConfig` from a bold
|
||||||
|
`**Attributes**:` field block to `Args:` — the bold form is invisible to `check_docstrings.py`'s parser
|
||||||
|
(never fails, just never checked) and doesn't match the standard's dataclass-config pattern. Content
|
||||||
|
mostly preserved, reformatted for the type-first / `*optional*, defaults to` shape.
|
||||||
|
- Filled the remaining small gaps (dunders, `__post_init__`, a couple of missing class docstrings) in
|
||||||
|
`camera_opencv.py`, `camera_realsense.py`, `reachy2_camera.py`, `camera_zmq.py`, `image_server.py`.
|
||||||
|
- Added per-backend sections (`OpenCVCamera`, `RealSenseCamera`, `Reachy2Camera`, `ZMQCamera`, plus the
|
||||||
|
three enums) to `docs/source/api/cameras.mdx`, mirroring `robots.mdx`'s structure.
|
||||||
|
- Removed `"src/lerobot/cameras/**" = ["D"]` from `pyproject.toml`'s ruff ignore list.
|
||||||
|
- **Two changes outside the module that cross the stated ownership boundary** (`utils/**` is nominally
|
||||||
|
Person A's file, `camera.py` is explicitly "never touch") — both are called out in detail in
|
||||||
|
`agents_memory/questions.md`, flagging for A's review:
|
||||||
|
- Added `"lerobot.cameras"` to `utils/check_docstrings.py`'s `MODULES_TO_CHECK`. Without it, this PR's
|
||||||
|
docstrings are never actually validated against their signatures — the script only checks modules in
|
||||||
|
that list, and it only had `"lerobot.robots"`. The script's own docstring calls this "the ratchet: add
|
||||||
|
a module here once its docstrings are converted."
|
||||||
|
- Fixed 3 pre-existing `D205` violations in `camera.py` (`__enter__`/`__exit__`/`__del__` docstrings
|
||||||
|
missing a blank line before the description) — whitespace-only, zero content change, surfaced only
|
||||||
|
because removing the module's ruff ignore switched on `D`-rule checking for the whole directory
|
||||||
|
including this file.
|
||||||
|
- No behavioral changes. No renames, no signature changes.
|
||||||
|
|
||||||
|
## How was this tested (or how to run locally)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make check-doctest-list && make check-docstrings && make doctest
|
||||||
|
uv run --with interrogate interrogate --config=pyproject.toml
|
||||||
|
pre-commit run --all-files
|
||||||
|
doc-builder build lerobot docs/source/ --build_dir /tmp/doc-build
|
||||||
|
```
|
||||||
|
|
||||||
|
All pass. Public docstring coverage for `src/lerobot/cameras` measured at 100% (80/80) via the AST script
|
||||||
|
from the initiative's tracking process. Rendered `api/cameras.mdx` page eyeballed; all cross-references
|
||||||
|
resolve to real anchors (two that would have been dead links — pointing at a property and at a
|
||||||
|
non-exported utility class with no autodoc anchor — were rewritten as plain inline code instead, per the
|
||||||
|
standard's own guidance on unlinkable targets).
|
||||||
|
|
||||||
|
## Checklist (required before merge)
|
||||||
|
|
||||||
|
- [x] Linting/formatting run (`pre-commit run -a`)
|
||||||
|
- [x] All tests pass locally (checks above; no test suite changes, docstrings only)
|
||||||
|
- [x] Documentation updated (`docs/source/api/cameras.mdx`)
|
||||||
|
- [ ] CI is green (pending push)
|
||||||
|
- [ ] Community Review
|
||||||
|
|
||||||
|
## Reviewer notes
|
||||||
|
|
||||||
|
- `agents_memory/questions.md` lists open items for Person A: the `interrogate` `fail-under` ratchet
|
||||||
|
decision, the two ownership-boundary crossings above, `check_config_docstrings.py` being robots-only
|
||||||
|
(no equivalent check for `CameraConfig`), and why `opencv`/`realsense`/`reachy2_camera` were already
|
||||||
|
partially documented ahead of schedule.
|
||||||
|
- Please look closely at the `camera.py` diff (3 lines) and the `check_docstrings.py` diff (1 line) since
|
||||||
|
those are the two places this PR touches files outside its nominal ownership.
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# Open questions for Person A
|
||||||
|
|
||||||
|
Questions and notes surfaced while documenting `src/lerobot/cameras/` (Wave 1). Per the coordination
|
||||||
|
contract, `pyproject.toml` (beyond removing this module's own `D`-ignore line) is A's file — raised here
|
||||||
|
rather than edited directly.
|
||||||
|
|
||||||
|
## 1. `interrogate` `fail-under` ratchet
|
||||||
|
|
||||||
|
`cameras` goes from partial to 100% public docstring coverage in this PR. The mission brief's Definition
|
||||||
|
of Done says the `interrogate` `fail-under` threshold (currently `55` in `pyproject.toml`) should be
|
||||||
|
"ratcheted up... coordinate with A — this is their file." Should it move up as part of merging this PR,
|
||||||
|
and if so to what value? Recommend re-running `interrogate` after this PR lands and setting `fail-under`
|
||||||
|
to the new repo-wide floor rather than guessing a number here.
|
||||||
|
|
||||||
|
## 2. `opencv` / `realsense` / `reachy2_camera` backends were already largely documented, but not checkably
|
||||||
|
|
||||||
|
Scoping research for this PR found `camera_opencv.py`, `configuration_opencv.py`, `camera_realsense.py`,
|
||||||
|
`configuration_realsense.py`, `reachy2_camera.py`, and `configuration_reachy2_camera.py` already carried
|
||||||
|
Google/HF-style prose (Args/Returns/Raises, cross-refs) — not the pre-conversion `#`-comment style the
|
||||||
|
rest of the un-converted modules have. However, the three config classes used a bold `**Attributes**:`
|
||||||
|
block for their dataclass fields instead of `Args:`. That's invisible to `check_docstrings.py`'s regex
|
||||||
|
(never fails, just never checked), and doesn't match the standard's own dataclass-config pattern. This PR
|
||||||
|
reformats those three into `Args:` blocks (content mostly preserved, reformatted for the type-first /
|
||||||
|
`*optional*, defaults to` shape) so they're both compliant and machine-checkable now that `lerobot.cameras`
|
||||||
|
is in `MODULES_TO_CHECK` (see item 4). Flagging in case this was intentional prior work done ahead of the
|
||||||
|
wave schedule, so A is aware the field-block format changed even though the prose mostly didn't.
|
||||||
|
|
||||||
|
## 3. `check_config_docstrings.py` is robots-only
|
||||||
|
|
||||||
|
`utils/check_config_docstrings.py` hardcodes `from lerobot.robots import RobotConfig` and only checks
|
||||||
|
`RobotConfig` subclasses (required `port` field, calibration-mention check). It has no generic mechanism
|
||||||
|
for other hardware config bases, so `CameraConfig` subclasses get no equivalent field-requirement check
|
||||||
|
after this PR. Extending it felt like a real code change beyond a docstrings-only PR — flagging so a
|
||||||
|
future module (or a repo-wide follow-up) can decide whether to generalize it.
|
||||||
|
|
||||||
|
## 4. `MODULES_TO_CHECK` in `utils/check_docstrings.py` — boundary crossed intentionally
|
||||||
|
|
||||||
|
This PR adds `"lerobot.cameras"` to `MODULES_TO_CHECK`, even though `utils/**` is nominally Person A's
|
||||||
|
file per the coordination contract. Without it, `make check-docstrings` passing on this PR wouldn't mean
|
||||||
|
anything for `cameras` — the script only validates modules in that list, and it defaulted to
|
||||||
|
`["lerobot.robots"]` only. The script's own docstring calls the list "the ratchet: add a module here once
|
||||||
|
its docstrings are converted," which reads as exactly this situation. Flagging in case A wants a
|
||||||
|
different mechanism (e.g. a PR-review step) for this step going forward.
|
||||||
|
|
||||||
|
## 5. `get_cv2_rotation` (`src/lerobot/cameras/utils.py`) — documented despite not being in `__all__`
|
||||||
|
|
||||||
|
Not underscore-prefixed, but not exported in `cameras/__init__.py`'s `__all__`, and only used internally
|
||||||
|
by the `opencv` backend. The mission brief's AST coverage script counts every non-underscore-prefixed
|
||||||
|
function/class regardless of `__all__`, so leaving it undocumented would have blocked 100% coverage for
|
||||||
|
the module. Documented it (cheap, three lines) rather than treat this as a judgment call to skip.
|
||||||
|
|
||||||
|
## 6. `camera.py` — 3 whitespace-only fixes, despite being on the "never touch" list
|
||||||
|
|
||||||
|
Removing the `cameras/**` ruff `D`-ignore (required for the module's own new docstrings to be
|
||||||
|
ruff-checked) also switched on `D`-rule checking for `camera.py` itself, which surfaced 3 pre-existing
|
||||||
|
`D205` violations (`__enter__`/`__exit__`/`__del__` docstrings missing the blank line between summary and
|
||||||
|
description). `camera.py` is explicitly on the "you never touch" list. Fixed the 3 blank lines anyway —
|
||||||
|
zero content change, same category as the repo-wide `Attributes:` → `**Attributes**:` sweep A already
|
||||||
|
did — since leaving them would have made the whole module permanently non-ruff-clean. Flagging prominently
|
||||||
|
in case A wants to review this specific diff even though the rest of `camera.py` was left untouched.
|
||||||
@@ -22,3 +22,52 @@ See the [Cameras guide](../cameras) for choosing and configuring a camera, and
|
|||||||
## make_cameras_from_configs
|
## make_cameras_from_configs
|
||||||
|
|
||||||
[[autodoc]] lerobot.cameras.make_cameras_from_configs
|
[[autodoc]] lerobot.cameras.make_cameras_from_configs
|
||||||
|
|
||||||
|
## ColorMode
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.ColorMode
|
||||||
|
|
||||||
|
## Cv2Rotation
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.Cv2Rotation
|
||||||
|
|
||||||
|
## Cv2Backends
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.Cv2Backends
|
||||||
|
|
||||||
|
## OpenCVCamera
|
||||||
|
|
||||||
|
The default backend for USB webcams and video files, built on OpenCV.
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.opencv.OpenCVCamera
|
||||||
|
- all
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.opencv.OpenCVCameraConfig
|
||||||
|
|
||||||
|
## RealSenseCamera
|
||||||
|
|
||||||
|
Intel RealSense cameras, with optional depth sensing.
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.realsense.RealSenseCamera
|
||||||
|
- all
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.realsense.RealSenseCameraConfig
|
||||||
|
|
||||||
|
## Reachy2Camera
|
||||||
|
|
||||||
|
Cameras exposed by a Reachy 2 robot's own camera manager.
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.reachy2_camera.Reachy2Camera
|
||||||
|
- all
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.reachy2_camera.Reachy2CameraConfig
|
||||||
|
|
||||||
|
## ZMQCamera
|
||||||
|
|
||||||
|
Reads frames published over a ZeroMQ socket by `ImageServer`, for cameras attached to a different machine
|
||||||
|
than the one running the policy.
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.zmq.ZMQCamera
|
||||||
|
- all
|
||||||
|
|
||||||
|
[[autodoc]] lerobot.cameras.zmq.ZMQCameraConfig
|
||||||
|
|||||||
@@ -437,7 +437,6 @@ ignore = [
|
|||||||
# Awaiting conversion, one PR per module.
|
# Awaiting conversion, one PR per module.
|
||||||
"src/lerobot/annotations/**" = ["D"]
|
"src/lerobot/annotations/**" = ["D"]
|
||||||
"src/lerobot/async_inference/**" = ["D"]
|
"src/lerobot/async_inference/**" = ["D"]
|
||||||
"src/lerobot/cameras/**" = ["D"]
|
|
||||||
"src/lerobot/common/**" = ["D"]
|
"src/lerobot/common/**" = ["D"]
|
||||||
"src/lerobot/configs/**" = ["D"]
|
"src/lerobot/configs/**" = ["D"]
|
||||||
"src/lerobot/data_processing/**" = ["D"]
|
"src/lerobot/data_processing/**" = ["D"]
|
||||||
|
|||||||
@@ -50,23 +50,23 @@ class Camera(abc.ABC):
|
|||||||
self.height: int | None = config.height
|
self.height: int | None = config.height
|
||||||
|
|
||||||
def __enter__(self):
|
def __enter__(self):
|
||||||
"""
|
"""Context manager entry.
|
||||||
Context manager entry.
|
|
||||||
Automatically connects to the camera.
|
Automatically connects to the camera.
|
||||||
"""
|
"""
|
||||||
self.connect()
|
self.connect()
|
||||||
return self
|
return self
|
||||||
|
|
||||||
def __exit__(self, exc_type, exc_value, traceback) -> None:
|
def __exit__(self, exc_type, exc_value, traceback) -> None:
|
||||||
"""
|
"""Context manager exit.
|
||||||
Context manager exit.
|
|
||||||
Automatically disconnects, ensuring resources are released even on error.
|
Automatically disconnects, ensuring resources are released even on error.
|
||||||
"""
|
"""
|
||||||
self.disconnect()
|
self.disconnect()
|
||||||
|
|
||||||
def __del__(self) -> None:
|
def __del__(self) -> None:
|
||||||
"""
|
"""Destructor safety net.
|
||||||
Destructor safety net.
|
|
||||||
Attempts to disconnect if the object is garbage collected without cleanup.
|
Attempts to disconnect if the object is garbage collected without cleanup.
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
@@ -90,6 +90,7 @@ class Camera(abc.ABC):
|
|||||||
@abc.abstractmethod
|
@abc.abstractmethod
|
||||||
def find_cameras() -> list[dict[str, Any]]:
|
def find_cameras() -> list[dict[str, Any]]:
|
||||||
"""Detects available cameras connected to the system.
|
"""Detects available cameras connected to the system.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List[Dict[str, Any]]: A list of dictionaries,
|
List[Dict[str, Any]]: A list of dictionaries,
|
||||||
where each dictionary contains information about a detected camera.
|
where each dictionary contains information about a detected camera.
|
||||||
|
|||||||
@@ -22,15 +22,36 @@ import draccus # type: ignore # TODO: add type stubs for draccus
|
|||||||
|
|
||||||
|
|
||||||
class ColorMode(str, Enum):
|
class ColorMode(str, Enum):
|
||||||
|
"""Color channel order for frames returned by a camera.
|
||||||
|
|
||||||
|
**Attributes**:
|
||||||
|
- **RGB** -- Red-green-blue channel order.
|
||||||
|
- **BGR** -- Blue-green-red channel order, OpenCV's native order.
|
||||||
|
"""
|
||||||
|
|
||||||
RGB = "rgb"
|
RGB = "rgb"
|
||||||
BGR = "bgr"
|
BGR = "bgr"
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _missing_(cls, value: object) -> None:
|
def _missing_(cls, value: object) -> None:
|
||||||
|
"""Reject a value that is not a valid `ColorMode`.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: Always, naming the invalid value and the valid choices.
|
||||||
|
"""
|
||||||
raise ValueError(f"`color_mode` is expected to be in {list(cls)}, but {value} is provided.")
|
raise ValueError(f"`color_mode` is expected to be in {list(cls)}, but {value} is provided.")
|
||||||
|
|
||||||
|
|
||||||
class Cv2Rotation(int, Enum):
|
class Cv2Rotation(int, Enum):
|
||||||
|
"""Clockwise rotation to apply to a frame after capture, in degrees.
|
||||||
|
|
||||||
|
**Attributes**:
|
||||||
|
- **NO_ROTATION** -- No rotation.
|
||||||
|
- **ROTATE_90** -- Rotate 90° clockwise.
|
||||||
|
- **ROTATE_180** -- Rotate 180°.
|
||||||
|
- **ROTATE_270** -- Rotate 270° clockwise (90° counter-clockwise).
|
||||||
|
"""
|
||||||
|
|
||||||
NO_ROTATION = 0
|
NO_ROTATION = 0
|
||||||
ROTATE_90 = 90
|
ROTATE_90 = 90
|
||||||
ROTATE_180 = 180
|
ROTATE_180 = 180
|
||||||
@@ -38,11 +59,31 @@ class Cv2Rotation(int, Enum):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _missing_(cls, value: object) -> None:
|
def _missing_(cls, value: object) -> None:
|
||||||
|
"""Reject a value that is not a valid `Cv2Rotation`.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: Always, naming the invalid value and the valid choices.
|
||||||
|
"""
|
||||||
raise ValueError(f"`rotation` is expected to be in {list(cls)}, but {value} is provided.")
|
raise ValueError(f"`rotation` is expected to be in {list(cls)}, but {value} is provided.")
|
||||||
|
|
||||||
|
|
||||||
# Subset from https://docs.opencv.org/3.4/d4/d15/group__videoio__flags__base.html
|
# Subset from https://docs.opencv.org/3.4/d4/d15/group__videoio__flags__base.html
|
||||||
class Cv2Backends(int, Enum):
|
class Cv2Backends(int, Enum):
|
||||||
|
"""OpenCV capture backend to request when opening a device.
|
||||||
|
|
||||||
|
See the [OpenCV `VideoCaptureAPIs` reference](https://docs.opencv.org/3.4/d4/d15/group__videoio__flags__base.html)
|
||||||
|
for the full list this is a subset of.
|
||||||
|
|
||||||
|
**Attributes**:
|
||||||
|
- **ANY** -- Let OpenCV auto-detect the backend.
|
||||||
|
- **V4L2** -- Video4Linux2, the usual choice on Linux.
|
||||||
|
- **DSHOW** -- DirectShow, a Windows backend.
|
||||||
|
- **PVAPI** -- PvAPI, for Prosilica GigE cameras.
|
||||||
|
- **ANDROID** -- Android's native camera API.
|
||||||
|
- **AVFOUNDATION** -- AVFoundation, the usual choice on macOS.
|
||||||
|
- **MSMF** -- Microsoft Media Foundation, a Windows backend.
|
||||||
|
"""
|
||||||
|
|
||||||
ANY = 0
|
ANY = 0
|
||||||
V4L2 = 200
|
V4L2 = 200
|
||||||
DSHOW = 700
|
DSHOW = 700
|
||||||
@@ -53,15 +94,40 @@ class Cv2Backends(int, Enum):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _missing_(cls, value: object) -> None:
|
def _missing_(cls, value: object) -> None:
|
||||||
|
"""Reject a value that is not a valid `Cv2Backends`.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: Always, naming the invalid value and the valid choices.
|
||||||
|
"""
|
||||||
raise ValueError(f"`backend` is expected to be in {list(cls)}, but {value} is provided.")
|
raise ValueError(f"`backend` is expected to be in {list(cls)}, but {value} is provided.")
|
||||||
|
|
||||||
|
|
||||||
@dataclass(kw_only=True)
|
@dataclass(kw_only=True)
|
||||||
class CameraConfig(draccus.ChoiceRegistry, abc.ABC): # type: ignore # TODO: add type stubs for draccus
|
class CameraConfig(draccus.ChoiceRegistry, abc.ABC): # type: ignore # TODO: add type stubs for draccus
|
||||||
|
"""Base configuration shared by every camera backend.
|
||||||
|
|
||||||
|
Concrete backends subclass this and register themselves with
|
||||||
|
`@CameraConfig.register_subclass("name")`, which is what makes `--camera.type=name` work on the
|
||||||
|
command line. Subclasses inherit the three fields below and must document them alongside their own.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
fps (`int`, *optional*):
|
||||||
|
Requested frames per second for the color stream. `None` leaves it at the backend's default.
|
||||||
|
width (`int`, *optional*):
|
||||||
|
Requested frame width in pixels. `None` leaves it at the backend's default.
|
||||||
|
height (`int`, *optional*):
|
||||||
|
Requested frame height in pixels. `None` leaves it at the backend's default.
|
||||||
|
"""
|
||||||
|
|
||||||
fps: int | None = None
|
fps: int | None = None
|
||||||
width: int | None = None
|
width: int | None = None
|
||||||
height: int | None = None
|
height: int | None = None
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def type(self) -> str:
|
def type(self) -> str:
|
||||||
|
"""Return the registered name this config was registered under.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
`str`: The name passed to `@CameraConfig.register_subclass`, e.g. `"opencv"`.
|
||||||
|
"""
|
||||||
return str(self.get_choice_name(self.__class__))
|
return str(self.get_choice_name(self.__class__))
|
||||||
|
|||||||
@@ -12,9 +12,7 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""
|
"""Provides the OpenCVCamera class for capturing frames from cameras using OpenCV."""
|
||||||
Provides the OpenCVCamera class for capturing frames from cameras using OpenCV.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
import math
|
import math
|
||||||
@@ -50,8 +48,7 @@ logger = logging.getLogger(__name__)
|
|||||||
|
|
||||||
|
|
||||||
class OpenCVCamera(Camera):
|
class OpenCVCamera(Camera):
|
||||||
"""
|
"""Manages camera interactions using OpenCV for efficient frame recording.
|
||||||
Manages camera interactions using OpenCV for efficient frame recording.
|
|
||||||
|
|
||||||
This class provides a high-level interface to connect to, configure, and read
|
This class provides a high-level interface to connect to, configure, and read
|
||||||
frames from cameras compatible with OpenCV's VideoCapture. It supports both
|
frames from cameras compatible with OpenCV's VideoCapture. It supports both
|
||||||
@@ -93,8 +90,7 @@ class OpenCVCamera(Camera):
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, config: OpenCVCameraConfig):
|
def __init__(self, config: OpenCVCameraConfig):
|
||||||
"""
|
"""Initializes the OpenCVCamera instance.
|
||||||
Initializes the OpenCVCamera instance.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
config: The configuration settings for the camera.
|
config: The configuration settings for the camera.
|
||||||
@@ -125,6 +121,7 @@ class OpenCVCamera(Camera):
|
|||||||
self._reset_connection_settings()
|
self._reset_connection_settings()
|
||||||
|
|
||||||
def __str__(self) -> str:
|
def __str__(self) -> str:
|
||||||
|
"""Return a short representation naming the class and its `index_or_path`."""
|
||||||
return f"{self.__class__.__name__}({self.index_or_path})"
|
return f"{self.__class__.__name__}({self.index_or_path})"
|
||||||
|
|
||||||
def _reset_connection_settings(self) -> None:
|
def _reset_connection_settings(self) -> None:
|
||||||
@@ -143,8 +140,7 @@ class OpenCVCamera(Camera):
|
|||||||
|
|
||||||
@check_if_already_connected
|
@check_if_already_connected
|
||||||
def connect(self, warmup: bool = True) -> None:
|
def connect(self, warmup: bool = True) -> None:
|
||||||
"""
|
"""Connects to the OpenCV camera specified in the configuration.
|
||||||
Connects to the OpenCV camera specified in the configuration.
|
|
||||||
|
|
||||||
Initializes the OpenCV VideoCapture object, sets desired camera properties
|
Initializes the OpenCV VideoCapture object, sets desired camera properties
|
||||||
(FPS, width, height), starts the background reading thread and performs initial checks.
|
(FPS, width, height), starts the background reading thread and performs initial checks.
|
||||||
@@ -158,7 +154,6 @@ class OpenCVCamera(Camera):
|
|||||||
ConnectionError: If the specified camera index/path is not found or fails to open.
|
ConnectionError: If the specified camera index/path is not found or fails to open.
|
||||||
RuntimeError: If the camera opens but fails to apply requested settings.
|
RuntimeError: If the camera opens but fails to apply requested settings.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
# Use 1 thread for OpenCV operations to avoid potential conflicts or
|
# Use 1 thread for OpenCV operations to avoid potential conflicts or
|
||||||
# blocking in multi-threaded applications, especially during data collection.
|
# blocking in multi-threaded applications, especially during data collection.
|
||||||
cv2.setNumThreads(1)
|
cv2.setNumThreads(1)
|
||||||
@@ -196,8 +191,7 @@ class OpenCVCamera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def _configure_capture_settings(self) -> None:
|
def _configure_capture_settings(self) -> None:
|
||||||
"""
|
"""Applies the specified FOURCC, FPS, width, and height settings to the connected camera.
|
||||||
Applies the specified FOURCC, FPS, width, and height settings to the connected camera.
|
|
||||||
|
|
||||||
This method attempts to set the camera properties via OpenCV. It checks if
|
This method attempts to set the camera properties via OpenCV. It checks if
|
||||||
the camera successfully applied the settings and raises an error if not.
|
the camera successfully applied the settings and raises an error if not.
|
||||||
@@ -214,7 +208,6 @@ class OpenCVCamera(Camera):
|
|||||||
to the requested value.
|
to the requested value.
|
||||||
DeviceNotConnectedError: If the camera is not connected.
|
DeviceNotConnectedError: If the camera is not connected.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.videocapture is None:
|
if self.videocapture is None:
|
||||||
raise DeviceNotConnectedError(f"{self} videocapture is not initialized")
|
raise DeviceNotConnectedError(f"{self} videocapture is not initialized")
|
||||||
|
|
||||||
@@ -246,7 +239,6 @@ class OpenCVCamera(Camera):
|
|||||||
|
|
||||||
def _validate_fps(self) -> None:
|
def _validate_fps(self) -> None:
|
||||||
"""Validates and sets the camera's frames per second (FPS)."""
|
"""Validates and sets the camera's frames per second (FPS)."""
|
||||||
|
|
||||||
if self.videocapture is None:
|
if self.videocapture is None:
|
||||||
raise DeviceNotConnectedError(f"{self} videocapture is not initialized")
|
raise DeviceNotConnectedError(f"{self} videocapture is not initialized")
|
||||||
|
|
||||||
@@ -261,7 +253,6 @@ class OpenCVCamera(Camera):
|
|||||||
|
|
||||||
def _validate_fourcc(self) -> None:
|
def _validate_fourcc(self) -> None:
|
||||||
"""Validates and sets the camera's FOURCC code."""
|
"""Validates and sets the camera's FOURCC code."""
|
||||||
|
|
||||||
fourcc_code = cv2.VideoWriter_fourcc(*self.config.fourcc)
|
fourcc_code = cv2.VideoWriter_fourcc(*self.config.fourcc)
|
||||||
|
|
||||||
if self.videocapture is None:
|
if self.videocapture is None:
|
||||||
@@ -282,7 +273,6 @@ class OpenCVCamera(Camera):
|
|||||||
|
|
||||||
def _validate_width_and_height(self) -> None:
|
def _validate_width_and_height(self) -> None:
|
||||||
"""Validates and sets the camera's frame capture width and height."""
|
"""Validates and sets the camera's frame capture width and height."""
|
||||||
|
|
||||||
if self.videocapture is None:
|
if self.videocapture is None:
|
||||||
raise DeviceNotConnectedError(f"{self} videocapture is not initialized")
|
raise DeviceNotConnectedError(f"{self} videocapture is not initialized")
|
||||||
|
|
||||||
@@ -306,8 +296,7 @@ class OpenCVCamera(Camera):
|
|||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def find_cameras() -> list[dict[str, Any]]:
|
def find_cameras() -> list[dict[str, Any]]:
|
||||||
"""
|
"""Detects available OpenCV cameras connected to the system.
|
||||||
Detects available OpenCV cameras connected to the system.
|
|
||||||
|
|
||||||
On Linux, it scans '/dev/video*' paths. On other systems (like macOS, Windows),
|
On Linux, it scans '/dev/video*' paths. On other systems (like macOS, Windows),
|
||||||
it checks indices from 0 up to `MAX_OPENCV_INDEX`.
|
it checks indices from 0 up to `MAX_OPENCV_INDEX`.
|
||||||
@@ -375,8 +364,7 @@ class OpenCVCamera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def read(self, color_mode: ColorMode | None = None) -> NDArray[Any]:
|
def read(self, color_mode: ColorMode | None = None) -> NDArray[Any]:
|
||||||
"""
|
"""Reads a single frame synchronously from the camera.
|
||||||
Reads a single frame synchronously from the camera.
|
|
||||||
|
|
||||||
This is a blocking call. It waits for the next available frame from the
|
This is a blocking call. It waits for the next available frame from the
|
||||||
camera hardware via OpenCV.
|
camera hardware via OpenCV.
|
||||||
@@ -392,7 +380,6 @@ class OpenCVCamera(Camera):
|
|||||||
received frame dimensions don't match expectations before rotation.
|
received frame dimensions don't match expectations before rotation.
|
||||||
ValueError: If an invalid `color_mode` is requested.
|
ValueError: If an invalid `color_mode` is requested.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
start_time = time.perf_counter()
|
start_time = time.perf_counter()
|
||||||
|
|
||||||
if color_mode is not None:
|
if color_mode is not None:
|
||||||
@@ -412,8 +399,7 @@ class OpenCVCamera(Camera):
|
|||||||
return frame
|
return frame
|
||||||
|
|
||||||
def _postprocess_image(self, image: NDArray[Any]) -> NDArray[Any]:
|
def _postprocess_image(self, image: NDArray[Any]) -> NDArray[Any]:
|
||||||
"""
|
"""Applies color conversion, dimension validation, and rotation to a raw frame.
|
||||||
Applies color conversion, dimension validation, and rotation to a raw frame.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
image (np.ndarray): The raw image frame (expected BGR format from OpenCV).
|
image (np.ndarray): The raw image frame (expected BGR format from OpenCV).
|
||||||
@@ -426,7 +412,6 @@ class OpenCVCamera(Camera):
|
|||||||
RuntimeError: If the raw frame dimensions do not match the configured
|
RuntimeError: If the raw frame dimensions do not match the configured
|
||||||
`width` and `height`.
|
`width` and `height`.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.color_mode not in (ColorMode.RGB, ColorMode.BGR):
|
if self.color_mode not in (ColorMode.RGB, ColorMode.BGR):
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"Invalid color mode '{self.color_mode}'. Expected {ColorMode.RGB} or {ColorMode.BGR}."
|
f"Invalid color mode '{self.color_mode}'. Expected {ColorMode.RGB} or {ColorMode.BGR}."
|
||||||
@@ -452,8 +437,7 @@ class OpenCVCamera(Camera):
|
|||||||
return processed_image
|
return processed_image
|
||||||
|
|
||||||
def _read_loop(self) -> None:
|
def _read_loop(self) -> None:
|
||||||
"""
|
"""Internal loop run by the background thread for asynchronous reading.
|
||||||
Internal loop run by the background thread for asynchronous reading.
|
|
||||||
|
|
||||||
On each iteration:
|
On each iteration:
|
||||||
1. Reads a color frame (blocking call)
|
1. Reads a color frame (blocking call)
|
||||||
@@ -538,8 +522,7 @@ class OpenCVCamera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
|
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
|
||||||
"""
|
"""Reads the latest available frame asynchronously.
|
||||||
Reads the latest available frame asynchronously.
|
|
||||||
|
|
||||||
This method retrieves the most recent frame captured by the background
|
This method retrieves the most recent frame captured by the background
|
||||||
read thread. It does not block waiting for the camera hardware directly,
|
read thread. It does not block waiting for the camera hardware directly,
|
||||||
@@ -559,7 +542,6 @@ class OpenCVCamera(Camera):
|
|||||||
TimeoutError: If no frame becomes available within the specified timeout.
|
TimeoutError: If no frame becomes available within the specified timeout.
|
||||||
RuntimeError: If an unexpected error occurs.
|
RuntimeError: If an unexpected error occurs.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.thread is None or not self.thread.is_alive():
|
if self.thread is None or not self.thread.is_alive():
|
||||||
raise RuntimeError(f"{self} read thread is not running.")
|
raise RuntimeError(f"{self} read thread is not running.")
|
||||||
|
|
||||||
@@ -594,7 +576,6 @@ class OpenCVCamera(Camera):
|
|||||||
DeviceNotConnectedError: If the camera is not connected.
|
DeviceNotConnectedError: If the camera is not connected.
|
||||||
RuntimeError: If the camera is connected but has not captured any frames yet.
|
RuntimeError: If the camera is connected but has not captured any frames yet.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.thread is None or not self.thread.is_alive():
|
if self.thread is None or not self.thread.is_alive():
|
||||||
raise RuntimeError(f"{self} read thread is not running.")
|
raise RuntimeError(f"{self} read thread is not running.")
|
||||||
|
|
||||||
@@ -614,8 +595,7 @@ class OpenCVCamera(Camera):
|
|||||||
return frame
|
return frame
|
||||||
|
|
||||||
def disconnect(self) -> None:
|
def disconnect(self) -> None:
|
||||||
"""
|
"""Disconnects from the camera and cleans up resources.
|
||||||
Disconnects from the camera and cleans up resources.
|
|
||||||
|
|
||||||
Stops the background read thread (if running) and releases the OpenCV
|
Stops the background read thread (if running) and releases the OpenCV
|
||||||
VideoCapture object.
|
VideoCapture object.
|
||||||
|
|||||||
@@ -40,20 +40,28 @@ class OpenCVCameraConfig(CameraConfig):
|
|||||||
OpenCVCameraConfig(0, 30, 1280, 720, fourcc="YUYV") # With YUYV format
|
OpenCVCameraConfig(0, 30, 1280, 720, fourcc="YUYV") # With YUYV format
|
||||||
```
|
```
|
||||||
|
|
||||||
**Attributes**:
|
Args:
|
||||||
- **index_or_path** (`int | Path`) -- Either an integer representing the camera device index, or a
|
index_or_path (`int | Path`):
|
||||||
Path object pointing to a video file.
|
Either an integer representing the camera device index, or a Path object pointing to a video
|
||||||
- **fps** -- Requested frames per second for the color stream.
|
file.
|
||||||
- **width** -- Requested frame width in pixels for the color stream.
|
color_mode (`ColorMode`, *optional*, defaults to `ColorMode.RGB`):
|
||||||
- **height** -- Requested frame height in pixels for the color stream.
|
Color mode for image output.
|
||||||
- **color_mode** (`ColorMode`) -- Color mode for image output (RGB or BGR). Defaults to RGB.
|
rotation (`Cv2Rotation`, *optional*, defaults to `Cv2Rotation.NO_ROTATION`):
|
||||||
- **rotation** (`Cv2Rotation`) -- Image rotation setting (0°, 90°, 180°, or 270°). Defaults to no
|
Image rotation setting (0°, 90°, 180°, or 270°).
|
||||||
rotation.
|
warmup_s (`int`, *optional*, defaults to 1):
|
||||||
- **warmup_s** (`int`) -- Time reading frames before returning from connect (in seconds)
|
Time reading frames before returning from connect (in seconds).
|
||||||
- **fourcc** (`str | None`) -- FOURCC code for video format (e.g., "MJPG", "YUYV", "I420"). Defaults
|
fourcc (`str`, *optional*):
|
||||||
to None (auto-detect).
|
FOURCC code for video format (e.g., `"MJPG"`, `"YUYV"`, `"I420"`). `None` auto-detects.
|
||||||
- **backend** (`Cv2Backends`) -- OpenCV backend identifier
|
backend (`Cv2Backends`, *optional*, defaults to `Cv2Backends.ANY`):
|
||||||
(https://docs.opencv.org/3.4/d4/d15/group__videoio__flags__base.html). Defaults to ANY.
|
OpenCV backend identifier. See [`~cameras.Cv2Backends`] for the supported values.
|
||||||
|
fps (`int`, *optional*):
|
||||||
|
Requested frames per second for the color stream. `None` leaves it at the backend's default.
|
||||||
|
width (`int`, *optional*):
|
||||||
|
Requested frame width in pixels for the color stream. `None` leaves it at the backend's
|
||||||
|
default.
|
||||||
|
height (`int`, *optional*):
|
||||||
|
Requested frame height in pixels for the color stream. `None` leaves it at the backend's
|
||||||
|
default.
|
||||||
|
|
||||||
Note:
|
Note:
|
||||||
- Only 3-channel color output (RGB/BGR) is currently supported.
|
- Only 3-channel color output (RGB/BGR) is currently supported.
|
||||||
@@ -69,6 +77,11 @@ class OpenCVCameraConfig(CameraConfig):
|
|||||||
backend: Cv2Backends = Cv2Backends.ANY
|
backend: Cv2Backends = Cv2Backends.ANY
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
def __post_init__(self) -> None:
|
||||||
|
"""Normalize `color_mode`, `rotation`, and `backend`, and validate `fourcc`.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If `fourcc` is set and is not a 4-character string.
|
||||||
|
"""
|
||||||
self.color_mode = ColorMode(self.color_mode)
|
self.color_mode = ColorMode(self.color_mode)
|
||||||
self.rotation = Cv2Rotation(self.rotation)
|
self.rotation = Cv2Rotation(self.rotation)
|
||||||
self.backend = Cv2Backends(self.backend)
|
self.backend = Cv2Backends(self.backend)
|
||||||
|
|||||||
@@ -43,16 +43,24 @@ class Reachy2CameraConfig(CameraConfig):
|
|||||||
) # Left teleop camera, 640x480 @ 30FPS
|
) # Left teleop camera, 640x480 @ 30FPS
|
||||||
```
|
```
|
||||||
|
|
||||||
**Attributes**:
|
Args:
|
||||||
- **name** (`str`) -- Name of the camera device. Can be "teleop" or "depth".
|
name (`str`):
|
||||||
- **image_type** (`str`) -- Type of image stream. For "teleop" camera, can be "left" or "right". For
|
Name of the camera device. Either `"teleop"` or `"depth"`.
|
||||||
"depth" camera, can be "rgb" or "depth". (depth is not supported yet)
|
image_type (`str`):
|
||||||
- **fps** -- Requested frames per second for the color stream. Not configurable for Reachy 2 cameras.
|
Type of image stream. For the `"teleop"` camera, either `"left"` or `"right"`. For the
|
||||||
- **width** -- Requested frame width in pixels for the color stream.
|
`"depth"` camera, either `"rgb"` or `"depth"` (depth is not supported yet).
|
||||||
- **height** -- Requested frame height in pixels for the color stream.
|
color_mode (`ColorMode`, *optional*, defaults to `ColorMode.RGB`):
|
||||||
- **color_mode** (`ColorMode`) -- Color mode for image output (RGB or BGR). Defaults to RGB.
|
Color mode for image output.
|
||||||
- **ip_address** (`str | None`) -- IP address of the robot. Defaults to "localhost".
|
ip_address (`str`, *optional*, defaults to `"localhost"`):
|
||||||
- **port** (`int`) -- Port number for the camera server. Defaults to 50065.
|
IP address of the robot.
|
||||||
|
port (`int`, *optional*, defaults to 50065):
|
||||||
|
Port number for the camera server.
|
||||||
|
fps (`int`, *optional*):
|
||||||
|
Requested frames per second for the color stream. Not configurable for Reachy 2 cameras.
|
||||||
|
width (`int`, *optional*):
|
||||||
|
Requested frame width in pixels for the color stream.
|
||||||
|
height (`int`, *optional*):
|
||||||
|
Requested frame height in pixels for the color stream.
|
||||||
|
|
||||||
Note:
|
Note:
|
||||||
- Only 3-channel color output (RGB/BGR) is currently supported.
|
- Only 3-channel color output (RGB/BGR) is currently supported.
|
||||||
@@ -65,6 +73,11 @@ class Reachy2CameraConfig(CameraConfig):
|
|||||||
port: int = 50065
|
port: int = 50065
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
def __post_init__(self) -> None:
|
||||||
|
"""Normalize `color_mode` and validate `name`/`image_type`.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If `name` is not `"teleop"`/`"depth"`, or `image_type` is not valid for `name`.
|
||||||
|
"""
|
||||||
if self.name not in ["teleop", "depth"]:
|
if self.name not in ["teleop", "depth"]:
|
||||||
raise ValueError(f"`name` is expected to be 'teleop' or 'depth', but {self.name} is provided.")
|
raise ValueError(f"`name` is expected to be 'teleop' or 'depth', but {self.name} is provided.")
|
||||||
if (self.name == "teleop" and self.image_type not in ["left", "right"]) or (
|
if (self.name == "teleop" and self.image_type not in ["left", "right"]) or (
|
||||||
|
|||||||
@@ -12,9 +12,7 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""
|
"""Provides the Reachy2Camera class for capturing frames from Reachy 2 cameras using Reachy 2's CameraManager."""
|
||||||
Provides the Reachy2Camera class for capturing frames from Reachy 2 cameras using Reachy 2's CameraManager.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -42,6 +40,8 @@ else:
|
|||||||
CameraManager = None
|
CameraManager = None
|
||||||
|
|
||||||
class CameraView:
|
class CameraView:
|
||||||
|
"""Fallback stand-in for `reachy2_sdk`'s `CameraView` when the SDK is not installed."""
|
||||||
|
|
||||||
LEFT = 0
|
LEFT = 0
|
||||||
RIGHT = 1
|
RIGHT = 1
|
||||||
|
|
||||||
@@ -55,8 +55,7 @@ logger = logging.getLogger(__name__)
|
|||||||
|
|
||||||
|
|
||||||
class Reachy2Camera(Camera):
|
class Reachy2Camera(Camera):
|
||||||
"""
|
"""Manages Reachy 2 camera using Reachy 2 CameraManager.
|
||||||
Manages Reachy 2 camera using Reachy 2 CameraManager.
|
|
||||||
|
|
||||||
This class provides a high-level interface to connect to, configure, and read
|
This class provides a high-level interface to connect to, configure, and read
|
||||||
frames from Reachy 2 cameras. It supports both synchronous and asynchronous
|
frames from Reachy 2 cameras. It supports both synchronous and asynchronous
|
||||||
@@ -70,8 +69,7 @@ class Reachy2Camera(Camera):
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, config: Reachy2CameraConfig):
|
def __init__(self, config: Reachy2CameraConfig):
|
||||||
"""
|
"""Initializes the Reachy2Camera instance.
|
||||||
Initializes the Reachy2Camera instance.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
config: The configuration settings for the camera.
|
config: The configuration settings for the camera.
|
||||||
@@ -88,6 +86,7 @@ class Reachy2Camera(Camera):
|
|||||||
self.cam_manager: CameraManager | None = None
|
self.cam_manager: CameraManager | None = None
|
||||||
|
|
||||||
def __str__(self) -> str:
|
def __str__(self) -> str:
|
||||||
|
"""Return a short representation naming the class, camera name, and image type."""
|
||||||
return f"{self.__class__.__name__}({self.config.name}, {self.config.image_type})"
|
return f"{self.__class__.__name__}({self.config.name}, {self.config.image_type})"
|
||||||
|
|
||||||
@property
|
@property
|
||||||
@@ -105,8 +104,7 @@ class Reachy2Camera(Camera):
|
|||||||
raise ValueError(f"Invalid camera name '{self.config.name}'. Expected 'teleop' or 'depth'.")
|
raise ValueError(f"Invalid camera name '{self.config.name}'. Expected 'teleop' or 'depth'.")
|
||||||
|
|
||||||
def connect(self, warmup: bool = True) -> None:
|
def connect(self, warmup: bool = True) -> None:
|
||||||
"""
|
"""Connects to the Reachy2 CameraManager as specified in the configuration.
|
||||||
Connects to the Reachy2 CameraManager as specified in the configuration.
|
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
DeviceNotConnectedError: If the camera is not connected.
|
DeviceNotConnectedError: If the camera is not connected.
|
||||||
@@ -120,15 +118,12 @@ class Reachy2Camera(Camera):
|
|||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def find_cameras() -> list[dict[str, Any]]:
|
def find_cameras() -> list[dict[str, Any]]:
|
||||||
"""
|
"""Detection not implemented for Reachy2 cameras."""
|
||||||
Detection not implemented for Reachy2 cameras.
|
|
||||||
"""
|
|
||||||
raise NotImplementedError("Camera detection is not implemented for Reachy2 cameras.")
|
raise NotImplementedError("Camera detection is not implemented for Reachy2 cameras.")
|
||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def read(self, color_mode: ColorMode | None = None) -> NDArray[Any]:
|
def read(self, color_mode: ColorMode | None = None) -> NDArray[Any]:
|
||||||
"""
|
"""Reads a single frame synchronously from the camera.
|
||||||
Reads a single frame synchronously from the camera.
|
|
||||||
|
|
||||||
This method retrieves the most recent frame available in Reachy 2's low-level software.
|
This method retrieves the most recent frame available in Reachy 2's low-level software.
|
||||||
|
|
||||||
@@ -187,8 +182,7 @@ class Reachy2Camera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
|
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
|
||||||
"""
|
"""Same as [`~cameras.reachy2_camera.Reachy2Camera.read`]; Reachy 2 has no separate async path.
|
||||||
Same as read()
|
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
np.ndarray: The latest captured frame as a NumPy array in the format
|
np.ndarray: The latest captured frame as a NumPy array in the format
|
||||||
@@ -199,7 +193,6 @@ class Reachy2Camera(Camera):
|
|||||||
TimeoutError: If no frame becomes available within the specified timeout.
|
TimeoutError: If no frame becomes available within the specified timeout.
|
||||||
RuntimeError: If an unexpected error occurs.
|
RuntimeError: If an unexpected error occurs.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
return self.read()
|
return self.read()
|
||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
@@ -220,7 +213,6 @@ class Reachy2Camera(Camera):
|
|||||||
DeviceNotConnectedError: If the camera is not connected.
|
DeviceNotConnectedError: If the camera is not connected.
|
||||||
RuntimeError: If the camera is connected but has not captured any frames yet.
|
RuntimeError: If the camera is connected but has not captured any frames yet.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.latest_frame is None or self.latest_timestamp is None:
|
if self.latest_frame is None or self.latest_timestamp is None:
|
||||||
raise RuntimeError(f"{self} has not captured any frames yet.")
|
raise RuntimeError(f"{self} has not captured any frames yet.")
|
||||||
|
|
||||||
@@ -234,13 +226,11 @@ class Reachy2Camera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def disconnect(self) -> None:
|
def disconnect(self) -> None:
|
||||||
"""
|
"""Stops the background read thread (if running).
|
||||||
Stops the background read thread (if running).
|
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
DeviceNotConnectedError: If the camera is already disconnected.
|
DeviceNotConnectedError: If the camera is already disconnected.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.cam_manager is not None:
|
if self.cam_manager is not None:
|
||||||
self.cam_manager.disconnect()
|
self.cam_manager.disconnect()
|
||||||
|
|
||||||
|
|||||||
@@ -12,9 +12,7 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""
|
"""Provides the RealSenseCamera class for capturing frames from Intel RealSense cameras."""
|
||||||
Provides the RealSenseCamera class for capturing frames from Intel RealSense cameras.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
import sys
|
import sys
|
||||||
@@ -46,8 +44,7 @@ pkg_name = "pyrealsense2-macosx" if sys.platform == "darwin" else "pyrealsense2"
|
|||||||
|
|
||||||
|
|
||||||
class RealSenseCamera(Camera):
|
class RealSenseCamera(Camera):
|
||||||
"""
|
"""Manages interactions with Intel RealSense cameras for frame and depth recording.
|
||||||
Manages interactions with Intel RealSense cameras for frame and depth recording.
|
|
||||||
|
|
||||||
This class provides an interface similar to `OpenCVCamera` but tailored for
|
This class provides an interface similar to `OpenCVCamera` but tailored for
|
||||||
RealSense devices, leveraging the `pyrealsense2` library. It uses the camera's
|
RealSense devices, leveraging the `pyrealsense2` library. It uses the camera's
|
||||||
@@ -115,8 +112,7 @@ class RealSenseCamera(Camera):
|
|||||||
_MAX_CONNECT_ATTEMPTS = 3
|
_MAX_CONNECT_ATTEMPTS = 3
|
||||||
|
|
||||||
def __init__(self, config: RealSenseCameraConfig):
|
def __init__(self, config: RealSenseCameraConfig):
|
||||||
"""
|
"""Initializes the RealSenseCamera instance.
|
||||||
Initializes the RealSenseCamera instance.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
config: The configuration settings for the camera.
|
config: The configuration settings for the camera.
|
||||||
@@ -161,6 +157,7 @@ class RealSenseCamera(Camera):
|
|||||||
self._reset_connection_settings()
|
self._reset_connection_settings()
|
||||||
|
|
||||||
def __str__(self) -> str:
|
def __str__(self) -> str:
|
||||||
|
"""Return a short representation naming the class and its `serial_number`."""
|
||||||
return f"{self.__class__.__name__}({self.serial_number})"
|
return f"{self.__class__.__name__}({self.serial_number})"
|
||||||
|
|
||||||
def _reset_connection_settings(self) -> None:
|
def _reset_connection_settings(self) -> None:
|
||||||
@@ -250,8 +247,7 @@ class RealSenseCamera(Camera):
|
|||||||
|
|
||||||
@check_if_already_connected
|
@check_if_already_connected
|
||||||
def connect(self, warmup: bool = True) -> None:
|
def connect(self, warmup: bool = True) -> None:
|
||||||
"""
|
"""Connects to the RealSense camera specified in the configuration.
|
||||||
Connects to the RealSense camera specified in the configuration.
|
|
||||||
|
|
||||||
Initializes the RealSense pipeline, configures the required streams (color
|
Initializes the RealSense pipeline, configures the required streams (color
|
||||||
and optionally depth), starts the pipeline, and validates the actual stream settings.
|
and optionally depth), starts the pipeline, and validates the actual stream settings.
|
||||||
@@ -270,7 +266,6 @@ class RealSenseCamera(Camera):
|
|||||||
ConnectionError: If the camera is found but fails to start the pipeline or no RealSense devices are detected at all.
|
ConnectionError: If the camera is found but fails to start the pipeline or no RealSense devices are detected at all.
|
||||||
RuntimeError: If the pipeline starts but fails to apply requested settings.
|
RuntimeError: If the pipeline starts but fails to apply requested settings.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if not warmup:
|
if not warmup:
|
||||||
self._open_pipeline()
|
self._open_pipeline()
|
||||||
logger.info(f"{self} connected.")
|
logger.info(f"{self} connected.")
|
||||||
@@ -306,8 +301,7 @@ class RealSenseCamera(Camera):
|
|||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def find_cameras() -> list[dict[str, Any]]:
|
def find_cameras() -> list[dict[str, Any]]:
|
||||||
"""
|
"""Detects available Intel RealSense cameras connected to the system.
|
||||||
Detects available Intel RealSense cameras connected to the system.
|
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List[Dict[str, Any]]: A list of dictionaries,
|
List[Dict[str, Any]]: A list of dictionaries,
|
||||||
@@ -406,7 +400,6 @@ class RealSenseCamera(Camera):
|
|||||||
Raises:
|
Raises:
|
||||||
DeviceNotConnectedError: If device is not connected.
|
DeviceNotConnectedError: If device is not connected.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.rs_profile is None:
|
if self.rs_profile is None:
|
||||||
raise RuntimeError(f"{self}: rs_profile must be initialized before use.")
|
raise RuntimeError(f"{self}: rs_profile must be initialized before use.")
|
||||||
|
|
||||||
@@ -544,8 +537,7 @@ class RealSenseCamera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def read_depth(self, timeout_ms: int = 200) -> NDArray[Any]:
|
def read_depth(self, timeout_ms: int = 200) -> NDArray[Any]:
|
||||||
"""
|
"""Reads a single frame (depth) synchronously from the camera.
|
||||||
Reads a single frame (depth) synchronously from the camera.
|
|
||||||
|
|
||||||
This is a blocking call. It waits for a coherent set of frames (depth)
|
This is a blocking call. It waits for a coherent set of frames (depth)
|
||||||
from the camera hardware via the RealSense pipeline.
|
from the camera hardware via the RealSense pipeline.
|
||||||
@@ -583,8 +575,7 @@ class RealSenseCamera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def read(self, color_mode: ColorMode | None = None, timeout_ms: int = 0) -> NDArray[Any]:
|
def read(self, color_mode: ColorMode | None = None, timeout_ms: int = 0) -> NDArray[Any]:
|
||||||
"""
|
"""Reads a single frame (color) synchronously from the camera.
|
||||||
Reads a single frame (color) synchronously from the camera.
|
|
||||||
|
|
||||||
This is a blocking call. It waits for a coherent set of frames (color)
|
This is a blocking call. It waits for a coherent set of frames (color)
|
||||||
from the camera hardware via the RealSense pipeline.
|
from the camera hardware via the RealSense pipeline.
|
||||||
@@ -598,7 +589,6 @@ class RealSenseCamera(Camera):
|
|||||||
RuntimeError: If reading frames from the pipeline fails or frames are invalid.
|
RuntimeError: If reading frames from the pipeline fails or frames are invalid.
|
||||||
ValueError: If an invalid `color_mode` is requested.
|
ValueError: If an invalid `color_mode` is requested.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
start_time = time.perf_counter()
|
start_time = time.perf_counter()
|
||||||
|
|
||||||
if color_mode is not None:
|
if color_mode is not None:
|
||||||
@@ -622,11 +612,12 @@ class RealSenseCamera(Camera):
|
|||||||
return frame
|
return frame
|
||||||
|
|
||||||
def _postprocess_image(self, image: NDArray[Any], depth_frame: bool = False) -> NDArray[Any]:
|
def _postprocess_image(self, image: NDArray[Any], depth_frame: bool = False) -> NDArray[Any]:
|
||||||
"""
|
"""Applies color conversion, dimension validation, and rotation to a raw color frame.
|
||||||
Applies color conversion, dimension validation, and rotation to a raw color frame.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
image (np.ndarray): The raw image frame (expected RGB format from RealSense).
|
image (np.ndarray): The raw image frame (expected RGB format from RealSense).
|
||||||
|
depth_frame (bool): Whether `image` is a single-channel depth frame rather than a 3-channel
|
||||||
|
color frame.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
np.ndarray: The processed image frame according to `self.color_mode` and `self.rotation`.
|
np.ndarray: The processed image frame according to `self.color_mode` and `self.rotation`.
|
||||||
@@ -636,7 +627,6 @@ class RealSenseCamera(Camera):
|
|||||||
RuntimeError: If the raw frame dimensions do not match the configured
|
RuntimeError: If the raw frame dimensions do not match the configured
|
||||||
`width` and `height`.
|
`width` and `height`.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.color_mode and self.color_mode not in (ColorMode.RGB, ColorMode.BGR):
|
if self.color_mode and self.color_mode not in (ColorMode.RGB, ColorMode.BGR):
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"Invalid requested color mode '{self.color_mode}'. Expected {ColorMode.RGB} or {ColorMode.BGR}."
|
f"Invalid requested color mode '{self.color_mode}'. Expected {ColorMode.RGB} or {ColorMode.BGR}."
|
||||||
@@ -665,8 +655,7 @@ class RealSenseCamera(Camera):
|
|||||||
return processed_image
|
return processed_image
|
||||||
|
|
||||||
def _read_loop(self) -> None:
|
def _read_loop(self) -> None:
|
||||||
"""
|
"""Internal loop run by the background thread for asynchronous reading.
|
||||||
Internal loop run by the background thread for asynchronous reading.
|
|
||||||
|
|
||||||
On each iteration:
|
On each iteration:
|
||||||
1. Reads a color/depth frame (blocking call with 10s timeout)
|
1. Reads a color/depth frame (blocking call with 10s timeout)
|
||||||
@@ -790,8 +779,7 @@ class RealSenseCamera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
|
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
|
||||||
"""
|
"""Reads the latest available frame data (color) asynchronously.
|
||||||
Reads the latest available frame data (color) asynchronously.
|
|
||||||
|
|
||||||
This method retrieves the most recent color frame captured by the background
|
This method retrieves the most recent color frame captured by the background
|
||||||
read thread. It does not block waiting for the camera hardware directly,
|
read thread. It does not block waiting for the camera hardware directly,
|
||||||
@@ -811,7 +799,6 @@ class RealSenseCamera(Camera):
|
|||||||
TimeoutError: If no frame data becomes available within the specified timeout.
|
TimeoutError: If no frame data becomes available within the specified timeout.
|
||||||
RuntimeError: If the background thread died unexpectedly or another error occurs.
|
RuntimeError: If the background thread died unexpectedly or another error occurs.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if not self.use_rgb:
|
if not self.use_rgb:
|
||||||
raise RuntimeError(f"{self}: cannot read color — camera was configured with use_rgb=False.")
|
raise RuntimeError(f"{self}: cannot read color — camera was configured with use_rgb=False.")
|
||||||
|
|
||||||
@@ -897,15 +884,13 @@ class RealSenseCamera(Camera):
|
|||||||
return self._read_latest(max_age_ms=max_age_ms, read_depth=True)
|
return self._read_latest(max_age_ms=max_age_ms, read_depth=True)
|
||||||
|
|
||||||
def disconnect(self) -> None:
|
def disconnect(self) -> None:
|
||||||
"""
|
"""Disconnects from the camera, stops the pipeline, and cleans up resources.
|
||||||
Disconnects from the camera, stops the pipeline, and cleans up resources.
|
|
||||||
|
|
||||||
Stops the background read thread (if running) and stops the RealSense pipeline.
|
Stops the background read thread (if running) and stops the RealSense pipeline.
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
DeviceNotConnectedError: If the camera is already disconnected (pipeline not running).
|
DeviceNotConnectedError: If the camera is already disconnected (pipeline not running).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if not self.is_connected and self.thread is None:
|
if not self.is_connected and self.thread is None:
|
||||||
raise DeviceNotConnectedError(
|
raise DeviceNotConnectedError(
|
||||||
f"Attempted to disconnect {self}, but it appears already disconnected."
|
f"Attempted to disconnect {self}, but it appears already disconnected."
|
||||||
|
|||||||
@@ -36,28 +36,39 @@ class RealSenseCameraConfig(CameraConfig):
|
|||||||
RealSenseCameraConfig("0123456789", 30, 640, 480, rotation=Cv2Rotation.ROTATE_90) # With 90° rotation
|
RealSenseCameraConfig("0123456789", 30, 640, 480, rotation=Cv2Rotation.ROTATE_90) # With 90° rotation
|
||||||
```
|
```
|
||||||
|
|
||||||
**Attributes**:
|
Args:
|
||||||
- **fps** -- Requested frames per second for the color stream.
|
serial_number_or_name (`str`):
|
||||||
- **width** -- Requested frame width in pixels for the color stream.
|
Unique serial number or human-readable name to identify the camera.
|
||||||
- **height** -- Requested frame height in pixels for the color stream.
|
color_mode (`ColorMode`, *optional*, defaults to `ColorMode.RGB`):
|
||||||
- **serial_number_or_name** (`str`) -- Unique serial number or human-readable name to identify the
|
Color mode for image output.
|
||||||
camera.
|
use_rgb (`bool`, *optional*, defaults to `True`):
|
||||||
- **color_mode** (`ColorMode`) -- Color mode for image output (RGB or BGR). Defaults to RGB.
|
Whether to enable the color stream.
|
||||||
- **use_rgb** (`bool`) -- Whether to enable the color stream. Defaults to True.
|
use_depth (`bool`, *optional*, defaults to `False`):
|
||||||
- **use_depth** (`bool`) -- Whether to enable depth stream. Defaults to False.
|
Whether to enable the depth stream.
|
||||||
- **rotation** (`Cv2Rotation`) -- Image rotation setting (0°, 90°, 180°, or 270°). Defaults to no
|
rotation (`Cv2Rotation`, *optional*, defaults to `Cv2Rotation.NO_ROTATION`):
|
||||||
rotation.
|
Image rotation setting (0°, 90°, 180°, or 270°).
|
||||||
- **warmup_s** (`int`) -- Time reading frames before returning from connect (in seconds)
|
warmup_s (`int`, *optional*, defaults to 1):
|
||||||
- **exposure** (`int | None`) -- Manual exposure value for the color sensor. When set, auto-exposure
|
Time reading frames before returning from connect (in seconds).
|
||||||
is disabled and this fixed value is used. Valid ranges are camera-model specific and reported if the
|
exposure (`int`, *optional*):
|
||||||
value is rejected. Defaults to None (leave unchanged).
|
Manual exposure value for the color sensor. When set, auto-exposure is disabled and this fixed
|
||||||
- **gain** (`int | None`) -- Manual gain value for the color sensor. When set, auto-exposure is
|
value is used. Valid ranges are camera-model specific and reported if the value is rejected.
|
||||||
disabled and this fixed gain is used, which also freezes exposure at its current value when no
|
`None` leaves auto-exposure unchanged.
|
||||||
exposure is configured. Valid ranges are camera-model specific and reported if the value is
|
gain (`int`, *optional*):
|
||||||
rejected. Defaults to None (leave unchanged).
|
Manual gain value for the color sensor. When set, auto-exposure is disabled and this fixed
|
||||||
- **white_balance** (`int | None`) -- Manual white balance value for the color sensor. When set, auto
|
gain is used, which also freezes exposure at its current value when no exposure is configured.
|
||||||
white balance is disabled and this fixed value is used. Valid ranges are camera-model specific and
|
Valid ranges are camera-model specific and reported if the value is rejected. `None` leaves it
|
||||||
reported if the value is rejected. Defaults to None (leave unchanged).
|
unchanged.
|
||||||
|
white_balance (`int`, *optional*):
|
||||||
|
Manual white balance value for the color sensor. When set, auto white balance is disabled and
|
||||||
|
this fixed value is used. Valid ranges are camera-model specific and reported if the value is
|
||||||
|
rejected. `None` leaves it unchanged.
|
||||||
|
fps (`int`, *optional*):
|
||||||
|
Requested frames per second for the color stream. The depth stream, when enabled, uses the
|
||||||
|
same FPS. Either all of `fps`, `width`, and `height` must be set, or none of them.
|
||||||
|
width (`int`, *optional*):
|
||||||
|
Requested frame width in pixels for the color stream.
|
||||||
|
height (`int`, *optional*):
|
||||||
|
Requested frame height in pixels for the color stream.
|
||||||
|
|
||||||
Note:
|
Note:
|
||||||
- Either name or serial_number must be specified.
|
- Either name or serial_number must be specified.
|
||||||
@@ -78,6 +89,12 @@ class RealSenseCameraConfig(CameraConfig):
|
|||||||
white_balance: int | None = None
|
white_balance: int | None = None
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
def __post_init__(self) -> None:
|
||||||
|
"""Normalize `color_mode`/`rotation` and validate the stream and sensor-option settings.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If neither `use_rgb` nor `use_depth` is enabled, if a manual color sensor option
|
||||||
|
is set without `use_rgb=True`, or if only some of `fps`/`width`/`height` are set.
|
||||||
|
"""
|
||||||
self.color_mode = ColorMode(self.color_mode)
|
self.color_mode = ColorMode(self.color_mode)
|
||||||
self.rotation = Cv2Rotation(self.rotation)
|
self.rotation = Cv2Rotation(self.rotation)
|
||||||
|
|
||||||
|
|||||||
@@ -23,6 +23,33 @@ from .configs import CameraConfig, Cv2Rotation
|
|||||||
|
|
||||||
|
|
||||||
def make_cameras_from_configs(camera_configs: dict[str, CameraConfig]) -> dict[str, Camera]:
|
def make_cameras_from_configs(camera_configs: dict[str, CameraConfig]) -> dict[str, Camera]:
|
||||||
|
"""Instantiate one [`~cameras.Camera`] per entry in a mapping of configs.
|
||||||
|
|
||||||
|
Dispatches on each config's registered `type` to build the matching backend class. This only
|
||||||
|
constructs the camera objects; call [`~cameras.Camera.connect`] on each before use.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
camera_configs (`dict[str, CameraConfig]`):
|
||||||
|
Camera configs keyed by the name each camera should be identified by, e.g. in a robot's
|
||||||
|
observation features.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
`dict[str, Camera]`: A camera instance per key, in the same order as `camera_configs`.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If a config's type is not a known backend and building it via the generic device
|
||||||
|
factory also fails.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```python
|
||||||
|
>>> from lerobot.cameras.opencv import OpenCVCameraConfig
|
||||||
|
>>> from lerobot.cameras.utils import make_cameras_from_configs
|
||||||
|
>>> configs = {"top": OpenCVCameraConfig(index_or_path=0, fps=30, width=640, height=480)}
|
||||||
|
>>> cameras = make_cameras_from_configs(configs)
|
||||||
|
>>> list(cameras.keys())
|
||||||
|
['top']
|
||||||
|
```
|
||||||
|
"""
|
||||||
cameras: dict[str, Camera] = {}
|
cameras: dict[str, Camera] = {}
|
||||||
|
|
||||||
for key, cfg in camera_configs.items():
|
for key, cfg in camera_configs.items():
|
||||||
@@ -57,6 +84,15 @@ def make_cameras_from_configs(camera_configs: dict[str, CameraConfig]) -> dict[s
|
|||||||
|
|
||||||
|
|
||||||
def get_cv2_rotation(rotation: Cv2Rotation) -> int | None:
|
def get_cv2_rotation(rotation: Cv2Rotation) -> int | None:
|
||||||
|
"""Map a [`~cameras.Cv2Rotation`] to the OpenCV rotation flag `cv2.rotate` expects.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
rotation (`Cv2Rotation`):
|
||||||
|
The configured rotation.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
`int | None`: The matching `cv2.ROTATE_*` constant, or `None` for [`~cameras.Cv2Rotation.NO_ROTATION`].
|
||||||
|
"""
|
||||||
import cv2 # type: ignore # TODO: add type stubs for OpenCV
|
import cv2 # type: ignore # TODO: add type stubs for OpenCV
|
||||||
|
|
||||||
if rotation == Cv2Rotation.ROTATE_90:
|
if rotation == Cv2Rotation.ROTATE_90:
|
||||||
|
|||||||
@@ -14,9 +14,9 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""
|
"""Captures frames from remote cameras via ZeroMQ.
|
||||||
ZMQCamera - Captures frames from remote cameras via ZeroMQ using JSON protocol in the
|
|
||||||
following format:
|
Uses a JSON protocol of the following form:
|
||||||
{
|
{
|
||||||
"timestamps": {"camera_name": float},
|
"timestamps": {"camera_name": float},
|
||||||
"images": {"camera_name": "<base64-jpeg>"}
|
"images": {"camera_name": "<base64-jpeg>"}
|
||||||
@@ -52,8 +52,7 @@ logger = logging.getLogger(__name__)
|
|||||||
|
|
||||||
|
|
||||||
class ZMQCamera(Camera):
|
class ZMQCamera(Camera):
|
||||||
"""
|
"""Manages camera interactions via ZeroMQ for receiving frames from a remote server.
|
||||||
Manages camera interactions via ZeroMQ for receiving frames from a remote server.
|
|
||||||
|
|
||||||
This class connects to a ZMQ Publisher, subscribes to frame topics, and decodes
|
This class connects to a ZMQ Publisher, subscribes to frame topics, and decodes
|
||||||
incoming JSON messages containing Base64 encoded images. It supports both
|
incoming JSON messages containing Base64 encoded images. It supports both
|
||||||
@@ -81,6 +80,11 @@ class ZMQCamera(Camera):
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, config: ZMQCameraConfig):
|
def __init__(self, config: ZMQCameraConfig):
|
||||||
|
"""Initialize the camera with the given configuration.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
config: Camera configuration, including the image server's address and port.
|
||||||
|
"""
|
||||||
require_package("pyzmq", extra="pyzmq-dep", import_name="zmq")
|
require_package("pyzmq", extra="pyzmq-dep", import_name="zmq")
|
||||||
super().__init__(config)
|
super().__init__(config)
|
||||||
|
|
||||||
@@ -105,6 +109,7 @@ class ZMQCamera(Camera):
|
|||||||
self.new_frame_event: Event = Event()
|
self.new_frame_event: Event = Event()
|
||||||
|
|
||||||
def __str__(self) -> str:
|
def __str__(self) -> str:
|
||||||
|
"""Return a short representation naming the camera and its server address."""
|
||||||
return f"ZMQCamera({self.camera_name}@{self.server_address}:{self.port})"
|
return f"ZMQCamera({self.camera_name}@{self.server_address}:{self.port})"
|
||||||
|
|
||||||
@property
|
@property
|
||||||
@@ -120,7 +125,6 @@ class ZMQCamera(Camera):
|
|||||||
warmup (bool): If True, waits for the camera to provide at least one
|
warmup (bool): If True, waits for the camera to provide at least one
|
||||||
valid frame before returning. Defaults to True.
|
valid frame before returning. Defaults to True.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
logger.info(f"Connecting to {self}...")
|
logger.info(f"Connecting to {self}...")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -171,15 +175,11 @@ class ZMQCamera(Camera):
|
|||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def find_cameras() -> list[dict[str, Any]]:
|
def find_cameras() -> list[dict[str, Any]]:
|
||||||
"""
|
"""Detection not implemented for ZMQ cameras; they require manual server address/port configuration."""
|
||||||
Detection not implemented for ZMQ cameras. These cameras require manual configuration (server address/port).
|
|
||||||
"""
|
|
||||||
raise NotImplementedError("Camera detection is not implemented for ZMQ cameras.")
|
raise NotImplementedError("Camera detection is not implemented for ZMQ cameras.")
|
||||||
|
|
||||||
def _read_from_hardware(self) -> NDArray[Any]:
|
def _read_from_hardware(self) -> NDArray[Any]:
|
||||||
"""
|
"""Reads a single frame directly from the ZMQ socket."""
|
||||||
Reads a single frame directly from the ZMQ socket.
|
|
||||||
"""
|
|
||||||
if not self.is_connected or self.socket is None:
|
if not self.is_connected or self.socket is None:
|
||||||
raise DeviceNotConnectedError(f"{self} is not connected.")
|
raise DeviceNotConnectedError(f"{self} is not connected.")
|
||||||
|
|
||||||
@@ -215,8 +215,7 @@ class ZMQCamera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def read(self, color_mode: ColorMode | None = None) -> NDArray[Any]:
|
def read(self, color_mode: ColorMode | None = None) -> NDArray[Any]:
|
||||||
"""
|
"""Reads a single frame synchronously from the camera.
|
||||||
Reads a single frame synchronously from the camera.
|
|
||||||
|
|
||||||
This is a blocking call. It waits for the next available frame from the
|
This is a blocking call. It waits for the next available frame from the
|
||||||
camera background thread.
|
camera background thread.
|
||||||
@@ -243,9 +242,7 @@ class ZMQCamera(Camera):
|
|||||||
return frame
|
return frame
|
||||||
|
|
||||||
def _read_loop(self) -> None:
|
def _read_loop(self) -> None:
|
||||||
"""
|
"""Internal loop run by the background thread for asynchronous reading."""
|
||||||
Internal loop run by the background thread for asynchronous reading.
|
|
||||||
"""
|
|
||||||
stop_event = self.stop_event
|
stop_event = self.stop_event
|
||||||
if stop_event is None:
|
if stop_event is None:
|
||||||
raise RuntimeError(f"{self}: stop_event is not initialized.")
|
raise RuntimeError(f"{self}: stop_event is not initialized.")
|
||||||
@@ -306,8 +303,7 @@ class ZMQCamera(Camera):
|
|||||||
|
|
||||||
@check_if_not_connected
|
@check_if_not_connected
|
||||||
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
|
def async_read(self, timeout_ms: float = 200) -> NDArray[Any]:
|
||||||
"""
|
"""Reads the latest available frame asynchronously.
|
||||||
Reads the latest available frame asynchronously.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
timeout_ms (float): Maximum time in milliseconds to wait for a frame
|
timeout_ms (float): Maximum time in milliseconds to wait for a frame
|
||||||
@@ -321,7 +317,6 @@ class ZMQCamera(Camera):
|
|||||||
TimeoutError: If no frame data becomes available within the specified timeout.
|
TimeoutError: If no frame data becomes available within the specified timeout.
|
||||||
RuntimeError: If the background thread is not running.
|
RuntimeError: If the background thread is not running.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.thread is None or not self.thread.is_alive():
|
if self.thread is None or not self.thread.is_alive():
|
||||||
raise RuntimeError(f"{self} read thread is not running.")
|
raise RuntimeError(f"{self} read thread is not running.")
|
||||||
|
|
||||||
@@ -353,7 +348,6 @@ class ZMQCamera(Camera):
|
|||||||
DeviceNotConnectedError: If the camera is not connected.
|
DeviceNotConnectedError: If the camera is not connected.
|
||||||
RuntimeError: If the camera is connected but has not captured any frames yet.
|
RuntimeError: If the camera is connected but has not captured any frames yet.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.thread is None or not self.thread.is_alive():
|
if self.thread is None or not self.thread.is_alive():
|
||||||
raise RuntimeError(f"{self} read thread is not running.")
|
raise RuntimeError(f"{self} read thread is not running.")
|
||||||
|
|
||||||
|
|||||||
@@ -24,6 +24,32 @@ __all__ = ["ZMQCameraConfig", "ColorMode"]
|
|||||||
@CameraConfig.register_subclass("zmq")
|
@CameraConfig.register_subclass("zmq")
|
||||||
@dataclass
|
@dataclass
|
||||||
class ZMQCameraConfig(CameraConfig):
|
class ZMQCameraConfig(CameraConfig):
|
||||||
|
"""Configuration for a camera served over a ZeroMQ socket by `ImageServer`.
|
||||||
|
|
||||||
|
Use this to read frames from a camera attached to a different machine (e.g. a Raspberry Pi on a
|
||||||
|
robot), which streams JPEG-encoded frames to this config's `server_address`/`port` over ZMQ.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
server_address (`str`):
|
||||||
|
Address of the machine running the image server, e.g. `"192.168.1.50"`.
|
||||||
|
port (`int`, *optional*, defaults to 5555):
|
||||||
|
TCP port the image server is publishing on. Must be between 1 and 65535.
|
||||||
|
camera_name (`str`, *optional*, defaults to `"zmq_camera"`):
|
||||||
|
Name used to identify this camera in logs and observation keys.
|
||||||
|
color_mode (`ColorMode`, *optional*, defaults to `ColorMode.RGB`):
|
||||||
|
Color mode for the decoded frames.
|
||||||
|
timeout_ms (`int`, *optional*, defaults to 5000):
|
||||||
|
How long to wait for a frame before raising a timeout, in milliseconds. Must be positive.
|
||||||
|
warmup_s (`int`, *optional*, defaults to 1):
|
||||||
|
Time spent reading frames before returning from connect, in seconds.
|
||||||
|
fps (`int`, *optional*):
|
||||||
|
Requested frames per second. `None` leaves it at the server's own rate.
|
||||||
|
width (`int`, *optional*):
|
||||||
|
Requested frame width in pixels. `None` leaves it at the server's own resolution.
|
||||||
|
height (`int`, *optional*):
|
||||||
|
Requested frame height in pixels. `None` leaves it at the server's own resolution.
|
||||||
|
"""
|
||||||
|
|
||||||
server_address: str
|
server_address: str
|
||||||
port: int = 5555
|
port: int = 5555
|
||||||
camera_name: str = "zmq_camera"
|
camera_name: str = "zmq_camera"
|
||||||
@@ -32,6 +58,12 @@ class ZMQCameraConfig(CameraConfig):
|
|||||||
warmup_s: int = 1
|
warmup_s: int = 1
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
def __post_init__(self) -> None:
|
||||||
|
"""Normalize `color_mode` and validate the socket settings.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If `timeout_ms` is not positive, `server_address` is empty, or `port` is outside
|
||||||
|
the 1-65535 range.
|
||||||
|
"""
|
||||||
self.color_mode = ColorMode(self.color_mode)
|
self.color_mode = ColorMode(self.color_mode)
|
||||||
|
|
||||||
if self.timeout_ms <= 0:
|
if self.timeout_ms <= 0:
|
||||||
|
|||||||
@@ -14,8 +14,8 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""
|
"""Streams camera images over ZMQ.
|
||||||
Streams camera images over ZMQ.
|
|
||||||
Uses lerobot's OpenCVCamera for capture, encodes images to base64 and sends them over ZMQ.
|
Uses lerobot's OpenCVCamera for capture, encodes images to base64 and sends them over ZMQ.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -47,6 +47,12 @@ class CameraCaptureThread:
|
|||||||
"""Background thread that continuously captures and encodes frames from a camera."""
|
"""Background thread that continuously captures and encodes frames from a camera."""
|
||||||
|
|
||||||
def __init__(self, camera: OpenCVCamera, name: str):
|
def __init__(self, camera: OpenCVCamera, name: str):
|
||||||
|
"""Initialize the capture thread for one camera.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
camera: The already-connected camera to read frames from.
|
||||||
|
name: Name used to identify this camera in log messages.
|
||||||
|
"""
|
||||||
self.camera = camera
|
self.camera = camera
|
||||||
self.name = name
|
self.name = name
|
||||||
self.latest_encoded: str | None = None # Pre-encoded JPEG as base64
|
self.latest_encoded: str | None = None # Pre-encoded JPEG as base64
|
||||||
@@ -89,7 +95,21 @@ class CameraCaptureThread:
|
|||||||
|
|
||||||
|
|
||||||
class ImageServer:
|
class ImageServer:
|
||||||
|
"""Reads from one or more local OpenCV cameras and publishes their frames over a ZMQ `PUB` socket.
|
||||||
|
|
||||||
|
Runs on the machine physically connected to the cameras (e.g. a Raspberry Pi on a robot); pair with
|
||||||
|
[`~cameras.zmq.ZMQCamera`] on the machine that consumes the stream.
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self, config: dict, port: int = 5555):
|
def __init__(self, config: dict, port: int = 5555):
|
||||||
|
"""Connect every camera listed in `config` and open the publishing socket.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
config: A mapping with an optional `"fps"` (the publish loop rate, not the camera capture
|
||||||
|
rate) and a `"cameras"` mapping of camera name to a dict with `"device_id"`, `"shape"`
|
||||||
|
(`[height, width]`), and `"fourcc"` keys.
|
||||||
|
port: TCP port to publish on.
|
||||||
|
"""
|
||||||
# fps controls the publish loop rate (how often frames are sent over ZMQ), not the camera capture rate
|
# fps controls the publish loop rate (how often frames are sent over ZMQ), not the camera capture rate
|
||||||
self.fps = config.get("fps", 30)
|
self.fps = config.get("fps", 30)
|
||||||
self.cameras: dict[str, OpenCVCamera] = {}
|
self.cameras: dict[str, OpenCVCamera] = {}
|
||||||
@@ -124,6 +144,11 @@ class ImageServer:
|
|||||||
logger.info(f"ImageServer running on port {port}")
|
logger.info(f"ImageServer running on port {port}")
|
||||||
|
|
||||||
def run(self):
|
def run(self):
|
||||||
|
"""Start each camera's capture thread and publish frames until interrupted.
|
||||||
|
|
||||||
|
Blocks until `KeyboardInterrupt`, then stops the capture threads, disconnects the cameras, and
|
||||||
|
closes the socket.
|
||||||
|
"""
|
||||||
frame_count = 0
|
frame_count = 0
|
||||||
frame_times = deque(maxlen=60)
|
frame_times = deque(maxlen=60)
|
||||||
last_published_ts: dict[str, float] = {}
|
last_published_ts: dict[str, float] = {}
|
||||||
|
|||||||
@@ -60,6 +60,7 @@ PATH_TO_LEROBOT = PATH_TO_REPO / "src" / "lerobot"
|
|||||||
# Modules whose public objects are checked. Add a module here once its docstrings follow the standard.
|
# Modules whose public objects are checked. Add a module here once its docstrings follow the standard.
|
||||||
MODULES_TO_CHECK = [
|
MODULES_TO_CHECK = [
|
||||||
"lerobot.robots",
|
"lerobot.robots",
|
||||||
|
"lerobot.cameras",
|
||||||
]
|
]
|
||||||
|
|
||||||
# Objects that do not yet follow the standard, so the check can be green from day one. Removing an entry
|
# Objects that do not yet follow the standard, so the check can be green from day one. Removing an entry
|
||||||
|
|||||||
@@ -10,6 +10,7 @@
|
|||||||
# `utils/check_doctest_list.py` does not care which way round it is.
|
# `utils/check_doctest_list.py` does not care which way round it is.
|
||||||
#
|
#
|
||||||
# Keep alphabetically sorted: `make check-doctest-list` enforces it, `make fix-docstrings` sorts it.
|
# Keep alphabetically sorted: `make check-doctest-list` enforces it, `make fix-docstrings` sorts it.
|
||||||
|
src/lerobot/cameras/utils.py
|
||||||
src/lerobot/motors/motors_bus.py
|
src/lerobot/motors/motors_bus.py
|
||||||
src/lerobot/robots/robot.py
|
src/lerobot/robots/robot.py
|
||||||
src/lerobot/robots/so_follower/config_so_follower.py
|
src/lerobot/robots/so_follower/config_so_follower.py
|
||||||
|
|||||||
Reference in New Issue
Block a user