mirror of
https://github.com/huggingface/lerobot.git
synced 2026-06-29 06:07:40 +00:00
3dd19d043e
* feat(depth): add depth quantization helpers and tests
* feat(video): add ffv1 to supported codecs
* feat(depth): persist depth metadata
* feat(depth): extend quantization tools to better fit the encoding/decoding pipeline
* feat(depth): plumb DepthEncoderConfig through LeRobotDataset and DatasetWriter
* feat(depth): wire StreamingVideoEncoder + writer to depth encoder
* feat(depth): wire DatasetReader to decode_depth_frames
* feat(cameras/realsense): expose async depth in metric meters
* feat(features): route 2D camera shapes to observation.depth.<key>
* feat(robots/so_follower): emit + populate depth keys when use_depth
* feat(record): plumb DepthEncoderConfig through lerobot-record
* feat(viz): render depth observations as rr.DepthImage in Viridis
* feat(depth maps writer): adding support for raw depth maps recording with image writer
* chore(format): format code
* feat(depth shape): ensuring depth maps shape is always including the channel
* feat(is_depth): simplifying is_depth nested name + legacy support
* fix(stop_event): fixing stop_event race condition in camera classes
* fix(plumbing): fixing missing parts in the depth maps pipeline
* chore(typos): fixing typos
* test(fix): fixing exisiting tests to still work with latest features
* tests(depth): adding new tests for depth integration validation
* feat(pix_fmt channels): use PyAv to check get pixel formats number of channels
* feat(refactor): refactor DepthEncoderConfig quantization pipeline, so that the methods do not live in the config class. Add pixel format - channels validation.Move the default pixel format for depth in the config file.
* fix(pre-commit): fixing mutable defautl value
* fix(info): fixing info metadata update when is_depth_map was set
* tests(typos): fixing typos in tests
* fix(realsense): fixing typo in realsense serial number
* fix(normalization): restricting 255 normalization to non depth/uint8 images only
* fix(typo): fixing typo
* fix(TIFF): add missing quantization and cleanup for TIFF files
* feat(batched dequantization): optimizing dequantize_depth for torch based batched dequantization
* feat(tools): adding depth support in LeRobotDataset edition tools
* test(aggregate): extending aggregation tests to depth frames
* test(cleaning): cleaning up tests
* fix(from_video_info): fixing early validation issue in from_video_info
* fix(typo): fixing typo
* fix(is_depth): adding missing doctrings and is_depth arguments in video decoding functions
Co-authored-by: Wensi (Vince) Ai <59036629+wensi-ai@users.noreply.github.com>
* fix(depth units): fixing depth units output for the realsense cameras
* feat(output unit): adding support for output unit specification at dataset reading/training time
Co-authored-by: Wensi (Vince) Ai <59036629+wensi-ai@users.noreply.github.com>
* test(depth): cleaning up depth tests
* test(depth encoding): updating and cleaning video/depth encoding tests
* chore(format): formatting code
* docs(depth): improving depth maps docs
* test(fix): fixing depth tests
* test(dataset tools): adding missing tests for new dataset edition tools features
* chore(format): formatting code
* fix(pyav check): fixing PyAV option validation for integer codec options by normalizing
numeric values before calling `is_integer()`
Co-authored-by: Wensi (Vince) Ai <59036629+wensi-ai@users.noreply.github.com>
* docs(mermaid): fixing mermaid diagram
* fix(rebase): rebase follow up corrections
* feat(dataset tools): adding missing docstrings and features for depth fill support in dataset edition tools
* docs(docstring): updating docstrings
* docs(dataset tools): updating docs
* fix(save images): fixing image saving in dataset tools
* fix(update video info): fixing update video info logic to match the recording and editing use cases
* test(reencode): fixing reencoding monkeypatch
* fix(review): add Claude review
* chore(format): format code
* fix(update video info): ditching the differentiated approahces for video info update - video info are always updated unless for preserved keys.
* chore(rebase): fixing rebase merge conflicts
* test(visualization): fixing visualization tests
* feat(docstrings): adding explicit docstring for encoding parameters. Docstrigns will now show up as description in the CLI --help.
* feat(mm as default): adding a global DEFAULT_DEPTH_UNIT variable setting mm as default depth unit
* fix(RGB <-> camera): renaming camera_encoder to rgb_encoder for clarity
* chore(TODO): removing deprecated TODO
* doc(write_u16_plane): improving docstrings for write_u16_plane
* feat(units): adding constants for depth frames units (m and mm)
* fix(spam): replacing spamming warning but a debug log
* feat(leagcy metadata): adding automatic metadata update for legacy 'video.is_depth_map' feature
* fix(copy&reindex): fixing metadat reshaping for single channel frames
* fix(ImageNet): excluding dpeth frames from ImageNet stats
* fix(PyAV container seek): fixing initial PyAV container seek to be robust againsy codec choice
* feat(lerobot-dataset-viz): adding support for depth in lerobot-dataset-viz
* fix(compress): removing rerun compression for DepthImages
* fix(signle channel squeeze): fixing single channel squeezing
* chore(format): format code
* fix(streaming): adding support for dequantization in streaming_dataset.py
* refactor(read depth): factorizing depth reading methods for realsense camera and adding support for depth-only usage
* chore(renaming): fixing missed RGBEncoderConfig renamings
* docs(renaming): reflecting renamings in a clearer way in the docs
* chore(annotation): excluding depth from the annotation pipeline
* feat(robots): adding depth support in compatible follower robots
* feat(LeSadKiwi): excluding LeKiwi from depth support (for now)
* chore(fail): removing misplaced file
* chore(fail): removing misplaced file
* fix(remove ffv1): removing ffv1 as it does not support MP4
* docs(cheat sheet): adding depth and video encoding to the cheat sheet
* fix(lossless): tuning depth encoding parameters for lossless depth storage
* test(fix): fixing failing tests
* depth(ZMQ): excluding ZMQ from depth support
* Revert "depth(ZMQ): excluding ZMQ from depth support"
This reverts commit b95cf4e4c2.
* fix(image transforms): excluding depth frames from images transforms
* fix(typo): typo
* fix(stats): fixing stats computation for depth frames
* fix(TIFF vs. pytorch): adding an extra uint16 to float32 conversion for depth maps stored as raw TIFF images
* fix(typos): fixing typos
* test(dtype): fixing stats computation typing tests
---------
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Wensi (Vince) Ai <59036629+wensi-ai@users.noreply.github.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: Wensi Ai <wsai@stanford.edu>
239 lines
6.3 KiB
Plaintext
239 lines
6.3 KiB
Plaintext
# EarthRover Mini Plus
|
|
|
|
<img
|
|
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Earth_Rover_Mini_5_240c9adc-4f9e-44b7-982f-5d1dc24af1d8.png.webp"
|
|
alt="EarthRover Mini Plus"
|
|
width="70%"
|
|
/>
|
|
|
|
The EarthRover Mini Plus is a fully open source mobile robot that connects through the cloud using the Frodobots SDK. This lets you control the robot and record datasets for training AI models.
|
|
|
|
## What You Need
|
|
|
|
### Hardware
|
|
|
|
- EarthRover Mini robot
|
|
- Computer with Python 3.12 or newer
|
|
- Internet connection
|
|
|
|
### Setting Up the Frodobots SDK
|
|
|
|
The robot needs the [Frodobots SDK](https://github.com/frodobots-org/earth-rovers-sdk) running on your computer. Here's how:
|
|
|
|
1. Download and install the SDK:
|
|
|
|
```bash
|
|
git clone https://github.com/frodobots-org/earth-rovers-sdk.git
|
|
cd earth-rovers-sdk
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
2. Save Credentials:
|
|
|
|
Write your .env variables with the SDK API key and bot name provided by the Frodobots team.
|
|
|
|
```bash
|
|
SDK_API_TOKEN=your_sdk_api_token_here
|
|
BOT_SLUG=your_bot_slug_here
|
|
CHROME_EXECUTABLE_PATH=/path/to/chrome_or_chromium
|
|
# Default value is MAP_ZOOM_LEVEL=18 https://wiki.openstreetmap.org/wiki/Zoom_levels
|
|
MAP_ZOOM_LEVEL=18
|
|
MISSION_SLUG=your_mission_slug_here
|
|
# Image quality between 0.1 and 1.0 (default: 0.8)
|
|
# Recommended: 0.8 for better performance
|
|
IMAGE_QUALITY=0.8
|
|
# Image format: jpeg, png or webp (default: png)
|
|
# Recommended: jpeg for better performance and lower bandwidth usage
|
|
IMAGE_FORMAT=jpeg
|
|
```
|
|
|
|
3. Start the SDK:
|
|
|
|
```bash
|
|
hypercorn main:app --reload
|
|
```
|
|
|
|
4. Open your web browser and go to `http://localhost:8000`, then click "Join"
|
|
|
|
The SDK gives you:
|
|
|
|
- Live video from front and rear cameras
|
|
|
|
> [!IMPORTANT]
|
|
> The SDK must be running before you can use the robot.
|
|
|
|
## Install LeRobot
|
|
|
|
Follow our [Installation Guide](./installation) to install LeRobot.
|
|
|
|
In addition to the base installation, install the EarthRover Mini with hardware dependencies:
|
|
|
|
```bash
|
|
pip install -e ".[hardware]"
|
|
```
|
|
|
|
## How It Works
|
|
|
|
The robot uses the internet to communicate:
|
|
|
|
- **Movement commands**: Sent through the SDK
|
|
- **Camera video**: Received from the SDK
|
|
- **Robot info**: Battery, location, speed from the SDK
|
|
|
|
You don't need to plug anything in - it all works through the SDK.
|
|
|
|
## Calibration
|
|
|
|
No calibration needed! The robot is ready to use as soon as the SDK is running.
|
|
|
|
## Controlling the Robot
|
|
|
|
You control the robot using your keyboard - just like playing a video game with WASD keys.
|
|
|
|
### Keyboard Controls
|
|
|
|
| Key | Action |
|
|
| --- | -------------------------------- |
|
|
| W | Move forward |
|
|
| S | Move backward |
|
|
| A | Turn left (with forward motion) |
|
|
| D | Turn right (with forward motion) |
|
|
| Q | Rotate left in place |
|
|
| E | Rotate right in place |
|
|
| X | Stop all movement |
|
|
| +/= | Increase speed |
|
|
| - | Decrease speed |
|
|
| ESC | Disconnect |
|
|
|
|
### Speed Settings
|
|
|
|
You can adjust how fast the robot moves:
|
|
|
|
- **Forward/backward speed**: Default is full speed (1.0)
|
|
- **Turning speed**: Default is full speed (1.0)
|
|
- **Speed changes**: Use +/- keys to adjust by 0.1 each time
|
|
|
|
### Try It Out
|
|
|
|
Test driving the robot before recording data:
|
|
|
|
```python
|
|
from lerobot.robots.earthrover_mini_plus import EarthRoverMiniPlus, EarthRoverMiniPlusConfig
|
|
from lerobot.teleoperators.keyboard import KeyboardRoverTeleop, KeyboardRoverTeleopConfig
|
|
|
|
# Initialize robot
|
|
robot_config = EarthRoverMiniPlusConfig()
|
|
robot = EarthRoverMiniPlus(robot_config)
|
|
|
|
# Initialize teleoperator
|
|
teleop_config = KeyboardRoverTeleopConfig(
|
|
linear_speed=1.0,
|
|
angular_speed=1.0,
|
|
speed_increment=0.1
|
|
)
|
|
teleop = KeyboardRoverTeleop(teleop_config)
|
|
|
|
# Connect
|
|
robot.connect()
|
|
teleop.connect()
|
|
|
|
# Teleoperate (use keyboard controls)
|
|
try:
|
|
while True:
|
|
action = teleop.get_action()
|
|
robot.send_action(action)
|
|
except KeyboardInterrupt:
|
|
pass
|
|
finally:
|
|
robot.disconnect()
|
|
teleop.disconnect()
|
|
```
|
|
|
|
> [!TIP]
|
|
> If you're using a Mac, you might need to give Terminal permission to access your keyboard for teleoperation. Go to System Preferences > Security & Privacy > Input Monitoring and check the box for Terminal.
|
|
|
|
## Recording Data
|
|
|
|
Once you can drive the robot well, you can start recording data to train AI models. The system records:
|
|
|
|
- **What you do**: How you move the robot (forward, backward, turning)
|
|
- **What the robot sees**:
|
|
- Videos from both cameras
|
|
- Robot speed and direction
|
|
- Battery level and location
|
|
- GPS position and signal
|
|
- Other sensor data
|
|
- **When it happened**: Timestamps for everything
|
|
|
|
### Setting Up Hugging Face
|
|
|
|
We use Hugging Face to store your data online. First, log in with your token from [Hugging Face settings](https://huggingface.co/settings/tokens):
|
|
|
|
```bash
|
|
hf auth login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential
|
|
```
|
|
|
|
Store your Hugging Face username:
|
|
|
|
```bash
|
|
HF_USER=$(hf auth whoami | awk -F': *' 'NR==1 {print $2}')
|
|
echo $HF_USER
|
|
```
|
|
|
|
### Start Recording
|
|
|
|
Use the standard recording command:
|
|
|
|
```bash
|
|
lerobot-record \
|
|
--robot.type=earthrover_mini_plus \
|
|
--teleop.type=keyboard_rover \
|
|
--dataset.repo_id=your_username/dataset_name \
|
|
--dataset.num_episodes=2 \
|
|
--dataset.fps=10 \
|
|
--dataset.single_task="Navigate around obstacles" \
|
|
--dataset.streaming_encoding=true \
|
|
--dataset.encoder_threads=2 \
|
|
# --dataset.rgb_encoder.vcodec=auto \
|
|
--display_data=true
|
|
```
|
|
|
|
Replace `your_username/dataset_name` with your Hugging Face username and a name for your dataset.
|
|
|
|
### What Gets Saved
|
|
|
|
Your dataset includes:
|
|
|
|
**Your Actions (2 features)**:
|
|
|
|
- `linear_velocity`: How much you moved forward/backward
|
|
- `angular_velocity`: How much you turned left/right
|
|
|
|
**Robot Observations (24 features)**:
|
|
|
|
- Front camera video
|
|
- Rear camera video
|
|
- Current speed
|
|
- Battery level
|
|
- Orientation
|
|
- GPS (latitude, longitude, signal strength)
|
|
- Network signal strength
|
|
- Vibration level
|
|
- Lamp state (on/off)
|
|
- Accelerometer (x, y, z)
|
|
- Gyroscope (x, y, z)
|
|
- Magnetometer (x, y, z)
|
|
- Wheel RPMs (4 wheels)
|
|
|
|
### Where Your Data Goes
|
|
|
|
On your computer: `~/.cache/huggingface/lerobot/{repo-id}`
|
|
|
|
After recording, your data automatically uploads to your Hugging Face page:
|
|
|
|
```bash
|
|
echo https://huggingface.co/datasets/${HF_USER}/earthrover-navigation
|
|
```
|
|
|
|
Your dataset will be tagged with `LeRobot` for community discovery.
|