diff --git a/docs/source/reachy2.mdx b/docs/source/reachy2.mdx index 85c4100ab..12196ef37 100644 --- a/docs/source/reachy2.mdx +++ b/docs/source/reachy2.mdx @@ -1,8 +1,10 @@ # Reachy 2 -## Prerequisites +## Setup -- Check the services images on your robot are meeting the minimal versions: +### Prerequisites + +- On your robot, check the services images are meeting the minimal versions: - **reachy2-core >= 1.7.5.2** - **webrtc >= 2.0.1.1** @@ -12,9 +14,9 @@ The teleoperation version must be **>=v1.2.0** Currently, the teleoperation is done through Pollen Robotics teleoperation, which is not included in LeRobot. We will work on teleoperators directly included soon! -We advise to have one computer running the teleoperation, while another one is running the record session with LeRobot. +We advise to have one computer running the teleoperation, while another one is running the recording session with LeRobot. -## Install LeRobot +### Install LeRobot Follow the [installation instructions](https://github.com/huggingface/lerobot#installation) to install LeRobot. @@ -24,20 +26,31 @@ Install LeRobot with Reachy 2 dependencies (reachy2-sdk is required): pip install -e ".[reachy2]" ``` -## Reachy 2 IP address +### Get Reachy 2 IP address Before starting teleoperation and data recording, you need to find the [robot IP address](https://docs.pollen-robotics.com/getting-started/setup-reachy2/connect-reachy2/). We strongly advise to get all your devices (computer, robot) plugged through ethernet cables. +### (Optional but recommended) Install pollen_data_acquisition_server -## Recording +The installation of the data acquisition server depends on the way you decide to manage your recording sessions for Reachy 2. +The easier way in currently to use this server, so that you can control the session directly from the VR teleoperation application. + +In your LeRobot environment, install the server from source: +```bash +git clone https://github.com/pollen-robotics/pollen_data_acquisition_server.git +cd pollen_data_acquisition_server +pip install -e . +``` + +## Step 1: Recording There are two ways of managing recording sessions using the teleoeperation application for Reachy 2: - using the LeRobot record script: LeRobot is in charge of the session management, and decides when to start and stop an episode. The teleoperation application is only used to control the robot's movements. - using the included data acquisition server for Reachy 2 (recommended): the teleoperation application does manage the session management, requesting LeRobot to start or stop an episode, in addition to controlling the robot's movements. -### Option 1: Teleoperation and lerobot.record +### Option 1: Using lerobot.record Reachy 2 is supported as the other robots by the record features. @@ -46,6 +59,7 @@ Here is an example of record, without mobile base: python -m lerobot.record \ --robot.type=reachy2 \ --robot.ip_address=192.168.0.200 \ + --robot.id=r2-0000 \ --robot.use_external_commands=true \ --robot.with_mobile_base=false \ --teleop.type=reachy2_fake_teleoperator \ @@ -63,18 +77,48 @@ python -m lerobot.record \ Then get into the teleoperation application and choose "Standard session". -### Option 2: Using data acquisition server (recommended) +### Option 2: Using Pollen data acquisition server (recommended) + +Make sure you have installed pollen_data_acquisition_server, as explained in the Setup section. Launch the data acquisition server to be able to manage your session directly from the teleoperation application: ```bash -python -m lerobot.data_acquisition_server \ +python -m data_acquisition_server.launch --display_data=true +``` + +Then get into the teleoperation application and choose "Data acquisition session". +You can then setup your session by following the screens displayed. + +### Choose which parts to use! + +From our first tests, we saw that recording all the joints with some policies while only some of them are moving does reduce the quality of the models. +That is why you can choose which parts to record / replay using the arguments `--robot.with_{part}=false`, +with part being one of : "mobile_base", "l_arm", "r_arm", "neck", "antennas". +By default, all parts are recorded. + +You can do the same for the cameras, where only the teleop cameras are recording by default. + +An extended setup for Reachy 2 would look like: +```bash +python -m lerobot.record \ --robot.type=reachy2 \ --robot.ip_address=192.168.0.200 \ --robot.use_external_commands=true \ - --robot.with_mobile_base=false \ + --robot.with_mobile_base=true \ + --robot.with_l_arm=true \ + --robot.with_r_arm=true \ + --robot.with_neck=true \ + --robot.with_antennas=true \ + --robot.with_left_teleop_camera=true \ + --robot.with_right_teleop_camera=true \ + --robot.with_torso_camera=true \ --teleop.type=reachy2_fake_teleoperator \ --teleop.ip_address=192.168.0.200 \ --teleop.with_mobile_base=false \ + --teleop.with_l_arm=true \ + --teleop.with_r_arm=true \ + --teleop.with_neck=true \ + --teleop.with_antennas=true \ --dataset.repo_id=pollen_robotics/record_test \ --dataset.single_task="Reachy 2 recording test" \ --dataset.num_episodes=1 \ @@ -85,13 +129,43 @@ python -m lerobot.data_acquisition_server \ --display_data=true ``` -### Choose your arguments! +## Step 2: Replay -From our first tests, we saw that recording all the joints with some policies while only some of them are moving does reduce the quality of the models. -So choose with parts to record / replay, using the arguments `--robot.with_{part}=false`, -with part being one of : "mobile_base", "l_arm", "r_arm", "neck", "antennas". -By default, all parts are recorded. +Make sure the robot is configured with the same parts as the dataset: -You can do the same for the cameras, where only the teleop cameras are recording by default. +```bash +python -m lerobot.replay \ + --robot.type=reachy2 \ + --robot.ip_address=192.168.0.200 \ + --robot.use_external_commands=false \ + --robot.with_mobile_base=false \ + --dataset.repo_id=pollen_robotics/record_test \ + --dataset.episode=0 + --display_data=true +``` -## Step 3: Record, Replay, Train +## Step 3: Train + +```bash +python -m lerobot.scripts.train \ + --dataset.repo_id=pollen_robotics/record_test \ + --policy.type=act \ + --output_dir=outputs/train/reachy2_test \ + --job_name=hopejr \ + --policy.device=mps \ + --wandb.enable=true \ + --policy.repo_id=pollen_robotics/record_test_policy +``` + +### Step 4: Evaluate + +```bash +python -m lerobot.record \ + --robot.type=reachy2 \ + --robot.ip_address=192.168.0.200 \ + --display_data=false \ + --dataset.repo_id=pollen_robotics/eval_record_test \ + --dataset.single_task="Evaluate reachy2 policy" \ + --dataset.num_episodes=10 \ + --policy.path=outputs/train/reachy2_test/checkpoints/last/pretrained_model +```