Abstract
The strawberry tourist farm faces two problems. The first problem is that visitors do not necessarily choose fully ripe strawberries. This is because fully ripe strawberries are easily damaged during transportation, and customers usually see less ripe strawberries in stores. As a result, visitors miss the opportunity to taste the most delicious strawberries, and farmers end up discarding strawberries. The second problem is that visitors sometimes damage the branches due to improper picking methods.
Therefore, we propose an augmented reality (AR) strawberry-picking support system called BerryScope to solve these two problems. Visitors wear an optical see-through Head-Mounted Display (HMD) and pick strawberries. Based on the images captured by the HMD camera, the system uses Deep Neural Network (DNN) technology to determine the ripeness of strawberries and overlay harvesting instructions for fully ripe strawberries in AR. To address the second problem, the system provides pre-harvest training using AR and points out incorrect picking methods during harvesting. For evaluation, we experimented on an actual strawberry farm with 15 participants aged 20 to 65 who had no strawberry-picking experience. As a result, we confirmed that using the proposed system enables users to select riper strawberries.
Figure 1: Strawberry Harvesting using the AR-based Support System BerryScore
Proposed Method
The proposed strawberry-picking support system has the following features:
- Determine the ripeness of strawberries from images captured by the HMD camera.
- Overlay harvesting instructions for ripe strawberries.
- Provide strawberry picking practice using virtual objects.
- Detect and point out incorrect picking methods.
This system uses the Microsoft HoloLens2 as an optical see-through HMD. Figure 2 shows the process flow until the harvesting instructions for ripe strawberries are displayed. First, the strawberries are captured using the HoloLens2, and the images are transferred to the server. On the server, the instance segmentation model QueryInst is used to detect strawberries and extract strawberry parts from the images. Next, ResNet18 is used as a classifier to determine the ripeness of the strawberries. Based on the results, the location information of the strawberries determined to be ripe is sent from the server to the HoloLens2. Then, the distance to the strawberries is measured using the HoloLens2 depth sensor. Combining the distance and the strawberry location information sent from the server, a bounding box for the strawberries is displayed in the 3D space.
Publication
S.Tamura, P. Buayai, X. Mao, “BerryScope: AR Strawberry Picking Aid,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Sydney, Australia, 2023pp.118-121, doi: 10.1109/ISMAR-Adjunct60411.2023.00032.