High-throughput image analysis framework for fruit detection, localization and measurement from video streams

Abstract

© 2019 ASABE Annual International Meeting. All rights reserved. Food crises and security issues are getting worse due to climate change and population growth. One of the solutions being sought is the use of efficient breeding systems, which requires accurate and detailed phenotyping of fruits and plants. However, traditional phenotyping methods are time consuming, labor intensive and prone to human error. Therefore, measuring the morphological and physiological parameters of fruits automatically is highly recommended. In this work, a high-throughput technique for fruits detection, localization and measurement from video streams using computer vision and deep neural networks is proposed. In contrast with other works that were developed for single type of fruits, a versatile method is proposed herein that can be applied for different types of fruits using a vision system to scan through plants row by row in a greenhouse. A real-time object detection algorithm using YOLOv2, a deep neural network-based detector, is used for fruit detection and localization on video frames with a hit rate of 84.98%. An individual fruit tracking algorithm is applied throughout the video stream to perform tracking of multiple fruits. The online tracking algorithm includes feature matching, optical flow and projective transformation optimized by occlusion handling techniques such as by applying threshold indices and denoising. On the other hand, the offline tracking algorithm uses a voting method to reduce the false alarms caused by the object detector. Finally, phenotyping information such as fruit counts, ripening stage, fruit size, and 2D spatial distribution maps were obtained. The proposed framework has demonstrated its efficacy in obtaining satisfactory phenotyping information that is useful for production management as well as its potential utilization in robotic operations.