Remote surveillance video activity recognition using spatiotemporal convolutional neural networks for greenhouse workload analysis
Journal
American Society of Agricultural and Biological Engineers Annual International Meeting, ASABE 2021
Journal Volume
3
Pages
1632-1639
Date Issued
2021
Author(s)
Abstract
With the advances of computer vision over the recent years, its applications have extended from handling static images to tackling the challenging domain of video analysis, with human activity recognition being among the core research topics. However, recognizing and recording the activity of agricultural workers through surveillance videos is still understudied. By workload efficiency analysis, the quality of agricultural produce can be determined and improved. In this work, an IoT system that utilizes MQTT messaging protocol to stream visual data collected from wireless monitoring nodes was developed. The video streams were displayed in a mobile APP for farm owners to remotely monitor their workplace and crop situation. A spatiotemporal convolutional network was trained to recognize the actions of the workers from the video streams, while the timestamp and duration were recorded in a remote server. Through vision-based analysis, workload data can be seamlessly extracted from the streaming footage, thereby eliminating the need for additional hardware setup, such as wearable sensors. The proposed algorithm was capable of achieving an average F1 score of 96.9%. The workload data analysis results could be beneficial for farm owners in reevaluating and improving their management strategy. Accurate detection and recording of farm activities will allow them to see if a particular activity had been performed appropriately and at the proper time. Furthermore, the usage of resources, including pesticide and water, can be tracked based on the frequency and duration of the activities. The proposed techniques and methods can be employed not only for agricultural applications, but also for other workplace monitoring applications. ? ASABE 2021 Annual International Meeting
Subjects
Activity recognition
Computer vision
Deep learning
Greenhouse management
Spatiotemporal convolutional neural network
Agricultural robots
Agriculture
Convolution
Convolutional neural networks
Data streams
Pattern recognition
Security systems
Video streaming
Agricultural workers
Convolutional networks
Human activity recognition
Management strategies
Messaging protocols
Monitoring applications
Remote surveillance
Wireless monitoring
Quality control
Type
conference paper