A Real-time Dynamic Hand Gesture Recognition System Based on 3D Depth Sensing and Fingertip Features

Interactions between humans and computers have long been restricted to the traditional means of keyboard and mouse. The concept, that movements from one’s fingers or hands provide new possibilities of human-computer interactions, is inspired by the gestural interface in sci-fi movie“Minority Report” and later proved to be plausible with the prevalence of multi-touch devices such as the Apple iPhone. The objective of this thesis is to develop a real time system capable of recognizing multi-touch hand gestures with a touchless interface by taking advantage of 3D sensing capabilities of Kinect, a novel yet affordable range sensor.
The system utilizes accurate 3D data and a depth-histogram in order to perform hand localization from any arbitrary background. K-means is used in 3D to determine the number of clusters representing hands found in the environment even in the occurrences of occlusions caused by hand overlaps. A variation of k-curvature extracts the location of fingertips from the hand contours. Based on the number of fingertips detected and their movements, a finite state machine is used to classify different multi-touch hand gestures performed by the user.
An evaluation of the system shows reliable accuracy of multi-touch gesture recognitions in a cluttered background under various lighting conditions while providing efficient real-time performance at 30 fps. In addition, the system offers users freedom in performing gestures since they are no longer restricted by the small sizes of the touch screen or the monitor of the device.