https://scholars.lib.ntu.edu.tw/handle/123456789/581395
Title: | Hand pose estimation in object-interaction based on deep learning for virtual reality applications | Authors: | Wu M.-Y Ting P.-W Tang Y.-H Chou E.-T LI-CHEN FU |
Keywords: | Convolutional neural networks; E-learning; Virtual reality; Data-driven approach; Hand pose estimations; Learning models; Object information; Object interactions; Object manipulating; Physical constraints; Training procedures; Deep learning | Issue Date: | 2020 | Journal Volume: | 70 | Source: | Journal of Visual Communication and Image Representation | Abstract: | Hand Pose Estimation aims to predict the position of joints on a hand from an image, and it has become popular because of the emergence of VR/AR/MR technology. Nevertheless, an issue surfaces when trying to achieve this goal, since a hand tends to cause self-occlusion or external occlusion easily as it interacts with external objects. As a result, there have been many projects dedicated to this field for a better solution of this problem. This paper develops a system that accurately estimates a hand pose in 3D space using depth images for VR applications. We propose a data-driven approach of training a deep learning model for hand pose estimation with object interaction. In the convolutional neural network (CNN) training procedure, we design a skeleton-difference loss function, which effectively can learn the physical constraints of a hand. Also, we propose an object-manipulating loss function, which considers knowledge of the hand-object interaction, to enhance performance. In the experiments we have conducted for hand pose estimation under different conditions, the results validate the robustness and the performance of our system and show that our method is able to predict the joints more accurately in challenging environmental settings. Such appealing results may be attributed to the consideration of the physical joint relationship as well as object information, which in turn can be applied to future VR/AR/MR systems for more natural experience. ? 2020 Elsevier Inc. |
URI: | https://www.scopus.com/inward/record.uri?eid=2-s2.0-85085254798&doi=10.1016%2fj.jvcir.2020.102802&partnerID=40&md5=c30c36749cd1629736a1ce1aefd46f9f https://scholars.lib.ntu.edu.tw/handle/123456789/581395 |
ISSN: | 10473203 | DOI: | 10.1016/j.jvcir.2020.102802 |
Appears in Collections: | 資訊工程學系 |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.