https://scholars.lib.ntu.edu.tw/handle/123456789/581395
標題: | Hand pose estimation in object-interaction based on deep learning for virtual reality applications | 作者: | Wu M.-Y Ting P.-W Tang Y.-H Chou E.-T LI-CHEN FU |
關鍵字: | Convolutional neural networks; E-learning; Virtual reality; Data-driven approach; Hand pose estimations; Learning models; Object information; Object interactions; Object manipulating; Physical constraints; Training procedures; Deep learning | 公開日期: | 2020 | 卷: | 70 | 來源出版物: | Journal of Visual Communication and Image Representation | 摘要: | Hand Pose Estimation aims to predict the position of joints on a hand from an image, and it has become popular because of the emergence of VR/AR/MR technology. Nevertheless, an issue surfaces when trying to achieve this goal, since a hand tends to cause self-occlusion or external occlusion easily as it interacts with external objects. As a result, there have been many projects dedicated to this field for a better solution of this problem. This paper develops a system that accurately estimates a hand pose in 3D space using depth images for VR applications. We propose a data-driven approach of training a deep learning model for hand pose estimation with object interaction. In the convolutional neural network (CNN) training procedure, we design a skeleton-difference loss function, which effectively can learn the physical constraints of a hand. Also, we propose an object-manipulating loss function, which considers knowledge of the hand-object interaction, to enhance performance. In the experiments we have conducted for hand pose estimation under different conditions, the results validate the robustness and the performance of our system and show that our method is able to predict the joints more accurately in challenging environmental settings. Such appealing results may be attributed to the consideration of the physical joint relationship as well as object information, which in turn can be applied to future VR/AR/MR systems for more natural experience. ? 2020 Elsevier Inc. |
URI: | https://www.scopus.com/inward/record.uri?eid=2-s2.0-85085254798&doi=10.1016%2fj.jvcir.2020.102802&partnerID=40&md5=c30c36749cd1629736a1ce1aefd46f9f https://scholars.lib.ntu.edu.tw/handle/123456789/581395 |
ISSN: | 10473203 | DOI: | 10.1016/j.jvcir.2020.102802 |
顯示於: | 資訊工程學系 |
在 IR 系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。