Learning to Perceive: Perceptual Resolution Enhancement for VR Display with Efficient Neural Network Processing

Even though the Virtual Reality (VR) industry is experiencing a rapid growth with ever-expanding demands today, VR applications have yet to provide a fully immersive experience. The insufficient resolution of the VR head-mounted display (HMD) hinders the user from further immersion into the virtual world. In this work, we attempt to enhance the immersive experience by improving the perceptual resolution of VR HMDs. We employ an efficient neuralnetwork-based approach with the proposed temporal integration loss function. By taking the temporal integration mechanism of the Human Visual System (HVS) into account, our network learns the perception process of the human eye, and temporally upsamples a sequence that in turn improves its perceived resolution. Specifically, we discuss a possible scenario where we deploy our approach on a VR system equipped with the eye-tracking technology, which could save up to 75% of the computational load. Compared with the state-of-the-art in terms of the inference time analysis and a user experiment, it shows that our approach runs around 1.89× faster and produces more favorable results. © 2021 IEEE.