STR-GQN: Scene Representation and Rendering for Unknown Cameras Based on Spatial Transformation Routing
Journal
Proceedings of the IEEE International Conference on Computer Vision
Pages
5946-5955
Date Issued
2021
Author(s)
Abstract
Geometry-aware modules are widely applied in recent deep learning architectures for scene representation and rendering. However, these modules require intrinsic camera information that might not be obtained accurately. In this paper, we propose a Spatial Transformation Routing (STR) mechanism to model the spatial properties without applying any geometric prior. The STR mechanism treats the spatial transformation as the message passing process, and the relation between the view poses and the routing weights is modeled by an end-to-end trainable neural network. Besides, an Occupancy Concept Mapping (OCM) framework is proposed to provide explainable rationals for scene-fusion processes. We conducted experiments on several datasets and show that the proposed STR mechanism improves the performance of the Generative Query Network (GQN). The visualization results reveal that the routing process can pass the observed information from one location of some view to the associated location in the other view, which demonstrates the advantage of the proposed model in terms of spatial cognition. © 2021 IEEE
SDGs
Other Subjects
Cameras; Deep learning; Message passing; Camera information; Camera-based; Learning architectures; Message passing process; Query networks; Routing mechanism; Routings; Scene representation; Spatial properties; Spatial transformation; Rendering (computer graphics)
Type
conference paper