Modular AR Framework for Vision-Language Tasks
Journal
ACM International Conference Proceeding Series
Pages
16-21
Date Issued
2020
Author(s)
Abstract
Mixed / augmented reality systems have become more and more sophisticated in recent years. However, they still lack any ability to reason about the surrounding world. On the other hand, computer vision research has made many advancements towards a more human-like reasoning process. This paper aims to bridge these 2 research areas by implementing a modular framework which interconnects an AR application with a deep learning based vision model. Finally, a few potential use cases of the proposed system are showcased. The developed framework allows the application to utilize a variety of Vision-Language (V+L) models, to gain additional understanding about the surrounding environment. The system is designed to be modular and expandable. It is able to connect any number of Python processes of the V+L models to Unity apps using AR technology. The system was evaluated in our university's smart home lab based on daily life use cases. With a further extension of the framework by additional downstream tasks provided by V+L models and other computer vision systems, this framework should find wider adoption in AR applications. The increasing ability of applications to comprehend visual common sense and natural conversations would enable more intuitive interactions with the user, who could perceive his device more as a virtual assistant and companion. ? 2020 ACM.
Event(s)
4th International Conference on Artificial Intelligence and Virtual Reality, AIVR 2020
Subjects
Automation; Deep learning; Virtual reality; AR application; Computer vision system; Intuitive interaction; Modular framework; Reality systems; Reasoning process; Surrounding environment; Virtual assistants; Computer vision
Type
conference paper