Transforming Multi-concept Attention into Video Summarization
Journal
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Journal Volume
12626 LNCS
Pages
498-513
Date Issued
2021
Author(s)
Abstract
Video summarization is among the challenging tasks in computer vision, which aims at identifying highlight frames or shots over a lengthy video input. In this paper, we propose a novel attention-based framework for video summarization with complex video data. Unlike previous works which only apply attention mechanism on the correspondence between frames, our multi-concept video self-attention (MC-VSA) model is presented to identify informative regions across temporal and concept video features, which jointly exploit context diversity over time and space for summarization purposes. Together with consistency between video and summary enforced in our framework, our model can be applied to both labeled and unlabeled data, making our method preferable to real-world applications. Extensive and complete experiments on two benchmarks demonstrate the effectiveness of our model both quantitatively and qualitatively, and confirms its superiority over the state-of-the-arts. ? 2021, Springer Nature Switzerland AG.
Subjects
Video recording; Attention mechanisms; Complete experiment; Labeled and unlabeled data; Real-world; State of the art; Video data; Video features; Video summarization; Computer vision
Type
conference paper