Seeing and hearing too: Audio representation for video captioning
Journal
2017 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2017 - Proceedings
Journal Volume
2018-January
Pages
381-388
Date Issued
2018
Author(s)
Abstract
Video captioning has been widely researched. Most related work takes into account only visual content in generating descriptions. However, auditory content such as human speech or environmental sounds contains rich information for describing scenes, but has yet to be widely explored for video captions. Here, we experiment with different ways to use this auditory content in videos, and demonstrate improved caption generation in terms of popular evaluation methods such as BLEU, CIDEr, and METEOR. We also measure the semantic similarities between generated captions and human-provided ground truth using sentence embeddings, and find that good use of multi-modal contents helps the machine to generate captions that are more semantically related to the ground truth. When analyzing the generated sentences, we find some ambiguous situations for which visual-only models yield incorrect results but that are resolved by approaches that take into account auditory cues.
Type
conference paper