AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Journal
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Part Of
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Start Page
6890
End Page
6894
ISBN (of the container)
979-835034485-1
Date Issued
2024-04-14
Author(s)
Yuan Tseng
Layne Berry
Yi-Ting Chen
I-Hsiang Chiu
Hsuan-Hao Lin
Max Liu
Puyuan Peng
Yi-Jen Shih
Hung-Yu Wang
Haibin Wu
Po-Yao Huang
Chun-Mao Lai
Shang-Wen Li
David Harwath
Yu Tsao
Abdelrahman Mohamed
Chi-Luen Feng
Abstract
Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information. However, current models often focus on a limited set of tasks, and generalization abilities of learned representations are unclear. To this end, we propose the AV-SUPERB benchmark that enables general-purpose evaluation of unimodal audio/visual and bimodal fusion representations on 7 datasets covering 5 audio-visual tasks in speech and audio processing. We evaluate 5 recent self-supervised models and show that none of these models generalize to all tasks, emphasizing the need for future study on improving universal model performance. In addition, we show that representations may be improved with intermediate-task fine-tuning and audio event classification with AudioSet serves as a strong intermediate task. We release our benchmark with evaluation code 1 and a model submission platform 2 to encourage further research in audio-visual learning.
Event(s)
2024 49th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2024
Publisher
IEEE
Type
conference paper