Audio Albert: A Lite Bert for Self-Supervised Learning of Audio Representation
Journal
2021 IEEE Spoken Language Technology Workshop, SLT 2021 - Proceedings
Pages
344-350
Date Issued
2021
Author(s)
Abstract
Self-supervised speech models are powerful speech representation extractors for downstream applications. Recently, larger models have been utilized in acoustic model training to achieve better performance. We propose Audio ALBERT, a lite version of the self-supervised speech representation model. We apply the lightweight representation extractor to two downstream tasks, speaker classification and phoneme classification. We show that Audio ALBERT achieves performance comparable with massive pre-trained networks in the downstream tasks while having 91% fewer parameters. Moreover, we design probing models to measure how much the latent representations can encode the speaker's and phoneme's information. We find that the representations encoded in internal layers of Audio ALBERT contain more information for both phoneme and speaker than the last layer, which is generally used for downstream tasks. Our findings provide a new avenue for using self-supervised networks to achieve better performance and efficiency. ? 2021 IEEE.
Subjects
Acoustic model trainings; Audio representation; Downstream applications; Light-weight representation; Phoneme classification; Representation model; Speaker classification; Supervised network
Type
conference paper