An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
Journal
2021 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021 - Proceedings
Pages
228-235
Date Issued
2021
Author(s)
Abstract
Self-supervised pretraining on speech data has achieved a lot of progress. High-fidelity representation of the speech signal is learned from a lot of untranscribed data and shows promising performance. Recently, there are several works focusing on evaluating the quality of self-supervised pretrained representations on various tasks with-out domain restriction, e.g. SUPERB. However, such evaluations do not provide a comprehensive comparison among many ASR benchmark corpora. In this paper, we focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models. We select sev-eral pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR. Without any modification of the back-end model archi-tectures or training strategy, some of the experiments with pretrained representations, e.g., WSJ, WSJ0-2mix with HuBERT, reach or out-perform current state-of-the-art (SOTA) recognition performance. Moreover, we further explore more scenarios for whether the pre-training representations are effective, such as the cross-language or overlapped speech. The scripts, configuratons and the trained mod-els have been released in ESPnet to let the community reproduce our experiments and improve them. © 2021 IEEE.
Subjects
End-to-End Speech Recognition; ESPnet; Representation Learning
SDGs
Other Subjects
Speech recognition; Comprehensive comparisons; End to end; End-to-end speech recognition; Espnet; High-fidelity; Performance; Pre-training; Representation learning; Speech data; Speech signals; Speech
Type
conference paper