Adapting Pretrained Speech Model for Mandarin Lyrics Transcription and Alignment
Journal
2023 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023
ISBN
9798350306897
Date Issued
2023-01-01
Author(s)
Abstract
The tasks of automatic lyrics transcription and lyrics alignment have witnessed significant performance improvements in the past few years. However, most of the previous works only focus on English in which large-scale datasets are available. In this paper, we address lyrics transcription and alignment of polyphonic Mandarin pop music in a low-resource setting. To deal with the data scarcity issue, we adapt pretrained Whisper model and fine-tune it on a monophonic Mandarin singing dataset. With the use of data augmentation and source separation model, results show that the proposed method achieves a character error rate of less than 18% on a Mandarin polyphonic dataset for lyrics transcription, and a mean absolute error of 0.071 seconds for lyrics alignment. Our results demonstrate the potential of adapting a pretrained speech model for lyrics transcription and alignment in low-resource scenarios.
Subjects
automatic lyrics alignment | Automatic lyrics transcription | data augmentation | model adaptation
Type
conference paper