Noise Robust Distillation of Self-Supervised Speech Models via Correlation Metrics
Part Of
2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops, ICASSPW 2024 - Proceedings開
Journal Volume
33
Start Page
495
End Page
499
ISBN (of the container)
979-835037451-3
Date Issued
2024-04-14
Author(s)
Fabian Ritter-Gutierrez
Kuan-Po Huang
Dianwen Ng
Jeremy H.M. Wong
Eng Siong Chng
Nancy F. Chen
Abstract
Compared to large speech foundation models, small student models exhibit degraded noise robustness. The student’s robustness can be improved by introducing noise at the inputs during pre-training. Despite this, using the standard distillation loss still yields a student with degraded performance. Thus, this paper proposes improving student robustness via distillation with correlation metrics. Teacher behavior is learned by maximizing the teacher and student cross-correlation matrix between their representations towards identity. Noise robustness is encouraged via the student’s self-correlation minimization. The proposed method consistently outperforms the previous approach on Intent Classification, Keyword Spotting, and Automatic Speech Recognition tasks on SUPERB Challenge.
Event(s)
49th IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops, ICASSPW 2024
Publisher
IEEE
Type
conference paper