https://scholars.lib.ntu.edu.tw/handle/123456789/580915
標題: | Understanding self-attention of self-supervised audio transformers | 作者: | Yang S.-W Liu A.T HUNG-YI LEE |
關鍵字: | Computer applications; Computer simulation; Attention mechanisms; Audio transformers; Model performance; Multiple strategy; Ranking strategy; Refinement techniques; Speech applications; Visualization tools; Speech communication | 公開日期: | 2020 | 卷: | 2020-October | 起(迄)頁: | 3785-3789 | 來源出版物: | Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH | 摘要: | Self-supervised Audio Transformers (SAT) enable great success in many downstream speech applications like ASR, but how they work has not been widely explored yet. In this work, we present multiple strategies for the analysis of attention mechanisms in SAT. We categorize attentions into explainable categories, where we discover each category possesses its own unique functionality. We provide a visualization tool for understanding multi-head self-attention, importance ranking strategies for identifying critical attention, and attention refinement techniques to improve model performance. ? 2020 ISCA |
URI: | https://www.scopus.com/inward/record.uri?eid=2-s2.0-85098186023&doi=10.21437%2fInterspeech.2020-2231&partnerID=40&md5=bf8a6377532666204cd096a2fefe832e https://scholars.lib.ntu.edu.tw/handle/123456789/580915 |
ISSN: | 2308457X | DOI: | 10.21437/Interspeech.2020-2231 |
顯示於: | 電機工程學系 |
在 IR 系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。