https://scholars.lib.ntu.edu.tw/handle/123456789/628556
標題: | Specific Expert Learning: Enriching Ensemble Diversity via Knowledge Distillation | 作者: | Kao, Wei-Cheng Xie, Hong-Xia Lin, Chih-Yang WEN-HUANG CHENG |
關鍵字: | Predictive models; Diversity reception; Task analysis; Boosting; Visualization; MIMICs; Knowledge engineering; Deep learning; ensemble diversity; knowledge distillation (KD); NETWORK | 公開日期: | 18-十一月-2021 | 出版社: | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC | 卷: | PP | 來源出版物: | IEEE transactions on cybernetics | 摘要: | In recent years, ensemble methods have shown sterling performance and gained popularity in visual tasks. However, the performance of an ensemble is limited by the paucity of diversity among the models. Thus, to enrich the diversity of the ensemble, we present the distillation approach--learning from experts (LFEs). Such method involves a novel knowledge distillation (KD) method that we present, specific expert learning (SEL), which can reduce class selectivity and improve the performance on specific weaker classes and overall accuracy. Through SEL, models can acquire different knowledge from distinct networks with various areas of expertise, and a highly diverse ensemble can be obtained afterward. Our experimental results demonstrate that, on CIFAR-10, the accuracy of the ResNet-32 increases 0.91% with SEL, and that the ensemble trained by SEL increases accuracy by 1.13%. Compared to state-of-the-art approaches, for example, DML only improves accuracy by 0.3% and 1.02% on single ResNet-32 and the ensemble, respectively. Furthermore, our proposed architecture also can be applied to ensemble distillation (ED), which applies KD on the ensemble model. In conclusion, our experimental results show that our proposed SEL not only improves the accuracy of a single classifier but also boosts the diversity of the ensemble model. |
URI: | https://scholars.lib.ntu.edu.tw/handle/123456789/628556 | ISSN: | 2168-2267 | DOI: | 10.1109/TCYB.2021.3125320 |
顯示於: | 資訊工程學系 |
在 IR 系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。