https://scholars.lib.ntu.edu.tw/handle/123456789/559003
標題: | Learning sparse neural networks through mixture-distributed regularization | 作者: | Huang, C.-T. Chen, J.-C. JA-LING WU |
關鍵字: | Computer vision; Concretes; Deep learning; Digital arithmetic; Learning algorithms; Mixtures; Floating point operations; Gradient estimates; Gradient-based optimization; Mixture distributions; Reparameterization; Sparse neural networks; State of the art; Structured sparsities; Neural networks | 公開日期: | 2020 | 卷: | 2020-June | 起(迄)頁: | 2968-2977 | 來源出版物: | IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops | 摘要: | L0-norm regularization is one of the most efficient approaches to learn a sparse neural network. Due to its discrete nature, differentiable and approximate regularizations based on the concrete distribution [31] or its variants are proposed as alternatives; however, the concrete relaxation suffers from high-variance gradient estimates and is limited to its own concrete distribution. To address these issues, in this paper, we propose a more general framework for relaxing binary gates through mixture distributions. With the proposed method, any mixture pair of distributions converging to δ(0) and δ(1) can be applied to construct smoothed binary gates. We further introduce a reparameterization method for the smoothed binary gates drawn from mixture distributions to enable efficient gradient gradient-based optimization under the proposed deep learning algorithm. Extensive experiments are conducted, and the results show that the proposed approach achieves better performance in terms of pruned architectures, structured sparsity and the reduced number of floating point operations (FLOPs) as compared with other state-of-the-art sparsity-inducing methods. © 2020 IEEE. |
URI: | https://www.scopus.com/inward/record.url?eid=2-s2.0-85090110791&partnerID=40&md5=2220db97f41daa817c1a4271c4b7d5a0 https://scholars.lib.ntu.edu.tw/handle/123456789/559003 |
DOI: | 10.1109/CVPRW50498.2020.00355 |
顯示於: | 電機工程學系 |
在 IR 系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。