Learning sparse neural networks through mixture-distributed regularization
Journal
IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
Journal Volume
2020-June
Pages
2968-2977
Date Issued
2020
Author(s)
Abstract
L0-norm regularization is one of the most efficient approaches to learn a sparse neural network. Due to its discrete nature, differentiable and approximate regularizations based on the concrete distribution [31] or its variants are proposed as alternatives; however, the concrete relaxation suffers from high-variance gradient estimates and is limited to its own concrete distribution. To address these issues, in this paper, we propose a more general framework for relaxing binary gates through mixture distributions. With the proposed method, any mixture pair of distributions converging to δ(0) and δ(1) can be applied to construct smoothed binary gates. We further introduce a reparameterization method for the smoothed binary gates drawn from mixture distributions to enable efficient gradient gradient-based optimization under the proposed deep learning algorithm. Extensive experiments are conducted, and the results show that the proposed approach achieves better performance in terms of pruned architectures, structured sparsity and the reduced number of floating point operations (FLOPs) as compared with other state-of-the-art sparsity-inducing methods. © 2020 IEEE.
Subjects
Computer vision; Concretes; Deep learning; Digital arithmetic; Learning algorithms; Mixtures; Floating point operations; Gradient estimates; Gradient-based optimization; Mixture distributions; Reparameterization; Sparse neural networks; State of the art; Structured sparsities; Neural networks
SDGs
Type
conference paper