Towards ℓ1Regularization for Deep Neural Networks: Model Sparsity Versus Task Difficulty
Journal
Proceedings - 2022 IEEE 9th International Conference on Data Science and Advanced Analytics, DSAA 2022
ISBN
9781665473309
Date Issued
2022-01-01
Author(s)
Abstract
Nowadays, numerous AI systems employ deep neural network models with excessive number of parameters to obtain superior performance on real-world applications. Such systems relies on high-performance GPUs to achieve real-time inference, especially for the ones that involve convolutional operations. This prohibits the practical deployment on resourcescarce edge devices. To reduce model size and inference time, model pruning for deep neural network models has been an active thread of research in the recent years. Among a significant amount of literature towards different model pruning strategies, ℓ1 regularization is commonly considered as a simple solution for sparse models. However, a fundamental issue is seldom addressed: "why ℓ1 regularization can be effective for model pruning". In this work, we provide a theoretical explanation showing that, for a specific type of neural network models with ℓ1 regularization, tasks of higher accuracy result in higher pruning ratio. Hence, network pruning could be exceptionally effective on high accuracy tasks. Based on the theoretical analysis, we demonstrate the effectiveness of ℓ1 regularization learning on two iconic computer vision tasks: (a) face detection and (b) image segmentation. The experiment results show that, with suitable ℓ1-regularized optimizations, even for a compact model like MobileNetV2, the model size can be reduced by an order of magnitude without significant losses on the accuracy. Furthermore, we show that with proper implementation of sparse convolution, the obtained sparse neural network models can achieve multiple times speed-ups not only in FLOPs, but also in actual inference time.
Subjects
deep learning | deep neural networks | machine learning | regularization | sparse training
Type
conference paper