An Adaptive Layer Expansion Algorithm for Efficient Training of Deep Neural Networks
Journal
Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020
Pages
420-425
Date Issued
2020
Author(s)
Abstract
In this paper, we propose an adaptive layer expansion algorithm to reduce the training time of deep neural networks without noticeable loss of accuracy. Neural networks have become much larger to improve accuracy. The size of such networks makes them time-consuming to train. Hence, we propose an adaptive layer expansion algorithm that reduces training time by dynamically adding nodes where they are necessary to improve the training efficiency while not losing accuracy. We start with a smaller model of only a fraction of parameters of the original model, then train the network and add nodes to specific layers determined by the stability of gradients. The algorithm repeatedly adds nodes until it reaches a threshold and trains the model until the accuracy converges. The experiment results indicate that our algorithm only uses a quarter of computation time of a full model and achieves 64.1% accuracy on MobileNet with dataset CIFAR100, which is only 2% less than the complete model. The algorithm stops adding nodes when it has only half of the parameters of the original model. As a result, this new model provides fast inference in those environments where both the computation power and memory storage are limited, such as mobile devices. ? 2020 IEEE.
Subjects
CNN; Deep Learning; Gradient Stability; Machine Learning; Model Acceleration
Other Subjects
Big data; Deep neural networks; Digital storage; Expansion; Computation power; Computation time; Fast inference; Layer expansion; Loss of accuracy; Memory storage; Original model; Training efficiency; Multilayer neural networks
Type
conference paper