Data Pinning and Back Propagation Memory Optimization for Deep Learning on GPU
Journal
Proceedings - 2018 6th International Symposium on Computing and Networking, CANDAR 2018
Pages
19-28
Date Issued
2018
Author(s)
Abstract
Many large deep neural network models have been proposed in recent years to achieve more accurate training results. The training of these large models requires a huge amount of memory and communication, which becomes a challenging issue in improving the performance of deep learning. In this paper, we analyze the data access pattern of training a deep neural network and propose a data pinning algorithm that reduces the data usage on GPU and the movement between a GPU and its CPU host. We show that to find an optimal data movement scheduling is NP-complete, and propose a dynamic programming that can find the optimal solution in pseudo polynomial time. That is, we observe the access pattern of the training of the deep neural network and propose specialized GPU data pinning algorithm that minimizes the unnecessary data movements. We then implement our dynamic programming and train very deep learning models. The experiments show that we can pin up to 20% more data into GPU memory than GeePS, a state of art deep learning framework. We also propose a memory reduction technique for back-propagation in deep learning. We analyze the access pattern of back propagation in deep learning and observe that gradient computation and weight update, two major steps usually done sequentially, can be partially overlapped and parallelized. In addition, we analyze the semantics of the computation and realize that by delaying the weight update we can avoid double buffering due to read/write conflicts in traditional naive parallel implementations. We then implement our techniques and observe up to 75% reduction in GPU memory usage.
SDGs
Type
conference paper