Communication Scheduling Optimization for Distributed Deep Learning Systems
Journal
Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS
Journal Volume
2018-December
Pages
739-746
Date Issued
2019
Author(s)
Abstract
Deep learning is an increasingly important technique that can solve complex problems. Due to the growth of data and model complexity, large-scale deep learning has became an important issue. Distributed deep learning is an efficient way to address these complexity issues in training a huge model. However, in a distributed environment network bandwidth becomes a performance bottleneck for deep learning. We propose various optimizations in reducing network usage by scheduling network request events properly, so as to reduce the total training time. These scheduling optimization only requires software innovation and without the need to upgrade physical network bandwidth, thus is economically competitive. The experiments indicate that our scheduler achieves up to 25 % speedup over traditional schedulers.
SDGs
Type
conference paper