02/02/2021

Step-Ahead Error Feedback for Distributed Training with Compressed Gradient

An Xu, Zhouyuan Huo, Heng Huang

Keywords:

Abstract Paper Similar Papers

Abstract: Although the distributed machine learning methods can speed up the training of large deep neural networks, the communication cost has become the non-negligible bottleneck to constrain the performance. To address this challenge, the gradient compression based communication-efficient distributed learning methods were designed to reduce the communication cost, and more recently the local error feedback was incorporated to compensate for the corresponding performance loss. However, in this paper, we will show that a new "gradient mismatch" problem is raised by the local error feedback in centralized distributed training and can lead to degraded performance compared with full-precision training. To solve this critical problem, we propose two novel techniques, 1) step ahead and 2) error averaging, with rigorous theoretical analysis. Both our theoretical and empirical results show that our new methods can handle the "gradient mismatch" problem. The experimental results show that we can even train faster with common gradient compression schemes than both the full-precision training and local error feedback regarding the training epochs and without performance loss.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38947848
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 1
 0
 0
 0
 4:45 
 0
 0
 0
 0
 14:03 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 13:59 
 0
 0
 0
 0
 14:30 
 0
 0
 0
 0
 5:21 
 0
 0
 0
 0
 5:10 
 0
 0
 0
 0
 13:21 
 1
 0
 0
 0
 1:01 
 0
 0
 0
 0
 14:01 
 0
 0
 0
 0
 3:16 
 0
 0
 0
 0
 16:42 
 0
 0
 0
 0
 13:43 
 0
 0
 0
 0
 3:01 
 0
 0
 0
 0
 3:16 
 0
 0
 0
 0
 4:57 
 0
 0
 0
 0
 4:53 
 0
 0
 0
 0
 19:50 
 0
 0
 0
 0
 14:38 
 0
 0
 0
 0
 15:17 
 0
 0
 0
 0
 7:48 
 0
 0
 0
 0
 14:59 
 0
 0
 0
 0
 12:34 
 0
 0
 0
 0
 3:01 
 0
 0
 0
 0
 14:04 
 0
 0
 0
 0
 1:01 
 1
 0
 0
 0
 14:15 
 0
 0
 0
 1
 14:05 
 0
 0
 0
 0
 15:34 
 0
 0
 0
 0
 22:37 
 0
 0
 0
 0
 4:13 
 1
 1
 0
 1
 5:20 
 0
 0
 0
 0
 10:49 
 0
 0
 0
 0
 3:29 
 0
 0
 0
 0
 3:05 
 0
 0
 0
 0
 18:14 
 0
 0
 0
 0
 19:12 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 14:07 
 0
 0
 0
 0
 11:05