05/04/2021

Pipelined Backpropagation at Scale: Training Large Models without Batches

Atli Kosson, Vitaliy Chiley, Abhi Venigalla, Joel Hestness, Urs Koster

Keywords:

Abstract Paper Similar Papers

Abstract: New hardware can substantially increase the speed and efficiency of deep neural network training. To guide the development of future hardware architectures, it is pertinent to explore the hardware and machine learning properties of alternative training algorithms. In this work we evaluate the use of small batch, fine-grained Pipelined Backpropagation, an asynchronous pipeline parallel training algorithm that has significant hardware advantages. We introduce two methods, Spike Compensation and Linear Weight Prediction, that effectively mitigate the downsides caused by the asynchronicity of Pipelined Backpropagation and outperform existing techniques in our setting. We show that appropriate normalization and small batch sizes can also aid training. With our methods, fine-grained Pipelined Backpropagation using a batch size of one can match the accuracy of SGD for multiple networks trained on CIFAR-10 and ImageNet. Simple scaling rules allow the use of existing hyperparameters for traditional training without additional tuning.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38952698
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at MLSYS 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 18:00 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 5:07 
 0
 0
 0
 0
 16:57 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 4:46 
 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 3:25 
 0
 0
 0
 0
 18:14 
 0
 0
 0
 0
 16:56 
 0
 0
 0
 0
 3:09 
 0
 0
 0
 0
 5:06 
 0
 0
 0
 0
 14:26 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 4:12 
 0
 0
 0
 0
 15:17 
 0
 0
 0
 0
 13:28 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 3:54 
 0
 0
 0
 0
 13:17 
 0
 0
 0
 0
 18:54 
 0
 0
 0
 0
 5:14 
 0
 0
 0
 0
 11:38 
 0
 0
 0
 0
 15:34 
 0
 0
 0
 0
 4:43 
 0
 0
 0
 0
 3:12 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 14:03 
 0
 0
 0
 0
 5:21 
 0
 0
 0
 0
 3:01 
 0
 0
 0
 0
 4:36 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 5:10 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 11:47 
 0
 0
 0
 0
 8:49 
 0
 0
 0
 0
 14:45 
 1
 0
 0
 0
 4:45 
 0
 0
 0
 0
 14:09