05/04/2021

PipeMare: Asynchronous Pipeline Parallel DNN Training

Bowen Yang, Jian Zhang, Jonathan Li, Christopher Re, Christopher Aberger, Christopher De Sa

Keywords:

Abstract Paper Similar Papers

Abstract: Pipeline parallelism when training neural networks enables models to be partitioned spatially, which can lead to overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing pipeline parallel training techniques sacrifice hardware efficiency by decreasing pipeline utilization or incurring extra memory costs. In this paper, we investigate to what extent these sacrifices will be necessary on the emerging class of new dataflow hardware accelerators. We devise PipeMare, a simple yet robust training method that tolerates asynchronous updates during pipeline parallel execution without sacrificing utilization or memory, which allows efficient use of fine-grained pipeline parallelism. Concretely, when tested on ResNet and Transformer networks, asynchrony enables PipeMare to use up to 2.7x less memory or get 14.3x higher pipeline utilization, with similar model quality, when compared to state-of-the-art synchronous pipeline parallel training techniques.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38952763
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at MLSYS 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 4:14 
 0
 0
 0
 0
 18:00 
 1
 0
 0
 0
 4:45 
 0
 0
 0
 0
 5:22 
 0
 0
 0
 0
 14:39 
 0
 0
 0
 0
 5:06 
 0
 0
 0
 0
 14:26 
 0
 0
 0
 0
 10:41 
 1
 0
 0
 0
 1:01 
 0
 0
 0
 0
 8:30 
 0
 0
 0
 0
 4:19 
 0
 0
 0
 0
 8:48 
 0
 0
 0
 0
 4:36 
 0
 0
 0
 0
 2:58 
 0
 0
 0
 0
 3:06 
 0
 0
 0
 0
 15:18 
 0
 0
 0
 1
 5:39 
 0
 0
 0
 0
 14:30 
 0
 0
 0
 0
 5:07 
 0
 0
 0
 0
 9:33 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 3:20 
 0
 0
 0
 0
 16:42 
 0
 0
 0
 0
 3:20 
 0
 0
 0
 0
 4:57 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 16:56 
 0
 0
 0
 0
 8:31 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 14:48 
 0
 0
 0
 0
 5:13 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 3:12 
 0
 0
 0
 0
 14:41 
 0
 0
 0
 0
 5:24 
 0
 0
 1
 0
 3:23 
 0
 0
 0
 0
 15:30 
 0
 0
 0
 0
 14:09 
 0
 0
 0
 0
 13:28 
 0
 0
 0
 0
 5:21