05/04/2021

Wavelet: Efficient DNN Training with Tick-Tock Scheduling

Guanhua Wang, Kehan Wang, Kenan Jiang, XIANGJUN LI, Ion Stoica

Keywords:

Abstract Paper Similar Papers

Abstract: DNNs have revolutionized across a wide range of applications, such as image classification, speech recognition and robotics control. As DNN models become more computationally expensive to train, parallel execution with multiple accelerators (e.g. GPUs) is adopted. System efficiency is a big issue when scaling out. However, as computation power increases, GPUs are under-utilized mainly due to limited local memory size. To address this memory bound, we present Wavelet, an efficient and generic approach that can fully utilize all the available on-device memory among GPUs involved in the distributed training job. Wavelet achieves near optimal on-device memory usage by adopting a simple scheduling scheme called Tick-Tock, which interleaves waves of peak memory usage among the accelerators. Evaluations on a variety of DNN models and tasks show that, Wavelet trains models up to 6.7x faster than commonly used parallelism techniques.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38952768
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at MLSYS 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:22 
 0
 0
 0
 0
 16:30 
 0
 0
 1
 0
 3:20 
 0
 0
 0
 0
 16:56 
 0
 0
 1
 0
 3:23 
 0
 0
 0
 0
 15:30 
 0
 0
 0
 0
 19:05 
 0
 0
 0
 0
 4:46 
 0
 0
 0
 0
 20:09 
 0
 0
 0
 0
 6:36 
 0
 0
 0
 0
 4:12 
 0
 0
 0
 0
 5:37 
 0
 0
 0
 0
 3:01 
 0
 0
 0
 0
 3:33 
 0
 0
 0
 0
 20:07 
 0
 0
 0
 0
 16:57 
 0
 0
 0
 0
 5:49 
 0
 0
 0
 0
 11:11 
 0
 0
 0
 0
 15:56 
 0
 1
 0
 1
 16:56 
 0
 0
 0
 0
 23:04 
 0
 0
 0
 0
 21:33 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 11:23 
 0
 0
 0
 0
 19:36 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 5:31 
 0
 0
 0
 0
 20:29 
 0
 0
 1
 0
 20:14 
 0
 0
 0
 0
 5:29 
 0
 0
 0
 0
 19:58 
 0
 0
 0
 0
 3:12 
 0
 0
 0
 0
 15:51 
 0
 0
 0
 0
 14:43 
 0
 0
 0
 0
 4:14 
 0
 0
 0
 0
 18:00 
 0
 0
 0
 0
 21:54 
 0
 0
 0
 0
 5:03 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 15:09