05/04/2021

Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters

Shaohuai Shi, Xianhao Zhou, Shutao Song, Xingyao Wang, Zilin Zhu, Xue Huang, Xinan Jiang, Feihu Zhou, Zhenyu Guo, Liqiang Xie, Rui Lan, Xianbin Ouyang, Yan Zhang, Jieqian Wei, Jing Gong, Weiliang Lin, Ping Gao, Peng Meng, Xiaomin Xu, Chenyang Guo, Bo Yang, Zhibo Chen, Yongjian Wu, Xiaowen Chu

Keywords:

Abstract Paper Similar Papers

Abstract: Distributed training techniques have been widely deployed in large-scale deep models training on dense-GPU clusters. However, on public cloud clusters, due to the moderate inter-connection bandwidth between instances, traditional state-of-the-art distributed training systems cannot scale well in training large-scale models. In this paper, we propose a new computing and communication efficient top-k sparsification communication library for distributed training. To further improve the system scalability, we optimize I/O by proposing a simple yet efficient multi-level data caching mechanism and optimize the update operation by introducing a novel parallel tensor operator. Experimental results on a 16-node Tencent Cloud cluster (each node with 8 Nvidia Tesla V100 GPUs) show that our system achieves 25%-40% faster than existing state-of-the-art systems on CNNs and Transformer. We finally break the record on DAWNBench on training ResNet-50 to 93% top-5 accuracy on ImageNet.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38952751
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at MLSYS 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:02 
 0
 0
 0
 0
 4:12 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 19:36 
 0
 0
 0
 0
 16:56 
 0
 0
 0
 0
 8:48 
 0
 0
 0
 0
 16:30 
 0
 0
 0
 0
 19:56 
 0
 0
 0
 0
 3:16 
 0
 0
 0
 0
 20:09 
 0
 0
 0
 0
 4:46 
 0
 0
 0
 0
 19:08 
 0
 0
 0
 0
 15:30 
 0
 0
 0
 0
 3:12 
 0
 0
 0
 0
 4:43 
 0
 0
 0
 0
 3:10 
 0
 0
 0
 0
 9:38 
 0
 0
 0
 0
 5:14 
 0
 0
 0
 0
 15:56 
 0
 0
 1
 0
 3:20 
 0
 0
 0
 0
 19:44 
 0
 0
 0
 1
 14:48 
 0
 0
 0
 0
 20:47 
 0
 0
 0
 0
 4:53 
 0
 0
 0
 0
 3:13 
 0
 0
 0
 0
 15:32 
 0
 0
 0
 0
 20:42 
 0
 0
 0
 0
 16:06 
 0
 0
 0
 0
 14:20 
 0
 0
 0
 0
 14:56 
 0
 0
 0
 0
 5:15 
 0
 0
 0
 0
 23:05 
 0
 0
 0
 0
 9:10 
 0
 0
 0
 0
 15:32 
 0
 0
 0
 0
 5:49 
 0
 0
 0
 0
 11:11 
 0
 0
 0
 0
 5:31 
 0
 0
 0
 0
 20:29 
 0
 0
 0
 0
 18:35 
 0
 0
 0
 0
 17:49