05/04/2021

Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and More

Shabnam Daghaghi, Nicholas Meisburger, Mengnan Zhao, Anshumali Shrivastava

Keywords:

Abstract Paper Similar Papers

Abstract: Deep learning implementations on CPUs (Central Processing Units) are gaining more traction. Enhanced AI capabilities on commodity x86 architectures are commercially appealing due to the reuse of existing hardware and virtualization ease. A notable work in this direction is the SLIDE system. SLIDE is a C++ implementation of a sparse hash table based back-propagation, which was shown to be significantly faster than GPUs in training hundreds of million parameter neural models. In this paper, we argue that SLIDE's current implementation is sub-optimal and does not exploit several opportunities available in modern CPUs. In particular, we show how SLIDE's computations allow for a unique possibility of vectorization via AVX (Advanced Vector Extensions)-512. Furthermore, we highlight opportunities for different kinds of memory optimization and quantizations. Combining all of them, we obtain up to 7x speedup in the computations on the same hardware. Our experiments are focused on large (hundreds of millions of parameters) recommendation and NLP models. Our work highlights several novel perspectives and opportunities for implementing randomized algorithms for deep learning on modern CPUs.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38952707
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at MLSYS 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 4:55 
 0
 0
 1
 0
 3:20 
 0
 0
 0
 0
 15:56 
 0
 0
 0
 0
 16:56 
 0
 0
 0
 0
 7:41 
 0
 0
 0
 0
 18:00 
 0
 0
 0
 0
 4:14 
 0
 0
 0
 0
 5:49 
 0
 0
 0
 0
 5:14 
 0
 0
 0
 0
 5:19 
 0
 0
 0
 0
 5:02 
 0
 0
 0
 0
 4:46 
 0
 0
 0
 0
 20:09 
 0
 0
 0
 0
 3:28 
 0
 0
 0
 0
 16:30 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 14:43 
 0
 0
 0
 0
 16:06 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 19:05 
 0
 0
 0
 0
 3:01 
 0
 0
 0
 0
 14:48 
 0
 0
 0
 0
 4:12 
 0
 0
 0
 0
 3:16 
 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 19:42 
 0
 0
 0
 0
 4:40 
 0
 0
 0
 0
 3:16 
 0
 0
 0
 0
 14:20 
 0
 0
 0
 0
 19:55 
 0
 0
 0
 0
 13:21 
 0
 0
 0
 0
 15:39 
 0
 0
 0
 0
 19:56 
 0
 0
 0
 0
 21:33 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 14:48 
 0
 0
 0
 0
 12:25 
 0
 0
 0
 0
 9:59 
 0
 0
 0
 0
 18:49