02/02/2021

Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks

Yoonho Boo, Sungho Shin, Jungwook Choi, Wonyong Sung

Keywords:

Abstract Paper Similar Papers

Abstract: The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve the performance of quantized networks. In this study, we propose stochastic precision ensemble training for QDNNs (SPEQ). SPEQ is a knowledge distillation training scheme; however, the teacher is formed by sharing the model parameters of the student network. We obtain the soft labels of the teacher by randomly changing the bit precision of the activation stochastically at each layer of the forward-pass computation. The student model is trained with these soft labels to reduce the activation quantization noise. The cosine similarity loss is employed, instead of the KL-divergence, for KD training. As the teacher model changes continuously by random bit-precision assignment, it exploits the effect of stochastic ensemble KD. SPEQ outperforms the existing quantization training methods in various tasks, such as image classification, question-answering, and transfer learning without the need for cumbersome teacher networks.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38949351
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:08 
 0
 0
 0
 0
 14:19 
 0
 0
 0
 0
 12:38 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 10:18 
 0
 0
 0
 0
 12:43 
 0
 0
 0
 0
 3:35 
 0
 0
 0
 0
 16:09 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 3:00 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 15:04 
 0
 0
 0
 0
 4:37 
 0
 0
 0
 0
 5:17 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 2:20 
 0
 0
 0
 0
 14:06 
 0
 0
 0
 0
 18:53 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 12:51 
 0
 0
 0
 0
 3:19 
 0
 0
 0
 0
 13:54 
 0
 0
 0
 0
 4:26 
 0
 0
 0
 0
 3:07 
 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 4:54 
 0
 0
 0
 0
 6:40 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 4:57 
 0
 0
 0
 0
 15:31 
 0
 0
 0
 0
 12:59 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 17:58 
 0
 0
 0
 0
 3:18 
 0
 0
 0
 0
 7:14 
 0
 0
 0
 0
 17:05 
 0
 0
 0
 0
 10:07 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 5:35