02/02/2021

Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis

Zi Wang

Keywords:

Abstract Paper Similar Papers

Abstract: Knowledge distillation (KD) has proved to be an effective approach for deep neural network compression, which learns a compact network (student) by transferring the knowledge from a pre-trained, over-parameterized network (teacher). In traditional KD, the transferred knowledge is usually obtained by feeding training samples to the teacher network to obtain the class probabilities. However, the original training dataset is not always available due to storage costs or privacy issues. In this study, we propose a novel data-free KD approach by modeling the intermediate feature space of the teacher with a multivariate normal distribution and leveraging the soft targeted labels generated by the distribution to synthesize pseudo samples as the transfer set. Several student networks trained with these synthesized transfer sets present competitive performance compared to the networks trained with the original training set and other data-free KD approaches.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948686
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:08 
 0
 0
 0
 0
 19:03 
 0
 0
 0
 0
 3:00 
 0
 0
 0
 0
 4:37 
 0
 0
 0
 0
 17:58 
 0
 0
 0
 0
 12:38 
 0
 0
 0
 0
 3:35 
 0
 0
 0
 0
 9:23 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 5:17 
 0
 0
 0
 0
 13:54 
 0
 0
 0
 0
 10:18 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 13:41 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 10:10 
 0
 0
 0
 0
 3:04 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 1:04 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 15:31 
 0
 0
 0
 0
 18:53 
 0
 0
 0
 0
 4:26 
 0
 0
 0
 0
 0:57 
 0
 0
 0
 0
 13:30 
 0
 0
 0
 0
 4:57 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 5:35 
 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 3:07 
 0
 0
 0
 0
 4:36 
 0
 0
 0
 0
 3:06 
 0
 0
 0
 0
 5:36 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 12:24 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 3:18 
 0
 0
 0
 0
 4:54 
 0
 0
 0
 0
 7:14