30/11/2020

Online Knowledge Distillation via Multi-branch Diversity Enhancement

Zheng Li, Ying Huang, Defang Chen, Tianren Luo, Ning Cai, Zhigeng Pan

Keywords:

Abstract Paper Similar Papers

Abstract: Knowledge distillation is an effective method to transfer the knowledge from the cumbersome teacher model to the lightweight student model. Online knowledge distillation uses the ensembled prediction results of multiple student models as soft targets to train each student model. However, the homogenization problem will lead to difficulty in further improving model performance. In this work, we propose a new distillation method to enhance the diversity among multiple student models. We introduce Feature Fusion Module (FFM), which improves the performance of the attention mechanism in the network by integrating rich semantic information contained in the last block of multiple student models. Furthermore, we use the Classifier Diversification(CD) loss function to strengthen the differences between the student models and deliver a better ensemble result. Extensive experiments proved that our method significantly enhances the diversity among student models and brings better distillation performance. We evaluate our method on three image classification datasets: CIFAR-10/100 and CINIC-10. The results show that our method achieves state-of-the-art performance on these datasets.

The video of this talk cannot be embedded. You can watch it here:
https://accv2020.github.io/miniconf/poster_862.html
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at ACCV 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 12:57 
 0
 0
 0
 0
 3:07 
 0
 0
 0
 0
 3:04 
 0
 0
 0
 0
 5:10 
 0
 0
 0
 0
 10:06 
 0
 0
 0
 0
 4:37 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 17:58 
 0
 0
 0
 0
 3:35 
 0
 0
 0
 0
 3:00 
 0
 0
 0
 0
 3:02 
 0
 0
 0
 0
 13:30 
 0
 0
 0
 0
 8:14 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 2:57 
 0
 0
 0
 0
 10:19 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 5:47 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 4:54 
 0
 0
 0
 0
 7:14 
 0
 0
 0
 0
 12:38 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 2:58 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 11:58 
 0
 0
 0
 0
 14:19 
 0
 0
 0
 0
 16:14 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 4:57 
 0
 0
 0
 0
 13:56 
 0
 0
 0
 0
 13:41 
 0
 0
 0
 0
 3:02 
 0
 0
 0
 0
 18:53 
 0
 0
 0
 0
 10:13 
 0
 0
 0
 0
 17:05 
 0
 0
 0
 0
 12:24 
 0
 0
 0
 0
 13:54 
 0
 0
 0
 0
 5:09