02/02/2021

Reinforced Multi-Teacher Selection for Knowledge Distillation

Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong, Yan Fu, Daxin Jiang

Keywords:

Abstract Paper Similar Papers

Abstract: In natural language processing (NLP) tasks, slow inference speed and huge footprints in GPU usage remain the bottleneck of applying pre-trained deep models in production. As a popular method for model compression, knowledge distillation transfers knowledge from one or multiple large (teacher) models to a small (student) model. When multiple teacher models are available in distillation, the state-of-the-art methods assign a fixed weight to a teacher model in the whole distillation. Furthermore, most of the existing methods allocate an equal weight to every teacher model. In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled. We systematically develop a reinforced method to dynamically assign weights to teacher models for different training instances and optimize the performance of student model. Our extensive experimental results on several NLP tasks clearly verify the feasibility and effectiveness of our approach.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38949352
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 9:03 
 0
 0
 0
 0
 14:18 
 0
 0
 0
 0
 9:23 
 0
 0
 0
 0
 12:38 
 0
 0
 0
 0
 8:19 
 0
 0
 0
 0
 11:05 
 0
 0
 0
 0
 12:24 
 0
 0
 0
 0
 3:02 
 0
 0
 0
 0
 13:41 
 0
 0
 0
 0
 17:53 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 17:11 
 0
 0
 0
 0
 10:58 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 12:59 
 0
 0
 0
 0
 14:06 
 0
 0
 0
 0
 3:21 
 0
 0
 0
 0
 10:10 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 3:52 
 0
 0
 0
 0
 7:58 
 0
 0
 0
 0
 15:11 
 0
 0
 0
 0
 3:19 
 0
 0
 0
 0
 4:54 
 0
 0
 0
 0
 5:10 
 0
 0
 0
 0
 18:53 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 15:31 
 0
 0
 0
 0
 15:07 
 0
 0
 0
 0
 2:57 
 0
 0
 0
 0
 2:20 
 0
 0
 0
 0
 12:57 
 0
 0
 0
 0
 4:21 
 0
 0
 0
 0
 5:35 
 0
 0
 0
 0
 10:07 
 0
 0
 0
 0
 19:13 
 0
 0
 0
 0
 3:19