08/12/2020

Query Distillation: BERT-based Distillation for Ensemble Ranking

Wangshu Zhang, Junhong Liu, Zujie Wen, Yafang Wang, Gerard de Melo

Keywords:

Abstract Paper Similar Papers

Abstract: Recent years have witnessed substantial progress in the development of neural ranking networks, but also an increasingly heavy computational burden due to growing numbers of parameters and the adoption of model ensembles. Knowledge Distillation (KD) is a common solution to balance the effectiveness and efficiency. However, it is not straightforward to apply KD to ranking problems. Ranking Distillation (RD) has been proposed to address this issue, but only shows effectiveness on recommendation tasks. We present a novel two-stage distillation method for ranking problems that allows a smaller student model to be trained while benefitting from the better performance of the teacher model, providing better control of the inference latency and computational burden. We design a novel BERT-based ranking model structure for list-wise ranking to serve as our student model. All ranking candidates are fed to the BERT model simultaneously, such that the self-attention mechanism can enable joint inference to rank the document list. Our experiments confirm the advantages of our method, not just with regard to the inference latency but also in terms of higher-quality rankings compared to the original teacher model.

The video of this talk cannot be embedded. You can watch it here:
https://underline.io/lecture/6113-query-distillation-bert-based-distillation-for-ensemble-ranking
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at COLING Workshops 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 10:19 
 0
 0
 0
 0
 5:36 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 10:18 
 0
 0
 0
 0
 3:07 
 0
 0
 0
 0
 19:03 
 0
 0
 0
 0
 12:38 
 0
 0
 0
 0
 17:58 
 0
 0
 0
 0
 2:58 
 0
 0
 0
 0
 16:14 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 3:35 
 0
 0
 0
 0
 13:54 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 5:08 
 0
 0
 0
 0
 3:52 
 0
 0
 0
 0
 5:17 
 0
 0
 0
 0
 3:09 
 0
 0
 0
 0
 11:01 
 0
 0
 0
 0
 6:40 
 0
 0
 0
 0
 15:28 
 0
 0
 0
 0
 3:00 
 0
 0
 0
 0
 16:09 
 0
 0
 0
 0
 4:37 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 18:53 
 0
 0
 0
 0
 5:10 
 0
 0
 0
 0
 3:27 
 0
 0
 0
 0
 17:11 
 0
 0
 0
 0
 13:28 
 0
 0
 0
 0
 15:13 
 0
 0
 0
 0
 15:22 
 0
 0
 0
 0
 11:58 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 14:19 
 0
 0
 0
 0
 3:03 
 0
 0
 0
 0
 12:53 
 0
 0
 0
 0
 12:43 
 0
 0
 0
 0
 5:01