08/12/2020

Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism

Pan Xie, Zhi Cui, Xiuying Chen, XiaoHui Hu, Jianwei Cui, Bin Wang

Keywords:

Abstract Paper Similar Papers

Abstract: Non-autoregressive models generate target words in a parallel way, which achieve a faster decoding speed but at the sacrifice of translation accuracy. To remedy a flawed translation by non-autoregressive models, a promising approach is to train a conditional masked translation model (CMTM), and refine the generated results within several iterations. Unfortunately, such approach hardly considers the sequential dependency among target words, which inevitably results in a translation degradation. Hence, instead of solely training a Transformer-based CMTM, we propose a Self-Review Mechanism to infuse sequential information into it. Concretely, we insert a left-to-right mask to the same decoder of CMTM, and then induce it to autoregressively review whether each generated word from CMTM is supposed to be replaced or kept. The experimental results (WMT14 En ↔ De and WMT16 En ↔ Ro) demonstrate that our model uses dramatically less training computations than the typical CMTM, as well as outperforms several state-of-the-art non-autoregressive models by over 1 BLEU. Through knowledge distillation, our model even surpasses a typical left-to-right Transformer model, while significantly speeding up decoding.

The video of this talk cannot be embedded. You can watch it here:
https://underline.io/lecture/6181-infusing-sequential-information-into-conditional-masked-translation-model-with-self-review-mechanism
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at COLING 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 11:35 
 0
 0
 0
 0
 11:56 
 0
 0
 0
 0
 18:21 
 0
 0
 0
 0
 14:43 
 0
 0
 0
 0
 13:44 
 0
 0
 0
 0
 14:09 
 0
 0
 0
 0
 11:34 
 0
 0
 0
 0
 17:06 
 0
 0
 0
 0
 5:02 
 0
 0
 0
 0
 17:41 
 0
 0
 0
 0
 14:42 
 0
 0
 0
 0
 10:34 
 0
 0
 0
 0
 8:21 
 0
 0
 0
 0
 10:57 
 0
 0
 0
 0
 14:56 
 0
 0
 0
 0
 14:24 
 0
 0
 0
 0
 14:51 
 0
 0
 0
 0
 12:19 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 11:12 
 0
 0
 0
 0
 12:07 
 0
 0
 0
 0
 9:41 
 0
 0
 0
 0
 12:32 
 0
 0
 0
 0
 15:08 
 0
 0
 0
 0
 11:02 
 0
 0
 0
 0
 3:04 
 0
 0
 0
 1
 4:49 
 0
 0
 0
 0
 15:33 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 9:58 
 0
 0
 0
 0
 17:37 
 0
 0
 0
 1
 3:25 
 0
 0
 0
 0
 12:57 
 0
 0
 0
 0
 22:15 
 0
 0
 0
 0
 10:18 
 0
 0
 0
 0
 11:47 
 0
 0
 0
 0
 10:04 
 0
 0
 0
 0
 9:03 
 0
 0
 0
 0
 14:59 
 0
 0
 0
 0
 5:01