02/02/2021

Listen, Understand and Translate: Triple Supervision Decouples End-to-end Speech-to-text Translation

Qianqian Dong, Rong Ye, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, Lei Li

Keywords:

Abstract Paper Similar Papers

Abstract: An end-to-end speech-to-text translation (ST) takes audio in a source language and outputs the text in a target language. Existing methods are limited by the amount of parallel corpus. Can we build a system to fully utilize signals in a parallel ST corpus? We are inspired by human understanding system which is composed of auditory perception and cognitive processing. In this paper, we propose Listen-Understand-Translate, (LUT), a unified framework with triple supervision signals to decouple the end-to-end speech-to-text translation task. LUT is able to guide the acoustic encoder to extract as much information from the auditory input. In addition, LUT utilizes a pre-trained BERT model to enforce the upper encoder to produce as much semantic information as possible, without extra data. We perform experiments on a diverse set of speech translation benchmarks, including Librispeech English-French, IWSLT English-German and TED English-Chinese. Our results demonstrate LUT achieves the state-of-the-art performance, outperforming previous methods. The code is available at https://github.com/dqqcasia/st.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38949368
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 15:12 
 0
 0
 0
 0
 7:16 
 0
 0
 0
 0
 8:53 
 0
 0
 0
 0
 14:47 
 0
 0
 0
 0
 11:45 
 0
 0
 0
 0
 5:13 
 0
 0
 0
 0
 2:41 
 0
 0
 0
 0
 5:40 
 0
 0
 0
 0
 19:58 
 0
 0
 0
 0
 4:37 
 0
 0
 0
 0
 14:19 
 0
 0
 0
 0
 5:17 
 0
 0
 0
 0
 5:19 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 6:12 
 0
 0
 0
 0
 13:54 
 0
 0
 0
 0
 5:44 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 5:15 
 0
 0
 0
 0
 10:44 
 0
 0
 0
 0
 15:14 
 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 16:08 
 0
 0
 0
 0
 10:01 
 0
 0
 0
 0
 15:07 
 0
 0
 0
 0
 13:28 
 0
 0
 0
 0
 14:09 
 0
 0
 0
 0
 4:50 
 0
 0
 0
 0
 11:37 
 0
 0
 0
 0
 19:30 
 0
 0
 0
 0
 14:39 
 0
 0
 0
 0
 14:47 
 0
 0
 0
 0
 14:43 
 0
 0
 0
 0
 11:49 
 0
 0
 0
 0
 8:43 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 12:23 
 0
 0
 0
 0
 15:23 
 0
 0
 0
 0
 18:11 
 0
 0
 0
 0
 5:51