02/02/2021

Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Grounding

Dexin Wang, Deyi Xiong

Keywords:

Abstract Paper Similar Papers

Abstract: Visual context provides grounding information for multimodal machine translation (MMT). However, previous MMT models and probing studies on visual features suggest that visual information is less explored in MMT as it is often redundant to textual information. In this paper, we propose an Object-level Visual Context modeling framework (OVC) to efficiently capture and explore visual information for multimodal machine translation. With detected objects, the proposed OVC encourages MMT to ground translation on desirable visual objects by masking irrelevant objects in the visual modality. We equip the proposed with an additional object-masking loss to achieve this goal. The object-masking loss is estimated according to the similarity between masked objects and the source texts so as to encourage masking source-irrelevant objects. Additionally, in order to generate vision-consistent target words, we further propose a vision-weighted translation loss for OVC. Experiments on MMT datasets demonstrate that the proposed OVC model outperforms state-of-the-art MMT models and analyses show that masking irrelevant objects helps grounding in MMT.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38947965
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 8:36 
 0
 0
 0
 0
 2:54 
 0
 0
 0
 0
 11:43 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 18:21 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 7:17 
 0
 0
 0
 0
 16:22 
 0
 0
 0
 0
 13:36 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 13:57 
 0
 0
 0
 0
 8:54 
 0
 0
 0
 0
 9:55 
 0
 0
 0
 0
 18:32 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 9:19 
 0
 0
 0
 0
 12:19 
 0
 0
 0
 0
 0:58 
 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 14:56 
 0
 0
 0
 0
 14:44 
 0
 0
 0
 0
 12:30 
 0
 0
 0
 0
 2:58 
 0
 0
 0
 0
 4:54 
 0
 0
 0
 0
 3:01 
 0
 0
 0
 0
 6:12 
 0
 0
 0
 0
 3:14 
 0
 0
 0
 0
 5:10 
 0
 0
 0
 0
 6:46 
 0
 0
 0
 0
 11:15 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 8:08 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 12:17 
 0
 0
 0
 0
 7:08 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 14:49 
 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 4:48