02/02/2021

RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER

Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, Fangsheng Weng

Keywords:

Abstract Paper Similar Papers

Abstract: Recently multimodal named entity recognition (MNER) has utilized images to improve the accuracy of NER in tweets. However, most of the multimodal methods use attention mechanisms to extract visual clues regardless of whether the text and image are relevant. Practically, the irrelevant text-image pairs account for a large proportion in tweets. The visual clues that are unrelated to the texts will exert uncertain or even negative effects on multimodal model learning. In this paper, we introduce a method of text-image relation propagation into the multimodal BERT model. We integrate soft or hard gates to select visual clues and propose a multitask algorithm to train and validate the effects of relation propagation on the MNER datasets. In the experiments, we deeply analyze the changes in visual attention before and after the use of relation propagation. Our model achieves state-of-the-art performance on the MNER datasets.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38947849
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 12:17 
 0
 0
 0
 0
 9:40 
 0
 0
 0
 0
 14:26 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 4:53 
 0
 0
 0
 0
 7:10 
 0
 0
 0
 0
 7:08 
 0
 0
 0
 0
 15:15 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 13:01 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 14:51 
 0
 0
 0
 1
 5:00 
 0
 0
 0
 0
 4:57 
 0
 0
 0
 0
 2:00 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 4:50 
 0
 0
 0
 0
 15:15 
 0
 0
 0
 0
 12:30 
 0
 0
 0
 0
 12:54 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 11:11 
 0
 0
 0
 0
 4:43 
 0
 0
 0
 0
 11:54 
 0
 0
 0
 0
 14:28 
 0
 0
 0
 0
 11:31 
 0
 0
 0
 0
 10:19 
 0
 0
 0
 0
 17:04 
 0
 0
 0
 0
 18:21 
 0
 0
 0
 0
 9:03 
 0
 0
 0
 0
 16:20 
 0
 0
 0
 0
 1:04 
 0
 0
 0
 0
 10:51 
 0
 0
 0
 0
 14:20 
 0
 0
 0
 0
 19:21 
 0
 0
 0
 0
 3:13 
 0
 0
 0
 0
 3:02 
 0
 0
 0
 0
 11:20 
 0
 0
 0
 0
 12:02 
 0
 0
 0
 0
 5:00