02/02/2021

Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual Guidance

Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, Guodong Zhou

Keywords:

Abstract Paper Similar Papers

Abstract: Multi-modal named entity recognition (MNER) aims to discover named entities in free text and classify them into pre-defined types with images. However, dominant MNER models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, which have the potential to refine multi-modal representation learning. To deal with this issue, we propose a unified multi-modal graph fusion (UMGF) approach for MNER. Specifically, we first represent the input sentence and image using a unified multi-modal graph, which captures various semantic relationships between multi-modal semantic units (words and visual objects). Then, we stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions to learn node representations. Finally, we achieve an attention-based multi-modal representation for each word and perform entity labeling with a CRF decoder. Experimentation on the two benchmark datasets demonstrates the superiority of our MNER model.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948191
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 9:34 
 0
 0
 0
 0
 13:01 
 0
 0
 0
 0
 16:10 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 6:35 
 0
 0
 0
 0
 6:02 
 0
 0
 0
 0
 2:56 
 0
 0
 0
 0
 6:53 
 0
 0
 0
 0
 15:24 
 0
 0
 0
 0
 6:55 
 0
 0
 0
 0
 12:13 
 0
 0
 0
 0
 10:48 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 11:50 
 0
 0
 0
 0
 10:03 
 0
 0
 0
 0
 15:08 
 0
 0
 0
 0
 10:38 
 0
 0
 0
 0
 7:02 
 0
 0
 0
 0
 10:43 
 0
 0
 0
 0
 16:52 
 0
 0
 0
 0
 2:57 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 9:14 
 0
 0
 0
 0
 12:05 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 17:12 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 10:52 
 0
 0
 0
 0
 14:58 
 0
 0
 0
 0
 16:54 
 0
 0
 0
 0
 10:56 
 0
 0
 0
 0
 15:21 
 0
 0
 0
 0
 10:02 
 0
 0
 0
 0
 6:04 
 0
 0
 0
 0
 16:34 
 0
 0
 0
 0
 10:33 
 0
 0
 0
 0
 12:20 
 0
 0
 0
 0
 16:29 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 6:41