08/12/2020

Bridge the Gap: High-level Semantic Planning for Image Captioning

Chenxi Yuan, Yang Bai, Chun Yuan

Keywords:

Abstract Paper Similar Papers

Abstract: Recent image captioning models have made much progress for exploring the multi-modal interaction, such as attention mechanisms. Though these mechanisms can boost the interaction, there are still two gaps between the visual and language domains: (1) the gap between the visual features and textual semantics, (2) the gap between the disordering of visual features and the ordering of texts. To bridge the gaps we propose a high-level semantic planning (HSP) mechanism that incorporates both a semantic reconstruction and an explicit order planning. We integrate the planning mechanism to the attention based caption model and propose the High-level Semantic PLanning based Attention Network (HS-PLAN). First, an attention based reconstruction module is designed to reconstruct the visual features with high-level semantic information. Then we apply a pointer network to serialize the features and obtain the explicit order plan to guide the generation. Experiments conducted on MS COCO show that our model outperforms previous methods and achieves the state-of-the-art performance of 133.4% CIDEr-D score.

The video of this talk cannot be embedded. You can watch it here:
https://underline.io/lecture/6251-bridge-the-gap-high-level-semantic-planning-for-image-captioning
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at COLING 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 3:19 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 3:01 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 14:13 
 0
 0
 0
 0
 13:26 
 0
 0
 0
 0
 17:44 
 0
 0
 0
 0
 10:02 
 0
 0
 0
 0
 16:54 
 0
 0
 0
 0
 11:17 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 10:54 
 0
 0
 0
 0
 12:02 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 16:59 
 0
 0
 0
 0
 10:51 
 0
 0
 0
 0
 10:19 
 0
 0
 0
 0
 12:37 
 0
 0
 0
 0
 9:40 
 0
 0
 0
 0
 7:26 
 0
 0
 0
 0
 2:20 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 17:04 
 0
 0
 0
 0
 16:24 
 0
 0
 0
 0
 13:13 
 0
 0
 0
 0
 8:13 
 0
 0
 0
 0
 13:32 
 0
 0
 0
 0
 3:10 
 0
 0
 0
 0
 7:08 
 0
 0
 0
 0
 14:39 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 3:21 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 10:13 
 0
 0
 0
 0
 9:49 
 0
 0
 0
 0
 3:11 
 0
 0
 0
 0
 12:51