02/02/2021

Dual-level Collaborative Transformer for Image Captioning

Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, Rongrong Ji

Keywords:

Abstract Paper Similar Papers

Abstract: Descriptive region features extracted by object detection networks have played an important role in the recent advancements of image captioning. However, they are still criticized for the lack of contextual information and fine-grained details, which in contrast are the merits of traditional grid features. In this paper, we introduce a novel Dual-Level Collaborative Transformer (DLCT) network to realize the complementary advantages of the two features. Concretely, in DLCT, these two features are first processed by a novel Dual-way Self Attenion (DWSA) to mine their intrinsic properties, where a Comprehensive Relation Attention component is also introduced to embed the geometric information. In addition, we propose a Locality-Constrained Cross Attention module to address the semantic noises caused by the direct fusion of these two features, where a geometric alignment graph is constructed to accurately align and reinforce region and grid features. To validate our model, we conduct extensive experiments on the highly competitive MS-COCO dataset, and achieve new state-of-the-art performance on both local and online test sets, i.e., 133.8% CIDEr on Karpathy split and 135.4% CIDEr on the official split.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948007
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 17:44 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 13:29 
 0
 0
 0
 0
 11:20 
 0
 0
 0
 0
 3:03 
 0
 0
 0
 0
 5:28 
 0
 0
 0
 0
 16:34 
 0
 0
 0
 0
 9:35 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 4:53 
 0
 0
 0
 0
 16:30 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 3:16 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 3:21 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 14:14 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 3:00 
 0
 0
 0
 0
 10:43 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 9:19 
 0
 0
 0
 0
 9:41 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 12:01 
 0
 0
 0
 0
 10:51 
 0
 0
 0
 0
 2:19 
 0
 0
 0
 0
 7:09 
 0
 0
 0
 0
 13:21