Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network

Abstract: Transformer-based architectures have shown great success in image captioning, where object regions are encoded and then attended into the vectorial representations to guide the caption decoding. However, such vectorial representations only contain region-level information without considering the global information reflecting the entire image, which fails to expand the capability of complex multi-modal reasoning in image captioning. In this paper, we introduce a Global Enhanced Transformer (termed GET) to enable the extraction of a more comprehensive global representation, and then adaptively guide the decoder to generate high-quality captions. In GET, a Global Enhanced Encoder is designed for the embedding of the global feature, and a Global Adaptive Decoder are designed for the guidance of the caption generation. The former models intra- and inter-layer global representation by taking advantage of the proposed Global Enhanced Attention and a layer-wise fusion module. The latter contains a Global Adaptive Controller that can adaptively fuse the global information into the decoder to guide the caption generation. Extensive experiments on MS COCO dataset demonstrate the superiority of our GET over many state-of-the-arts.

22/11/2021

Alaaeldin Ali, Hugo Touvron, Mathilde Caron and
Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, Herve Jegou

Zilong Zhong, Zhong Qiu Lin, Rene Bidart and
Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei-Shi Zheng, Jonathan Li, Alexander Wong

Jie Li, Kai Han, Peng Wang and
Yu Liu, Xia Yuan

Keywords Paper

semantic scene completion, dense voxel prediction, shape completion, semantic segmentation, rgb-d, anisotropic convolution, voxel-wise receptive fields, 3d convolution

1:01

14/06/2020

image captioning, image description, editing captions, sequence editing, copy mechanism, adaptive copy mechanism, selecting mechanism, copy lstm

1:01

14/06/2020

Zhaowen Li, Zhiyang Chen, Fan Yang and
Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, Jinqiao Wang

network architecture, attention, self-supervision, weakly-supervised object localization, interpretability, classification, cutmix, cnn