02/02/2021

Confidence-aware Non-repetitive Multimodal Transformers for TextCaps

Zhaokai Wang, Renda Bao, Qi Wu, Si Liu

Keywords:

Abstract Paper Similar Papers

Abstract: When describing an image, reading text in the visual scene is crucial to understand the key information. Recent work explores the TextCaps task, i.e. image captioning with reading Optical Character Recognition (OCR) tokens, which requires models to read text and cover them in generated captions. Existing approaches fail to generate accurate descriptions because of their (1) poor reading ability; (2) inability to choose the crucial words among all extracted OCR tokens; (3) repetition of words in predicted captions. To this end, we propose a Confidence-aware Non-repetitive Multimodal Transformers (CNMT) to tackle the above challenges. Our CNMT consists of a reading, a reasoning and a generation modules, in which Reading Module employs better OCR systems to enhance text reading ability and a confidence embedding to select the most noteworthy tokens. To address the issue of word redundancy in captions, our Generation Module includes a repetition mask to avoid predicting repeated word in captions. Our model outperforms state-of-the-art models on TextCaps dataset, improving from 81.0 to 93.0 in CIDEr. Our source code is publicly available.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38947895
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 11:26 
 0
 0
 0
 0
 11:52 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 6:32 
 0
 0
 0
 0
 11:36 
 0
 0
 0
 0
 15:58 
 0
 0
 0
 0
 11:13 
 0
 0
 0
 0
 15:05 
 0
 0
 0
 0
 4:47 
 0
 0
 0
 0
 7:06 
 0
 0
 0
 0
 18:21 
 0
 0
 0
 0
 14:51 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 1
 12:13 
 0
 0
 0
 0
 12:34 
 0
 0
 0
 0
 17:31 
 0
 0
 0
 0
 13:47 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 9:16 
 0
 0
 0
 0
 12:13 
 0
 0
 0
 0
 2:56 
 0
 0
 0
 0
 15:34 
 0
 0
 0
 0
 14:37 
 0
 0
 0
 0
 11:45 
 0
 0
 0
 0
 8:42 
 0
 0
 0
 0
 12:30 
 0
 0
 0
 0
 12:46 
 0
 0
 0
 0
 6:53 
 0
 0
 0
 0
 7:01 
 0
 0
 0
 0
 11:51 
 0
 0
 0
 0
 6:27 
 0
 0
 0
 1
 3:25 
 0
 0
 0
 0
 13:11 
 0
 0
 1
 1
 4:58 
 0
 0
 0
 0
 9:44 
 0
 0
 0
 0
 13:07 
 0
 0
 0
 0
 8:54 
 0
 0
 0
 0
 12:17 
 0
 0
 0
 0
 17:06 
 0
 0
 0
 0
 11:25