08/12/2020

A hierarchical approach to vision-based language generation: from simple sentences to complex natural language

Simion-Vlad Bogolin, Ioana Croitoru, Marius Leordeanu

Keywords:

Abstract Paper Similar Papers

Abstract: Automatically describing videos in natural language is an ambitious problem, which could bridge our understanding of vision and language. We propose a hierarchical approach, by first generating video descriptions as sequences of simple sentences, followed at the next level by a more complex and fluent description in natural language. While the simple sentences describe simple actions in the form of (subject, verb, object), the second-level paragraph descriptions, indirectly using information from the first-level description, presents the visual content in a more compact, coherent and semantically rich manner. To this end, we introduce the first video dataset in the literature that is annotated with captions at two levels of linguistic complexity. We perform extensive tests that demonstrate that our hierarchical linguistic representation, from simple to complex language, allows us to train a two-stage network that is able to generate significantly more complex paragraphs than current one-stage approaches.

The video of this talk cannot be embedded. You can watch it here:
https://underline.io/lecture/6221-a-hierarchical-approach-to-vision-based-language-generation-from-simple-sentences-to-complex-natural-language
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at COLING 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 4:50 
 0
 0
 0
 0
 18:15 
 0
 0
 0
 0
 7:22 
 0
 0
 0
 0
 7:40 
 1
 0
 0
 0
 3:25 
 0
 0
 0
 0
 6:14 
 0
 0
 0
 0
 16:57 
 0
 0
 0
 0
 16:17 
 0
 0
 0
 0
 5:02 
 0
 0
 0
 0
 8:46 
 0
 0
 0
 0
 16:24 
 0
 0
 0
 0
 15:58 
 0
 0
 0
 0
 4:08 
 0
 0
 0
 0
 9:52 
 0
 0
 0
 0
 8:29 
 0
 0
 0
 0
 7:00 
 0
 0
 0
 0
 3:24 
 0
 0
 0
 0
 11:51 
 0
 0
 0
 0
 14:19 
 0
 0
 0
 0
 10:56 
 0
 0
 0
 0
 15:03 
 0
 0
 0
 0
 2:50 
 0
 0
 0
 0
 10:10 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 11:11 
 0
 0
 0
 0
 9:53 
 0
 0
 0
 0
 5:02 
 0
 0
 0
 0
 4:33 
 0
 0
 0
 0
 3:25 
 0
 0
 0
 0
 4:34 
 0
 0
 0
 0
 6:37 
 0
 0
 0
 0
 14:19 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 13:05 
 0
 0
 0
 1
 14:34 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 6:56 
 0
 0
 0
 0
 1:00