19/10/2020

Multimodal knowledge graph for deep learning papers and code

Amar Viswanathan Kannan, Dmitriy Fradkin, Ioannis Akrotirianakis, Tugba Kulahcioglu, Arquimedes Canedo, Aditi Roy, Shih-Yuan Yu, Malawade Arnav, Mohammad Abdullah Al Faruque

Keywords: multimodal information retrieval, scientific knowledge graphs, knowledge graphs, scientific knowledge graph exploration, deep learning

Abstract Paper Similar Papers

Abstract: Keeping up with the rapid growth of Deep Learning (DL) research is a daunting task. While existing scientific literature search systems provide text search capabilities and can identify similar papers, gaining an in-depth understanding of a new approach or an application is much more complicated. Many publications leverage multiple modalities to convey their findings and spread their ideas - they include pseudocode, tables, images and diagrams in addition to text, and often make publicly accessible their implementations. It is important to be able to represent and query them as well. We utilize RDF Knowledge graphs (KGs) to represent multimodal information and enable expressive querying over modalities. In our demo we present an approach for extracting KGs from different modalities, namely text, architecture images and source code. We show how graph queries can be used to get insights into different facets (modalities) of a paper, and its associated code implementation. Our innovation lies in the multimodal nature of the KG we create. While our work is of direct interest to DL researchers and practitioners, our approaches can also be leveraged in other scientific domains.

The video of this talk cannot be embedded. You can watch it here:
https://dl.acm.org/doi/10.1145/3340531.3417439#sec-supp
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at CIKM 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 15:42 
 0
 0
 0
 0
 11:03 
 0
 0
 0
 0
 11:04 
 0
 0
 0
 0
 11:05 
 0
 0
 0
 0
 16:27 
 0
 0
 0
 0
 19:10 
 0
 0
 0
 0
 19:41 
 0
 0
 0
 0
 4:31 
 0
 0
 0
 0
 16:51 
 0
 0
 0
 0
 4:49 
 0
 0
 0
 0
 11:57 
 0
 0
 0
 0
 18:29 
 0
 0
 0
 0
 9:41 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 16:38 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 13:40 
 0
 0
 0
 0
 12:01 
 0
 0
 0
 0
 3:25 
 0
 0
 0
 0
 11:38 
 0
 0
 0
 0
 8:04 
 0
 0
 0
 0
 5:50 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 5:12 
 0
 0
 0
 0
 2:48 
 0
 0
 0
 0
 11:49 
 0
 0
 0
 0
 8:44 
 0
 0
 0
 0
 15:06 
 0
 0
 0
 0
 5:15 
 0
 0
 0
 0
 11:41 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 13:54 
 0
 0
 0
 0
 11:55 
 0
 0
 0
 0
 9:55 
 0
 0
 0
 0
 4:49 
 0
 0
 0
 0
 12:35 
 0
 0
 0
 0
 14:36 
 0
 0
 0
 0
 22:48 
 0
 0
 0
 0
 13:40 
 0
 0
 0
 0
 4:57