23/08/2020

Multimodal learning with incomplete modalities by knowledge distillation

Qi Wang, Liang Zhan, Paul Thompson, Jiayu Zhou

Keywords: knowledge distillation, multimodal learning, incomplete modalities

Abstract Paper Similar Papers

Abstract: Multimodal learning aims at utilizing information from a variety of data modalities to improve the generalization performance. One common approach is to seek the common information that is shared among different modalities for learning, whereas we can also fuse the supplementary information to leverage modality-specific information. Though the supplementary information is often desired, most existing multimodal approaches can only learn from samples with complete modalities, which wastes a considerable amount of data collected. Otherwise, model-based imputation needs to be used to complete the missing values and yet may introduce undesired noise, especially when the sample size is limited. In this paper, we proposed a framework based on knowledge distillation, utilizing the supplementary information from all modalities, and avoiding imputation and noise associated with it. Specifically, we first train models on each modality independently using all the available data. Then the trained models are used as teachers to teach the student model, which is trained with the samples having complete modalities. We demonstrate the effectiveness of the proposed method in extensive empirical studies on both synthetic datasets and real-world datasets.

The video of this talk cannot be embedded. You can watch it here:
https://dl.acm.org/doi/10.1145/3394486.3403234#sec-supp
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at KDD 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 3:26 
 0
 0
 0
 0
 19:13 
 0
 0
 0
 0
 14:18 
 0
 0
 0
 0
 14:18 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 18:53 
 0
 0
 0
 0
 2:49 
 0
 0
 0
 0
 10:10 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 5:18 
 0
 0
 0
 0
 5:35 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 12:24 
 0
 0
 0
 0
 6:46 
 0
 0
 0
 0
 9:03 
 0
 0
 0
 0
 11:05 
 0
 0
 0
 0
 3:27 
 0
 0
 0
 0
 4:08 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 8:29 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 2:22 
 0
 0
 0
 0
 15:11 
 0
 0
 0
 0
 8:28 
 0
 0
 0
 0
 3:02 
 0
 0
 0
 0
 9:23 
 0
 0
 0
 0
 13:31 
 0
 0
 0
 0
 4:54 
 0
 0
 0
 0
 18:14 
 0
 0
 0
 0
 14:27 
 0
 0
 0
 0
 5:53 
 0
 0
 0
 0
 8:19 
 0
 0
 0
 0
 3:52 
 0
 0
 0
 1
 15:24 
 0
 0
 0
 0
 5:36 
 0
 0
 0
 0
 14:58 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 7:41