02/02/2021

Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation

Yan-Bo Lin, Yu-Chiang Frank Wang

Keywords:

Abstract Paper Similar Papers

Abstract: Human perceives rich auditory experience with distinct sound heard by ears. Videos recorded with binaural audio particular simulate how human receives ambient sound. However, a large number of videos are with monaural audio only, which would degrade the user experience due to the lack of ambient information. To address this issue, we propose an audio spatialization framework to convert a monaural video into a binaural one exploiting the relationship across audio and visual components. By preserving the left-right consistency in both audio and visual modalities, our learning strategy can be viewed as a self-supervised learning technique, and alleviates the dependency on a large amount of video data with ground truth binaural audio data during training. Experiments on benchmark datasets confirm the effectiveness of our proposed framework in both semi-supervised and fully supervised scenarios, with ablation studies and visualization further support the use of our model for audio spatialization.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38949188
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 9:52 
 0
 0
 0
 0
 15:47 
 0
 0
 0
 0
 8:43 
 0
 0
 0
 0
 13:42 
 0
 0
 0
 0
 3:34 
 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 6:53 
 1
 1
 0
 0
 11:59 
 0
 0
 0
 0
 5:06 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 3:08 
 0
 0
 0
 1
 9:39 
 0
 0
 0
 0
 4:41 
 0
 0
 0
 0
 5:18 
 0
 0
 0
 0
 14:30 
 0
 0
 0
 0
 15:23 
 0
 0
 0
 0
 13:54 
 0
 0
 0
 0
 2:46 
 0
 0
 0
 0
 4:56 
 0
 0
 0
 0
 12:12 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 17:02 
 0
 0
 0
 0
 10:18 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 14:06 
 0
 0
 0
 0
 14:36 
 0
 0
 0
 0
 2:46 
 0
 0
 0
 0
 9:48 
 0
 0
 0
 0
 14:43 
 0
 0
 0
 0
 2:52 
 0
 0
 0
 0
 15:07 
 0
 0
 0
 0
 3:24 
 0
 0
 0
 0
 5:48 
 0
 0
 0
 0
 11:24 
 0
 0
 0
 0
 4:32 
 0
 0
 0
 0
 13:22 
 0
 0
 0
 0
 3:22