16/11/2020

PatchBERT: Just-in-Time, Out-of-Vocabulary Patching

Sangwhan Moon, Naoaki Okazaki

Keywords: natural processing, downstream tasks, mitigation, large models

Abstract Paper Similar Papers

Abstract: Large scale pre-trained language models have shown groundbreaking performance improvements for transfer learning in the domain of natural language processing. In our paper, we study a pre-trained multilingual BERT model and analyze the OOV rate on downstream tasks, how it introduces information loss, and as a side-effect, obstructs the potential of the underlying model. We then propose multiple approaches for mitigation and demonstrate that it improves performance with the same parameter count when combined with fine-tuning.

 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at EMNLP 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:23 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 6:16 
 0
 0
 0
 0
 4:16 
 0
 0
 0
 0
 5:07 
 0
 0
 0
 0
 9:58 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 13:09 
 0
 0
 0
 0
 11:40 
 0
 0
 0
 0
 5:12 
 0
 0
 0
 0
 4:11 
 0
 0
 0
 0
 10:55 
 0
 0
 0
 0
 11:30 
 0
 0
 0
 1
 10:52 
 0
 0
 0
 0
 6:56 
 0
 0
 0
 0
 9:21 
 0
 0
 0
 0
 12:06 
 0
 0
 0
 0
 13:53 
 0
 0
 0
 0
 12:17 
 0
 0
 0
 0
 6:10 
 0
 0
 0
 0
 11:27 
 0
 0
 0
 0
 2:38 
 0
 0
 0
 0
 5:42 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 9:08 
 0
 0
 0
 0
 12:00 
 0
 0
 0
 0
 7:10 
 0
 0
 0
 0
 12:03 
 0
 0
 0
 0
 6:03 
 0
 0
 0
 0
 12:38 
 0
 0
 0
 0
 2:49 
 0
 0
 0
 0
 14:56 
 0
 0
 0
 0
 11:16 
 0
 0
 0
 0
 11:43 
 0
 0
 0
 0
 11:36 
 0
 0
 0
 0
 5:04 
 0
 0
 0
 0
 16:33 
 0
 0
 0
 0
 6:06 
 0
 0
 0
 0
 15:00