02/02/2021

IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization

Wenxuan Zhou, Bill Yuchen Lin, Xiang Ren

Keywords:

Abstract Paper Similar Papers

Abstract: Fine-tuning pre-trained language models (PTLMs), such as BERT and its better variant RoBERTa, has been a common practice for advancing performance in natural language understanding (NLU) tasks. Recent advance in representation learning shows that isotropic (i.e., unit-variance and uncorrelated) embeddings can significantly improve performance on downstream tasks with faster convergence and better generalization. The isotropy of the pre-trained embeddings in PTLMs, however, is relatively under-explored. In this paper, we analyze the isotropy of the pre-trained [CLS] embeddings of PTLMs with straightforward visualization, and point out two major issues: high variance in their standard deviation, and high correlation between different dimensions. We also propose a new network regularization method, isotropic batch normalization (IsoBN) to address the issues, towards learning more isotropic representations in fine-tuning by dynamically penalizing dominating principal components. This simple yet effective fine-tuning method yields about 1.0 absolute increment on the average of seven NLU tasks.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38949013
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 11:43 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 4:44 
 0
 0
 0
 0
 4:59 
 0
 0
 0
 0
 9:56 
 0
 0
 0
 0
 11:00 
 0
 0
 0
 0
 11:47 
 0
 0
 0
 0
 5:13 
 0
 0
 0
 0
 11:50 
 0
 0
 0
 0
 5:21 
 0
 0
 0
 0
 4:16 
 0
 0
 0
 0
 13:17 
 0
 0
 0
 0
 11:27 
 0
 0
 0
 0
 5:17 
 0
 0
 0
 0
 7:33 
 0
 0
 0
 0
 16:44 
 0
 0
 0
 0
 7:54 
 0
 0
 0
 0
 14:34 
 0
 0
 0
 0
 7:53 
 0
 0
 0
 0
 14:19 
 0
 0
 0
 0
 17:06 
 0
 0
 0
 0
 10:58 
 0
 0
 0
 0
 4:48 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 10:04 
 0
 0
 0
 0
 4:43 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 15:04 
 0
 0
 0
 0
 10:53 
 0
 0
 0
 0
 8:28 
 0
 0
 0
 0
 11:43 
 0
 0
 0
 0
 3:09 
 0
 0
 0
 0
 11:24 
 0
 0
 0
 0
 14:45 
 0
 0
 0
 0
 6:26 
 0
 0
 0
 0
 12:23 
 0
 0
 0
 1
 9:03 
 0
 0
 0
 0
 14:27 
 0
 0
 0
 0
 14:53 
 0
 0
 0
 0
 17:37