25/07/2020

Symmetric regularization based BERT for pair-wise semantic reasoning

Weidi Xu, Xingyi Cheng, Kunlong Chen, Taifeng Wang

Keywords: BERT, natural language inference

Abstract Paper Similar Papers

Abstract: The ability of semantic reasoning over the sentence pair is essential for many natural language understanding tasks, e.g., natural language inference and machine reading comprehension. A recent significant improvement in these tasks comes from BERT. As reported, the next sentence prediction (NSP) in BERT is of great significance for downstream problems with sentence-pair input. Despite its effectiveness, NSP still lacks the essential signal to distinguish between entailment and shallow correlation. To remedy this, we propose to augment the NSP task to a multi-class categorization task, which includes previous sentence prediction (PSP). This task encourages the model to learn the subtle semantics, thereby improves the ability of semantic understanding. Furthermore, by using a smoothing technique, the scopes of NSP and PSP are expanded into a broader range which includes close but nonsuccessive sentences. This simple method yields remarkable improvement against vanilla BERT. Our method consistently improves the performance on the NLI and MRC benchmarks by a large margin, including the challenging HANS dataset.

The video of this talk cannot be embedded. You can watch it here:
https://dl.acm.org/doi/10.1145/3397271.3401309#sec-supp
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at SIGIR 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 4:43 
 0
 0
 0
 0
 13:01 
 0
 0
 0
 0
 12:23 
 0
 0
 0
 0
 11:43 
 0
 0
 0
 0
 14:09 
 0
 0
 0
 0
 9:11 
 0
 0
 0
 0
 12:24 
 0
 0
 0
 0
 12:24 
 0
 0
 0
 0
 11:49 
 0
 0
 0
 0
 12:01 
 0
 0
 0
 0
 18:55 
 0
 0
 0
 0
 6:53 
 0
 0
 0
 0
 16:40 
 0
 0
 0
 0
 3:51 
 0
 0
 0
 0
 4:38 
 0
 0
 0
 0
 16:25 
 0
 0
 0
 0
 11:55 
 0
 0
 0
 0
 12:11 
 0
 0
 0
 0
 10:21 
 0
 0
 0
 0
 11:00 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 12:22 
 0
 0
 0
 0
 13:17 
 0
 0
 0
 0
 6:40 
 0
 0
 0
 0
 11:00 
 0
 0
 0
 0
 10:15 
 0
 0
 0
 0
 19:20 
 0
 0
 0
 0
 14:56 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 7:09 
 0
 0
 0
 0
 12:03 
 0
 0
 0
 0
 12:03 
 0
 0
 0
 0
 0:59 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 9:45 
 0
 0
 0
 0
 14:50 
 0
 0
 0
 0
 17:21 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 17:06 
 0
 0
 0
 0
 6:38