02/02/2021

Semantics Altering Modifications for Evaluating Comprehension in Machine Reading

Viktor Schlegel, Goran Nenadic, Riza Batista-Navarro

Keywords:

Abstract Paper Similar Papers

Abstract: Advances in NLP have yielded impressive results for the task of machine reading comprehension (MRC), with approaches having been reported to achieve performance comparable to that of humans. In this paper, we investigate whether state-of-the-art MRC models are able to correctly process Semantics Altering Modifications (SAM): linguistically-motivated phenomena that alter the semantics of a sentence while preserving most of its lexical surface form. We present a method to automatically generate and align challenge sets featuring original and altered examples. We further propose a novel evaluation methodology to correctly assess the capability of MRC systems to process these examples independent of the data they were optimised on, by discounting for effects introduced by domain shift. In a large-scale empirical study, we apply the methodology in order to evaluate extractive MRC models with regard to their capability to correctly process SAM-enriched data. We comprehensively cover 12 different state-of-the-art neural architecture configurations and four training datasets and find that -- despite their well-known remarkable performance -- optimised models consistently struggle to correctly process semantically altered data.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38947840
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 15:18 
 0
 0
 0
 0
 17:06 
 0
 0
 0
 0
 10:32 
 0
 0
 0
 0
 11:47 
 0
 0
 0
 0
 11:48 
 0
 0
 0
 0
 0:59 
 0
 0
 0
 0
 12:45 
 0
 0
 0
 0
 8:53 
 0
 0
 0
 0
 7:10 
 0
 0
 0
 0
 11:42 
 0
 0
 0
 0
 9:08 
 0
 0
 0
 0
 14:47 
 0
 0
 0
 0
 5:07 
 0
 0
 0
 0
 6:16 
 0
 0
 0
 0
 11:43 
 0
 0
 0
 0
 12:23 
 0
 0
 0
 0
 14:20 
 0
 0
 0
 0
 6:33 
 0
 0
 0
 0
 11:11 
 0
 0
 0
 0
 15:15 
 0
 0
 0
 0
 6:53 
 0
 0
 0
 0
 16:25 
 0
 0
 0
 0
 14:20 
 0
 0
 0
 0
 10:08 
 0
 0
 0
 0
 5:21 
 0
 0
 0
 0
 4:33 
 0
 0
 0
 0
 5:16 
 0
 0
 0
 0
 13:17 
 0
 0
 0
 0
 11:24 
 0
 0
 0
 0
 11:09 
 0
 0
 0
 0
 3:06 
 0
 0
 0
 0
 4:50 
 0
 0
 0
 0
 15:21 
 0
 0
 0
 0
 19:17 
 0
 0
 0
 0
 11:58 
 0
 0
 0
 0
 11:39 
 0
 0
 0
 0
 3:51 
 0
 0
 0
 0
 15:00 
 0
 0
 0
 0
 9:45 
 0
 0
 0
 0
 15:10