08/12/2020

An Analysis of Dataset Overlap on Winograd-Style Tasks

Ali Emami, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung

Keywords:

Abstract Paper Similar Papers

Abstract: The Winograd Schema Challenge (WSC) and variants inspired by it have become important benchmarks for common-sense reasoning (CSR). Model performance on the WSC has quickly progressed from chance-level to near-human using neural language models trained on massive corpora. In this paper, we analyze the effects of varying degrees of overlaps that occur between these corpora and the test instances in WSC-style tasks. We find that a large number of test instances overlap considerably with the pretraining corpora on which state-of-the-art models are trained, and that a significant drop in classification accuracy occurs when models are evaluated on instances with minimal overlap. Based on these results, we provide the WSC-Web dataset, consisting of over 60k pronoun disambiguation problems scraped from web data, being both the largest corpus to date, and having a significantly lower proportion of overlaps with current pretraining corpora.

The video of this talk cannot be embedded. You can watch it here:
https://underline.io/lecture/6166-an-analysis-of-dataset-overlap-on-winograd-style-tasks
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at COLING 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 11:09 
 0
 0
 0
 0
 16:26 
 0
 0
 0
 0
 15:39 
 0
 0
 0
 0
 11:51 
 0
 0
 0
 0
 18:24 
 0
 0
 0
 0
 13:06 
 0
 0
 0
 0
 5:59 
 0
 0
 0
 0
 12:23 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 12:32 
 0
 0
 0
 0
 8:36 
 0
 0
 0
 0
 5:14 
 0
 0
 0
 0
 12:35 
 0
 0
 0
 0
 15:08 
 0
 0
 0
 0
 12:12 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 19:11 
 0
 0
 0
 0
 20:00 
 0
 0
 0
 0
 5:17 
 0
 0
 0
 0
 15:25 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 11:00 
 0
 0
 0
 0
 18:23 
 0
 0
 0
 0
 5:08 
 0
 0
 0
 0
 5:42 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 6:48 
 0
 0
 0
 0
 11:45 
 0
 0
 0
 0
 16:19 
 0
 0
 0
 0
 14:58 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 16:14 
 0
 0
 0
 0
 5:03 
 0
 0
 0
 0
 11:18 
 0
 0
 0
 0
 11:02 
 0
 0
 0
 0
 14:22 
 0
 0
 0
 0
 0:59 
 0
 0
 0
 0
 3:14 
 0
 0
 0
 0
 6:05 
 0
 0
 0
 0
 8:46