08/12/2020

Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction

Tong Zhu, Haitao Wang, Junjie Yu, Xiabing Zhou, Wenliang Chen, Wei Zhang, Min Zhang

Keywords:

Abstract Paper Similar Papers

Abstract: In recent years, distantly-supervised relation extraction has achieved a certain success by using deep neural networks. Distant Supervision (DS) can automatically generate large-scale annotated data by aligning entity pairs from Knowledge Bases (KB) to sentences. However, these DS-generated datasets inevitably have wrong labels that result in incorrect evaluation scores during testing, which may mislead the researchers. To solve this problem, we build a new dataset NYTH, where we use the DS-generated data as training data and hire annotators to label test data. Compared with the previous datasets, NYT-H has a much larger test set and then we can perform more accurate and consistent evaluation. Finally, we present the experimental results of several widely used systems on NYT-H. The experimental results show that the ranking lists of the comparison systems on the DS-labelled test data and human-annotated test data are different. This indicates that our human-annotated data is necessary for evaluation of distantly-supervised relation extraction.

The video of this talk cannot be embedded. You can watch it here:
https://underline.io/lecture/6376-towards-accurate-and-consistent-evaluation-a-dataset-for-distantly-supervised-relation-extraction
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at COLING 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 15:59 
 0
 0
 0
 0
 10:29 
 0
 0
 0
 0
 5:21 
 0
 0
 0
 0
 10:19 
 0
 0
 0
 0
 5:12 
 0
 0
 0
 0
 5:12 
 0
 0
 0
 0
 12:23 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 15:20 
 0
 0
 0
 0
 13:37 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 3:19 
 0
 0
 0
 0
 3:21 
 0
 0
 0
 0
 3:25 
 0
 0
 0
 0
 11:24 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 12:53 
 0
 0
 0
 0
 17:40 
 0
 0
 0
 0
 7:02 
 0
 0
 0
 0
 5:02 
 0
 0
 0
 0
 5:07 
 0
 0
 0
 0
 15:04 
 0
 0
 0
 0
 4:53 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 15:11 
 1
 0
 0
 0
 5:11 
 0
 0
 0
 0
 11:27 
 0
 0
 0
 0
 4:33 
 0
 0
 0
 0
 12:51 
 0
 0
 0
 0
 10:38 
 0
 0
 0
 0
 12:46 
 0
 0
 0
 1
 3:14 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 4:58 
 1
 1
 1
 1
 12:34 
 0
 0
 0
 0
 14:43 
 0
 0
 0
 0
 9:28 
 0
 0
 0
 0
 11:56 
 0
 0
 0
 0
 12:02 
 0
 0
 0
 0
 10:57