02/02/2021

Debiasing Evaluations That Are Biased by Evaluations

Jingyan Wang, Ivan Stelmakh, Yuting Wei, Nihar B. Shah

Keywords:

Abstract Paper Similar Papers

Abstract: It is common to evaluate a set of items by soliciting people to rate them. For example, universities ask students to rate the teaching quality of their instructors, and conference organizers ask authors of submissions to evaluate the quality of the reviews. However, in these applications, students often give a higher rating to a course if they receive higher grades in a course, and authors often give a higher rating to the reviews if their papers are accepted to the conference. In this work, we call these external factors the "outcome" experienced by people, and consider the problem of mitigating these outcome-induced biases in the given ratings when some information about the outcome is available. We formulate the information about the outcome as a known partial ordering on the bias. We propose a debiasing method by solving a regularized optimization problem under this ordering constraint, and also provide a carefully designed cross-validation method that adaptively chooses the appropriate amount of regularization. We provide theoretical guarantees on the performance of our algorithm, as well as experimental evaluations.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38947990
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 8:11 
 0
 0
 0
 0
 16:15 
 0
 0
 0
 0
 10:10 
 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 12:12 
 0
 0
 0
 0
 12:00 
 0
 0
 0
 0
 9:15 
 0
 0
 0
 0
 14:20 
 0
 0
 0
 0
 1:33 
 0
 0
 0
 0
 3:05 
 0
 0
 0
 0
 2:42 
 0
 0
 0
 0
 8:04 
 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 8:28 
 0
 0
 0
 0
 14:55 
 0
 0
 0
 0
 19:13 
 0
 0
 0
 0
 14:37 
 0
 0
 0
 0
 17:59 
 0
 0
 0
 0
 2:58 
 0
 0
 0
 0
 9:54 
 0
 0
 0
 0
 15:44 
 0
 0
 0
 0
 20:17 
 0
 0
 0
 0
 10:19 
 0
 0
 0
 0
 10:42 
 0
 0
 0
 0
 15:00 
 0
 0
 0
 0
 12:24 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 14:00 
 0
 0
 0
 0
 3:21 
 0
 0
 0
 0
 4:54 
 0
 0
 0
 0
 8:49 
 0
 0
 0
 0
 14:20 
 0
 0
 0
 0
 14:11 
 0
 0
 0
 0
 15:28 
 0
 0
 0
 0
 15:41 
 0
 0
 0
 0
 12:59 
 0
 0
 0
 0
 19:42 
 0
 0
 0
 0
 3:03 
 0
 0
 0
 0
 3:25