02/02/2021

Active Bayesian Assessment of Black-Box Classifiers

Disi Ji, Robert L. Logan, Padhraic Smyth, Mark Steyvers

Keywords:

Abstract Paper Similar Papers

Abstract: Recent advances in machine learning have led to increased deployment of black-box classifiers across a wide variety of applications. In many such situations there is a critical need to both reliably assess the performance of these pre-trained models and to perform this assessment in a label-efficient manner (given that labels may be scarce and costly to collect). In this paper, we introduce an active Bayesian approach for assessment of classifier performance to satisfy the desiderata of both reliability and label-efficiency. We begin by developing inference strategies to quantify uncertainty for common assessment metrics such as accuracy, misclassification cost, and calibration error. We then propose a general framework for active Bayesian assessment using inferred uncertainty to guide efficient selection of instances for labeling, enabling better performance assessment with fewer labels. We demonstrate significant gains from our proposed active Bayesian approach via a series of systematic empirical experiments assessing the performance of modern neural classifiers (e.g., ResNet and BERT) on several standard image and text classification datasets.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948759
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 14:23 
 0
 0
 0
 0
 11:24 
 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 18:24 
 0
 0
 0
 0
 8:43 
 0
 0
 0
 0
 20:16 
 0
 0
 0
 0
 10:07 
 0
 0
 0
 0
 14:14 
 0
 0
 0
 0
 5:18 
 0
 0
 0
 0
 16:11 
 0
 0
 0
 0
 3:13 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 20:12 
 0
 0
 0
 0
 13:57 
 0
 0
 0
 0
 12:16 
 0
 0
 0
 0
 13:22 
 0
 0
 0
 0
 11:27 
 0
 0
 0
 0
 15:23 
 0
 0
 0
 0
 14:02 
 0
 0
 0
 0
 13:37 
 0
 0
 0
 0
 11:00 
 0
 0
 0
 0
 15:00 
 0
 0
 0
 0
 8:59 
 0
 0
 0
 0
 4:33 
 0
 0
 0
 0
 15:41 
 0
 0
 0
 0
 4:32 
 0
 0
 0
 0
 4:25 
 0
 0
 0
 0
 5:12 
 0
 0
 0
 0
 2:28 
 0
 0
 0
 0
 11:41 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 4:28 
 0
 0
 0
 0
 3:24 
 0
 0
 0
 0
 5:26 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 3:25 
 0
 0
 0
 0
 7:02 
 0
 0
 0
 0
 19:55 
 0
 0
 0
 0
 19:21 
 0
 0
 0
 1
 5:13