05/04/2021

Accounting for Variance in Machine Learning Benchmarks

Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, Pascal Vincent

Keywords:

Abstract Paper Similar Papers

Abstract: Strong empirical evidence that one machine-learning algorithm A outperforms another one B, ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process and all sources of variation, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly machine learning benchmark. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that a biased estimator with more source of variation will give better results, closer to the ideal estimator at a 51× reduction in compute cost. Using this we perform a detailed study on the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for future performance comparisons.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38952709
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at MLSYS 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 19:40 
 0
 0
 0
 0
 4:46 
 0
 0
 0
 0
 5:17 
 0
 0
 0
 0
 15:07 
 0
 0
 0
 0
 4:31 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 5:19 
 0
 0
 0
 0
 3:04 
 0
 0
 0
 0
 19:34 
 0
 0
 0
 0
 17:53 
 0
 0
 0
 0
 4:46 
 0
 0
 0
 0
 3:13 
 0
 0
 0
 0
 13:22 
 0
 0
 0
 0
 16:45 
 0
 0
 0
 0
 5:03 
 0
 0
 0
 0
 5:26 
 0
 0
 0
 0
 4:25 
 0
 0
 0
 0
 19:36 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 2:59 
 0
 0
 0
 0
 11:27 
 0
 0
 0
 0
 14:50 
 0
 0
 0
 0
 8:04 
 1
 0
 0
 0
 4:45 
 0
 0
 0
 0
 8:59 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 15:22 
 0
 0
 0
 0
 13:10 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 14:10 
 0
 0
 0
 0
 9:55 
 0
 0
 0
 0
 15:07 
 0
 0
 0
 0
 4:49 
 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 3:01 
 0
 0
 0
 0
 13:59 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 10:48 
 0
 0
 0
 0
 17:00 
 0
 0
 0
 0
 3:12