Black Magic in Deep Learning: How Human Skill Impacts Network Training

07/09/2020

Black Magic in Deep Learning: How Human Skill Impacts Network Training

Kanav Anand, Ziqi Wang, Marco Loog, Jan van Gemert

Keywords: deep learning, hyperparameter tuning, human factor, reproducibility

Abstract Paper Similar Papers

Abstract: How does a user's prior experience with deep learning impact network performance? We present an initial study based on 31 participants with different levels of experience. Their task is to perform hyperparameter optimization for a given deep learning architecture. The results show a strong positive correlation between the participant's experience and the final performance. They additionally indicate that an experienced participant finds better solutions using fewer resources on average. The data suggests furthermore that participants with no prior experience follow random strategies in their pursuit of optimal hyperparameters. Our study reveals the human factor in scientific reproducibility and in comparisons of state of the art results in deep learning.

0

0

0

0

Share

This is an embedded video. Talk and the respective paper are published at BMVC 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment

no comments yet

Similar Papers

18/07/2021

Meta-learning Hyperparameter Performance Prediction with Neural Processes

Ying WEI, Peilin Zhao, Junzhou Huang

Keywords Paper

Algorithms, Supervised Learning

0

0

0

0

5:07

18/07/2021

Fast Algorithms for Stackelberg Prediction Game with Least Squares Loss

jiali wang, He Chen, Rujun Jiang and
Xudong Li, Zihao Li

Keywords Paper

Optimization, Non-Convex Optimization

0

0

0

0

4:42

12/07/2020

Which Tasks Should Be Learned Together in Multi-task Learning?

Trevor Standley, Amir Zamir, Dawn Chen and
Leonidas Guibas, Jitendra Malik, Silvio Savarese

Keywords Paper

Transfer, Multitask and Meta-learning

0

0

0

0

15:07

06/12/2020

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

Stefano Sarao Mannelli, Eric Vanden-Eijnden, Lenka Zdeborová

Keywords Paper

Algorithms -> Model Selection and Structure Learning; Algorithms -> Representation Learning; Theory -> Computational Complexity, Reinforcement Learning and Planning -> Markov Decision Processes

0

0

0

0

3:22

06/12/2021

Model-Based Domain Generalization

Alexander Robey, George J. Pappas, Hamed Hassani

Keywords Paper

theory, deep learning, optimization, robustness, domain adaptation

0

0

0

0

15:08

06/12/2021

On Episodes, Prototypical Networks, and Few-Shot Learning

Steinar Laenen, Luca Bertinetto

Keywords Paper

machine learning, generative model, meta learning, few shot learning

0

0

0

0

14:46

06/12/2020

Submodular Meta-Learning

Arman Adibi, Aryan Mokhtari, Hamed Hassani

Keywords Paper

0

0

0

0

3:17

06/12/2021

Towards Enabling Meta-Learning from Target Models

Su Lu, Han-Jia Ye, Le Gan, De-Chuan Zhan

Keywords Paper

meta learning, few shot learning

0

0

0

0

11:12

03/05/2021

Complex Query Answering with Neural Link Predictors

Erik Arakelyan, Daniel Daza, Pasquale Minervini, Michael Cochez

Keywords Paper

neural link prediction, complex query answering

0

0

0

0

15:28

06/12/2020

Learning to Mutate with Hypergradient Guided Population

Zhiqiang Tao, Yaliang Li, Bolin Ding and
Ce Zhang, Jingren Zhou, Yun Fu

Keywords Paper

0

0

0

0

3:17

03/05/2021

Entropic gradient descent algorithms and wide flat minima

Fabrizio Pittorino, Carlo Lucibello, Christoph Feinauer and
Gabriele Perugini, Carlo Baldassi, Elizaveta Demyanenko, Riccardo Zecchina

Keywords Paper

flat minima, belief-propagation, statistical physics, entropic algorithms

0

0

0

0

5:38

06/12/2020

Curriculum learning for multilevel budgeted combinatorial problems

Adel Nabli, Margarida Carvalho

Keywords Paper

0

0

0

0

3:21

03/08/2020

Semi-bandit Optimization in the Dispersed Setting

Travis Dick, Wesley Pegden, Maria-Florina Balcan

Keywords Paper

0

0

0

0

8:04

03/05/2021

Few-Shot Bayesian Optimization with Deep Kernel Surrogates

Martin Wistuba, Josif Grabocka

Keywords Paper

automl, bayesian optimization, metalearning, few-shot learning

0

0

0

0

5:18

06/12/2021

Uniform Sampling over Episode Difficulty

Sébastien Arnold, Guneet Dhillon, Avinash Ravichandran, Stefano Soatto

Keywords Paper

meta learning, few shot learning

0

0

0

0

13:08

18/07/2021

On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting

Shunta Akiyama, Taiji Suzuki

Keywords Paper

Theory, Deep learning Theory

0

0

0

0

5:17

06/12/2021

How Powerful are Performance Predictors in Neural Architecture Search?

Colin White, Arber Zela, Robin Ru and
Yang Liu, Frank Hutter

Keywords Paper

deep learning, optimization

0

0

0

0

15:02

06/12/2020

Understanding spiking networks through convex optimization

Allan Mancoo, Sander Keemink, Christian Machens

Keywords Paper

0

0

0

0

3:21

06/12/2021

Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms

Alexander Camuto, George Deligiannidis, Murat Erdogdu and
Mert Gurbuzbalaban, Umut Simsekli, Lingjiong Zhu

Keywords Paper

theory, deep learning, optimization

0

0

0

0

14:36

02/02/2021

Towards Faster Deep Collaborative Filtering via Hierarchical Decision Networks

Yu Chen, Sinno Jialin Pan

Keywords Paper

0

0

0

0

15:28

18/07/2021

Neural Architecture Search without Training

Joe Mellor, Jack Turner, Amos Storkey, Elliot Crowley

Keywords Paper

Deep Learning, Architectures

0

0

0

1

20:37

06/12/2020

An analytic theory of shallow networks dynamics for hinge loss classification

Franco Pellegrini, Giulio Biroli

Keywords Paper

, Deep Learning -> Optimization for Deep Networks

0

0

0

0

3:11

06/12/2020

Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers

Kiwon Um, Robert Brand, Yun (Raymond) Fei and
Philipp Holl, Nils Thuerey

Keywords Paper

0

0

0

0

3:13

19/08/2021

A Comparative Survey: Benchmarking for Pool-based Active Learning

Xueying Zhan, Huan Liu, Qing Li, Antoni B. Chan

Keywords Paper

Machine learning, General

0

0

0

0

15:13

06/12/2021

Efficient Training of Retrieval Models using Negative Cache

Erik Lindgren, Sashank Reddi, Ruiqi Guo, Sanjiv Kumar

Keywords Paper

deep learning, machine learning

0

0

0

0

10:41

18/07/2021

PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees

Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, Andreas Krause

Keywords Paper

Algorithms, Multitask, Transfer, and Meta Learning

1

1

0

0

5:46

14/09/2020

GIKT: A Graph-based Interaction Model for Knowledge Tracing

Yang Yang, Jian Shen, Yanru Qu and
Yunfei Liu, Kerong Wang, Yaoming Zhu, Weinan Zhang, Yong Yu

Keywords Paper

knowledge tracing, graph neural network, information interaction

0

0

0

0

11:34

04/07/2020

Uncertainty-Aware Curriculum Learning for Neural Machine Translation

Yikai Zhou, Baosong Yang, Derek F. Wong and
Yu Wan, Lidia S. Chao

Keywords Paper

Neural Translation, assessment difficulty, translation tasks, Uncertainty-Aware Learning

0

0

0

0

8:20

26/04/2020

You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings

Daniel Ruffinelli, Samuel Broscheit, Rainer Gemulla

Keywords Paper

knowledge graph embeddings, hyperparameter optimization

0

0

0

0

4:56

13/04/2021

Contrastive learning of strong-mixing continuous-time stochastic processes

Bingbin Liu, Pradeep Ravikumar, Andrej Risteski

Keywords Paper

0

0

0

0

2:57

14/09/2020

A Taxonomy of Interactive Online Machine Learning Strategies

Agnes Tegen, Paul Davidsson, Jan A. Persson

Keywords Paper

interactive machine learning, online learning, active learning

0

0

0

0

14:20

06/12/2021

Deep Learning Through the Lens of Example Difficulty

Robert Baldock, Hartmut Maennel, Behnam Neyshabur

Keywords Paper

deep learning

0

0

0

0

12:54

06/12/2021

L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimization

Jiaqi Gu, Hanqing Zhu, Chenghao Feng and
Zixuan Jiang, Ray Chen, David Pan

Keywords Paper

deep learning, optimization

0

0

0

0

12:07

13/04/2021

Towards understanding the behaviors of optimal deep active learning algorithms

Yilun Zhou, Adithya Renduchintala, Xian Li and
Sida Wang, Yashar Mehdad, Asish Ghoshal

Keywords Paper

0

0

0

0

3:00

18/07/2021

Training Data Subset Selection for Regression with Controlled Generalization Error

Durga S, Rishabh Iyer, Ganesh Ramakrishnan, Abir De

Keywords Paper

, Algorithms, Online Learning, Algorithms, Supervised Learning

0

0

0

0

4:15

26/04/2020

Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees

Binghong Chen, Bo Dai, Qinjie Lin and
Guo Ye, Han Liu, Le Song

Keywords Paper

learning to plan, representation learning, learning to design algorithm, reinforcement learning, meta learning

0

0

0

0

4:59

02/02/2021

Physarum Powered Differentiable Linear Programming Layers and Applications

Zihang Meng, Sathya N. Ravi, Vikas Singh

Keywords Paper

0

0

0

0

16:57

06/12/2021

MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge

Geng Yuan, Xiaolong Ma, Wei Niu and
Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, Siyue Wang, Minghai Qin, Bin Ren, Yanzhi Wang, Sijia Liu, Xue Lin

Keywords Paper

deep learning, reinforcement learning and planning

0

0

0

0

15:00

18/07/2021

Learning Bounds for Open-Set Learning

Zhen Fang, Jie Lu, Anjin Liu and
Feng Liu, Guangquan Zhang

Keywords Paper

Theory, Statistical Learning Theory

0

0

0

0

5:17

14/06/2020

HyperSTAR: Task-Aware Hyperparameters for Deep Networks

Gaurav Mittal, Chang Liu, Nikolaos Karianakis and
Victor Fragoso, Mei Chen, Yun Fu

Keywords Paper

auto ml, hyperparameter optimization, meta learning, task aware, hyperband, hyperparameters, warm start, image classication, resnet, shufflenet

0

0

0

0

4:58