Characterizing Deep Gaussian Processes via Nonlinear Recurrence Systems

02/02/2021

Characterizing Deep Gaussian Processes via Nonlinear Recurrence Systems

Anh Tong, Jaesik Choi

Keywords:

Abstract Paper Similar Papers

Abstract: Recent advances in Deep Gaussian Processes (DGPs) show the potential to have more expressive representation than that of traditional Gaussian Processes (GPs). However, there exists a pathology of deep Gaussian processes that their learning capacities reduce significantly when the number of layers increases. In this paper, we present a new analysis in DGPs by studying its corresponding nonlinear dynamic systems to explain the issue. Existing work reports the pathology for the squared exponential kernel function. We extend our investigation to four types of common stationary kernel functions. The recurrence relations between layers are analytically derived, providing a tighter bound and the rate of convergence of the dynamic systems. We demonstrate our finding with a number of experimental results.

The video of this talk cannot be embedded. You can watch it here:

https://slideslive.com/38949228

(Link will open in new window)

0

0

0

0

Share

This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment

no comments yet

Similar Papers

06/12/2021

Robust Regression Revisited: Acceleration and Improved Estimation Rates

Arun Jambulapati, Jerry Li, Tselil Schramm, Kevin Tian

Keywords Paper

theory, optimization

0

0

0

0

14:22

12/07/2020

Better depth-width trade-offs for neural networks through the lens of dynamical systems

Evangelos Chatziafratis, Ioannis Panageas, Sai Ganesh Nagarajan

Keywords Paper

Deep Learning - Theory

0

0

0

0

16:21

06/12/2020

Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians

Juhan Bae, Roger Grosse

Keywords Paper

0

0

0

0

3:20

26/08/2020

Bayesian Image Classification with Deep Convolutional Gaussian Processes

Vincent Dutordoir, Mark van der Wilk, Artem Artemev, James Hensman

Keywords Paper

0

0

0

0

16:29

06/12/2021

Collapsed Variational Bounds for Bayesian Neural Networks

Marcin Tomczak, Siddharth Swaroop, Andrew Foong, Richard Turner

Keywords Paper

deep learning, optimization, generative model

0

0

0

0

5:44

14/06/2020

What Deep CNNs Benefit From Global Covariance Pooling: An Optimization Perspective

Qilong Wang, Li Zhang, Banggu Wu and
Dongwei Ren, Peihua Li, Wangmeng Zuo, Qinghua Hu

Keywords Paper

global covariance pooling, deep cnns, loss lipschitzness, gradient predictiveness, second-order optimization, faster convergence, stronger robustness, generalization

0

0

0

0

1:01

06/12/2020

Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties

Jakob Lindinger, David Reeb, Christoph Lippert, Barbara Rakitsch

Keywords Paper

0

0

0

0

3:21

06/12/2020

Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks

Umut Simsekli, Ozan Sener, George Deligiannidis, Murat Erdogdu

Keywords Paper

Deep Learning -> Supervised Deep Networks, Deep Learning -> Embedding Approaches

0

0

0

0

3:32

26/04/2020

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Jian Li, Xuanyuan Luo, Mingda Qiao

Keywords Paper

learning theory, generalization, nonconvex learning, stochastic gradient descent, Langevin dynamics

0

0

0

0

4:50

18/07/2021

LAMDA: Label Matching Deep Domain Adaptation

Trung Le, Tuan Nguyen, Nhat Ho and
Hung Bui, Dinh Phung

Keywords Paper

Theory, Deep learning Theory

0

0

0

1

5:14

12/07/2020

Extrapolation for Large-batch Training in Deep Learning

Tao LIN, Lingjing Kong, Sebastian Stich, Martin Jaggi

Keywords Paper

Deep Learning - Algorithms

0

0

0

0

13:21

06/12/2021

Post-Contextual-Bandit Inference

Aurelien Bibaut, Maria Dimakopoulou, Nathan Kallus and
Antoine Chambaz, Mark van der Laan

Keywords Paper

bandits, causality

0

0

0

0

20:05

06/12/2020

Improved Analysis of Clipping Algorithms for Non-convex Optimization

Bohang Zhang, Jikai Jin, Cong Fang, Liwei Wang

Keywords Paper

0

0

0

0

3:16

06/12/2021

Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning

Ekdeep S Lubana, Robert Dick, Hidenori Tanaka

Keywords Paper

deep learning

0

0

0

0

8:28

06/12/2021

Structured Dropout Variational Inference for Bayesian Neural Networks

Son Nguyen, Duong Nguyen, Khai Nguyen and
Khoat Than, Hung Bui, Nhat Ho

Keywords Paper

deep learning, generative model

0

0

0

0

11:28

06/12/2021

Global Filter Networks for Image Classification

Yongming Rao, Wenliang Zhao, Zheng Zhu and
Jiwen Lu, Jie Zhou

Keywords Paper

machine learning, robustness, transformers, vision

0

0

0

0

9:28

03/05/2021

Accelerating Convergence of Replica Exchange Stochastic Gradient MCMC via Variance Reduction

Wei Deng, Qi Feng, Georgios Karagiannis and
Guang Lin, Faming Liang

Keywords Paper

Markov jump process, uncertainty quantification, generalized Girsanov theorem, change of measure, stochastic gradient Langevin dynamics, parallel tempering, replica exchange, Dirichlet form, variance reduction

0

0

0

0

5:19

02/02/2021

Semi-Supervised Learning with Variational Bayesian Inference and Maximum Uncertainty Regularization

Kien Do, Truyen Tran, Svetha Venkatesh

Keywords Paper

0

0

0

0

16:56

22/11/2021

Parameter Efficient Dynamic Convolution via Tensor Decomposition

Zejiang Hou, Sun-Yuan Kung

Keywords Paper

dynamic convolution, input-dependent reparameterization, parameter efficiency, tensor decomposition

0

0

0

0

3:58

02/02/2021

Training Spiking Neural Networks with Accumulated Spiking Flow

Hao Wu, Yueyi Zhang, Wenming Weng and
Yongting Zhang, Zhiwei Xiong, Zheng-Jun Zha, Xiaoyan Sun, Feng Wu

Keywords Paper

0

0

0

0

16:45

26/08/2020

Linear Dynamics: Clustering without identification

Chloe Hsu, Michaela Hardt, Moritz Hardt

Keywords Paper

0

0

0

0

14:37

06/12/2021

Revisiting ResNets: Improved Training and Scaling Strategies

Irwan Bello, William Fedus, Xianzhi Du and
Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, Barret Zoph

Keywords Paper

machine learning, vision, semi-supervised learning

0

0

0

0

13:59

02/02/2021

Kernel-convoluted Deep Neural Networks with Data Augmentation

Minjin Kim, Young-geun Kim, Dongha Kim and
Yongdai Kim, Myunghee Cho Paik

Keywords Paper

0

0

0

0

15:32

26/04/2020

Stochastic AUC Maximization with Deep Neural Networks

Mingrui Liu, Zhuoning Yuan, Yiming Ying, Tianbao Yang

Keywords Paper

Stochastic AUC Maximization, Deep Neural Networks

0

0

0

0

4:58

26/04/2020

Improved Sample Complexities for Deep Neural Networks and Robust Classification via an All-Layer Margin

Colin Wei, Tengyu Ma

Keywords Paper

deep learning theory, generalization bounds, adversarially robust generalization, data-dependent generalization bounds

0

0

0

0

5:30

18/07/2021

Leveraging Non-uniformity in First-order Non-convex Optimization

Jincheng Mei, Yue Gao, Bo Dai and
Csaba Szepesvari, Dale Schuurmans

Keywords Paper

Theory, RL, Decisions and Control Theory

0

0

0

0

4:49

06/12/2021

On the Role of Optimization in Double Descent: A Least Squares Study

Ilja Kuzborskij, Csaba Szepesvari, Omar Rivasplata and
Amal Rannen-Triki, Razvan Pascanu

Keywords Paper

theory, deep learning, optimization

0

0

0

0

13:48

05/04/2021

An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems

Ahmed M. Abdelmoniem, Ahmed Elzanaty Elzanaty, Mohamed-Slim Alouini , Marco Canini

Keywords Paper

0

0

0

0

4:13

05/04/2021

An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems

Ahmed M. Abdelmoniem, Ahmed Elzanaty Elzanaty, Mohamed-Slim Alouini , Marco Canini

Keywords Paper

0

0

0

0

22:37

14/06/2020

Channel Attention Based Iterative Residual Learning for Depth Map Super-Resolution

Xibin Song, Yuchao Dai, Dingfu Zhou and
Liu Liu, Wei Li, Hongdong Li, Ruigang Yang

Keywords Paper

depth, super-resolution, channel attention, residual learning

0

0

0

0

1:00

06/12/2021

Representation Learning Beyond Linear Prediction Functions

Ziping Xu, Ambuj Tewari

Keywords Paper

theory, deep learning, optimization, representation learning, few shot learning

0

0

0

0

11:00

12/07/2020

Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent

Yunwen Lei, Yiming Ying

Keywords Paper

Learning Theory

0

0

0

0

13:15

06/12/2020

Walsh-Hadamard Variational Inference for Bayesian Deep Learning

Simone Rossi, Sebastien Marmin, Maurizio Filippone

Keywords Paper

0

0

0

0

2:59

13/04/2021

Multi-fidelity high-order gaussian processes for physical simulation

Zheng Wang, Wei Xing, Robert Kirby, Shandian Zhe

Keywords Paper

0

0

0

0

3:34

18/07/2021

Large-Scale Meta-Learning with Continual Trajectory Shifting

JWoong Shin, Hae Beom Lee, Boqing Gong, Sung Ju Hwang

Keywords Paper

Algorithms, Multitask, Transfer, and Meta Learning

0

0

0

0

6:14

03/05/2021

Lipschitz Recurrent Neural Networks

N. Benjamin Erichson, Omri Azencot, Alejandro Queiruga and
Liam Hodgkinson, Michael W Mahoney

Keywords Paper

dynamical systems, differential equations, recurrent neural networks

0

0

0

0

5:37

06/12/2021

Heavy Ball Neural Ordinary Differential Equations

Hedi Xia, Vai Suliafu, Hangjie Ji and
Tan Nguyen, Andrea Bertozzi, Stanley Osher, Bao Wang

Keywords Paper

deep learning, optimization, machine learning, vision

0

0

0

0

4:08

06/12/2020

CSER: Communication-efficient SGD with Error Reset

Cong Xie, Shuai Zheng, Sanmi Koyejo and
Indranil Gupta, Mu Li, Haibin Lin

Keywords Paper

0

0

0

0

3:12

07/09/2020

Neural Network Quantization with Scale-Adjusted Training

Qing Jin, Linjie Yang, Zhenyu Liao, Xiaoning Qian

Keywords Paper

neural network quantization, over-fitting, regularization

0

0

0

0

5:02