Neural network integral representations with the ReLU activation function

20/07/2020

Neural network integral representations with the ReLU activation function

Armenak Petrosyan, Anton Dereventsov, Clayton G. Webster

Keywords:

Abstract Paper Similar Papers

Abstract: In this effort, we derive a formula for the integral representation of a shallow neural network with the ReLU activation function. We assume that the outer weighs admit a finite $L_1$-norm with respect to Lebesgue measure on the sphere. For univariate target functions we further provide a closed-form formula for all possible representations. Additionally, in this case our formula allows one to explicitly solve the least $L_1$-norm neural network representation for a given function.

0

0

0

0

Share

This is an embedded video. Talk and the respective paper are published at MSML 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment

no comments yet

Similar Papers

12/07/2020

Constructive universal distribution generation through deep ReLU networks

Dmytro Perekrestenko, Stephan Müller, Helmut Bölcskei

Keywords Paper

Deep Learning - Theory

0

0

0

0

15:27

06/12/2021

Deep Networks Provably Classify Data on Curves

Tingran Wang, Sam Buchanan, Dar Gilboa, John Wright

Keywords Paper

theory, deep learning, optimization, machine learning, kernel methods

0

0

0

0

14:50

04/08/2021

A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Partial Differential Equations

Yulong Lu, Jianfeng Lu, Min Wang

Keywords Paper

0

0

0

0

12:38

03/05/2021

Universal approximation power of deep residual neural networks via nonlinear control theory

Paulo Tabuada, Bahman Gharesifard

Keywords Paper

nonlinear control theory, Deep residual neural networks, universal approximation

0

0

0

0

4:48

03/05/2021

Large-width functional asymptotics for deep Gaussian neural networks

Daniele Bracale, Stefano Favaro, Sandra Fortini, Stefano Peluchetti

Keywords Paper

deep learning theory, stochastic process, Gaussian process, infinitely wide neural network

0

0

0

0

4:48

18/07/2021

Revealing the Structure of Deep Neural Networks via Convex Duality

Tolga Ergen, Mert Pilanci

Keywords Paper

Theory, Deep learning Theory

0

0

0

0

5:35

12/07/2020

Second-Order Provable Defenses against Adversarial Attacks

Sahil Singla, Soheil Feizi

Keywords Paper

Adversarial Examples

0

0

0

0

12:45

06/12/2021

Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II

Yossi Arjevani, Michael Field

Keywords Paper

theory, deep learning, optimization

0

0

0

0

8:40

13/04/2021

The spectrum of fisher information of deep networks achieving dynamical isometry

Tomohiro Hayase, Ryo Karakida

Keywords Paper

0

0

0

0

3:10

03/05/2021

A unifying view on implicit bias in training linear neural networks

Chulhee (Charlie) Yun, Shankar Krishnan, Hossein Mobahi

Keywords Paper

convergence, implicit bias, gradient flow, implicit regularization, gradient descent

0

0

0

0

5:24

03/05/2021

Deep Networks and the Multiple Manifold Problem

Sam Buchanan, Dar Gilboa, John Wright

Keywords Paper

low-dimensional structure, overparameterized neural networks, deep learning

0

0

0

0

5:14

18/07/2021

Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks

Quynh Nguyen, Marco Mondelli, Guido Montufar

Keywords Paper

Theory, Deep learning Theory

0

0

0

0

5:04

22/06/2020

Estimating normalizing constants for log-concave distributions: Algorithms and lower bounds

Rong Ge, Holden Lee, Jianfeng Lu

Keywords Paper

sampling algorithm, partition function, normalizing constant, log-concave distribution, multilevel Monte Carlo, Langevin Monte Carlo

0

0

0

0

26:11

26/08/2020

Convex Geometry of Two-Layer ReLU Networks: Implicit Autoencoding and Interpretable Models

Tolga Ergen, Mert Pilanci

Keywords Paper

0

0

0

0

14:07

06/12/2020

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová

Keywords Paper

0

0

0

0

3:21

06/12/2021

Analysis of one-hidden-layer neural networks via the resolvent method

Vanessa Piccolo, Dominik Schröder

Keywords Paper

theory, deep learning

0

0

0

0

11:28

09/07/2020

Universal Approximation with Deep Narrow Networks

Patrick Kidger, Terry J Lyons

Keywords Paper

Neural networks/deep learning, Regression

0

0

0

0

13:40

26/04/2020

A closer look at the approximation capabilities of neural networks

Kai Fong Ernest Chong

Keywords Paper

deep learning, approximation, universal approximation theorem

0

0

0

0

5:06

06/12/2020

A Universal Approximation Theorem of Deep Neural Networks for Expressing Probability Distributions

Yulong Lu, Jianfeng Lu

Keywords Paper

0

0

0

0

2:55

06/12/2021

DNN-based Topology Optimisation: Spatial Invariance and Neural Tangent Kernel

Benjamin Dupuis, Arthur Jacot

Keywords Paper

deep learning, optimization

0

0

0

0

9:39

26/04/2020

Functional vs. parametric equivalence of ReLU networks

Mary Phuong, Christoph H. Lampert

Keywords Paper

ReLU networks, symmetry, functional equivalence, over-parameterization

0

0

0

0

5:15

26/04/2020

Neural tangent kernels, transportation mappings, and universal approximation

Ziwei Ji, Matus Telgarsky, Ruicheng Xian

Keywords Paper

Neural Tangent Kernel, universal approximation, Barron, transport mapping

0

0

0

0

4:48

18/07/2021

Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

Berfin Simsek, François Ged, Arthur Jacot and
Francesco Spadaro, Clement Hongler, Wulfram Gerstner, Johanni Brea

Keywords Paper

Theory, Algorithms, Representation Learning, Algorithms, Large Scale Learning; Applications, Natural Language Processing; Deep Learning, Efficient Inference Methods;

0

0

0

0

5:05

06/12/2021

Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activations

Pranjal Awasthi, Alex Tang, Aravindan Vijayaraghavan

Keywords Paper

theory, deep learning

0

0

0

0

14:31

06/12/2021

Towards a Unified Information-Theoretic Framework for Generalization

Mahdi Haghifam, Gintare Karolina Dziugaite, Shay Moran, Dan Roy

Keywords Paper

graph learning

0

0

0

0

11:51

02/02/2021

Submodular Span, with Applications to Conditional Data Summarization

Lilly Kumari, Jeff Bilmes

Keywords Paper

0

0

0

0

16:22

18/07/2021

Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks

Greg Yang, Edward Hu

Keywords Paper

Theory, Deep learning Theory

0

0

0

0

5:22

03/05/2021

Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms

Arda Sahiner, Tolga Ergen, John M Pauly, Mert Pilanci

Keywords Paper

convolutional neural networks, convex duality, copositive programming, nonnegative PCA, semi-nonnegative matrix factorization, computational complexity, global optima, semi-infinite duality, theory, convex optimization, neural networks

0

0

0

0

6:08

06/12/2020

Provably Efficient Neural GTD for Off-Policy Learning

Hoi-To Wai, Zhuoran Yang, Zhaoran Wang, Mingyi Hong

Keywords Paper

0

0

0

0

3:40

06/12/2020

On the Tightness of Semidefinite Relaxations for Certifying Robustness to Adversarial Examples

Richard Zhang

Keywords Paper

0

0

0

0

3:13

06/12/2020

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

Raphaël Berthier, Francis Bach, Pierre Gaillard

Keywords Paper

Optimization -> Non-Convex Optimization, Deep Learning -> Optimization for Deep Networks

0

0

0

0

3:05

12/07/2020

Accelerated Stochastic Gradient-free and Projection-free Methods

Feihu Huang, Lue Tao, Songcan Chen

Keywords Paper

Optimization - Non-convex

0

0

0

0

13:05

06/12/2021

Adversarial Examples in Multi-Layer Random ReLU Networks

Peter Bartlett, Sebastien Bubeck, Yeshwanth Cherapanamjeri

Keywords Paper

theory, adversarial robustness and security

0

0

0

0

10:49

06/12/2021

Implicit Regularization in Matrix Sensing via Mirror Descent

Fan Wu, Patrick Rebeschini

Keywords Paper

optimization

0

0

0

0

9:35

08/07/2020

Contraction: a Unified Perspective of Correlation Decay and Zero-Freeness of 2-Spin Systems

Shuai Shao and Yuxin Sun

Keywords Paper

2-Spin system, Correlation decay, Zero-freeness, Phase transition, Contraction

0

0

0

0

25:04

06/12/2020

Quantitative Propagation of Chaos for SGD in Wide Neural Networks

Valentin De Bortoli, Alain Durmus, Xavier Fontaine, Umut Simsekli

Keywords Paper

0

0

0

0

3:29

26/04/2020

Pure and Spurious Critical Points: a Geometric Study of Linear Networks

Matthew Trager, Kathlén Kohn, Joan Bruna

Keywords Paper

Loss landscape, linear networks, algebraic geometry

0

0

0

0

5:22

26/04/2020

SNODE: Spectral Discretization of Neural ODEs for System Identification

Alessio Quaglino, Marco Gallieri, Jonathan Masci, Jan Koutník

Keywords Paper

Recurrent neural networks, system identification, neural ODEs

0

0

0

0

5:00

18/07/2021

Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks

Hao Liu, Minshuo Chen, Tuo Zhao, Wenjing Liao

Keywords Paper

Applications, Computer Vision, , Theory, Deep learning Theory

0

0

0

0

5:14

18/07/2021

Bayesian Deep Learning via Subnetwork Inference

Erik Daxberger, Eric Nalisnick, James Allingham and
Javier Antorán, Jose Miguel Hernandez-Lobato

Keywords Paper

, Reinforcement Learning and Planning, Multi-Agent RL, Deep Learning, Bayesian Deep Learning

0

0

0

0

5:18