04/08/2021

When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?

Niladri S Chatterji, Philip M Long, Peter Bartlett

Keywords:

Abstract Paper Similar Papers

Abstract: We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero, and prove bounds on the rate of convergence. Our analysis applies for smoothed approximations to the ReLU, such as Swish and the Huberized ReLU, proposed in previous applied work. We provide two sufficient conditions for convergence. The first is simply a bound on the loss at initialization. The second is a data separation condition used in prior analyses.

 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at COLT 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 8:12 
 0
 0
 0
 1
 3:14 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 14:22 
 0
 0
 0
 0
 15:10 
 0
 0
 0
 0
 3:37 
 0
 0
 0
 0
 15:15 
 0
 0
 0
 0
 9:07 
 0
 0
 0
 0
 13:55 
 0
 0
 0
 0
 3:27 
 0
 0
 0
 0
 11:27 
 0
 0
 0
 0
 15:27 
 0
 0
 0
 0
 15:40 
 0
 0
 0
 0
 10:02 
 0
 0
 0
 0
 3:30 
 0
 0
 0
 0
 2:53 
 0
 0
 0
 0
 2:30 
 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 2:52 
 0
 0
 0
 0
 3:07 
 0
 0
 0
 0
 15:29 
 0
 0
 0
 0
 14:15 
 0
 0
 0
 0
 13:49 
 0
 0
 0
 0
 13:13 
 0
 0
 0
 0
 11:47 
 0
 0
 0
 0
 3:21 
 0
 0
 0
 0
 14:55 
 0
 0
 0
 0
 17:35 
 0
 0
 0
 0
 15:44 
 0
 0
 0
 0
 15:32 
 0
 0
 0
 0
 15:56 
 0
 0
 0
 0
 5:37 
 0
 0
 0
 0
 14:46 
 0
 0
 0
 0
 15:47 
 0
 0
 0
 0
 3:14 
 0
 0
 0
 0
 4:48 
 0
 0
 0
 0
 3:31 
 0
 1
 1
 0
 12:25 
 0
 0
 0
 0
 4:56