02/02/2021

Gradient Descent Averaging and Primal-dual Averaging for Strongly Convex Optimization

Wei Tao, Wei Li, Zhisong Pan, Qing Tao

Keywords:

Abstract Paper Similar Papers

Abstract: Averaging scheme has attracted extensive attention in deep learning as well as traditional machine learning. It achieves theoretically optimal convergence and also improves the empirical model performance. However, there is still a lack of sufficient convergence analysis for strongly convex optimization. Typically, the convergence about the last iterate of gradient descent methods, which is referred to as individual convergence, fails to attain its optimality due to the existence of logarithmic factor. In order to remove this factor, we first develop gradient descent averaging (GDA), which is a general projection-based dual averaging algorithm in the strongly convex setting. We further present primal-dual averaging for strongly convex cases (SC-PDA), where primal and dual averaging schemes are simultaneously utilized. We prove that GDA yields the optimal convergence rate in terms of output averaging, while SC-PDA derives the optimal individual convergence. Several experiments on SVMs and deep learning models validate the correctness of theoretical analysis and effectiveness of algorithms.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948051
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 14:30 
 0
 0
 0
 0
 13:18 
 0
 0
 0
 0
 14:47 
 0
 0
 0
 0
 3:02 
 0
 0
 0
 0
 3:12 
 0
 0
 0
 0
 4:49 
 0
 0
 0
 0
 12:07 
 0
 0
 0
 0
 3:05 
 0
 0
 0
 0
 3:11 
 0
 0
 0
 0
 5:16 
 0
 0
 0
 0
 5:13 
 0
 0
 1
 1
 16:17 
 0
 0
 0
 0
 14:05 
 0
 0
 0
 0
 14:58 
 0
 0
 0
 0
 19:14 
 0
 0
 0
 0
 3:02 
 0
 0
 0
 0
 18:12 
 0
 0
 0
 0
 10:21 
 0
 0
 0
 0
 5:15 
 0
 0
 0
 0
 12:59 
 0
 0
 0
 0
 2:58 
 0
 0
 0
 0
 5:07 
 0
 0
 0
 0
 5:47 
 0
 0
 0
 0
 3:26 
 0
 0
 0
 0
 15:14 
 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 3:11 
 0
 0
 0
 0
 5:03 
 0
 0
 0
 0
 2:53 
 0
 0
 0
 0
 13:08 
 0
 0
 0
 0
 15:53 
 0
 0
 0
 0
 13:48 
 0
 0
 0
 1
 3:28 
 0
 0
 0
 0
 5:10 
 0
 0
 0
 0
 14:56 
 0
 0
 0
 0
 14:10 
 0
 0
 0
 0
 13:21 
 0
 0
 0
 0
 3:12 
 0
 0
 0
 0
 15:10 
 0
 0
 0
 0
 3:11