02/02/2021

A Trace-restricted Kronecker-Factored Approximation to Natural Gradient

Kaixin Gao, Xiaolei Liu, Zhenghai Huang, Min Wang, Zidong Wang, Dachuan Xu, Fan Yu

Keywords:

Abstract Paper Similar Papers

Abstract: Second-order optimization methods have the ability to accelerate convergence by modifying the gradient through the curvature matrix. There have been many attempts to use second-order optimization methods for training deep neural networks. In this work, inspired by diagonal approximations and factored approximations such as Kronecker-factored Approximate Curvature (KFAC), we propose a new approximation to the Fisher information matrix (FIM) called Trace-restricted Kronecker-factored Approximate Curvature (TKFAC), which can hold the certain trace relationship between the exact and the approximate FIM. In TKFAC, we decompose each block of the approximate FIM as a Kronecker product of two smaller matrices and scaled by a coefficient related to trace. We theoretically analyze TKFAC's approximation error and give an upper bound of it. We also propose a new damping technique for TKFAC on convolutional neural networks to maintain the superiority of second-order optimization methods during training. Experiments show that our method has better performance compared with several state-of-the-art algorithms on some deep network architectures.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948405
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 3:13 
 0
 0
 0
 0
 12:27 
 0
 0
 0
 0
 12:15 
 0
 0
 0
 0
 14:59 
 0
 0
 0
 0
 14:59 
 0
 0
 0
 0
 11:18 
 0
 0
 0
 0
 5:07 
 0
 0
 0
 0
 14:18 
 0
 0
 0
 0
 3:05 
 0
 0
 0
 0
 14:46 
 0
 0
 0
 0
 5:25 
 0
 0
 0
 0
 3:20 
 0
 0
 0
 0
 12:59 
 0
 0
 0
 0
 10:02 
 0
 0
 0
 1
 12:45 
 0
 0
 0
 0
 13:48 
 0
 0
 0
 0
 17:10 
 0
 0
 0
 0
 3:18 
 0
 0
 0
 0
 9:51 
 0
 0
 0
 0
 5:10 
 0
 0
 0
 0
 9:59 
 0
 0
 0
 0
 4:48 
 0
 0
 0
 0
 8:56 
 0
 0
 0
 0
 5:17 
 0
 0
 0
 0
 18:48 
 0
 0
 0
 0
 3:09 
 0
 0
 0
 0
 13:21 
 0
 0
 0
 0
 18:46 
 0
 0
 0
 0
 9:58 
 0
 0
 0
 0
 11:14 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 15:08 
 0
 0
 0
 0
 14:42 
 0
 0
 0
 0
 10:30 
 0
 0
 0
 0
 14:55 
 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 14:09 
 0
 0
 0
 0
 5:02 
 0
 0
 0
 0
 6:00