02/02/2021

On Convergence of Gradient Expected Sarsa(λ)

Long Yang, Gang Zheng, Yu Zhang, Qian Zheng, Pengfei Li, Gang Pan

Keywords:

Abstract Paper Similar Papers

Abstract: We study the convergence of Expected Sarsa(λ) with function approximation. We show that with off-line es- timate (multi-step bootstrapping) to ExpectedSarsa(λ) is unstable for off-policy learning. Furthermore, based on convex-concave saddle-point framework, we propose a con- vergent Gradient Expected Sarsa(λ) (GES(λ)) algorithm. The theoretical analysis shows that the proposed GES(λ) converges to the optimal solution at a linear convergence rate under true gradient setting. Furthermore, we develop a Lyapunov function technique to investigate how the step- size influences finite-time performance of GES(λ). Addition- ally, such a technique of Lyapunov function can be poten- tially generalized to other gradient temporal difference algo- rithms. Finally, our experiments verify the effectiveness of our GES(λ). For the details of proof, please refer to https: //arxiv.org/pdf/2012.07199.pdf.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948258
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:02 
 0
 0
 0
 0
 10:02 
 0
 0
 0
 0
 2:53 
 0
 0
 0
 1
 3:14 
 0
 0
 0
 0
 15:15 
 0
 0
 0
 0
 2:28 
 0
 1
 1
 0
 12:25 
 0
 0
 0
 0
 3:26 
 0
 0
 0
 0
 3:09 
 0
 0
 0
 0
 8:30 
 0
 0
 0
 0
 3:21 
 0
 0
 0
 0
 2:46 
 0
 0
 0
 0
 14:28 
 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 11:47 
 0
 0
 0
 0
 3:27 
 0
 0
 0
 0
 17:35 
 0
 0
 0
 0
 3:00 
 0
 0
 0
 0
 4:50 
 1
 1
 0
 0
 9:31 
 0
 0
 0
 0
 3:14 
 0
 0
 0
 0
 4:51 
 0
 0
 0
 0
 2:52 
 0
 0
 0
 0
 14:13 
 0
 0
 0
 0
 14:49 
 0
 0
 0
 0
 4:23 
 0
 0
 0
 0
 9:44 
 0
 0
 0
 0
 3:20 
 0
 0
 0
 0
 8:36 
 0
 0
 0
 0
 2:44 
 0
 0
 0
 0
 3:11 
 0
 0
 0
 0
 3:24 
 0
 0
 0
 0
 14:43 
 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 4:42 
 0
 0
 0
 0
 16:50 
 0
 0
 0
 0
 15:53 
 0
 0
 0
 0
 13:41 
 0
 0
 0
 0
 3:05 
 0
 0
 0
 0
 14:55