02/02/2021

Expected Eligibility Traces

Hado van Hasselt, Sephora Madjiheurem, Matteo Hessel, David Silver, André Barreto, Diana Borsa

Keywords:

Abstract Paper Similar Papers

Abstract: The question of how to determine which states and actions are responsible for a certain outcome is known as the credit assignment problem and remains a central research question in reinforcement learning and artificial intelligence. Eligibility traces enable efficient credit assignment to the recent sequence of states and actions experienced by the agent, but not to counterfactual sequences that could also have led to the current state. In this work, we introduce expected eligibility traces. Expected traces allow, with a single update, to update states and actions that could have preceded the current state, even if they did not do so on this occasion. We discuss when expected traces provide benefits over classic (instantaneous) traces in temporal-difference learning, and show that some- times substantial improvements can be attained. We provide a way to smoothly interpolate between instantaneous and expected traces by a mechanism similar to bootstrapping, which ensures that the resulting algorithm is a strict generalisation of TD(λ). Finally, we discuss possible extensions and connections to related ideas, such as successor features.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38949366
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 9:48 
 0
 0
 0
 0
 20:06 
 0
 0
 0
 0
 14:06 
 0
 0
 0
 0
 12:40 
 0
 0
 0
 0
 15:00 
 0
 0
 0
 0
 12:57 
 0
 0
 0
 0
 14:41 
 0
 0
 0
 0
 21:04 
 0
 0
 0
 0
 8:58 
 1
 0
 0
 0
 5:06 
 0
 0
 0
 0
 15:12 
 1
 0
 0
 0
 15:47 
 0
 0
 0
 0
 13:45 
 0
 0
 0
 0
 11:24 
 0
 0
 0
 0
 5:43 
 0
 0
 0
 0
 10:00 
 0
 0
 0
 0
 6:51 
 0
 0
 0
 0
 4:17 
 0
 0
 0
 0
 3:15 
 0
 0
 0
 0
 16:45 
 0
 0
 0
 0
 10:28 
 0
 0
 0
 0
 17:51 
 0
 0
 0
 0
 3:02 
 0
 0
 0
 0
 5:36 
 0
 0
 0
 0
 11:27 
 0
 0
 0
 0
 5:02 
 1
 0
 0
 0
 8:31 
 0
 0
 0
 0
 11:27 
 1
 0
 0
 0
 12:13 
 0
 0
 0
 0
 2:44 
 0
 0
 0
 0
 2:30 
 0
 0
 0
 0
 5:18 
 0
 0
 0
 0
 11:18 
 0
 0
 0
 0
 5:13 
 0
 0
 0
 0
 3:19 
 0
 0
 0
 0
 16:11 
 0
 0
 0
 0
 15:19 
 0
 0
 0
 0
 6:30 
 0
 0
 0
 0
 5:18 
 0
 0
 0
 0
 4:51