02/02/2021

Advice-Guided Reinforcement Learning in a non-Markovian Environment

Daniel Neider, Jean-Raphael Gaglione, Ivan Gavran, Ufuk Topcu, Bo Wu, Zhe Xu

Keywords:

Abstract Paper Similar Papers

Abstract: We study a class of reinforcement learning tasks in which the agent receives its reward for complex, temporally-extended behaviors sparsely. For such tasks, the problem is how to augment the state-space so as to make the reward function Markovian in an efficient way. While some existing solutions assume that the reward function is explicitly provided to the learning algorithm (e.g., in the form of a reward machine), the others learn the reward function from the interactions with the environment, assuming no prior knowledge provided by the user. In this paper, we generalize both approaches and enable the user to give advice to the agent, representing the user’s best knowledge about the reward function, potentially fragmented, partial, or even incorrect. We formalize advice as a set of DFAs and present a reinforcement learning algorithm that takes advantage of such advice, with optimal con- vergence guarantee. The experiments show that using well- chosen advice can reduce the number of training steps needed for convergence to optimal policy, and can decrease the computation time to learn the reward function by up to two orders of magnitude.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948141
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 14:17 
 0
 0
 0
 0
 13:11 
 0
 0
 0
 0
 14:14 
 0
 0
 0
 0
 11:47 
 0
 0
 0
 0
 14:52 
 0
 0
 0
 0
 10:52 
 0
 0
 0
 0
 12:21 
 0
 0
 0
 0
 5:19 
 0
 0
 0
 0
 7:50 
 0
 0
 0
 0
 5:14 
 0
 0
 0
 0
 7:57 
 0
 0
 0
 0
 15:21 
 0
 0
 0
 0
 3:19 
 0
 0
 0
 0
 5:12 
 0
 0
 0
 0
 3:59 
 0
 0
 0
 0
 15:18 
 0
 0
 0
 0
 3:09 
 0
 0
 0
 0
 14:31 
 0
 0
 0
 0
 14:47 
 0
 0
 0
 0
 4:31 
 0
 0
 0
 0
 3:20 
 0
 0
 0
 0
 15:53 
 0
 0
 0
 0
 15:40 
 0
 0
 0
 0
 9:50 
 0
 0
 0
 0
 5:04 
 0
 0
 0
 0
 10:21 
 0
 0
 0
 0
 3:13 
 0
 0
 0
 0
 3:16 
 0
 0
 0
 0
 5:58 
 0
 0
 0
 0
 4:10 
 0
 0
 0
 0
 3:09 
 0
 0
 0
 0
 14:49 
 0
 0
 0
 0
 9:41 
 0
 0
 0
 0
 9:55 
 0
 0
 0
 0
 14:57 
 0
 0
 0
 0
 5:09 
 0
 0
 0
 0
 5:20 
 0
 0
 0
 0
 9:06 
 0
 0
 0
 0
 2:42 
 0
 0
 0
 0
 4:38