Lipschitz Lifelong Reinforcement Learning

02/02/2021

Lipschitz Lifelong Reinforcement Learning

Erwan Lecarpentier, David Abel, Kavosh Asadi, Yuu Jinnai, Emmanuel Rachelson, Michael L. Littman

Keywords:

Abstract Paper Similar Papers

Abstract: We consider the problem of knowledge transfer when an agent is facing a series of Reinforcement Learning (RL) tasks. We introduce a novel metric between Markov Decision Processes and establish that close MDPs have close optimal value functions. Formally, the optimal value functions are Lipschitz continuous with respect to the tasks space. These theoretical results lead us to a value-transfer method for Lifelong RL, which we use to build a PAC-MDP algorithm with improved convergence rate. Further, we show the method to experience no negative transfer with high probability. We illustrate the benefits of the method in Lifelong RL experiments.

The video of this talk cannot be embedded. You can watch it here:

https://slideslive.com/38948232

(Link will open in new window)

1

1

1

0

Share

This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment

no comments yet

Similar Papers

06/12/2021

Learning Markov State Abstractions for Deep Reinforcement Learning

Cameron Allen, Neev Parikh, Omer Gottesman, George Konidaris

Keywords Paper

reinforcement learning and planning, contrastive learning, representation learning

0

0

0

0

12:31

03/08/2020

No-regret Exploration in Contextual Reinforcement Learning

Aditya Modi, Ambuj Tewari

Keywords Paper

0

0

0

0

8:19

18/07/2021

Provably Efficient Algorithms for Multi-Objective Competitive RL

Tiancheng Yu, Yi Tian, Jingzhao Zhang, Suvrit Sra

Keywords Paper

Theory, RL, Decisions and Control Theory

0

0

0

0

17:04

06/12/2020

Learning from Aggregate Observations

Yivan Zhang, Nontawat Charoenphakdee, Zhenguo Wu, Masashi Sugiyama

Keywords Paper

0

0

0

0

3:21

12/07/2020

Sequential Transfer in Reinforcement Learning with a Generative Model

Andrea Tirinzoni, Riccardo Poiani, Marcello Restelli

Keywords Paper

Reinforcement Learning - General

0

0

0

0

10:54

06/12/2020

A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions

Wei Deng, Guang Lin, Faming Liang

Keywords Paper

0

0

0

0

3:26

06/12/2021

Average-Reward Learning and Planning with Options

Yi Wan, Abhishek Naik, Rich Sutton

Keywords Paper

reinforcement learning and planning

0

0

0

0

14:35

18/07/2021

Cross-Gradient Aggregation for Decentralized Learning from Non-IID Data

Yasaman Esfandiari, Sin Yong Tan, Zhanhong Jiang and
Aditya Balu, Ethan Herron, Chinmay Hegde, Soumik Sarkar

Keywords Paper

Optimization, Distributed and Parallel Optimization

1

0

0

1

5:15

06/12/2021

Gradient Starvation: A Learning Proclivity in Neural Networks

Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio and
Aaron Courville, Doina Precup, Guillaume Lajoie

Keywords Paper

theory, deep learning, optimization, robustness

0

0

0

0

10:52

06/12/2021

Offline Reinforcement Learning as One Big Sequence Modeling Problem

Michael Janner, Qiyang Li, Sergey Levine

Keywords Paper

reinforcement learning and planning, transformers, language

0

0

0

0

9:48

26/04/2020

Sharing Knowledge in Multi-Task Deep Reinforcement Learning

Carlo D'Eramo, Davide Tateo, Andrea Bonarini and
Marcello Restelli, Jan Peters

Keywords Paper

Deep Reinforcement Learning, Multi-Task

0

0

0

0

4:27

18/07/2021

Stochastic Sign Descent Methods: New Algorithms and Better Theory

Mher Safaryan, Peter Richtarik

Keywords Paper

Optimization, Distributed and Parallel Optimization

0

0

0

0

5:12

18/07/2021

A New Representation of Successor Features for Transfer across Dissimilar Environments

Majid Abdolshah, Hung Le, Thommen Karimpanal George and
Sunil Gupta, Santu Rana, Svetha Venkatesh

Keywords Paper

Reinforcement Learning and Planning

0

0

0

0

5:43

06/12/2021

Adversarial Intrinsic Motivation for Reinforcement Learning

Ishan Durugkar, Mauricio Tec, Scott Niekum, Peter Stone

Keywords Paper

reinforcement learning and planning, generative model

0

0

0

0

13:11

18/07/2021

Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time

Weichen Wang, Jiequn Han, Zhuoran Yang, Zhaoran Wang

Keywords Paper

Theory

0

0

0

0

5:03

18/07/2021

A Wasserstein Minimax Framework for Mixed Linear Regression

Theo Diamandis, Yonina Eldar, Alireza Fallah and
Farzan Farnia, Asuman Ozdaglar

Keywords Paper

Algorithms, Multimodal Learning

0

0

0

0

25:41

26/08/2020

Finite-Time Analysis of Decentralized Temporal-Difference Learning with Linear Function Approximation

Jun Sun, Gang Wang, Georgios B. Giannakis and
Qinmin Yang, Zaiyue Yang

Keywords Paper

0

0

0

0

17:07

18/07/2021

Variational Auto-Regressive Gaussian Processes for Continual Learning

Sanyam Kapoor, Theofanis Karaletsos, Thang Bui

Keywords Paper

Probabilistic Methods, Gaussian Processes and Bayesian non-parametrics

0

0

0

0

5:12

03/05/2021

CPR: Classifier-Projection Regularization for Continual Learning

Sungmin Cha, Hsiang Hsu, Taebaek Hwang and
Flavio Calmon, Taesup Moon

Keywords Paper

regularization, wide local minima, continual learning

0

0

0

1

5:21

18/07/2021

Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap

Gokul Swamy, Sanjiban Choudhury, J. Bagnell, Steven Wu

Keywords Paper

Theory, RL, Decisions and Control Theory

0

0

0

0

5:12

06/12/2020

Decentralized Langevin Dynamics for Bayesian Learning

Anjaly Parayil, He Bai, Jemin George, Prudhvi Gurram

Keywords Paper

0

0

0

0

3:17

06/12/2021

A Max-Min Entropy Framework for Reinforcement Learning

Seungyul Han, Youngchul Sung

Keywords Paper

optimization, reinforcement learning and planning

0

0

0

0

14:35

03/05/2021

Learning Robust State Abstractions for Hidden-Parameter Block MDPs

Amy Zhang, Shagun Sodhani, Khimya Khetarpal, Joelle Pineau

Keywords Paper

bisimulation, block mdp, hidden-parameter mdp, multi-task reinforcement learning

0

0

0

0

4:17

06/12/2020

Generalised Bayesian Filtering via Sequential Monte Carlo

Ayman Boustati, Omer Deniz Akyildiz, Theo Damoulas, Adam Johansen

Keywords Paper

0

0

0

0

2:52

06/12/2021

Inverse Reinforcement Learning in a Continuous State Space with Formal Guarantees

Gregory Dexter, Kevin Bello, Jean Honorio

Keywords Paper

theory, reinforcement learning and planning

0

0

0

0

14:49

18/07/2021

Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees

Kishan Panaganti, Dileep Kalathil

Keywords Paper

Theory, RL, Decisions and Control Theory

0

0

0

0

5:15

06/12/2020

Deep Reinforcement and InfoMax Learning

Bogdan Mazoure, Remi Tachet des Combes, Thang Doan and
Philip Bachman, R Devon Hjelm

Keywords Paper

0

0

0

0

3:15

26/04/2020

Frequency-based Search-control in Dyna

Yangchen Pan, Jincheng Mei, Amir-massoud Farahmand

Keywords Paper

Model-based reinforcement learning, search-control, Dyna, frequency of a signal

0

0

0

0

4:32

12/07/2020

Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning

Qing Li, Siyuan Huang, Yining Hong and
Yixin Chen, Ying Nian Wu, Song-Chun Zhu

Keywords Paper

Applications - Computer Vision

0

0

0

0

14:01

04/07/2020

Effective Estimation of Deep Generative Language Models

Tom Pelsmaeker, Wilker Aziz

Keywords Paper

Estimation Models, parameterisation models, posterior collapse, language modelling

0

0

0

0

12:19

13/04/2021

Optimal quantisation of probability measures using maximum mean discrepancy

Onur Teymur, Jackson Gorham, Marina Riabiz, Chris Oates

Keywords Paper

0

0

0

0

2:57

02/02/2021

Dynamic Automaton-Guided Reward Shaping for Monte Carlo Tree Search

Alvaro Velasquez, Brett Bissey, Lior Barak and
Andre Beckus, Ismail Alkhouri, Daniel Melcer, George Atia

Keywords Paper

0

0

0

0

18:52

13/04/2021

Adversarially robust estimate and risk analysis in linear regression

Yue Xing, Ruizhi Zhang, Guang Cheng

Keywords Paper

0

0

0

0

3:03

13/04/2021

Logistic q-learning

Joan Bas-Serrano, Sebastian Curi, Andreas Krause, Gergely Neu

Keywords Paper

0

0

0

0

2:44

13/04/2021

Convergence and accuracy trade-offs in federated learning and meta-learning

Zachary Charles, Jakub Konečný

Keywords Paper

0

0

0

0

3:04

06/12/2020

Latent Bandits Revisited

Joey Hong, Branislav Kveton, Manzil Zaheer and
Yinlam Chow, Amr Ahmed, Craig Boutilier

Keywords Paper

0

0

0

0

3:11

06/12/2021

On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning

Alireza Fallah, Kristian Georgiev, Aryan Mokhtari, Asuman Ozdaglar

Keywords Paper

theory, optimization, reinforcement learning and planning, meta learning

0

1

1

0

12:25

26/08/2020

Finite-Time Error Bounds for Biased Stochastic Approximation with Applications to Q-Learning

Gang Wang, Georgios B. Giannakis

Keywords Paper

0

0

0

0

14:03

04/07/2020

Estimating the influence of auxiliary tasks for multi-task learning of sequence tagging tasks

Fynn Schröder, Chris Biemann

Keywords Paper

multi-task tasks, MTL, TL, MTL setups

0

0

0

0

12:02

26/04/2020

PAC Confidence Sets for Deep Neural Networks via Calibrated Prediction

Sangdon Park, Osbert Bastani, Nikolai Matni, Insup Lee

Keywords Paper

PAC, confidence sets, classification, regression, reinforcement learning

0

0

0

0

5:02