Published on Tue Nov 19 2019

Implicit Generative Modeling for Efficient Exploration

Neale Ratzlaff, Qinxun Bai, Li Fuxin, Wei Xu

Efficient exploration remains a challenging problem in reinforcement learning. A commonly used approach for exploring environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration.

0
0
0
Abstract

Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration. In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent's belief of the environment dynamics. Each random draw from our generative model is a neural network that instantiates the dynamic function, hence multiple draws would approximate the posterior, and the variance in the future prediction based on this posterior is used as an intrinsic reward for exploration. We design a training algorithm for our generative model based on the amortized Stein Variational Gradient Descent. In experiments, we compare our implementation with state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches based on an ensemble of dynamic models. In challenging exploration tasks, our implicit generative model consistently outperforms competing approaches regarding data efficiency in exploration.

Tue May 31 2016
Artificial Intelligence
VIME: Variational Information Maximizing Exploration
Scalable and effective exploration remains a key challenge in reinforcement learning (RL) Most contemporary RL relies on simple heuristics such as epsilon-greedy exploration or adding Gaussian noise to the controls. This paper introduces Variational Information Maximizing Exploration (VIME)
0
0
0
Tue Nov 29 2016
Artificial Intelligence
Exploration for Multi-task Reinforcement Learning with Deep Generative Models
Exploration in multi-task reinforcement learning is critical in training agents to deduce the underlying MDP. Many of the existing exploration frameworks assume a single stationary MDP and are not suitable for system identification. We present a novel method to facilitate exploration using deep generative models.
0
0
0
Mon Jun 11 2018
Artificial Intelligence
The Potential of the Return Distribution for Exploration in RL
This paper studies the potential of the return distribution for exploration in deterministic reinforcement learning (RL) environments. We study network losses and propagation mechanisms for Gaussian, Categorical and Gaussian mixture distributions. Combined with exploration policies that leverage this return distribution, we solve a randomized Chain
0
0
0
Sat Dec 21 2019
Machine Learning
Can Agents Learn by Analogy? An Inferable Model for PAC Reinforcement Learning
Model-based reinforcement learning algorithms make decisions by building and utilizing a model of the environment. We propose a new model-based method called Greedy Inference Model (GIM) that infers the unknown dynamics from known dynamics. In other words, GIM can "learn by analogy"
0
0
0
Tue Feb 20 2018
Neural Networks
Meta-Reinforcement Learning of Structured Exploration Strategies
Exploration is a fundamental challenge in reinforcement learning (RL) We explore how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based adaptation algorithm to learn exploration strategies from prior experience.
0
0
0
Mon Oct 29 2018
Neural Networks
Model-Based Active Exploration
Efficient exploration is an unsolved problem in Reinforcement Learning. This paper introduces an efficient active exploration algorithm. It uses an ensemble of forward models to plan to observe novel events.
0
0
0