Published on Wed Mar 24 2021

Discriminator Augmented Model-Based Reinforcement Learning

Behzad Haghgoo, Allan Zhou, Archit Sharma, Chelsea Finn
0
0
0
Abstract

By planning through a learned dynamics model, model-based reinforcement learning (MBRL) offers the prospect of good performance with little environment interaction. However, it is common in practice for the learned model to be inaccurate, impairing planning and leading to poor performance. This paper aims to improve planning with an importance sampling framework that accounts and corrects for discrepancy between the true and learned dynamics. This framework also motivates an alternative objective for fitting the dynamics model: to minimize the variance of value estimation during planning. We derive and implement this objective, which encourages better prediction on trajectories with larger returns. We observe empirically that our approach improves the performance of current MBRL algorithms on two stochastic control problems, and provide a theoretical basis for our method.

Tue Dec 24 2019
Machine Learning
Learning to Combat Compounding-Error in Model-Based Reinforcement Learning
Model-based reinforcement learning can fail catastrophically if the model is inaccurate. An algorithm should ideally be able to trust an imperfect model over a reasonably long planning horizon. The proposed method can successfully adapt the planning horizon to account for state-dependent model accuracy.
0
0
0
Sat Apr 17 2021
Artificial Intelligence
Planning with Expectation Models for Control
0
0
0
Sat Oct 12 2019
Machine Learning
Regularizing Model-Based Planning with Energy-Based Models
Model-based reinforcement learning could enable sample-efficient learning. We show that off-policy training of an energy estimator can be effectively used to regularize planning with pre-trained dynamics models.
0
0
0
Mon May 14 2012
Artificial Intelligence
Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search
Bayesian model-based reinforcement learning is a formally elegant approach to learning optimal behaviour under model uncertainty. Finding the resulting Bayes-optimal policies is notoriously taxing, since the search space becomes monstrous. In this paper we introduce a tractable, sample-based method for roughly approximating
0
0
0
Thu Aug 15 2019
Artificial Intelligence
Model-based Lookahead Reinforcement Learning
Model-based Reinforcement Learning (MBRL) allows data-efficient learning which is required in real world applications such as robotics. MBRL does not achieve the final performance of state-of-the-art Model-free Rein reinforcement Learning (MFRL) methods.
0
0
0
Wed Dec 12 2012
Machine Learning
Reinforcement Learning with Partially Known World Dynamics
Most problems have both hidden state and unknown dynamics. Partially observable Markov decision processes (POMDPs) allow for the modeling of both.
0
0
0