Published on Wed Mar 24 2021

On Imitation Learning of Linear Control Policies: Enforcing Stability and Robustness Constraints via LMI Conditions

Aaron Havens, Bin Hu

In this paper, we formulate the.imitation learning of linear policies as a constrained optimization problem. We present efficient methods which can be used to enforce stability and.robustness constraints during the learning processes.

1
0
1
Abstract

When applying imitation learning techniques to fit a policy from expert demonstrations, one can take advantage of prior stability/robustness assumptions on the expert's policy and incorporate such control-theoretic prior knowledge explicitly into the learning process. In this paper, we formulate the imitation learning of linear policies as a constrained optimization problem, and present efficient methods which can be used to enforce stability and robustness constraints during the learning processes. Specifically, we show that one can guarantee the closed-loop stability and robustness by posing linear matrix inequality (LMI) constraints on the fitted policy. Then both the projected gradient descent method and the alternating direction method of multipliers (ADMM) method can be applied to solve the resulting constrained policy fitting problem. Finally, we provide numerical results to demonstrate the effectiveness of our methods in producing linear polices with various stability and robustness guarantees.

Tue Mar 30 2021
Machine Learning
Learning Robust Feedback Policies from Demonstrations
0
0
0
Thu May 26 2016
Artificial Intelligence
Model-Free Imitation Learning with Policy Optimization
In imitation learning, an agent learns how to behave in an environment with an unknown cost function by mimicking expert demonstrations. Existing imitation learning algorithms typically involve solving a sequence of planning or reinforcement learning problems. We develop alternative model-free algorithms for finding a stochastic policy that performs at
0
0
0
Sat May 26 2018
Machine Learning
Fast Policy Learning through Imitation and Reinforcement
Imitation learning (IL) is a set of tools that leverage expert demonstrations to quickly learn policies. If the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL) We aim to provide an algorithm that combines the best aspects of RL and
0
0
0
Fri Jan 11 2019
Artificial Intelligence
On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator
We study the global convergence of generative adversarial imitation learning for linear quadratic regulators. We hope our results may serve as a small step toward understanding and taming the instability in imitation learning as well as in more general non-convex-concave alternating minimax optimization.
0
0
0
Tue Aug 10 2021
Machine Learning
Imitation Learning by Reinforcement Learning
Imitation Learning algorithms learn a policy from demonstrations of expert behavior. We show that, for deterministic experts,imitation learning can be done by reduction to reinforcement learning, which is commonly considered more difficult.
0
0
0
Tue Jun 23 2020
Artificial Intelligence
Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
Adversarial Imitation Learning alternates between learning a discriminator and a generator's policy. The alternated optimization is known to be delicate in practice. We propose to remove the burden of the policy optimization steps by leveraging a novel discriminator formulation.
0
0
0
Mon Nov 16 2020
Machine Learning
Enforcing robust control guarantees within neural network policies
Robust control methods often yield simple controllers that perform poorly in the average (non-worst) case. In contrast, nonlinear control methods using deep learning have achieved state-of-the-art performance. We propose combining the strengths of these two approaches.
1
34
162
Fri Nov 16 2018
Machine Learning
An Algorithmic Perspective on Imitation Learning
This article is the first of a two-part series on the subject of imitation learning. The second part of the series will focus on the development of new tools to help students learn more effectively.
0
0
0
Tue Nov 02 2010
Artificial Intelligence
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Sequential prediction problems such as imitation learning violate the common.i.d. assumptions made in statistical learning. We show that any such no regret algorithm, combined with additional reduction assumptions, must find a policy with good performance.
0
0
0
Fri Jun 10 2016
Artificial Intelligence
Generative Adversarial Imitation Learning
We propose a new general framework for directly extracting a policy from data. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks. From this we derive a model-free imitation learning algorithm that obtains significant performance gains.
0
0
0
Thu Feb 18 2021
Machine Learning
On the Sample Complexity of Stability Constrained Imitation Learning
We show that a surprisingly granular connection can be made between the underlying expert system's incremental gain stability and the sample-complexity of an imitation learning task. As a special case, we delineate a class of systems for which the number of trajectories needed to achieve sublinear
1
0
0
Thu Nov 07 2019
Machine Learning
Model-free Reinforcement Learning with Robust Stability Guarantee
Reinforcement learning is showing great potentials in robotics applications, including autonomous driving, robot manipulation and locomotion. With complex uncertainties in the real-world environment, it is difficult to guarantee successful generalization and sim-to-real transfer of learned policies theoretically.
0
0
0