Model-Based Reinforcement Learning for Learning Deterministic Policies

Abstract

Reinforcement Learning (RL) is a trial and error process where a robot interacts with its environment using a stochastic policy. For example, it is realized by adding Gaussian noise to a deterministic policy. Therefore, a straightforward application of RL to robot control tasks is often problematic because the stochastic policy does not produce smooth behaviors. We propose model-based reinforcement learning for learning a deterministic policy to overcome this issue. First, we formulate the RL algorithm with entropy regularization of the model. In this formulation, the robot explores the environment based on the simulated environmental uncertainty. We utilize the stochastic value gradient method for this formulation. Then, we derive a model learning algorithm inspired by density ratio estimation. Our proposed method is evaluated on three benchmark tasks provided by the DeepMind Control Suite, and the experimental results show that our method can produce smooth behaviors and outperform the other baselines.