Offline Model-Based Imitation Learning with Entropy Regularization of Model and Policy

Abstract

Model-Based Entropy-Regularized Imitation Learning (MB-ERIL) is an online model-based generative adversarial imitation learning method that introduces entropy regularization of policy and state transition model. Online-MB-ERIL learns the policy and model from expert data, learner's data, and generated data. Costly interactions with an actual environment are needed to obtain the first two datasets, while the policy and model quickly generate the last one. This report discusses an offline learning setting without using the second data obtained from the interaction between the policy and the actual environment. Next, we propose Offline-MB-ERIL, which introduces the idea of Positive and Unlabeled data learning. Given sub-optimal data, Offline-MB-ERIL can recover policy and model efficiently using them as unlabeled data. Through a vision-based arm-reaching task, we show that Offline-MB-ERIL can better use suboptimal data than Online-MB-ERIL.