振舞複製による敵対的模倣学習高速化に関する考察

抄録

Imitation learning is a popular method to obtain policies on autonomous robots given expert demonstrations. Recently, adversarial imitation learning methods, such as generative adversarial imitation learning (GAIL), have achieved great successes even on complex continuous control tasks. However, GAIL as well as its variants require a huge amount of environment interactions that often take impractically long time for training the robot. An intuitive way to reduce the number of interactions is initializing a policy by behavioral cloning (BC) before performing GAIL as pointed out in [1]. However, Sasaki et al reports that the BC initialization does not lead to reduce the number of interactions at all, rather significantly harms the imitation results. In this paper, we further analyze the BC initialization to figure out why the results are opposed to the intuition. Experimental results show that one of the cause of failure due to the BC initialization is that BC vanishes gradients of objective functions for the adversarial imitation learning algorithms, even though the objective differs from that of BC.