The Impact of Demonstration Injection Timing from Pre-trained Agents on Learning Efficiency in Deep Reinforcement Learning

Abstract

Although deep reinforcement learning is attracting attention in various fields of modern society, many challenges remain for its practical application. The use of expert data generated by pre-trained agents has been studied as one of the solutions to these challenges, but the optimal timing of such intervention has not been sufficiently examined. In this study, using the Cart Pole environment and the DDQN algorithm, we analyzed the effect of utilizing expert data at different learning stages and investigated how the timing of expert intervention influences learning performance. Specifically, we introduced expert demonstrations generated by a trained agent at various learning phases and parameter settings, and then evaluated the agent’s performance. The experimental results confirmed that the timing of expert intervention significantly affects both the improvement of agent performance and the acceleration of learning convergence. In particular, it was suggested that using expert data from the early learning stage does not necessarily lead to optimal results, highlighting the importance of appropriate intervention timing.