RL Agents with Commonsense Knowledge from Scene Graphs

Abstract

Text-based games are becoming commonly used in reinforcement learning and natural language processing. To challenge these games, it is effective to make the search of the action space more efficient by giving common sense. However, such knowledge has only been available from textual information with a high level of abstraction in previous works. In this paper, we propose to employ common sense reasoning obtained from visual datasets such as scene graph datasets. In general, images convey more comprehensive information compared with text for humans. This property enables to extract commonsense relationship knowledge more useful for acting effectively in a game. We also experimented with our proposed methods on a text-based game task that requires commonsense reasoning. Our experimental results showed that our proposed methods have higher and competitive performance than existing state-of-the-art methods.