Simultaneous Generation of Daily Life Videos and Event-Centric Knowledge Graphs in the Home

Toward the Knowledge Graph Reasoning Challenge for Social Issues

Abstract

As society ages, it is expected that AI systems will be developed that can assist with household chores and prevent accidents. However, there is a lack of datasets that map the semantic relationship between daily life videos and their contents, hindering the development of AI technology that integrates visual and symbolic information. We have previously developed "VirtualHome2KG," a system that generates videos of daily life using a virtual space and generates event-centered knowledge graphs of their contents. In this paper, we extended VirtualHome2KG to generate a wider variety of data. Specifically, (1) more than 30 types of executable actions have been added, and (2) a function for generating 2D bounding boxes for objects in images have been added. This has enabled us to construct a paired dataset of video and knowledge graphs for more complex inference tasks that are closer to those in the real world than before. Furthermore, we have opened this to the public and held the "Knowledge Graph Reasoning Challenge for Social Issues,'' calling for systems that can detect and explain dangerous situations in the home. In this paper, we describe our expectations for social implementation of this technology by bringing together various approaches.