The 2nd International Knowledge Graph Reasoning Challenge: Application of LLM to Predicting Behavior from Multimodal Data about Daily Life

Abstract

We held the final presentation of the 2nd International Knowledge Graph Inference Challenge on February 8, 2024 as a workshop in conjunction with the International Conference on Semantic Computing. The main task of the Challenge was to obtain statistics about actions, objects, and locations from videos and knowledge graphs generated using a 3D simulator of parts of daily life. The unique feature of this challenge is that it provides data with a missing part of the knowledge graph, and it is necessary to compensate for the missing information by extracting information from the video and predicting using machine learning on the knowledge graph. In this presentation, we provide an overview of the dataset and tasks of this inference challenge and introduce the four submissions. Since several of the submissions used multimodal LLMs, we will compare them and also discuss the challenges and expectations for current multimodal LLMs in this task.