Extraction of Scene-Specific Co-Occurrence Information Using Large Language Models and Its Application to Robot Scene Understanding

Abstract

To enable a robot to act appropriately in its operational space, it is crucial to understand the relationships between objects specific to a given context. This is because the arrangement and associations of objects determine their functionality and purpose within a scene. By accurately capturing these relationships, a robot can comprehend the intent of a scene and effectively plan and execute tasks. This study proposes a method for extracting scene-specific co-occurrence information from large language models (LLMs). While LLMs provide extensive co-occurrence knowledge, their accuracy declines in specific contexts, necessitating additional fine-tuning for real-world applications. Our approach extracts scene-specific co-occurrence information based on object placement by incorporating the surrounding objects’ information when generating co-occurrence data for objects A and B. This method generates contextually appropriate co-occurrence information without additional training, making it suitable for specific scenes and environments. By emphasizing the functional relationships formed by object groups, we demonstrate its high effectiveness in applications such as scene understanding.