Theory of Mind and Reflection-Based Sampling with LLMs for Decentralized Sequential Decision-Making by Embodied Multi-Agents

Abstract

Realizing joint action based on Theory of Mind (ToM) is an important step for enabling large language models (LLMs) to collaborate effectively with humans. However, autonomously generating action plans that take into account the perspectives and beliefs of others remains a challenging problem for LLMs. In this study, we aim to achieve autonomous joint action with embodied LLM agents equipped with ToM, and propose a decentralized sequential decision-making method, ToM-JAC (Joint Action Control with Theory of Mind), which plans and adapts actions online. We demonstrate the effectiveness of the proposed method through two-agent joint action tasks in a Minecraft environment.