MetaProxy: Bridging Agents and Environments in Commercial Metaverse Platforms through Remote Intermediary Monitoring

Abstract

Metaverse platforms have emerged as central hubs for virtual communication and entertainment, attracting large numbers of users. Integrating autonomous agent avatars into these spaces could enable diverse applications, including educational support, psychological assistance, and enhanced entertainment experiences. However, commercial metaverse platforms typically lack external Application Programming Interfaces (APIs) for agent control, creating significant implementation barriers for researchers and developers. This paper presents a network proxy module that monitors client-server communication in commercial metaverse platforms, enabling avatar control without platform-side modifications. Our approach extracts spatial information from network packets and controls avatars via keyboard-mouse emulation, requiring no changes to the underlying platform architecture. We validated our method by implementing a Large Language Model (LLM)-integrated agent avatar on Cluster, a commercial metaverse platform. The experimental evaluation demonstrated the agent’s ability to respond to user proximity, comments, and emotional expressions with an average response time of 1.34 seconds.