Integration of Large-scale Language Model and World Models in Robot Control

Abstract

In general, robot controls are limited to pre-programmed actions, lacking adaptability to the environment and ease of use for humans. This paper proposes a method that combines a large-scale language model for code generation with a world model for robot control, allowing autonomous robot behavior from natural language sentences. The Language Model Program (LMP) generated by the large-scale language model uses the latent space of the world model learned by NewtonianVAE to enable high-freedom and high-abstraction control. This combination of large-scale language models and world models is important in the context of neuro-symbolic AI, which combines latent and symbolic representations.