An Approach to Language Instruction Operation by Verbalizing Control Commands

Abstract

We aim to operate robots by generating robot action strategies corresponding to the real-world environment based on ambiguous instructions in natural language with a large language model, and mapping these strategies to robot control commands. The environment in which the robot exists and instructions for the robot are given in natural language as prompts, and the robot's action strategy is generated by Chain-of-Thought inference. By the generated action strategy to the control commands expressed in the language, the robot can select the appropriate command to execute according to the generated strategy. By processing the entire sequence of generating action strategies for the robot based on the real-world environment and corresponding to the robot's commands, even when the same instructions are given, in natural language, we aim to realize flexible control of robots that change their behavior according to the situation based on the meaning of words.