Cooking Recognition Planning Action Robot System Considering the Change of Food Condition from the Description of Cooking Recipe Using Large-scale Foundation Models

Abstract

In the cooking task, the goal is to make the dish complete by changing the state of the ingredients by following the recipe description. Therefore, it is desirable for robots to be able to plan cooking tasks based on recipes written in natural language and to be able to recognize changes in the state of ingredients. In this study, we propose a robot system that uses human knowledge and common sense contained in the vast amount of language-related data on the Internet by applying large-scale foundation models that have been actively developed in recent years to robots, and executes cooking based on recipe descriptions while considering changes in the state of ingredients. and executes cooking based on recipe descriptions while considering changes in the state of food ingredients. We have confirmed the effectiveness of the proposed system through experiments by converting recipes into cooking function expressions using a language model and by recognizing changes in the state of food ingredients based on language descriptions by time-series use of a vision-language model.