Article ID: 2025EAP1197
In recent years, methods that enable robots to generate actions based on human instructions have attracted significant attention. In cooking tasks in particular, robots must determine whether to continue linguistically specified actions, such as “frying” or “mixing,” based on the visually observed state of the ingredients. This study focuses on the decline in learning performance caused by the characteristics of training data of cooking task monitoring with CLIP models, which are capable of training correspondences between images and texts. In conventional task monitoring datasets, all frames—from the beginning to the end of the cooking process—are paired with a single recipe text, such as “Fry the chicken until golden brown.” In other words, one recipe text is associated with multiple images representing different ingredient states, resulting in highly ambiguous data. For example, in the initial frames, the ingredients have not yet been cooked to the state described by the recipe text, resulting in a mismatch between the image and the text and therefore low-quality paired data. Because CLIP models are trained under the assumption that the image and text in each pair are semantically aligned, training with such mismatched data degrades learning quality and leads to poor task-monitoring performance. To address the problem of degraded learning caused by ambiguous training data, we propose CookClip, a training method that enables the model to accurately learn the relationships between recipe texts and temporally sequential cooking images. This approach enhances the CLIP model's ability to monitor cooking tasks more effectively. Furthermore, we constructed a dataset called Time-series Cooking Image, specialized for cooking task monitoring. It consists of sequences of cooking images and corresponding recipe texts, and is used for both training and evaluation. This dataset includes image sequences corresponding to cooking initiation, successful completion, and failed outcomes—features that were not present in previous datasets. We evaluated the task completion detection performance of the proposed method, based on the error between the correct completion frame range and the predicted frame. A smaller error indicates that the model more accurately captures the temporal progress of cooking and successfully trains the correspondence between the recipe texts and cooking images. The error between the correct frame range and the output frame was 3,077 frames with the original CLIP, while it was reduced to 51 frames with the proposed method, confirming the effectiveness of our training approach. The dataset and construction method are publicly available at: http://isl.sist.chukyo-u.ac.jp/archives/TSCI.