IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences
Online ISSN : 1745-1337
Print ISSN : 0916-8508
CookClip: Learning the Relationship between Time-Series Cooking Process Images and Recipes using CLIP model for Cooking Task Monitoring
Rina TAGAMI, Hiroki KOBAYASHI, Shuichi AKIZUKI, Manabu HASHIMOTO
Author information
JOURNAL FREE ACCESS Advance online publication

Article ID: 2025EAP1197

Details
Abstract

In recent years, methods that enable robots to generate actions based on human instructions have attracted significant attention. In cooking tasks in particular, robots must determine whether to continue linguistically specified actions, such as “frying” or “mixing,” based on the visually observed state of the ingredients. This study focuses on the decline in learning performance caused by the characteristics of training data of cooking task monitoring with CLIP models, which are capable of training correspondences between images and texts. In conventional task monitoring datasets, all frames—from the beginning to the end of the cooking process—are paired with a single recipe text, such as “Fry the chicken until golden brown.” In other words, one recipe text is associated with multiple images representing different ingredient states, resulting in highly ambiguous data. For example, in the initial frames, the ingredients have not yet been cooked to the state described by the recipe text, resulting in a mismatch between the image and the text and therefore low-quality paired data. Because CLIP models are trained under the assumption that the image and text in each pair are semantically aligned, training with such mismatched data degrades learning quality and leads to poor task-monitoring performance. To address the problem of degraded learning caused by ambiguous training data, we propose CookClip, a training method that enables the model to accurately learn the relationships between recipe texts and temporally sequential cooking images. This approach enhances the CLIP model's ability to monitor cooking tasks more effectively. Furthermore, we constructed a dataset called Time-series Cooking Image, specialized for cooking task monitoring. It consists of sequences of cooking images and corresponding recipe texts, and is used for both training and evaluation. This dataset includes image sequences corresponding to cooking initiation, successful completion, and failed outcomes—features that were not present in previous datasets. We evaluated the task completion detection performance of the proposed method, based on the error between the correct completion frame range and the predicted frame. A smaller error indicates that the model more accurately captures the temporal progress of cooking and successfully trains the correspondence between the recipe texts and cooking images. The error between the correct frame range and the output frame was 3,077 frames with the original CLIP, while it was reduced to 51 frames with the proposed method, confirming the effectiveness of our training approach. The dataset and construction method are publicly available at: http://isl.sist.chukyo-u.ac.jp/archives/TSCI.

Content from these authors
© 2026 The Institute of Electronics, Information and Communication Engineers
Previous article Next article
feedback
Top