SARI: A Stage-aware Recognition Method for Ingredients Changing Appearance in Cooking Image Sequences

Abstract

Recognizing ingredients in cooking images is a challenging task due to the significant visual changes that ingredients undergo throughout the cooking process. As ingredients are prepared, cooked, and served, their appearances vary greatly between the beginning, intermediate, and finishing stages. Traditional object recognition methods, which assume constant object appearances, struggle with this variability and are often not good at accurately identifying ingredients at different cooking stages. To address this challenge, we propose a stage-aware recognition method specifically designed for dynamically changing ingredients in cooking images. Our approach introduces two techniques: 1. Stage-Wise Model Learning: This technique involves training separate models for each stage of the cooking process. By adapting models to specific stages, we can better capture the distinct visual characteristics of ingredients as their appearances change. 2. Stage-Aware Curriculum Learning: This technique begins training with data from the beginning cooking stages and progressively incorporates data from later stages. This gradual approach helps the model adapt to the evolving appearances of ingredients. Our experimental results, using our published dataset, demonstrate that our stage-aware methods significantly outperform models trained without stage considerations, achieving higher accuracy in ingredient recognition.