Fine-Grained Cooking Activity Recognition System With Adversarial Learning to Track Sequential Transformation of Ingredients

Abstract

Recently, automatic instructional manual creation systems from videos have been focused on for supporting beginners. An automatic recipe creation system is one of them and it is created by recognizing cooking activities and objects having relation to the activities. To realize a more useful system, we need to recognize the activities from coarse to fine-grained, such as from “cut” to “slicing”, “cutting into small pieces”, etc. However, the recognition of such fine-grained cooking activities is a very challenging task because we utilize the same utensil such as kitchen knives, and employ similar hand motions among the activities. To solve this problem, we focus on the sequential transformation of ingredients. By using this information, in this paper, we introduce a new GAN (generative adversarial network) -based network model to recognize fine-grained cooking activities and investigate the effectiveness of the model by comparing it with a spatio-temporal network model.