Clustering time-series data of cooking using object detection

Abstract

This paper presents an appliance of object detection with deep learning for a series of cooking video frames and an analysis of these time-series detection data. Our final goal is offering personalized kitchens. We have observed and analyzed cooking behaviors with visual observation and spread sheet software. However this conventional method was too tough to perform with sufficient precision and subject numbers. Furthermore, it was difficult to analyze from various viewpoints. So, we introduced object detection with deep learning as a tool for automation of the method. We applied object detection for a series of cooking video frames which contains about 90 objects, extracted these data and analyzed with Python, and found that the time-series data counting number of detected objects clustered 3 patterns in sink area. This discovery contributes to understanding of cooking patterns which leads to offering personalized kitchens.