Crowdsourcing Data Collection and Automatic Annotation of Infant’s Behaviors

Abstract

This paper describes a novel approach toward automatic annotation of infant’s behaviors following the collection of action descriptions in the crowdsourcing. Videos containing actions are publicly opened, and are manually annotated by internet users in the crowdsourcing platform. This results in a large training dataset of infant’s actions and their relavant descriptions. Relation between the actions and their descriptions are extracted by a probabilistic framework with two modules; semantic module and syntax module. The semantic module trains the association of the words from the action, and the syntax module trains the word sequence with grammatical consistency. This framework allows for automatic conversion of action observation to annotations. Our proposed approach was tested on actions performed by an infant and its validity was demonstrated.