Self-supervised Learning Method for Behavior Prediction during Dialogue Based on Temporal Consistency

Abstract

Developing a communication agent that can mutually interact with a human has been expected. To realize the agent, real-time situation recognition and motion generation are necessary. The human-human interaction data is utilized to develop the recognition and the generation model. However, a cost of giving a certain label to the data is expensive, i.e., the number of labeled data becomes small. To cope with the small dataset problem, one of the approaches is to obtain the pre-trained weight by self-supervised learning. In this research, we propose estimating the amount of time-shift by “lag operation” as a task for self-supervised learning. The observed data is not isolated during the interaction between two people, and using both observed information from two people makes an estimation model reduce the uncertainty of situation detection. By exploiting these properties of interaction data, the time index of data of one person is shifted, i.e., the entrainment of two data is broken. This operation is called a “lag operation”, and estimating the amount of time-shift is defined as the pre-training task. We apply this pre-training to the prediction experiment that estimates near-future laughing during a conversation. The result shows the accuracy of the laughing prediction is improved by 1.3 points, and the lag operation is an effect for predicting the change of interaction situation.