A Method for Detecting Concealed Nervousness in Spoken Dialog Using Facial and Acoustic Features

Abstract

Early detection of psychological distress is important for preventing serious mental health issues. This includes situations where individuals hide their stress from others or are not aware of their own condition. Traditional methods mainly rely on physiological signals such as skin temperature and heart rate, which require contact-based sensors. In this study, we propose a method for detecting concealed nervousness in spoken dialogue using facial and acoustic features obtained from non-contact devices such as cameras and microphones. We labeled the data based on self-reported nervousness levels and trained a machine learning model. The combined use of facial expressions and speech features achieved an accuracy of 0.74. We also examined individual differences in how nervousness appears. Some people tend to show tension in facial expressions, while others show it in vocal tone. Statistical analysis showed that relaxed facial muscles reduce the visibility of nervousness in expressions, and facing the conversation partner makes nervousness more detectable in speech.