Analysis of Job Interview Training Feedback System Effectiveness Based on a Multimodal Machine Learning Model

Abstract

We built a humanoid agent system for VR experiences and collected a job interview data corpus. The data corpus includes annotations of interview skill scores graded by third-party experts and self-efficacy annotations by the interviewees, for each question-answer. The data corpus contains various kinds of multimodal data, including audio, biological (i.e., physiological), gaze, and language data. In this study, we developed a feedback system for automated job interview training and analyzed the impact of the feedback. The feedback system utilizes a machine learning model that uses acoustic and linguistic features. In the control group, feedback was provided using a book. The results of the comparison of the effects of the proposed system and the book suggested that the proposed feedback system could suppress the self-confidence of the group that tended to overestimate their performance when compared with the book.