Assessment of target speakerness by non-acquaintance through dialogue comparison

Abstract

Several dialogue system studies have attempted to replicate a specific speaker desired by users. Typically, to assess the ``speakerness'' of a specific speaker subjectively, evaluators should be familiar with the target speaker. However, when using a corpus collected from the web and crowdsourcing, it becomes challenging to find evaluators familiar with the target speaker. To address this issue, we propose a novel method for assessing target speakerness through dialogue comparison that can be utilized by non-acquainted evaluators. We evaluate the effectiveness of this method using both expert annotators and non-expert crowdworkers, discussing its validity as a subjective evaluation tool for speakerness. Additionally, we train and examine the performance of a baseline model for assessment of target speakerness through dialogue comparison.