Analysis of Subjective Evaluation for Fine-tuning Methods of Transformer encoder-decoder based Conversational Systems

Abstract

In recent years, several high-performance conversational systems based on the transformer encoder-decoder model have been proposed. Natural response generation is achieved by increasing the system scale (model parameters, amount of training data, etc.). While previous studies have analyzed the relationship between the system size and decoding method on the subjective evaluation of dialogues, they have not analyzed the differences among Fine-tune corpora. In addition, conventional analysis has focused only on overall naturalness and superiority, and has not sufficiently analyzed the relationship with multifaceted and detailed impressions. We evaluate and analyze the impressions of human dialogues in different Fine-tune corpora, system sizes, and the use of additional information.