Development of an Affirmative/Negative Intention Estimator for YES-NO Question Answers Using a Large-scale Language Model

Abstract

In dialogue systems, it is especially important to ask questions that can be answered with "yes," "no," or "I don't know" intentions (YES-NO questions) in order to confirm the user's intentions and status, and to accurately interpret the intention of the user's response. In this study, we aimed to estimate the response intention to these YES-NO questions with high accuracy. Specifically, we created a dialogue corpus (Japanese Yes-No question-answer pairs), designed several intention estimators using a large-scale language model, and compared and evaluated their accuracy. As a result, the GPT model with the addition of an all-combining layer showed the highest estimation accuracy, achieving an accuracy of 91\%. Comparison among the estimators also confirmed a tendency to mis-estimate "I don't know" when using prompt programming with the GPT model. This study contributes to the ability of dialogue systems to improve their ability to estimate the intent of interrogative sentences and provides new metrics and insights into the performance evaluation of machine learning models.