Toward Automatic Evaluation of Haiku using GPT

Abstract

Haiku generation and evaluation using deep learning receive lots of attention in recent years. While there are existing studies about haiku generation, haiku evaluation takes time and effort, and quality of haiku evaluation can vary based on evaluators and the time of evaluation. Large language models like GPT, with massive pre-training data and model parameters, are expected to evaluate haikus more accurately than existing language models. Therefore, this research investigates the possibility of automatic evaluation of haikus using GPT. The objective is to automate haiku evaluation using large language models, targeting reduction of human workload and standardization of haiku evaluation. Experimental results show that GPT can learn haiku evaluation using few-shot learning, and that GPT-4 performed more accurate haiku evaluation than GPT-3.5-turbo.