Serendipity Assessment in Recommender Systems Using Large Language Models

Abstract

Serendipity-oriented recommender systems have been proposed to prevent over-specialization in user preferences. However, evaluating serendipity is challenging due to its reliance on subjective user emotions. We try to address this issue by leveraging the rich knowledge of large language models (LLMs), which can perform a variety of tasks. As a first step, this study investigates the alignment between serendipity assessments made by LLMs and those by humans. Specifically, using GPT-3.5 we assess the serendipity of recommended items based on users' evaluation history. We evaluate the accuracy of assessment made by the LLM using an annotated benchmark dataset. Experimental results indicate that our method outperforms the baseline method, showing improvements of up to 0.6, 4.9 and 1.5 points in Accuracy, Precision and Macro-F1-score.