Evaluation of RAG-Based Dialogue Systems using LLM in Customer Support and Efforts Towards Business Implementation

Abstract

In customer support, the advancement of Large Language Models (LLMs) has led to the development of dialogue systems based on the Retrieval Augmented Generation (RAG) that answer based on frequently asked questions. However, presenting the direct outputs of LLMs to users poses challenges, including issues like hallucination, necessitating thorough pre-validation. In this context, we introduce a method where an LLM-simulated user interacts with the RAG-based dialogue system, and the resulting dialogue data is evaluated using the LLM. Furthermore, we present a demo created for the business implementation of the dialogue system and efforts towards improving the prompts.