Classification of Utterances that Lead to Dialogue Breakdowns in Chat-oriented Dialogue Systems

Abstract

This study proposes a taxonomy of errors in chat-oriented dialogue systems. Previously, two taxonomies were proposed, one theory-driven and the other data-driven. The former suffers from the fact that dialogue theories for human conversation are often not appropriate for categorizing errors made by chat-oriented dialogue systems. The latter has limitations in that it can only cope with system errors for which data exist. This paper integrates these two taxonomies to create a comprehensive taxonomy of errors in chat-oriented dialogue systems. It was determined that, with our integrated taxonomy, errors can be reliably annotated with a higher Fleiss’ kappa compared with the previously proposed taxonomies.