Abstract
Recent advancements in AI, and generative AI in particular, have boosted chatbots’
ability to maintain a conversation. This is especially critical for the field of English
language education where chatbots can serve as conversation partners. However, how
well can modern chatbots sustain a conversation with learners of English, as compared to
the chatbots studied a decade ago? This research focuses on evaluating the linguistic
ability of AI chatbots in text-based interactions with 64 Japanese undergraduate students
who are non-native English speakers. In particular, the study examines three AI chatbots
with different algorithms to find out how effectively these three chatbots can deliver errorfree,
contextually appropriate responses even despite errors in users’ messages, and what
linguistic challenges English language learners encounter when engaging with chatbots.
The study revealed a relatively high response accuracy of AI chatbots towards non-native
English speech. However, there was a significant difference in response accuracy among
the three AI chatbots due to the nuances in their algorithms, with generative AI chatbot
showing the highest response quality among the three. Despite the significant
improvement of chatbots’ linguistic ability, the participants still reported instances of
confusion and frustration due to occasional inappropriate responses.
