Evaluation of Large Language Models' Foreign Language Teaching Ability: An Experimental Study Focusing on Pedagogical Grammar

Abstract

This study examines the potential applications of large language models (LLMs) in language education. To evaluate their ability to identify the compatibility between grammatical items and example sentences, we designed a task and conducted experiments. Using multiple LLMs, we compared their performance based on accuracy, false negative rate (FN rate), false positive rate (FP rate), and Balanced Score. Additionally, we confirmed that synthetic data could serve as a practical alternative. Future research should focus on developing high-quality synthetic data generation methods and expanding their applicability. The findings of this study are expected to contribute to the establishment of benchmarks for evaluating the grammatical competence of LLMs in natural language.