Evaluating Summary Writing by Japanese EFL Learners Using a Constructed Response Analysis Tool

Abstract

This study aims to evaluate summaries by Japanese English as a Foreign Language (EFL) learners and identify their characteristics based on three indices: lexical overlap (the commonality between specific words and phrases between a summary and the original text), semantic overlap (the degree of semantic relatedness between a summary and its source text), and lexical sophistication (the diversity and complexity of the vocabulary used in a summary). These indices were calculated using the Constructed Response Analysis Tool (CRAT), an automated English writing assessment tool. Japanese EFL learners’ summaries were analyzed using CRAT, and correlation and decision-tree analyses were conducted to investigate relationships between the indices. Subsequently, decision-tree analyses were conducted to identify the indices that had a significant impact on the summary scores. The important indices for Japanese EFL learners included verb-unigram overlap percentage for lexical overlap, Latent Semantic Analysis similarity or synonym overlap for semantic overlap, and psycholinguistic indices for lexical sophistication. The findings using objective indices from CRAT could inform Japanese EFL teachers’ feedback and summary instructions, contributing to the development of EFL learners’ summary writing and teachers’ evaluation efficiency.