Reliability and Optimal Designs for Measuring Accuracy in L2 Writing With a Weighted Clause Ratio

Abstract

　　Prior research has proposed a new index for assessing written accuracy known as weighted clause ratio (WCR). However, the descriptors used to determine WCR scores are vague and open to multiple interpretations. Additionally, prior research has not investigated the impact that various factors (e.g., rater) might have on the WCR score variances and reliability. Optimal measurement designs have not also been investigated. Thus, this study revised the rating scale descriptors used to determine WCR scores and examined the index’s reliability using generalizability theory. The impact of factors that affect the score variances and optimal measurement designs were also investigated. English essays written by 100 Japanese English language learners from the International Corpus Network of Asian Learners of English were evaluated for this purpose. The results indicated that the reliability of the evaluation was high (G coefficient = 0.91), although only some factors affected the variances of WCR scores. Consequently, optimal measurement conditions were found under which written accuracy can be reliably measured using WCR. Results indicate that the revised rating scale of WCR can be applied to essays written by Japanese EFL learners. The study results can also serve as a guideline for assessing accuracy using WCR.