A Systematic Review of Research Designs and Tests Used for Quantification of Treatment Effects in ARELE 13–28

Abstract

　　The present study reviewed the quality of quantitative research in Annual Review of English Language Education in Japan (ARELE) volumes 13 to 28 from two perspectives: research designs and test traits. In articles on foreign language pedagogy including ARELE, quantitative data have been used as scientific evidence to determine if a teaching method is effective. This requires us to revisit the types of research designs and tests that have been employed to quantify the efficacy of instructions used in English education in Japan. In this systematic review, we targeted research that claimed a causal relationship between a pedagogical intervention and its outcome and coded 398 ARELE articles according to research designs and test traits for a meta-analysis. Results showed that 59% of primary studies adopted less appropriate designs involving threats to internal validity. Similarly, research that used less appropriate tests was more likely to overestimate and/or underestimate the treatment effects. Although our research community has continuously provided information about statistical reforms, such as statistical power and statistical indices to be reported, findings further suggest that we should revisit methods of designing our quantitative research.