Applying Three-Arm Design for Assessing Distance Learning

抄録

The COVID-19 pandemic has transformed teaching methods, shifting from traditional face-to-face teaching to distance learning. To explore the effectiveness of digital distance teaching in STEM subjects, this study uses mathematical statistics as a case study to analyze and compare student learning outcomes across three different digital distance teaching methods. While conventional assessment of differences in mean values among three sample groups often adopts analysis of variance (ANOVA), ANOVA requires adherence to two major assumptions: normality and homogeneity of variances, which may not be satisfied in practical applications. However, in clinical trials, superiority or noninferiority tests in a three-arm design are often conducted to confirm the effectiveness of a new drug compared to a control group (old drug or placebo). In a three-arm design, superiority or noninferiority tests can be applied under conditions of variance heterogeneity, which represents the primary distinction from the ANOVA approach. We employed superiority and noninferiority tests in the three-arm design to assess student learning effectiveness across three distance digital teaching methods, utilizing Fieller’s and bootstrap methods. An intensive simulation study revealed Fieller’s method performed satisfactorily. Fieller’s method both adequately controlled the empirical type I error rate and was uniformly more powerful than the bootstrap method. Accordingly, Fieller’s method yields stable test results in small samples, making it suitable for scenarios with limited sample sizes in educational settings. Finally, the proposed application methods are illustrated using real-world data.