Reliability and Validity of Multiple-Choice Questions Measuring Higher-Order Thinking Skills in University English Entrance Examinations across Two Years

抄録

This study investigates the validity and reliability of multiple-choice questions (MCQs) designed to assess analytical higher-order thinking skills (HOTS) in university English entrance exams. Using actual exam data from 2024 (N=121) and 2025 (N=141), the study evaluates whether a limited number of MCQs can effectively maintain sufficient validity (difficulty and discrimination indices) and provide meaningful evidence of reliability (KR-20 coefficient). Results revealed that MCQs from both years had appropriate difficulty and high discrimination, indicating good overall quality. However, the small item pool contributed to lower reliability, especially in 2025, due to frequent perfect scores. Increased lexical complexity and sentence length in 2025 did not correlate with higher difficulty or discrimination. These findings highlight the necessity of clear, well-structured items rather than superficial textual complexity, providing practical guidance for designing effective HOTS MCQ assessments under realistic constraints.