Who Select Attractors Including the Words in Key Sentences?

—Focusing on Different Types of English Reading Items—

Abstract

We investigated choice ratios of attractors that are the incorrect choices reflective of typical errors made by students during reading comprehension tests by examining the effects of lexical overlaps between key sentences related to each choice and an attractor. We also investigated the effects of question types represented by lower- and upper-level questions on attractor decisions. The former questions asked test takers only to identify key sentences and evaluate options, whereas the latter asked them to grasp the structure or the gist of a paragraph or a passage. Undergraduate student test takers (N = 460) participated. They were given one of eight booklets. Experimental items consisted of one key and three attractors: a negation, an antonym, and a causal misunderstanding. Estimates and generated quantities in a Bayesian hierarchical model obtained via Gibbs sampling indicated that for lower-level questions test takers with low proficiency selected attractors with overlapping words, whereas those with high proficiency chose attractors with negations or antonyms in the non-overlapping condition. In contrast, for upper-level questions less proficient students chose attractors in the non-overlapping condition and proficient students selected attractors with negations or antonyms in the overlapping condition. These results suggest that examining attractors in multiple-choice tests could enable us to develop optimal items and to qualify test items.