Abstract
This paper investigated the effects of the choice between two different strategies for solving TOEIC(R) listening test part 4 type questions on item difficulties. The strategies focused were the preview-questions and answer-while-listening strategy and the listen-without-question-preview and answer-after-listening strategies. A group of 64 Japanese EFL under- or post-graduate learners solved 30 TOEIC part 4 type listening problems using either of the strategies. Rasch-based common item equating located these items on a single difficulty dimension with measurement error information. Even though a τ test indicated no significant difference between the means, correlational analyses and 95% control line analyses suggested that the choice of different strategies did make a difference in the difficulty of each item as well as the exact nature of the construct tapped. Test-takers' perception of these strategies were also explored by examining their verbal comments, which indicated unanimous preference for question preview but divided opinions regarding while-listening answering. Based on these results, we argue that the current structure of TOEIC listening test part 4 poses a serious threat to its own test validity.
