DIF analyses on the Business Japanese Proficiency Test: Comparisons across gender and districts of residence

Abstract

The purpose of this study is to show differential item functioning (DIF) of gender and resident (in Japan or not) on the Business Japanese proficiency Test. In the first study, 2,444 examinees, the students of Japanese as a foreign language class, took 160 item test of business Japanese proficiency. Three empirical methods were used to detect DIF: namely, the Mantel-Haenszel method, ANOVA method and item response theory method. As a result, 20 gender DIF items and 67 residential DIF items were found by at least one of the methods mentioned above. Three gender DIF items and 14 residential DIF (in Japan) items were detected by all the three methods. In the second study, 60 items consisting of DIF and DIF free items were shown to nine experts (5 females and 4 males) of item writing and their subjective impression of items concerning DIF were analyzed. It was found that the experts used five beliefs about gender DIF and five beliefs about residential DIF when they rated items. 12 items were judged to have gender DIF and 54 items were judged to have residential DIF by at least one expert. The comparative study showed that the result of the statistical DIF analysis and the subjective prediction did not coincide much: The experts overlooked gender DIF items and they were very sensitive to residential DIF items.