Validating a Performance Assessment of Intercultural Understanding

Abstract

Measures of intercultural understanding are part of a growing body of language testing scholarship focused around defining L2 proficiency as a sociocognitive process, that is, as the interplay between learner-internal language knowledge and learner-external cultural factors. The performance assessment described in this paper used an analytic rubric to assess the sociocognitive variable of intercultural understanding through an interview task. To demonstrate the relationship between the descriptors on the rubric and authentic task performance, we analyzed transcripts of the interview task itself, along with official feedback reports about the test from raters and administrators. To create a probabilistic model of rater agreement, we used Many-Facet Rasch Analysis to evaluate the extent to which a high score on this task indicates an examinee's ability to function well in intercultural situations, and a low score indicates the opposite. The results provide some evidence for a unidimensional construct related to intercultural understanding that can be measured using an interview task.