Abstract
Group oral test formats have recently been introduced to the range of oral performance tests because the assessment of L2 learners' authentic conversational competence is considered important in the current era of globalization. However, these tests are not widely administered in Japan and only a few studies dealing with group oral interaction have been carried out to date. This study explored three facets - the severity of raters, the difficulty of rating categories, and participants' speaking ability - when Japanese learners of English encountered oral interaction in a small group. The Common European Framework of Reference (CEFR; Council of Europe, 2001) was utilized for the rating in this study. The analysis was executed by means of multi-faceted Rasch measurement, which analyzed sources of variation and estimated "measures" from the raw scores given by raters. The raters were considered as self-consistent but showed significant differences among themselves as well as with other research. Utilizing multi-faceted analysis and building a consensus among raters by having them identify certain attributes in the speakers are recommended approaches to control such variances. The CEFR rating categories were proved to be valid; however, more detailed rating scales should be created for the situation in which all participants are Japanese, as the speaking ability of most of the participants was judged as low.
