Online system to have users reflect upon their rating tendency on speaking skills

Abstract

This study discusses a new online system that allows Japanese teachers of English (JTEs) to reflect on their internal speaking assessment criteria. The user (an English teacher) watches three movie clips of student speaking performances and then rates them using vocabulary, accuracy, fluency, and pronunciation scales. The system then employs the Many-facet Rasch Measurement (MFRM) to immediately analyze the user ratings by accessing a database developed from JTE mass speech sample ratings data (approximately 7,500 ratings). Sixteen English teacher trainees were analyzed using both the new system and FACETS (an MFRM statistical software), and the results were compared. It was confirmed that the newly developed system was able to accurately detect any deviations in the user ratings.