Abstract
This study aims to investigate inter-rater reliability of free composition ratings by different methods. An experimental research was conducted to find out at least how many raters are required in order to obtain reliable ratings of free composition. In the present research, 36 essays written by Japanese high school students were scored by 14 raters by three different methods: 1) impressionistic method (six scales), 2) holistic method (six scales), 3) analytic method (five subordinate criteria). All possible combinations of raters from two raters to 14 raters were prepared and inter-rater reliability for ratings by different number of raters of all the combinations was calculated for each method. The result showed that in order to obtain reliable ratings of free composition, at least 5 raters are required for impressionistic method, 6 raters for holistic method, and 3 for analytic method.
