Abstract
This comparative study examines the effects of a rater-training session in relation to the degree of differences in raters' severity, consistency, and biased interactions between raters who received training before rating (trained raters) and raters without a training session beforehand (untrained raters). Ten raters (five trained and five untrained) rated a total of 40 scripts (20 scripts for each of accuracy and communicability tasks); the analyses were done using FACETS. In addition, questionnaire responses were analyzed to investigate raters' views regarding the effectiveness of the training. The results showed that 1) the raters were not equally severe; untrained raters were more severe than trained raters, 2) both trained and untrained raters behaved consistently in scoring; untrained raters were assumed to be less consistent, 3) a clear distinction cannot be made among trained and untrained raters as a group in terms of rater-task interactions, 4) untrained raters might be more biased in rater-subject interactions than trained raters; this is shown by the untrained raters' inconsistency in scoring. These results imply that rater training more effectively improves raters' scoring consistency than scoring severity.
