The purpose of this study is to investigate the relationship between the oral test scores marked by Japanese language teachers and actual utterance data in formative evaluation. Teachers evaluated 7 criterias; vocabulary, grammar, pronunciation, repetition, silence, quantity of speech, and sentence type. The utterance data was quantitatively analyzed at morpheme level and was investigated in correlation with the scores. The results show that repetition, silence, quantity of speech, and the sentence type significantly correlate with the utterance data. However grammar and pronunciation do not correlate significantly with the utterance data and appear to be influenced more by qualitative aspects rather than quantitative ones.