A preliminary study of automated scoring in crystallized intelligence scale

Abstract

Crystallized intelligence is a knowledge which is acquired from the experiences, education, and culture. Crystallized intelligence is often evaluated by the standardized test named Wechsler Adult Intelligence Scale (WAIS). WAIS evaluated participant’s knowledge about language comprehension like similarity (how presented two words are related in meanings). Though the test scores were evaluated by the trained judge, there is the possibility to automate the test scoring. In this study, we focus on the ability to find a similarity of two concepts which is an aspect of crystallized intelligence, and we conduct a preliminary test of whether the measure of crystallized intelligence – named Japanese version of Semantic Similarity Test – can be scored automatically. A BERT based model which classifies participant’s responses to three class label (2: perfectly captures the relationship of two words, 1: partially captures the relationship of two words, 0: bad response) was trained in two different approaches (one trained by participants answers and their scores, and the other trained by rubric sentence and their scores) and compared the accuracy and f1 score. Results showed the latter model overwhelmed the performances of the former. We discussed the background of the results (the difference of the quality of sentences and the quantity of the pseudo labels).