To become an advanced second language listener, a learner needs to have good speech perception ability. Previous research that focused on measuring this ability often utilized integrated-skills tasks (e.g., repetition tasks and dictation tasks), but their validity and reliability are questionable because learners’ productive skills affect their task performance. This study attempted to develop an original discrete-point task called the word count task, in which learners count and report the number of words in blanks. To evaluate the task’s validity and reliability, two comparable studies with dictation tasks were conducted with university students in Japan. The second study, which was revised based on the first study, revealed that the reliability coefficient of the word count task expressed by Cronbach’s αreached .85, slightly exceeding the reliability of the dictation tasks. Moreover, correlations with dictation tasks were found to be significantly positive with moderate to strong relationships, meaning the word count task demonstrated sufficient criterion-related validity. Moreover, the listening strategy survey conducted to explore cognitive processes involved in the task showed that phonological processing is more dominant than meaning processing in the word count task. These findings seem to corroborate the applicability of the word count task to research and classroom assessment, but further research is necessary to reevaluate its validity using other methods mentioned in this study.

This study is aimed at developing a Japanese listening test for cognitive diagnostic assessment at the CEFR B1 level. The paper describes improvements to the attributes and test items based on the values of guessing and slip parameters estimated by the DINA model and the retrospective verbal report method, and discusses whether these methods are effective for developing cognitive diagnostic assessments. First, CEFR materials were analyzed, and nine attributes were identified. Next, a Japanese listening test consisting of 27 items measuring these attributes was developed, and a 12 × 27 Q-matrix was constructed. Two surveys were then conducted. Survey 1 obtained answers to the test from 297 learners of the Japanese language. Survey 2 obtained retrospective verbal reports from 13 learners to understand the answering process. The test items were then analyzed based on the estimated parameter values and the findings from the retrospective verbal reports. The attributes were modified, some of the test items were eliminated, and the Q-matrix was reconstructed. When re-estimation was performed after revision, the parameter values were improved. It was confirmed that the interpretation conducted by the retrospective verbal reports was valid and the effectiveness of our methods for developing a new cognitive diagnostic assessment was demonstrated.

The current study aimed to reveal similarities and differences between a holistic and an analytic rubric used in assessing speaking performance in a paired oral test. To this end, speaking performances of 110 Japanese university students produced in paired oral interaction were evaluated by raters, holistically and analytically. The comparisons made between the two rubrics using many-facet Rasch measurement showed that both worked effectively, with the analytic rubric working slightly better in terms of a better global fit, a better test-taker and task separation, higher test-taker and task reliability, smaller standard errors, and a smaller percentage of test takers with overfit. Correlation and regression analysis indicated a strong relationship between the two (r = .84) and the Interactive communication and Fluency analytic criteria substantially explained holistic scores (adjusted R² = .71). Results suggest that teachers can obtain similar results with either rubric type and, if they select an analytic one, a priority would be to include Interactive communication and Fluency criteria.