This study is aimed at developing a Japanese listening test for cognitive diagnostic assessment at the CEFR B1 level. The paper describes improvements to the attributes and test items based on the values of guessing and slip parameters estimated by the DINA model and the retrospective verbal report method, and discusses whether these methods are effective for developing cognitive diagnostic assessments. First, CEFR materials were analyzed, and nine attributes were identified. Next, a Japanese listening test consisting of 27 items measuring these attributes was developed, and a 12 × 27 Q-matrix was constructed. Two surveys were then conducted. Survey 1 obtained answers to the test from 297 learners of the Japanese language. Survey 2 obtained retrospective verbal reports from 13 learners to understand the answering process. The test items were then analyzed based on the estimated parameter values and the findings from the retrospective verbal reports. The attributes were modified, some of the test items were eliminated, and the Q-matrix was reconstructed. When re-estimation was performed after revision, the parameter values were improved. It was confirmed that the interpretation conducted by the retrospective verbal reports was valid and the effectiveness of our methods for developing a new cognitive diagnostic assessment was demonstrated.
