This paper examines the purposes, effects, roles, and responsibilities of language assessment. The purposes of assessment include the two types of testing decisions: norm-referenced decisions (including aptitude, proficiency, and placement assessments) and criterion-referenced decisions (including diagnosis, progress, and achievement assessments). The effects of assessment are the positive and negative washback effects of tests on language curriculum. To clarify, the concept of superordinate tests is introduced here and discussed in relationship to the washback effect. The roles of assessment are the many ways in which assessments affect curriculum development, particularly in analyzing students', needs, setting goals and objectives, developing program level tests, producing materials, delivering instruction, and evaluating program effectiveness. The responsibilities of assessment include (a) Messick's notions of the evidential and consequential bases of test interpretation and use and (b) Cronbach's ideas on the functional, political, economic, and explanatory perspectives of questions about testing. As examples, the university entrance examinations in Japan are considered in terms of their purposes, effects, roles, and responsibilities within the larger context of Japanese education and society.

View full abstract

In English language teaching in junior and senior high schools, development of listening comprehension tests focusing on oral communication abilities is needed due to the shift of the course of study toward a more communication-oriented approach (Monbusho 1989). Our previous studies (Kinoshita et al. 1996, 1998a, 1998b) reported the formation of a listening comprehension test focusing on oral communication abilities and presented an analysis of the test which was administered to Japanese and Korean high school students. The results of our previous analysis (Kinoshita et al. 1998a, 1998b) indicated that there were no significant differences between the scores of Japanese and Korean test takers, though some differences were found to be statistically significant in the performance of certain aspects of listening skills. Our previous studies, however, could not identify clearly why some differences happened because several conditions of the test takers differed. This study deals with only the test data taken from the test takers under the same condition and reanalyzes the data based on the Classical Test Theory and Item Response Theory. As a result, we find that some test items are biased by cultural differences between Japan and Korea and some differences are due to the differences of the instruction itself. To test listening comprehension abilities focusing on oral communication, it is important to take account of the cultural differences among the test takers at a high school level. Finally, our results suggest that the way to teach listening in Japan needs to be revised because Japanese test takers show poorer performance in test items focusing on everyday situations.

View full abstract

The purpose of this research was to obtain statistical information on test components included in English-language dentistry tests administered over a two year period. Two different types of test items were analyzed in order to creat more appropriate test questions. Both types consisted of multiple-choice questions. Type A was composed of one English-language term as a stem and four Japanese-language equivalents as options. A distractor could be selected in the case of an exam sitter not knowing the correct answer. On the other hand, type B included one Japanese term as a stem and four English equivalents as options. All items in both tests were analyzed using classical test theory (CTT) and item response theory (IRT). Results demonstrated a negatively skewed distribution pattern and the curve was sharply peaked (leptokurtic distribution) in type A. In type B, however, the data spread out in a flat curve (platykurtic distribution). It was statistically proved that easy items in type A were more numerous and the number of considerably difficult items increased in type B.

View full abstract

This paper proposes a new classification scheme for reading test items and reports on its usefulness and limitations when actually applied to three reading tests. In this scheme, an item is placed in one of nine categories based on its position on two dimensions. The first dimension concerns the size of the relevant text portion. If an item can be correctly answered by processing one sentence or a smaller portion, it will be placed in the category SENTENCE; if an item requires two successive sentences to be processed, it will be categorized as LOCAL, and if two non-successive sentences or three or more successive sentences need to be comprehended, the item will be termed GLOBAL. The second dimension concerns the type of cognitive activity required to respond correctly to an item. If an item can be answered by merely comprehending the explicit meaning of the portion and if key words used in the question and the relevant text are identical, the item will be placed in the category VERBATIM; if the answer is explicitly stated but in paraphrase, it will be termed PARAPHRASE; and if the implicit meaning of the relevant part needs to be comprehended to answer the item, it will be classified as IMPLICIT. Thus, each dimension having three categories, the result will be a 9-category (3 categories times 3 categories) taxonomy. Four testing experts classified three reading tests, consisting of 35 items each, by this classification scheme. The results indicated somewhat less than optimal inter-classifier reliability, when the VERBATIM/PARAPHRASE/IMPLICIT dimension was involved, in particular. To compensate for the low agreement rate, Rasch Rating Scale Model (Wright & Masters 1982) was applied to spot 'misfit' ratings to be cyclically deleted and to compute the final position (in Rasch logits) of each item on each dimension. The resultant scatter plots graphically illustrated the characteristics of each reading test, implying a high practical utility of the proposed taxonomy.

View full abstract