A test specification (spec) is a generative document from which equivalent language test items or tasks can be produced. There are many formats for specs, but all share two elements: sample(s) of the items/tasks and guiding language that describes the sample(s). Through consensus-building and feedback, specs evolve and stabilize. A complete illustrative test spec is presented, based on workshops held in Japan in late 2011. A new problem in spec-driven test development is posed in this paper: releasability, which refers to whether a spec should be shared outside of the test development team, and if so, when and in what form. The illustrative spec is again used to explore releasability. A number of theoretical questions are posed about spec release, and future research about spec release is encouraged.

View full abstract

Korea's experiment of using an internet-based speaking test of English for university admission qualifications is unprecedented and hence still controversial. In this paper, how to test and rate test-takers' speaking performance of English by internet is briefly discussed, and the possibilities and problems of using this speaking test for university admission qualifications are discussed.

View full abstract

The purposes of this study are (1) to longitudinally investigate the percentages of Japanese university English majors who make (a) good progress, (b) little progress, and (c) negative progress in vocabulary learning during specific years in their university studies; and (2) to describe the longitudinal developmental patterns of the students' vocabulary sizes during their first two years of university. At the beginning of their freshman year (1st year), sophomore year (2nd year), and junior year (3rd year), 109 university students took one of the three parallel forms of Mochizuki's (1998) Vocabulary Size Test (VST). The tests were always administered in April, when the academic year begins in Japan. By comparing the standard errors of difference (SEdiffs) across years, the author examined the progress of students in each year to investigate what percentage of the participants had made good progress, little progress, or negative progress in vocabulary learning. The author also sought to identify developmental patterns in the first half (i.e., the first two years) of students' time at the university. The results showed that 38.5% of the participants made good progress, 51.4% little progress, and 10.1% negative progress in their vocabulary size during their freshman year. These numbers were 57.8%, 26.6%, and 15.6%, respectively, for the sophomore year. Nine longitudinal developmental patterns were found. The most frequent pattern (31.2%) is Type (→↑), in which students made no progress as freshmen but successfully increased their vocabulary size as sophomores. The other patterns are also described in detail. The study revealed that even English majors did not always increase their vocabulary size in the course of one year.

View full abstract

This comparative study examines the effects of a rater-training session in relation to the degree of differences in raters' severity, consistency, and biased interactions between raters who received training before rating (trained raters) and raters without a training session beforehand (untrained raters). Ten raters (five trained and five untrained) rated a total of 40 scripts (20 scripts for each of accuracy and communicability tasks); the analyses were done using FACETS. In addition, questionnaire responses were analyzed to investigate raters' views regarding the effectiveness of the training. The results showed that 1) the raters were not equally severe; untrained raters were more severe than trained raters, 2) both trained and untrained raters behaved consistently in scoring; untrained raters were assumed to be less consistent, 3) a clear distinction cannot be made among trained and untrained raters as a group in terms of rater-task interactions, 4) untrained raters might be more biased in rater-subject interactions than trained raters; this is shown by the untrained raters' inconsistency in scoring. These results imply that rater training more effectively improves raters' scoring consistency than scoring severity.

View full abstract

Testing second language speaking via computer is a fast-growing area in second language assessment. The purpose of this article is threefold: to familiarize readers with the state of computer technology as it can be applied to speaking tests by discussing three speaking tests: the speaking section of the Test of English as a Foreign Language Internet-based test (TOEFL iBT), the Versant English Test (VET), and the Basic English Skill Test Plus (BEST Plus); to summarize the evidence for these tests' validity for potential end-users to evaluate them; and to provide critical issues to consider for developing a computer-delivered speaking test and potential areas for future research.

View full abstract

This study investigated the relationship between the TOEIC Bridge and TOEIC test scores, in particular, the extent to which the TOEIC Bridge test scores can predict the TOEIC test scores. The participants in this study were 292 non-English major students who took both the TOEIC Bridge and TOEIC tests in 2009. They were first-year students enrolled in a private university in Western Japan. Their scores on both tests were statistically examined using regression analysis. The results of the study showed that (1) the scores of the TOEIC Bridge and TOEIC tests were moderately correlated and (2) the TOEIC Bridge scores significantly predicted the TOEIC scores. Equations for estimating the TOEIC scores using the TOEIC Bridge scores were also specified, from which a comparison of the predicted TOEIC scores from the ETS study and the present study was constructed. The results of the comparison showed that the predicted scores from the two studies had similar intercepts and slopes for a certain range of TOEIC Bridge scores, but that the predicted scores diverged above this range.

View full abstract

Among non-Japanese residents who moved to Japan with the aim of finding employment, there are many people who have lived in Japan for a long period with few chances to learn Japanese. Due to the fact that there are many non-Japanese residents whose Japanese reading and writing skills, in particular, are low, it is difficult to use the type of Japanese tests that are currently being set. Therefore, this research utilizes data on language ability and self-evaluation obtained from the Target Persons Assessment of the Japanese Language Learning Support System for Toyota City (JLLSSTC) Proficiency Test, which was developed to target the region's non-Japanese residents. These assessment examinations are composed of questions that test very basic Japanese language abilities; moreover, learners have the advantage of being able to take the test in their native language. This research focuses on non-Japanese residents living in regions with concentrated levels of foreigners, and clarifies the features of their Japanese language abilities and their self-evaluation of those abilities. In particular, we first applied external criteria to the actual assessed Japanese ability levels of 185 respondents. Then, based on this criteria, we were able to extract a more than moderate correlation between performance test results and the various self-evaluation items of listening, speaking, conversational exchange, reading and writing through the results of analysis of 30 self-evaluation items. Based on these results we can conclude that the high correlation found between the total points of the various self-evaluated skills and the classification of assessment results suggests that self-evaluations could become an alternative to placement tests.

View full abstract

The purpose of this study is to explore how learners adjust their listening strategies when listening to inauthentic and authentic materials. Japanese university students listened to inauthentic and authentic materials and answered questionnaires regarding their listening strategy use. Exploratory factor analyses were conducted on two sets of data: data after listening to authentic materials and those after listening to inauthentic materials. Four factors were extracted from the data of each type of material. Three common factors, which were labeled as, Top-down Strategy, Phonology-based Strategy, and Bottom-up Strategy, were found in both materials. However, the items in each factor were slightly different between the inauthentic and the authentic materials. In one of the four factors, categorized as Metacognitive Strategy, different variables were observed depending on the material types. In the inauthentic listening, the items were related to the metacognitive strategies during or after listening, while in the authentic listening the items in the Metacognitive Strategy were about strategies used mainly before listening. Some adjustment of learners' listening strategies was observed between the inauthentic and the authentic materials based on the instruction before listening, and learners adjusted their strategies when they found out the difficulty levels of the listening passages.

View full abstract

This research aims for more valid and efficient in-house pre- and post-testing systems for program evaluation in university language programs. In order to avoid conducting two paper-and-pencil tests for over 5,000 students, about 14% of all students were given a post-test within a large university language program. As our first stage, we attempted to find the best fitting system to equate these pre- and post-tests data. Two IRT (Item Response Theory) models, a two parameter model (2PL), and a three parameter model (3PL) were chosen and combined with five different equating methods, these being mean-mean (MM), mean-sigma (MS), Haebara (HB), Stocking & Lord (SL), and CALR (CR). Eventually ten combinations of IRT model and equating methods were examined for evaluation. The best fitting one was found to be 2PL-CR, while the least fitting one was 3PL-MS. Afterward, the average gain in ability (θ) of about 700 students who took both tests was calculated with both the best and least fitting combinations. The largest difference in the average gain of θ between the two was θ=0.143. The results indicated that we need to keep applying 2PL-CR for accuracy and consistency so that we can compare the average gain of students' θ between two or more school years to monitor changes. As further evidence of program efficacy, regression analyses were run to look into the relationships between the gain of θ and the survey results of students' needs. This was done in order to reflect the voices of the students who gained more ability. The overall research results indicate that pre- and post-testing systems, when conducted appropriately, can yield reliable and efficient student evaluation data within large university language programs.

View full abstract

This study aims to examine the effects of topic choice on written exams for Japanese second language (JSL) students. For this study, 56 JSL students each wrote two discursive compositions (one on a topic of their choice, and one on a given topic), which were used for the Examination for Japanese University Admission for International Students (EJU). The 112 compositions were scored by two raters who received 8-hour rating training. The results showed a 0.68 correlation and that the differences were not statistically significant between the choice topic compositions and the non-choice topic compositions. Although the score of the choice topic compositions should be higher than the score of non-choice topic compositions, only 30% of the students received higher scores with the choice topic compositions. Twenty-seven percent of the students received higher scores with the non-choice topic compositions. For comparison, students were classified into a high-score group and a low-score group. Topic choice appears to have the high-score group at an advantage over the low-score group. These results indicate that topic choice may have an effect on JSL writing scores.

View full abstract

Spelling tests have been used to diagnose learners' spelling acquisition in English speaking countries (L1). Previous studies have focused mainly on (a) spelling acquisition and (b) the relationships among spelling tests and other types of test (e.g., writing tests, proficiency tests, or cognitive function tests). Therefore, the spelling test formats or the reason for using those spelling tests were varied. On the other hand, only a small number of studies focused on L2 learners or EFL learners spelling acquisition. In other words, studies on spelling acquisition have not received much attention in L2 or EFL. The purpose of this study was to diagnose Japanese university students' spelling knowledge (which is closely related to depth of vocabulary knowledge) with recall (production) tests (i.e., story form, timed dictation, list form, L2 to L1 translation task) and recognition tests (i.e., multiple-choice, matching task). In order to identify learners' weaknesses in spelling knowledge, several tasks had to be developed. After quantitatively analyzing scores on spelling tests, the same data were qualitatively analyzed using implicational scaling (Hatch & Lazaraton, 1991) to clarify acquisition order of English spelling among Japanese university students.

View full abstract