Samejima's(1969) graded response model is useful for analyzing test items scored as grade-based. This paper presents two item characteristic curve(item true score) methods of estimating equating coefficients for the item parameters of the graded response model under the common-item design. The first is one in which the criterion function is defined on the both target and new scales like the Haebara's criterion function for the dichotomous response model. The second is one in which the criterion function is defined only on the target scale. The results of simulation experiments, which implemented horizontal and vertical equatings, suggest that the item characteristic curve, the test characteristic curve and the Mean & Sigma methods are preferable to the category characteristic curve and the Mean & Mean methods in terms of standard errors.

View full abstract

This study looked at the effects of declarative knowledge, (information that consists of facts consciously learned in a formal classroom), on procedural knowledge, (knowledge for real use of declarative knowledge), by examining the correlation between the two different types of knowledge. The study also examined the correlation of declarative knowledge with reading in which students presumably use different knowledge intrinsically to just declarative knowledge or procedural knowledge alone. We believe investigation into the relationships of various types of knowledge is crucial to improve the practice of measurement and evaluation, or Language Testing Practice in English education in Japan. The result showed that there was a moderate degree of correlation between declarative knowledge and procedural knowledge from the point of view of fluency and complexity in a writing task, and little correlation between declarative knowledge and accuracy in writing. In terms of the correlation between declarative knowledge and reading, there was little correlation. The possible reasons for the findings, pedagogical implications, and suggestions are discussed.

View full abstract

This study examines the differences between the test-taking strategies used by higher and lower proficiency level learners at Japanese high school in answering three types of reading comprehension tests. The participants were 113 second-year and 79 third-year senior high school students. Written recall and retrospective interview were adopted to gather protocols of test-takers. The data were classified and analyzed according to the two lists of test-taking strategies. The main results were (1) both higher and lower levels of students tend to decide what strategies they would use to read the passage and answer the questions based on the types of the reading comprehension sections, (2) higher level students tend to adopt the strategies to find correct choices by understanding the passage, whereas lower level students are inclined to utilize the strategies to acquire correct choices without comprehending the passage (test-wiseness strategies), (3) higher level students tend to use the test-wiseness strategies in answering the integrated type of reading comprehension sections called 'sougou-mondai,' (4) the percentage of reaching correct answers with the use of test-wiseness strategies in most questions of the three types of reading comprehension tests were over 25%.

View full abstract

The present study attempts to produce a revised version of the Mochizuki Vocabulary Size Test (Mochizuki, 1998). There is room for further improvement of this test, though it has been used as a reliable vocabulary size test for Japanese learners of English. The present study employs FACETS, which is Item Response Theory software, and identifies 21 items to be discarded. Then more suitable new items are chosen based on the JACET List of 8000 Basic Words (JACET, 2003). It was found that the new version is as reliable as the original VST and moreover, the modified version has no misfit items.

View full abstract

This article reports findings from retrospective verbal reports of test-takers. Data included retrospective protocols of 9 examinees on 24 listening comprehension items. Three kinds of test forms were designed for the experiment. These tests are different from each other in styles of presenting multiple-choice options. 3 examinees took a listening test consisting of items whose multiple-choice options were presented in written format, 3 examinees took a test consisting of items whose options were presented in recorded format, and 3 examinees took a test.consisting of items whose options were presented in written format after playing dialogue recorded. Research questions were as follows: (i) why items with written options were less difficult than items with recorded ones in the case of items that test-takers can get an answer by scanning a part of a dialogue; and (ii) whether any other differences of retrospective verbal reports are observed between items with written options and items with recorded ones. The results of the analysis were as follows: (i) test-takers of a test consisting of items with recorded options could not get a keyword for an answer because a keyword was an unfamiliar word or they could not assume what an answer was like after listening to a question, although test-takers of a test consisting of items with written options could get correctly a keyword, or scan a part of a dialogue; (ii) test-takers of a test consisting of items with recorded options sometimes forgot what options were, therefore chose an wrong answer although they understood correctly; and (iii) test-takers of a test consisting of items with written options had difficulty to read multiple choices of complicated sentences during listening.

View full abstract

This paper reports on a development and assessment of a scale of motivation for learning English, targeted on use with Japanese college students, using self-determination theory as the theoretical framework. Study one explored the factor structure of the original data (N=510) through exploratory factor analysis, and yielded six subscales. Confirmatory factor analysis supported an adequate fit of the factor structure with the covariance matrix of the data. Reliability assessment indicated moderate to high internal consistency. A series of correlational analyses among the subscales provided support for the construct validity of the scale. Study two was performed with different participants (N=277) in order to further investigate the reliability and validity of the scale. Reliability was found to be at an adequate level except for the introjected regulation subscale. Inter-subscale correlations were in general agreement with the theoretical continuum, indicating acceptable construct validity. Concurrent validity was assessed through t tests between English-major students and non-English-major students, and through correlations of the subscales with the participants' will to learn English. Both proved an adequate accordance of the scale with the self-determination continuum. Comparisons with other studies indicated that the scale shares important features with other self-determination-based scales of motivation. Particularly, it was found that identified regulation correlated more negatively with amotivation and more positively with will to learn English than did intrinsic motivation, as evidenced by other studies.

View full abstract

Although presenting passages divided with slashes at chunk boundaries, sometimes called 'phrase reading' or 'slash reading,' is a widely used method for raising reading speed in classes, its effect on comprehension is not clear since some studies have shown positive effects but others have not. This may be explained from the aspect of working memory, which concurrently conducts both processing and maintenance. In this study, four types of reading conditions were allotted to 87 Japanese university students. These were: a whole passage, a chunk-cued passage with no pause, a chunk-cued passage with pauses at every chunk, and a chunk-cued passage with pauses at the end of sentences. The current study compared the correlations with memory span for both the non-chunked and chunked conditions. The results showed: (a) in the chunk-cued condition, the load on memory was larger than that of the non-chunked condition, and (b) if pauses were inserted between chunks, the memory load became less.

View full abstract

This paper will consider the issue of test practicality by explaining an item banking system being developed for an in-house proficiency test at University of Tsukuba. We administer English language placement and year-end achievement tests to all first-year university students every year. In lieu of the demand for many test forms and the need for precise assessment of the students each year, all the test forms consist of items calibrated using Item Response Theory (IRT) and some items are repeatedly used on different test forms. Under these circumstances, efficient item management in the form of an item bank has been increasingly in demand. Since there was little information about practical item banking systems, we decided to use Microsoft(R). Access software to develop a user-friendly item bank as part of the in-house test reform project.

View full abstract

This study particularly aims at examining the content characteristics of TOEIC and its relevancy to language curricula in English as Foreign Language (EFL) contexts in Japan. The study consists of two phases. In the first phase of the study, the content analysis of the TOEIC was conducted and the results were compared with those of the Daigaku Nyushi Center Shiken (National Center for University Entrance Examination) and two forms of a private university's entrance examination. In the second phase of the study, 136 university students' test performances on the TOEIC were examined based on Classical Test Theory (CTT) and Rasch Measurement Model (RMM). Also, factors which might affect test-takers performance were examined in terms of test method effects and item characteristics. Even though the results of the study are sample-dependent and generalizability is very limited, the findings were useful to pinpoint the strengths and weaknesses of the students' test performance. For example, low-English proficient learners of the group find the following three items difficult: (a) items that have no explicit clues in stems or response options; (b) items that include domain specific vocabulary; (c) items that require test-takers to make inferences based on the information spread across texts or other types of prompts.

View full abstract