In this paper I discuss an approach to language assessment development and use that is elaborated in Bachman and Palmer (2010) and Bachman and Damböck (Forthcoming). I will first discuss the genesis of our approach, some limitations of other approaches to validity/validation, and the basic premises of our approach. I will then discuss the uses of language assessments and the notion of accountability. I will then briefly describe an “assessment use argument” (AUA), as a series of claims and warrants, along with the qualities of the claims in an AUA. Finally, after a brief summary, I will address some questions that have been asked about our approach.

View full abstract

In the Korea’s educational settings, assessment has been done mainly by multiple choice question tests. It is because securing the objectivity and reliability of the tests tops the priority list. As a result, now even in elementary schools the MCQ test has become a standard method of assessment. This poses no-small problems in school education: Students’ passive attitude is systematically encouraged in schools, and creativity-fostering is systematically blocked. If passive attitude, instead of creativity, keeps being encouraged, school education may cease to have hope for the future, because creativity seems to be one of the key competences that are needed for the future society. For this presentation, a research was conducted using a delphi-type questionnaire and an interview-type discussion to 47 highly-experienced elementary English teachers. The subjects were asked to present their views about creativity and what they were doing for creativity-fostering and how the creativity fostering English education could be made feasible in the ordinary English classes. Their opinions expressed in the delphi-type questionnaire was fully elucidated by an interview-type group discussion with the researcher. On the basis of the research, it was investigated and argued why and how creativity could be fostered by school assessment in a typical EFL context.

View full abstract

This study examined what linguistic variables affecting the cognitive process in reading comprehension determine the difficulty of Eiken reading passages. Using Coh-Metrix, a corpus analysis of Eiken first-grade to third-grade passages was run to compute lexical (word frequency and lexical diversity), syntactic (syntactic similarity), and meaning construction indices (argument overlap and occurrence of causal connectives and verbs). A stepwise discriminant function analysis showed that surface-level linguistic variables (i.e., lexical and syntactic indices) were stronger predictors in the discrimination of Eiken test grades than the linguistic variables affecting higher-level language processing. To verify whether these results corresponded with Japanese EFL learners’ reading performance, Japanese university students completed recall tasks after reading second-grade and third-grade passages. A stepwise multiple regression analysis found that word frequency, lexical diversity, and syntactic similarity indices explained their recall productions. Consistent with the corpus analysis, the meaning construction indices did not explain the recall performances. These findings suggest that the difficulty of Eiken reading passages have probably been designed to measure learners’ lower-level language processing abilities, such as word recognition and syntactic parsing.

View full abstract

This study produced six types of tests to measure knowledge of multiple meanings of one word. Despite the abundance of vocabulary research for some decades, the amount of research on words with multiple meanings is scarce compared to research on other aspects of vocabulary, such as vocabulary size and collocation. This is largely due to the lack of tests for measuring learners’ knowledge of words with multiple meanings. Such tests have not been developed thus far; hence, this study tried to find better tests for words with multiple meanings. Using seven target verbs, three test formats with either sentential or collocational contexts were prepared. The test formats were selection (selecting the correct expressions), translation (translating the target English expressions into Japanese), and pairing (finding pairs that have the same meaning among the target words). These formats were compared in terms of reliability, discriminability, difficulty, and correlation coefficients with estimated vocabulary size and reading ability. About half of the participants took these tests in the sentential context, and the others answered the questions in the collocational context. The results showed that the translation tests had the highest reliability in both contexts. The translation format also had the highest discriminability in the sentential context, but the pairing test had the highest discriminability in the collocation context. In terms of difficulty, the translation tests were affected by the length of the context more than the selection and the pairing tests were. The correlation coefficients for vocabulary size and reading ability were highest in the translation tests for both the sentence and the collocation groups. Considering these results as well as test practicality, this study concluded that the translation test in the sentential context was the best test for knowledge of words with multiple meanings out of the six different tests.

View full abstract

The present study describes the first step in the development and validation of a task-based reading performance test. We designed 6 information transfer task test items in which test-takers were required to transfer what they comprehended from passages (e.g., reading a travel schedule and communicating it to circle members via email). To validate extrapolations on the construct validity of our task-based reading performance test, this study examined the reliability of the test scores by performing a generalizability theory study and a qualitative analysis of rating approaches. In particular, we considered 3 factors (task characteristics, number of raters, and type of rating scale) that affect the reliability of observed scores to obtain an appropriate rating scale and procedure. Over 3 weeks of English classes, 122 Japanese university students completed the 6 different reading tasks. Their reading task outcomes were scored by 6 raters using either a task-dependent or task-independent rating scale. A generalizability study suggested that the 2 types of rating scale could be used alternatively, but qualitative analysis revealed that the 2 rating procedures differed in scoring of local errors associated with detailed information, appropriate reorganization of passage contents, and appropriateness of sociolinguistic elements. Moreover, a decision study demonstrated that the reliability of the observed scores was strongly affected by the number of tasks. To obtain a strictly high-reliability coefficient (.80), 7 tasks by 3 raters are desirable using the task-independent rating scale, while 9 tasks by 3 raters are necessary using the task-dependent rating scale. This study suggested the applicability of task-based reading performance tests and points to be noted for the test implementation from the viewpoints of test material development, scoring procedures, and possible washback effects on teaching and learning of English as a foreign language.

View full abstract

The aim of this study is to observe the effects of text length and input mode on speaking performances in retelling tasks. A total of 53 participants performed their speaking skills for four different task condition test sets. Their performances were analyzed in terms of complexity, accuracy, and fluency; in addition to those data, the score of objective evaluations were compared. The results revealed that the participants could perform more words – that is to say, the number and length of utterances – under the reading input condition. On the other hand, under the listening condition, the participants could perform their speaking skills more fluently. The factor of text length partially affects the number and length of utterances and fluency, and basically the participants could not perform well under the Long Listening condition. The indices of accuracy and complexity do not have an interaction, except the Guiraud index, but overall the text length and input modes do not affect the accuracy and complexity.

View full abstract