Development and Validation of a 15-Item Japanese Health Knowledge Test

Background Health literacy affects the acquisition of health knowledge and is thus linked to health outcomes. However, few scales have been developed to assess the level of health knowledge among the general public. Methods The 15-item Japanese Health Knowledge Test (J-HKT) was developed by using item response theory to score an item pool. We examined the construct validity of the J-HKT in relation to health literacy items, and analyzed the sociodemographic and behavioral factors associated with poor health knowledge. Results We enrolled 1040 adult participants (mean age, 57 years; women, 52%). The 15 items that best identified people with poor health knowledge were selected. For all items on the J-HKT, the information function curves had a peak in the negative spectrum of the latent trait. As compared with participants reporting high levels of income, educational attainment, and literacy, those with low levels of income, education, and literacy had a lower total score on the J-HKT. As compared with non/light drinkers, moderate and heavy drinkers had lower total scores on the J-HKT. Conclusions The J-HKT may prove useful in measuring health knowledge among the general public, and in identifying and characterizing those with poor health knowledge.


INTRODUCTION
A growing body of evidence supports the impact of low health literacy on the health of individuals 1 ; therefore, recent attention has focused on the elucidation of potential causal pathways linking low health literacy to poor health. 2,3 Among the mechanisms that mediate the influence of health literacy on the health of individuals, the effect of health literacy on health knowledge may be one of the most consistent and critical factors. [4][5][6] It has been proposed that low health literacy leads to poor health knowledge and, ultimately, to worse health outcomes, because people with low health literacy have difficulty in acquiring the health knowledge necessary to navigate the healthcare system and to practice effective self-care.
Patients with poor knowledge of illness prevention and chronic diseases have lower adherence to medical instructions and are more likely to have high-risk health behaviors. [7][8][9][10] Thus, these individuals are less likely to utilize healthcare services, such as recommended vaccination and health screening programs. [11][12][13] In addition, during both acute and chronic illnesses, the quality of self-care is poor among those with limited knowledge, which may manifest in the greater use of potentially harmful complementary or alternative medicine. 14 Many studies have evaluated the relationship between health literacy and health knowledge. 4,5,7,8,15,16 These have mostly focused on patients with specific illnesses, such as asthma, diabetes, congestive heart failure, hypertension, and human immunodeficiency virus infection; few have evaluated the association between low health literacy and poor health knowledge in the general public. In patients with chronic diseases, the relationship between health literacy and health knowledge of a particular disease has been confirmed. 4,5,7,8,15,16 In order to better understand the relationship between health literacy and health knowledge, and to help target education and guide disease prevention for the general public, it would be useful to examine the relationship between health knowledge and health literacy in the general public. However, this objective cannot be realized without a tool to assess general health knowledge. Such a tool would be particularly useful if it were short, if it could differentiate among people at the low end of the health knowledge spectrum, and if it could be administered in a mode other than in-person interview. Thus, in the present study, we used nominal categories modeling of item response theory (IRT) analysis to develop a test of general health knowledge for Japanese adults. To evaluate construct validity, we examined the association between this health knowledge test and health literacy. In addition, we identified the sociodemographic and health behavioral factors that were significantly associated with poor health knowledge.

Study participants
The data for this study were collected from responses to a national cross-sectional online survey conducted from 3 July to 8 July 2008. Institutional review board approval was obtained from the National Institute of Japanese Language. Japan was divided into 10 regions: Hokkaido, Tohoku, Kanto, Tokai, Keihin, Hokuriku, Kyouhanshin, Chugoku, Shikoku, and Kyushu. The number of potential participants was determined within each region from a panel of people registered by Yahoo JAPAN Co. (Tokyo, Japan) by means of probability sampling proportionate to age and sex, using Japanese national census data of population distributions for people aged 30-90 years in 2007. People younger than 30 years were excluded because our aims included evaluation of the potential association between health knowledge and final educational attainment. In addition, health care workers, such as physicians, nurses, hospital workers, and public health workers, were excluded. No gifts or payments were given for participating in the survey.

Data collection
The survey gathered demographic and socioeconomic data, as well as responses to the questionnaire for health literacy and the test of health knowledge. Demographic data included age, sex, annual income, education, and occupation. Regarding annual income, cutoff points of 2, 4, 6, and 8 million Japanese Yen (JY) were used to generate 5 income categories (the average exchange rate for 1 US dollar in July 2008 was about 100 JY). We used these income cutoffs because the National Tax Agency regards an income of 2 million JY as the cutoff level for low-wage workers and reports the income distribution in this fashion. For educational attainment, 5 categories were used (did not graduate high school, high school graduate, vocational school, short-term college, and undergraduate/postgraduate degree). For occupational status, 5 categorical levels were included: working full-time, homemaker, working part-time, retired, and not currently working. Survey items also assessed current and past smoking, current alcohol use, and chronic conditions (cancer, cardiovascular disease, hypertension, diabetes, arthritis, asthma or chronic obstructive pulmonary disease, and depression), as previously described. 2 Current alcohol consumption was categorized into 3 categories: non/light, moderate, and heavy. Non/light drinkers were defined as those who drank less than once a week; moderate and heavy users included those who drank at least once a week. In addition, heavy users were defined as those who drank in a day ≥3 glasses of beer, ≥540 ml of Japanese sake (nihonshu), three-quarters of a bottle or more of wine, or ≥180 ml of whisky. All remaining participants were defined as moderate users. Health literacy was measured by self-report using 2 validated screening questions. 17,18 Specifically, we asked: "How often do you have problems learning about your medical condition because of difficulty understanding written information?" (Item 1: "Problems learning") and "How often do you have someone help you read hospital materials?" (Item 2: "Help reading"). The 5-point Likert response scale was, "Never", "Occasionally", "Sometimes", "Often", or "Always". These 2 items have been shown to predict scores on commonly used English-language measures of health literacy: the Short Test of Functional Health Literacy in Adults (STOFHLA) and the Rapid Estimate of Adult Literacy in Medicine (REALM). 17,18 Due to the linguistic differences between English and Japanese, English-language instruments for measuring health literacy cannot be simply translated. Thus, we used these 2 self-report items as surrogate measures of health literacy. [17][18][19] Development of the Japanese Health Knowledge Test (J-HKT) The first phase of development included item generation by a group of experts in healthcare, literacy, linguistics, and mass media. This 25-member group included physicians, nurses, pharmacists, linguists, journalists, university researchers in communication, and representatives of patient advocate groups. Each item was developed with a single correct response among a list of 4 choices. When providing the item test to the study participants, they were advised that there was a 2-minute time limit for each item. Each item was scored as correct or incorrect.
In the second phase of development, the 48-item pool was shortened using item response theory (IRT) analysis, specifically the nominal categories model. This model was proposed by Bock 20 as an extension of IRT analysis for nominally scored items. As compared with the use of a graded categories model or a binary logistic model, the nominal categories model is more effective in examining the full spectrum of contributions for each item and the possible responses in an instrument. For this purpose, we used a sample size large enough to meet the requirements of nominal categories modeling.
In the nominal categories model, the response probability p ijk that respondent i with a latent trait θ i response to category k (k = 1, 2,…, K j ) of item j is described as follows 21 : where K j denotes the number of the category of item j. We cannot interpret the parameters of the categories independently in the nominal categories model because the equation defined for a response probability to the category contains other parameters. Thus, in order to estimate item parameters, Okubo suggested that a restriction be imposed as follows 22 : The role of the alpha parameter is that of a slope in the linear function. A larger slope implies that the item clearly discriminates the latent trait θ i , while a smaller slope implies low discrimination. The role of the gamma parameter is that of an intercept. A larger intercept gamma suggests that the item is difficult to solve, while a smaller intercept gamma suggests it is easy to solve. Next, the item response category characteristic curve (IRCCC) is determined by the relative relations among parameters; thus, each parameter cannot be interpreted alone. The usual method to analyze the characteristics of items is to draw the IRCCC by using the estimated parameters. The IRCCC is a multinomial logistic regression curve whose independent variable is a factor-in this case, health knowledge.
Item information functions were then generated for each item. Item information function curves were derived from the response probabilities from the IRCCCs. The standard error of measurement curve can be calculated as the reciprocal of the square root of the item information function. Item information functions describe responses at different levels of a latent trait-health knowledge in this study. A combination of all items together was used to generate the test information function, and an item reduction procedure was performed based on the item information functions. Participants with a score that was ≥1 standard deviation lower than the mean were classified as having a low score.
Phase 3 of development sought to support the validity of the J-HKT. The face validity of the J-HKT was confirmed by the aforementioned expert panel. Next, for construct validity, we hypothesized that health literacy would be associated with improved J-HKT scores and thus the association between literacy and J-HKT scores was examined by using the nonparametric test for trend across ordered groups developed by Cuzick. 23 Associations between sociodemographic characteristics and J-HKT scores were evaluated by a logistic regression model that included age and sex, as well as additional variables found to be significant in univariate analyses. Statistical analyses were performed using R version 2.6.6 (R Foundation for Statistical Computing) and STATA 10.0 (College Station, Texas, USA), and graphics were generated using Mathematica version 6.0 (Wolfram Research, Illinois, USA). A 2-tailed P value <0.05 was considered statistically significant.

RESULTS
Of 2500 subjects randomly selected from the online panel, 1074 participated in the study (response rate, 43.0%). Among these, after deleting data from participants working in the health care industry, data for 1040 persons were available for our analysis and were considered as the final sample. Table 1 shows the sociodemographic characteristics of all participants; 52% were women and the mean age was 57 years (range, 30-90). The initial item pool contained 48 items that covered knowledge of body parts, diseases, hospitals, drugs, healthcare systems, health policy, and home care. The expert panel considered these 48 items to have adequate content validity, and to represent the range of patient knowledge required to understand common medical problems. Based on the item information functions of the IRT analysis for health knowledge testing in the 1040 participants, a 15-item J-HKT was produced from the initial 48-item pool ( Table 2 and Supplement).
Regarding each response to individual items of the J-HKT, all IRCCCs of the J-HKT satisfied the assumption of monotonicity, ie, scores for each item were higher among participants with a higher overall J-HKT score. For most items, a greater number of intersections of probability curves of item responses was shifted to the negative spectrum of the latent trait. Figure 1 shows the item information function for individual items of the J-HKT.
To better discriminate between people with poor health knowledge and those with intermediate or higher levels of health knowledge, 15 items with the highest information function at −0.85 (those with the lowest percentile of 20% of overall scores in all participants) of latent trait θ i were included in the J-HKT. Thus, we chose items able to differentiate among people at the low end of health knowledge; as such, the curves for all items of the J-HKT show a peak of the information functions in the negative spectrum of the latent trait. Figure 2 shows a histogram of total scores for the 15-item J-HKT. The mean score was 4.7 and the standard deviation was 1.6; the median score was 5.0 and the mode was 4.0. The score is normally distributed, with a skewness of −0.37 and a kurtosis of −0.38. Figures 3 and 4 show the proportions of participants with poor health knowledge, by responses to the 2 health literacy items ("Problems learning" and "Help reading"). There were statistically significant associations between responses to the health literacy items and total score on the J-HKT (ie, construct validity). Figure 5 shows the item information function curve of the 15-item J-HKT, and Figure 6 depicts the standard error curve of the item information function of the 15-item J-HKT (the standard error is the reciprocal of the item information function). Table 3 shows the distributions of total score on the J-HKT by sociodemographic characteristics, smoking, and alcohol use. Overall, 36% of participants had a score of 0-3, which was defined as poor health knowledge (ie, more than 1 standard deviation below the mean, 4.7 − 1.6 = 3.1). Age, sex, and employment status were not associated with test scores; however, participants with low income and low educational attainment were more likely to have a lower score on the J-HKT. Although smoking status was not associated with J-HKT score, those with higher current alcohol use had a lower total score on the J-HKT. Table 4 presents the results of the logistic regression model for poor health knowledge on the J-HKT (0-3, yes versus no) adjusted for age, sex, income, education, and current alcohol use. Compared with those with an income >8 million JY, those with income ≥2 and <4 million JY were more likely to have poor health knowledge (odds ratio [OR], 1.68; 95% confidence interval [CI], 1.08-2.62) and those with an income <2 million JY were also more likely to have poor health knowledge (1.84; 1.02-3.31). In addition, as compared with university degree holders, those who had not graduated high school were also more likely to have poor health knowledge (2.08; 1.05-4.14). Regarding current alcohol use, as compared with non/light drinkers, poor health knowledge was more likely among moderate drinkers (1.53; 1.12-2.09) and heavy drinkers (2.28; 1. 16-4.47).    . Proportion of participants with poor health knowledge, by response to item 1 ("Problems learning") on the health literacy test. The question was, "How often do you have problems learning about your medical condition because of difficulty understanding written information?" The 5-point Likert response scale was, "Always" (1), "Often" (2), "Sometimes" (3), "Occasionally" (4), and "Never" (5). Participants with lower literacy represented a higher proportion of those with a low score on the Japanese Health Knowledge Test.

DISCUSSION
Using nominal categories modeling of item response theory analysis, we developed the 15-item J-HKT for Japanese adults. The instrument had a good ability to discriminate among those with poor health knowledge. In addition, items on the J-HKT and health literacy instruments were significantly correlated. The proportion of respondents with a low score on the J-HKT was higher among those with low literacy, which provides evidence of construct validity. Further, fully 36% of the participants had poor health knowledge (defined as a score of 0-3 of a possible 15 on the J-HKT). Finally, we found that poor health knowledge was    associated with low income, low educational attainment, and heavier current use of alcohol.
We used nominal categories modeling to elucidate the individual discriminating power and the effect of item position in the initial 48-item pool. This allowed us to identify items with good psychometric characteristics for inclusion in the 15-item J-HKT. Therefore, it is likely that we successfully developed a test that performs well in assessing health knowledge level among people with moderately poor health knowledge.
We chose to focus the discriminating capacity of this test at the low end of health knowledge, for several reasons. First, people with the lowest levels of health knowledge are those who have the worst health outcomes. 16,24,25 An increase in health knowledge among people who already have relatively greater knowledge is desirable, but is not likely to provide the biggest health impact. Next, focusing health resources on people with poor health knowledge is a means of minimizing health disparities. 26 People with poor health knowledge are likely to have more complex illnesses, and management of complex illnesses requires proper adherence to regimens via active patient involvement in treatment, which is more likely when illnesses are better understood. 5 Several limitations should be noted. First, the results of our study were based on an online survey. A high proportion of Japanese adults use the internet, and while this mode of testing is much less expensive and much more convenient than inperson household interviews, it is possible that people in the sampling frame were younger, wealthier, and more educated than the general public. 27 As such, caution should be used in extrapolating our results.
Similarly, while the participation rate in this project is satisfactory for online research, it is likely that the participants were different from nonparticipants. Different methods for sampling the general population or patient populations with experience of frequent visits to clinicians (eg, due to chronic illness) might result in different distributions of J-HKT scores. There might also have been issues related to differential item functioning between participants and nonparticipants. 27 Although this paper presents a careful psychometric evaluation of the 15-item J-HKT, additional research is needed to ensure appropriate calibration.
Third, since this was an online survey, we do not know if the participants had help or discussed the questions with anyone else. The online panel registration system required a personal identification number and password, and did not allow participants to test more than once. However, participants had to read the questions, and poor reading skill may have resulted in an incorrect answer for an item that would have been answered correctly had it been read aloud. Further research in the form of a test-retest evaluation is needed to determine if the results of verbal administration differ from those of the written test. 28 Fourth, based on the item information functions of the IRT analysis for each response to individual items, the content of several responses must be improved. For instance, on item 10, no participants selected responses 3 or 4, and, on item 1, nearly all participants selected response 3. Moreover, several items will require revision because of dynamic changes in the public's awareness of health information, due to rapid turnover in health-related knowledge in this era of rapid technological advance.
In summary, the current study described an online test of health knowledge among Japanese. We carefully evaluated the psychometric properties of this test and produced an instrument that can accurately discriminate among participants with poor health knowledge. The J-HKT is a convenient and valid measure of health knowledge, and can be used for the general Japanese public. Japanese public health practitioners and clinicians can easily use this quick test for the purposes of health education and disease prevention.

SUPPLEMENTS
How to answer the Items below Below are some medical terms that you may encounter or have encountered on various occasions in medical settings or situations. Please select the sentence that you think best describes each term. The objective of this test is to evaluate your awareness of healthcare terms. It is NOT a test to determine the number of correct responses. Please answer each Item based on your knowledge, even if you are unfamiliar with the terms. Item 1 Please select the sentence that best describes the term "Tumor."