Short- and Long-term Reliability of Information on Previous Illness and Family History as Compared with that on Smoking and Drinking Habits in Questionnaire Surveys

To assess the reliability of responses to questionnaires regarding previous illness, family history of cancer, and smoking and drinking habits, we repeated questionnaire surveys four times at intervals of 2 weeks and 1 year (short-term), and 4.5 years (long-term) among 440 subjects aged 40-69. The reliability was assessed using kappa statistic. Kappa was calculated both for complete data and data including missing values. The changes of mode of pre-after paired responses were also investigated. Our results from complete data showed both short- and long-term reliabilities of replies regarding smoking or drinking were excellent (mean kappa 0.85-0.99). The reliability of previous illness was excellent except for stroke for short-term intervals (mean kappa 0.85 -1.00), but varied depending on the kinds of illness with long-term intervals (mean kappa -0.01-0.75). Responses to family history had fair to excellent short-term reliability (mean kappa 0.54-0.85). Inclusion of missing value as an independent category reduced reliability remarkably. Subjects stating absence of medical history were more likely to have missing values for this item than subjects with some history. In conclusion, the reliability for information given on previous illness was as good as that on smoking and drinking for a short interval, but was lower for a long-interval probably due to the development of new cases. The reliability of a family history on cancer was slightly poorer than that of individual’s previous cancer or other illnesses and that of smoking and drinking even for a short interval.


INTRODUCTION
Self-administered questionnaires have been frequently used in epidemiological studies to obtain data concerning exposure variables 1).However, the answers might not be reliable enough, especially when subjects must recall past events such as medical history.Reliability in this instance refers to the degree to which the results obtained by a measurement procedure can be replicated 2).Random variation associated with completing the questionnaire, true change in behavior and experience over time, and recall bias, i.e., reduced preciseness of memory due to time lag, are considered the main factors affecting reliability of questionnaires [3][4][5].Missing values generally receive less attention by researchers or were treated as "never" answ ers in various reliability studies 4, 6, a .Various studies have evaluated the short-term reliability of data concerning disease history, and smoking and drinking habits measured on two occasions one year apart or less 1,[6][7][8] .A few studies have reported the long-term reliability of responses regarding medical history or smoking habits , and the change in reliability of alcohol consumption 4,9, lo) The present study was done, first , to assess the short-and long-term test-retest reliabilities of responses to questions regarding previous illness and family history on cancer by comparing it with that of smoking and drinking habits.The second objective was to investigate the intra-individual changes of paired responses to the items listed above, and the third, to evaluate the effects of missing values on reliability.

MATERIALS AND METHODS
This study was based on information obtained from questionnaire surveys repeated four times as shown in Fig. 1.A baseline survey was carried out as part of the national collaborative cohort study of cancer for 6,742 residents aged 40 to 69 years in a rural town of northeastern Japan in July and August 1988 by a mailed self-administered questionnaire (Q1).In early February 1993, a subsequent questionnaire (Q2) was mailed to 463 subjects who had responded to Q1 and resided in limited districts in the town.Two weeks later, about half of the responders to Q2 were randomly selected and surveyed with a similar questionnaire (Q3), and 214 subjects responded.Finally, 460 subjects who responded to Q1 were surveyed (Q4) again in March 1994, and 447 of them responded.Subjects themselves answered all four questionnaires.Details of this study have been reported elsewhere 3).440 subjects who responded to Q1, Q2 and Q4 were used for analyses.Among them, 172 were men and 268 were women.The length of the interval was 2 weeks between Q2 and Q3 (210 subjects); one year between Q2 and Q4 (440 subjects); and 4.5 years between Q1 and Q2 (440 subjects).The questionnaire used in the four surveys covered health conditions and dietary consumption.The following items which showed identical contents in all four surveys were selected for analyses in this study: 1) smoking and drinking habits, 2) previous and current illness: stroke, myocardial infarction, diabetes, and cancer, and 3) family history on cancer during the past five years.Subjects with smoking or drinking habits were those who currently smoked or drank at each of the four survey points, or who had quit these habits prior to or during the investigation period.Subjects whose parent or sibling had developed one of the following cancers during the previous five years were considered to have a positive family history for cancer: stomach, colorectal, lung, liver, breast, uterus cancer or cancer of other sites.
The test-retest reliability of responses to the aforementioned items was measured with kappa statistic computed by comparing two paired answers in questionnaires on different occasions.Kappa (K) is defined as the agreement beyond chance divided by the amount of agreement possible beyond chance 11) As in most studies, kappa values of greater than 0.75 were taken to represent excellent agreement beyond chance, values between 0.40 and 0.75, fair agreement, and those less than 0.40, poor agreement7,8, [12][13][14] For two surveys conducted over 1 year apart, subjects who answered 'No'or did not respond in the earlier questionnaire but answered 'Yes' in the latter, were assumed to be newly developed cases, and these inconsistent data were treated as consistent responses.Kappa values were thus calculated additionally.To investigate the effects of a missing value on reliability, first, we calculated the kappa statistic for only the subjects who had complete data for both earlier and later questionnaires.Then, we calculated kappa statistic including those with missing values as an independent category of `No Response' according to 3 X 3 cross-tabulation.To further study the effects of missing values on reliability, we also investigated the number of subjects who answered 'Yes' or 'No', and those who did not respond to each item.
Previous illness with cancer was not asked in the baseline survey, and the questionnaires Q2 to Q4 asked about cancer occurring among family members during the previous five years.Therefore, data concerning these questions were analyzed only for Q2 vs. Q3 (2-week interval) and Q2 vs. Q4 (1year interval).All analyses used SPSS statistical package (SPSS for Windows, 10.0, SPSS Inc., USA).Statistical significance was set at p <0.05 and the stability of the estimates reflected by 95% confidence intervals.

RESULTS
Table I shows the reliability of responses by kappa values .
When individuals with missing data were excluded , the kappa value was 0.87 or higher for short-term reliability with both smoking and drinking habits , and still remained over 0.85 for long-term reliability.The kappa values of all previous illnesses examined for short-term reliability reached 0 .85or higher, except for stroke (k=0.5 with a one-year interval  1 were kappa statistics obtained by considering the assumed new cases to be consistent responses.Compared with the previous results obtained from considering all these inconsistent answers as they are, kappa values for short-term reliability was greater than 0.80 in all items.The kappa values for long-term reliability were kept as high as 0.99 and 0.90 for smoking and drinking, and 0.62 or higher for previous illness, respectively. Table 2 shows the percentage and number of subjects with missing values on one occasion.The average percentage was lowest for drinking and smoking habits at 7.9-10 .9%,and was slightly higher at 20.5-29.4% for previous illness and family history than for these habits.The numbers and rates of subjects with missing values on at least one of the questionnaires were   also obtained and shown in Table 2.The average rate of missing values for drinking habit was lowest (14.4%), followed by that of smoking habit (18.6%).The average rate was higher and approximately 35% for previous illness and siblings' history of cancer, reaching the highest level of approximately 44% in the parents' history of cancer.
Table 3 shows the distribution of subjects according to the answering mode to the paired data of pre-after questionnaires for short-term and long-term survey intervals.The number of subjects decreased in Missing-Missing situation and increased in Yes-No, No-Yes, and No-Missing situations, along with the increase in the interval.The subjects whose answer was missing in one occasion answered more frequently 'No' than ' YES' in another occasion for any item.

DISCUSSION
In this study, the responses to smoking and drinking habits, and previous illness showed excellent short-term reliability.This finding was in accordance with those of other investigators 1, 6, 8, 10) Although the short-term reliability for responding to a family history of cancer only had a fair to excellent agreement, a similar study conducted by Tsubono et al. indicated that the reliability of responses to a food frequency question naire was about the same 3) It is clear that smoking or drinking habits can be easily defined and determined by subjects and that such habits relate very closely to their daily life.Previous illness, however, may not always be recalled accurately, especially when the disease has been cured and the time lag becomes longer.Recalling the family history would be even worse, since fewer people live together with their siblings or elderly parents.Wolk et al. revealed that the consistency of reports of foods consumed during adolescence between subjects and their siblings was not high with a mean Pearson correlation of 0.27 7).In addition, it may be possible that people may not be willing to speak openly about their family history of cancer.
Reliability of stroke and myocardial infarction information was fair and excellent, respectively, within one year, but decreased seriously with a long-term interval.People suffering from stroke may be forgetful of the nature of the disease.Besides, these findings would suggest that people might forget acute events even though they were serious ones.On the contrary, history of diabetes shows a high long-term reliability.This might be explained by the fact that people with some chronic diseases such as diabetes are notified of same in an annual physical examination.
In order to eliminate the effects of real changes of behaviors or health status 6,8), subjects who answered 'No' or gave no response in the earlier questionnaire but replied 'Yes' in the latter with a 1-or 4.5-year interval were assumed to have a consistent answer.Little effect on reliability was detected in smoking and drinking.However, a greater effect could be found in previous illness and family history of cancer.This Table 3. Distribution of the number of subjects according to the mode of paired responses on two occasions at short-and long-term intervals.§: Previous illness including stroke, myocardial infarction , and diabetes.
might be due to the low prevalence of these diseases , which results in few cases in this study and small changes in case number would have a great effect on reliability .Obviously such an approach overestimates the reliability of responses .It is likely that a true kappa value should lie somewhere in between the results obtained from these two approaches 6) .One might raise issues concerning the different prevalence of smoking and drinking by gender affecting kappa values .However, because of the small number of subjects in our study , sex-specific kappa values in terms of smoking and drinking were not computed.Lipsitz et al. reported that subjects who were obese at the time of the first questionnaire might be more likely to be missing at the second time 15).This finding is partly in accordance with ours, though, in this study, few subjects who answered ' Yes' in an earlier questionnaire showed no response in a later one.Subjects who gave no answer at one time were more likely to answer 'No' than 'Yes' at another time for any item .Further, it was shown that subjects without previous illness and a family history of cancer had a tendency not to respond to the questions when survey intervals became longer.
Many epidemiological studies often suffer from missing val-ues 16, 17)When subjects with missing values were included in our analyses, kappa values dramatically decreased for all but smoking and drinking habits .A much higher prevalence of smoking and drinking habits than other items might have resulted in better reliability 18) .The higher missing value rates of previous illness and family history compared to smoking and drinking habits might have also contributed to the remarkable decrease in the reliability of these items .A common practice in epidemiological analysis is to eliminate all individuals for whom information is missing (complete case analysis) 19).However, the subjects with complete data are more likely a biased subsample of all subjects , and some nonresponses to items are often non-random in the sense that they occur more frequently in certain subgroups 16,17,19) Although an alternative approach often used in epidemiological analysis is to create an additional category for missing data , it is quite difficult to determine whether or not it is proper to take a missing value as one category in a reliability study 19) .Considering a great difference of reliability brought about by including or excluding the missing values, we have to be cautious in interpreting the reliability of medical history obtained from only the respondents.
It is likely that missing values consist of at least two components, one is from those who really did not know the answers and another, from those who showed no concerns on the items and/or were not willing to answer questions.It might be worth creating an additional category of choice 'Unclear' for items like individual or family history that may easily be affected by recall bias.This approach might be able to distinguish subjects from those who take indifferent attitude to health survey and, therefore, reduce missing value rates and improve the reliability.In addition, the survey method other than mailing selfadministered questionnaire, such as face-to-face interview would be useful in reducing such a high rate of missing value.
The limitations in this study were small sample and information for disease validation could not be obtained.However, our study has several advantages.First, a self-administered questionnaire survey was conducted four times for the same subjects at different intervals.This made it possible not only to assess the reliability of questionnaire answers but also to detect trends in reliability.Second, comparing the reliability of previous illness and family history of cancer with smoking and drinking habits would give us a clear and direct picture to understand to what degree such information is reliable.Third, to our knowledge, this is the first time missing values were made a category in a reliability study.Finally, This study was part of the national collaborative cohort study of cancer and therefore could provide useful information on the reliability of responses to questions for the whole study.Further investigation is required to reassess the reliability as well as validity of the same items using a much larger population.

Figure 1 .
Figure 1.The process of self-administered questionnaire surveys given four times between July 1988 and March 1994 *: p<0 .05;**: p<0.01, by statistics test for unweighted kappa .§ 1: The upper rows represent numbers excluding subjects with missing value; the lower rows include all subjects .§2 : K represents kappa values.§3: 95% confidence interval for = ± 1.96 (SE ).§4: Kappa values in parentheses were the results obtained when possible new cases were taken as consistent responses .§ 5: Myocardial infarction.
, for smoking and drinking.The kappa value for short-term reliability was no more than 0.44 for any previous illness examined (2-week interval), and decreased further to approximately 0.30 (1-year interval).The kappa values for long-term reliability were nearly 0 for all items in previous illness.Reliability of family history of cancer shows results close to that of previous illness.The values in parentheses in Table analyses, the kappa value decreased in all items .The kappa values decreased to 0.70 and 0 .68 for short-term reliability, and further decreased to 0.62 and 0 .64 for long-term reliability, respectively

Table 2 .
Percentage (%) and number (n) of subjects with missing values in one questionnaire survey (Q1-Q4) , and in earlier and/or later surveys at four different intervals, by item.§ 1: The percentage was obtained by excluding Q1. §2: The percentage was obtained by excuding values in long-term interval.§3: MI represents myocardial infarction.§4: The percentage was obtained by excluding previous and family history of cancer.