Examining “Web Public Opinion Survey” Based on Random Sampling from the Basic Resident Register [Part II]

A Comparison of Response Distributions between Web and Mail Surveys

Abstract

With low response rates posing a challenge to public opinion surveys, the NHK Broadcasting Culture Research Institute conducted an experimental survey to explore the potential of “Web Public Opinion Survey”—a method collecting responses online from respondents randomly selected from the Basic Resident Register after sending advance letters by post. The design and the response rate of the experimental survey were reported in the June 2018 issue of this journal. For this paper, the authors compared the difference in response distributions between the web survey and the “comparative mail survey,” which was conducted for a comparison purpose, to investigate the quality of responses in web surveys. The findings include the following.- To see the difference in response distributions between the web survey and the comparative mail survey, the response ratios for all 447 options were compared. Around 90% of the differences were within 4 points. This suggests there is little difference by method on the whole.- For grid question formats, response options at both ends of the scale were less likely to be chosen and options in the middle were more likely to be chosen in the web survey than in the comparative mail survey.- Regarding open-ended questions, multiple answer questions, questions answered by scores, and sensitive questions, no major difference by survey method was found.These results indicate that web surveys are as valid as conventional mail surveys. However, some questions did have a considerable difference in response distributions, and continuous investigations will be needed. In addition, no effect was observed in terms of the main purpose of introducing web surveys, which was to improve response rates of young respondents. The authors intend to conduct additional surveys, especially with specific groups that are likely to respond to web surveys, to examine the validity and explore further the potential of the web method.