Social Desirability Bias in on-Line Surveys: A Comparison Among Different Sources of Respondents

On-line surveys have been commonly used due to its advantages of lower cost and higher efficiency. It also has a higher possibility to reach hidden population that are less contacted by in-person or telephone interviews. However, lack of sampling frame, non-probability samples, and coverage issues often result in lower data quality and skewed response distributions. Researchers have suggested to increase sample size or use various channels for recruitment to improve heterogeneity of the respondents. Recent studies have collected emails from probability samples as an alternative for recruiting on-line survey respondents. However, it remains unclear whether the data quality of the latter is better than that of the former.

Concerning for data quality, social desirability bias is found to be an important issue. Social desirability (SD) bias is related to how survey questions are measured, whether an interviewer is involved, pace of cognition process during interviews, sensitivity of survey questions, etc. The results of SD measures can be an indicator of data quality.

This study aims to compare data quality from different resources of respondents in an on-line survey. These sources include probability samples using previously collected emails from cross-sectional surveys and from a large-scale panel survey. The third sample was recruited using both on-line and off-line advertisement and flyers. Given the different levels of rapport developed between respondents and the survey institute, it is expected that probability sample from the panel survey has the highest level of SD bias, while the non-probability sample has the lowest. However, the distributions show the opposite findings. Social-demographic characteristics, as well as life style, are also compared to understand the dis/similarity among different sources of samples of on-line surveys. The findings are expected to contribute to the field of survey methodology for further understand on-line samples.