Response Bias in Web/Paper General Population Surveys: More Than a Decade of Experiments to Switch from Face-to-Face to Web/Paper in Switzerland

Several societal developments currently challenge general population surveys, such as budget constraints and the respondents’ more active lifestyle, which leads to a lower contact success rate and higher costs in interviewer-based survey designs. At the same time, digitalization increasingly penetrates society. General population surveys are therefore pushed to switch to self-completion, mainly web/paper. Such a mode change raises questions as to whether and how this change affects representation bias. In Switzerland, FORS fields several international general population surveys and has conducted experiments using different designs for switching mode from face-to-face (f2f) to web/paper during the last decade.

In this presentation, we will summarise results of a series of studies focusing on representation bias using experiments fielded in context of the European Social Survey (ESS) 2012, the European Values Study (EVS) 2017 and the International Social Survey Programme (ISSP). We will first present a framework to analyse the risk of representation bias that considers several types of use of the survey data as well as several sources of benchmark data. We then present empirical results from different experiments applying this framework. Based on the EVS 2017, we present results from a complex parallel experiment fielding a f2f survey and four different web/paper designs.

Using data from a national probabilistic multi-wave web/paper survey, MOSAiCH 2020, that includes the ISSP module in the first wave, we analyse effects of different incentives on representation bias and report on effects of attrition on representation bias of a COVID-19 survey panel following up MOSAiCH 2020.

We find that all designs are feasible and provide generally comparable results, but that representation biases do occur across all modes and differ according to the type of use of survey data. Therefore, measures to reduce risks of representation bias should be tailored to the substantial analysis.