Assessing the Quality of Nonprobability Online Panels. the Italian Case.

Online panels are increasingly used in social research. The advantages of online panels are undisputed (i.e., fast data collection, lower costs). However, online panels – and in particular the nonprobability ones – have a number of limitations, due to coverage error and self-selection into the panel. The key issue is the quality of the estimates produced using nonprobability online panels that result not to be representative of the population they intend to represent (AAPOR, 2010; Callegaro et al., 2014). Despite the relevance of this issue, there are few studies that addressed this topic; these studies have found strong evidence for bias and have shown that adjustments strategies may not be effective in compensating for bias. The overall aim of the work is to assess the quality of the estimates produced using an Italian nonprobability online panel and investigate the effectiveness of different post-survey adjustment strategies (i.e., post-stratification, propensity score adjustments) in reducing the bias.

We compare estimates from the online panel and two “gold standards”, computing percentage differences and appropriate statistical tests. When appropriate, we use regression analysis. We then weight the data implementing different post-survey adjustment strategies. As gold standards we use the Multipurpose Survey “Aspects of Everyday Living”, that is a probability-based survey that collects a wealth of information on the socio-economic characteristics and opinions of the Italian population (N=44.974; RR: 78.9%), and administrative data from the Italian Registry Offices. We also use data from the Italian online panel Opinione.net, that is a nonprobability online panel, established in 2011 (see also http://opinione.net/).