Comparing the Quality of Personal-Register and Non-Personal-Register Samples in Cross-National Interviewer-Administrated Surveys Using Internal Criteria of Representativeness - Comparative Analysis Based on European Social Survey Data

This presentation examines demographic representativeness of different types of probabilistic samples based on the results of 130 different surveys form 7 rounds of European Social Survey. Focusing on the distinction between personal-register (i.e., individual name) and non-personal-register (i.e., household or address) samples, it demonstrates that the latter exhibit systematically larger gender- and age-biases. Expanding upon a ‘gold standard’ evaluation (Groves 2006), an analysis based on internal criteria (Sodeur 1996, Kohler 2007) leads to the conclusion that the inferior quality of surveys involving interviewer-driven within-household selection of target respondents results from the impact of illegitimate substitutions. Such protocol-defying selection of individuals with higher levels of readiness and availability only superficially improves survey outcome rates while yielding samples of actually inferior quality. The internal-criteria approach provides a straightforward and undemanding way of monitoring representativeness of samples, and proves especially handy when it comes to large cross-country projects as it requires no data external to the survey results, and allows for comparing surveys regardless of possible differences in sampling frames, sampling design and fieldwork execution procedures.

[1] Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646-675.

[2] Kohler, U. (2007). Surveys from inside: An assessment of unit nonresponse bias with internal criteria. Survey Research Methods, 1(2), 55-67.

[3] Sodeur, W. (1997). Interne Kriterien zur Beurteilung von Wahrscheinlichkeitsauswahlen. ZA-Information/Zentralarchiv für Empirische Sozialforschung, 41, 58-82.