Comparability of International Survey Projects and Ex-Post Harmonization: Data Documentation and Quality Controls

This paper proposes methodology to assess and account for variation in survey quality as reflected in the general documentation of extant international survey projects. Poorly documented data reduce user confidence, while inconsistencies between codebook and technical reports on one hand, and the data records in the computer files, on the other, make the data difficult to use. Researchers harmonizing cross-national surveys ex-post encounter these problems early on, since existing projects vary substantially in how transparent and accurate the descriptions of survey design and implementation, and of data coding, are. To facilitate greater inter-survey comparability in these dimensions, we develop a schema of data quality indicators that takes into account accumulated knowledge from the Total Survey Error, Total Survey Quality, and Total Quality Management frameworks. We apply the schema to 1,721 national surveys in 22 international projects and construct metadata composed of three sets of control variables: variables describing the quality of data documentation; variables capturing inconsistencies between survey documentation and data records; and variables revealing errors in data records. We analyze the extent to which these control variables correlate with substantive variables such as trust in institutions, interest in politics, or attending demonstrations. We propose that researchers can use the metadata in statistical analyses to partial out the effect of variation in source data quality on variables of substantive interest.