Factorial Surveys in Social Psychology: External Validation of a Factorial Survey with Longitudinal and Administrative Data

In Factorial surveys (FSs) the experimental variation allows assessing the causal impact of dimensions on the evaluations, while at the same time the multifactorial design forces respondents to make judgments based on trade-offs. Moreover, the hypothetical nature of the stimuli enables researchers to focus on rarely observed events. The experimental design provides a high amount of internal validity, while the implementation in a survey is assumed to ensure a high amount of external validity. Given this advantages, factorial surveys have a long tradition in many disciplines of social sciences, including social psychology, to measure attitudes, norms, social definitions, and behavioral intentions.

However, from an empirical point of view, it is still unclear to what extent factorial survey show external validity, and in particular, to what extent behavioral intentions measured by a factorial survey help to predict real behavior. Hypothetical decisions and stated preferences might differ from real decisions for many reasons, including discrepancies between planned and real behavior. So far, there are only few empirical studies focusing on the validity of factorial survey results. Most of them suffer from severe limitations such as mixing up different concepts of validity and using different samples for stated and revealed preferences.

To overcome this research gap, in this presentation first theoretical reasons are provided why stated and revealed preferences might differ. Second, results from a validation study based on job-related mobility decisions are presented. The study was performed in a large-scale German panel study, which offers the possibility to utilize non-reactive, administrative data on job search behavior. All in all, the research tries to gain better knowledge about the validity of FSs, but also the designs needed to allow validation of results: Do stated and revealed preferences diverge? If yes, for what reasons? Which design features help to validate FS research?