Factorial Surveys in Social Psychology: Testing within Respondent Variance Homogeneity in Factorial Surveys

That the residual variance of responses net of experimental treatments and additional indicators is homogenous is a fundamental assumption of factorial surveys. Therefore, it has become common practice to control the variance heterogeneity between individuals induced by differing response situations using random effects when analyzing factorial surveys. But based on theories about personality differences between individuals it is expectable that the residual variance is also heterogeneous within respondents. Specifically, we hypothesize that the variance component within respondents increases with their expressiveness. Moreover, since expressiveness is a general personality trait it is plausible that the type of variance heterogeneity it induces constitutes a common type of measurement error.

We model the within respondent variance of factorial surveys using mixed-effect location scale analysis and test the influence of expressiveness on this variance component. As test cases we use two factorial surveys on tertiary students’ preferences with respect to internships. In both surveys expressiveness is measured using the standard items of the respective Big Five sub-scale. One of the samples used consists of respondents experienced in answering surveys while the other sample is composed of inexperienced individuals in this regard, enabling us to assess the effect of practice on expressiveness induced measurement error. Furthermore, we check how the influence of expressiveness changes if we control other sources of measurement error in factorial surveys (like e.g., violations of the interval scaling assumption regarding response behavior). In addition, we evaluate the impact of expressiveness based measurement error on estimated individual preferences. First results indicate support for our central hypothesis, the within respondent variance of factorial survey answers is larger for students with a higher expressiveness score.