Assessing the Measurement of Science Knowledge in the Wellcome Monitor Surveys: An Item Factor Analysis

The Wellcome Trust Monitor Surveys (2009 and 2012) are fielded to two samples : adults (aged 18+), and young people (aged 14-18). The survey contains multiple items designed to assess science knowledge and engagement with biomedicine. Analysis of the first Monitor Survey yielded a curvilinear relationship between age and high and low levels of scientific knowledge. High scorers were lowest among those aged 65 years and over. The proportion of high scorers for the middle age groups was around three in ten (of those aged 35-49 and 50-64). For low scorers, the reverse pattern holds true, and for young people, 14-16 year olds were less likely to be high scorers than 17-18 year olds. A possibility is that a given knowledge item has a different probability of generating a correct response from different age groups even if each age group has the same underlying level of knowledge. This study will evaluate the performance of the literacy items and the extent of any age-related bias. The results will feed into the design of the next wave of the survey but also speak more generally to the question of how to evaluate science knowledge in heterogeneous populations. I propose a single group IFA-IRT model to evaluate the performance of the items for a pooled sample of respondents, and a multigroup IFA-IRT model to identify the presence of differential item functioning (DIF) by age group.  Briefly, the procedure is to fit a series of IFA models to the data, assessing the adequacy of fit for the scales within each subpopulation. The second stage is to look at evidence for DIF/measurement non-invariance on any of the test items with a series of increasingly restrictive nested models. Finally, results from the multigroup analysis suggest that partial measurement invariance across age groups holds.