Comparing Modularity Scores across Different Social Networks: Cautions, Illustrations, and Suggestions

In the thriving field of computational social sciences, scholars often employ techniques and methods developed by natural or computer scientists to model social phenomena. One salient example is the recently emerging practice of social scientists comparing modularity scores across different social networks to evaluate the variation of network-module-related substantive concepts such as the levels of consensus, polarization, or community boundary rigidity. Although the rationale for this practice is comprehensible, we caution that it may suffer from various conceptual and empirical issues. Conceptually, modularity scores may misalign with the substantive concepts in which social scientists are interested. Empirically, the commonly used modularity metrics are highly sensitive to various network characteristics that are irrelevant to those substantive concepts. We illustrate these conceptual and analytical problems with toy examples and systematic simulations. We demonstrate the practical significance of these lessons by re-examining an empirical study that uses modularity scores to evaluate the evolution of the rigidity of occupational mobility in the US labor market. We conclude with a summary of these lessons and offer practical suggestions for future applications.