Compliance and Comparability in a Cross-National Survey

In this paper, we analyze data from the Generations and Gender Programme (GGP) to examine whether comparability in cross-national data leads to greater scientific activity and outputs. Launched in 2000 by the United Nations Economic Commission for Europe (UNECE), the GGP is a longitudinal comparative survey of 18-79 year olds in 20 countries in Europe and beyond, run by a consortium of research institutions. It is based on a relatively decentralized management model and relies on considerable post-hoc harmonization of data. The international “core” questionnaire is either adapted to the different national contexts or partly incorporated into existing surveys. Using data from the surveys administration, we examine the quality of compliance and standardization in the GGP and whether this affects data usage. Firstly, we analyze the extent to which instruments from the core questionnaire were fielded within each of the countries in the GGP. The results show that on average across countries, 66% of instruments in the core questionnaire were captured but this varies widely from 23% to 84%. Secondly, we take administrative data from the GGP website to capture the number of times each country dataset is downloaded, and we study whether those countries with greater compliance are more likely to be used in scientific analysis. We supplement this with the number of times a country dataset is used in peer reviewed comparative publications and cited in scientific articles. The paper concludes with some reflections on the analysis of compliance and usage in assessing the interest of comparative projects.