Options and limitations in measuring the impact of research grants—evidence from Denmark and Norway

Competitive grant schemes are set up with the intention of improving research performance. It may, however, be difficult to find evidence of the intervention impact of research grants for ex post evaluations of grant schemes. Based on data on applicants to Danish and Norwegian open mode grant schemes—research projects as well as post doc fellowships—this article applies difference in difference analysis to study to what extent research grants are likely to affect the publication and citation rates of the principle investigators (PIs). The results show higher increases in the number of publications for grant recipients than for rejected applicants, while increases in mean normalized citation rates were not significantly higher for the successful applicants. In other words, the grants seem to have increased productivity, e.g. by helping PIs to add staff to their research teams, but not to have influenced the importance of the research as measured by average citations. However, along with increases in the number of publications also came a greater increase in the number of highly cited papers for grant recipients than for rejected applicants. In sum, the analyses indicate that the measurement of grant impact is sensitive to how research performance is defined and tested using bibliometric indicators. Furthermore, the applicants’ complex landscape of multiple projects and grants makes it difficult to isolate the output of a single grant. Hence, using bibliometrics to measure the impact of smaller grant schemes and smaller grants may often yield inconclusive results.