An evaluation of the National Institutes of Health Early Stage Investigator policy: Using existing data to evaluate federal policy

To assist new scientists in the transition to independent research careers, the National Institutes of Health (NIH) implemented an Early Stage Investigator (ESI) policy beginning with applications submitted in 2009. During the review process, the ESI designation segregates applications submitted by investigators who are within 10 years of completing their terminal degree or medical residency from applications submitted by more experienced investigators. Institutes/centers can then give special consideration to ESI applications when making funding decisions. One goal of this policy is to increase the probability of newly emergent investigators receiving research support. Using optimal matching to generate comparable groups pre- and post-policy implementation, generalized linear models were used to evaluate the ESI policy. Due to a lack of control group, existing data from 2004 to 2008 were leveraged to infer causality of the ESI policy effects on the probability of funding applications from 2011 to 2015. This article addresses the statistical necessities of public policy evaluation, finding administrative data can serve as a control group when proper steps are taken to match the samples. Not only did the ESI policy stabilize the proportion of NIH funded newly emergent investigators but also, in the absence of the ESI policy, 54% of newly emergent investigators would not have received funding. This manuscript is important to Research Evaluation as a demonstration of ways in which existing data can be modeled to evaluate new policy, in the absence of a control group, forming a quasi-experimental design to infer causality when evaluating federal policy.