IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/26841.html
   My bibliography  Save this paper

Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program

Author

Listed:
  • Donna K. Ginther
  • Misty L. Heggeness

Abstract

The scientific community is engaged in an active debate on the value of its peer-review system. Does peer review actually serve the role we envision for it—that of helping government agencies predict what ideas have the best chance of contributing to scientific advancement? Many federal agencies use a two-step review process that includes programmatic discretion in selecting awards. This process allows us to determine whether success in a future independent scientific-research career is more accurately predicted by peer-review recommendations or discretion by program staff and institute leaders. Using data from a prestigious training program at the National Institute of Health (NIH), the Ruth L. Kirschstein National Research Service Award (NRSA), we provide evidence on the efficacy of peer review. We find that, despite all current claims to the contrary, the existing peer-review system works as intended. It more closely predicts high-quality science and future research independence than discretion. We discover also that regression discontinuity, the econometric method typically used to examine the effect of scientific funding, does not fit many scientific-funding models and should only be used with caution when studying federal awards for science.

Suggested Citation

  • Donna K. Ginther & Misty L. Heggeness, 2020. "Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program," NBER Working Papers 26841, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:26841
    Note: LS PR
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w26841.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    2. Benavente, José Miguel & Crespi, Gustavo & Figal Garone, Lucas & Maffioli, Alessandro, 2012. "The impact of national research funds: A regression discontinuity approach to the Chilean FONDECYT," Research Policy, Elsevier, vol. 41(8), pages 1461-1475.
    3. Pierre Azoulay & Joshua S Graff Zivin & Danielle Li & Bhaven N Sampat, 2019. "Public R&D Investments and Private-sector Patenting: Evidence from NIH Funding Rules," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 86(1), pages 117-152.
    4. Stephen A Gallo & Afton S Carpenter & David Irwin & Caitlin D McPartland & Joseph Travis & Sofie Reynders & Lisa A Thompson & Scott R Glisson, 2014. "The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies," PLOS ONE, Public Library of Science, vol. 9(9), pages 1-9, September.
    5. Guido W. Imbens, 2015. "Matching Methods in Practice: Three Examples," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 373-419.
    6. Stephen A Gallo & Joanne H Sullivan & Scott R Glisson, 2016. "The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications," PLOS ONE, Public Library of Science, vol. 11(10), pages 1-18, October.
    7. Ayoubi, Charles & Pezzoni, Michele & Visentin, Fabiana, 2019. "The important thing is not to win, it is to take part: What if scientists benefit from participating in research grant competitions?," Research Policy, Elsevier, vol. 48(1), pages 84-97.
    8. Danielle Li, 2017. "Expertise versus Bias in Evaluation: Evidence from the NIH," American Economic Journal: Applied Economics, American Economic Association, vol. 9(2), pages 60-92, April.
    9. Sabrina T. Howell, 2017. "Financing Innovation: Evidence from R&D Grants," American Economic Review, American Economic Association, vol. 107(4), pages 1136-1164, April.
    10. Pierre Azoulay & Joshua S. Graff Zivin & Gustavo Manso, 2013. "National Institutes of Health Peer Review: Challenges and Avenues for Reform," Innovation Policy and the Economy, University of Chicago Press, vol. 13(1), pages 1-22.
    11. Misty L. Heggeness & Donna K. Ginther & Maria I. Larenas & Frances D. Carter-Johnson, 2018. "The Impact of Postdoctoral Fellowships on a Future Independent Career in Federally Funded Biomedical Research," NBER Working Papers 24508, National Bureau of Economic Research, Inc.
    12. repec:nas:journl:v:115:y:2018:p:2952-2957 is not listed on IDEAS
    13. Massimo G. Colombo & Luca Grilli & Lucia Piscitello & Cristina Rossi-Lamastra (ed.), 2011. "Science and Innovation Policy for the New Knowledge Economy," Books, Edward Elgar Publishing, number 13746.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kok, Holmer & Faems, Dries & de Faria, Pedro, 2022. "Pork Barrel or Barrel of Gold? Examining the performance implications of earmarking in public R&D grants," Research Policy, Elsevier, vol. 51(7).
    2. Lawson, Cornelia & Salter, Ammon, 2023. "Exploring the effect of overlapping institutional applications on panel decision-making," Research Policy, Elsevier, vol. 52(9).
    3. Bol, Thijs & de Vaan, Mathijs & van de Rijt, Arnout, 2022. "Gender-equal funding rates conceal unequal evaluations," Research Policy, Elsevier, vol. 51(1).
    4. Azoulay, Pierre & Greenblatt, Wesley H. & Heggeness, Misty L., 2021. "Long-term effects from early exposure to research: Evidence from the NIH “Yellow Berets”," Research Policy, Elsevier, vol. 50(9).
    5. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Working Papers 26889, National Bureau of Economic Research, Inc.
    6. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Chapters, in: Innovation and Public Policy, pages 117-150, National Bureau of Economic Research, Inc.
    7. Cruz-Castro, Laura & Sanz-Menendez, Luis, 2021. "What should be rewarded? Gender and evaluation criteria for tenure and promotion," Journal of Informetrics, Elsevier, vol. 15(3).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ginther, Donna K. & Heggeness, Misty L., 2020. "Administrative discretion in scientific funding: Evidence from a prestigious postdoctoral training program✰," Research Policy, Elsevier, vol. 49(4).
    2. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Working Papers 26889, National Bureau of Economic Research, Inc.
    3. Elizabeth Webster & Alfons Palangkaraya & Paul H. Jensen & Russell Thomson, 2024. "The effects of university-industry collaboration subsidies on firm performance," Melbourne Institute Working Paper Series wp2024n14, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.
    4. Margaret K. Kyle, 2019. "The Alignment of Innovation Policy and Social Welfare: Evidence from Pharmaceuticals," NBER Chapters, in: Innovation Policy and the Economy, Volume 20, pages 95-123, National Bureau of Economic Research, Inc.
    5. Kok, Holmer & Faems, Dries & de Faria, Pedro, 2022. "Pork Barrel or Barrel of Gold? Examining the performance implications of earmarking in public R&D grants," Research Policy, Elsevier, vol. 51(7).
    6. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Chapters, in: Innovation and Public Policy, pages 117-150, National Bureau of Economic Research, Inc.
    7. Wagener, Andreas & Zenker, Juliane, 2018. "Decoupled but not neutral: The effects of stochastic transfers on investment and incomes in rural Thailand," TVSEP Working Papers wp-008, Leibniz Universitaet Hannover, Institute for Environmental Economics and World Trade, Project TVSEP.
    8. Raphael Calel & Jonathan Colmer & Antoine Dechezleprêtre & Matthieu Glachant, 2025. "Do Carbon Offsets Offset Carbon?," American Economic Journal: Applied Economics, American Economic Association, vol. 17(1), pages 1-40, January.
    9. John Van Reenen, 2022. "Innovation and Human Capital Policy," NBER Chapters, in: Innovation and Public Policy, pages 61-83, National Bureau of Economic Research, Inc.
    10. Tymon Słoczyński, 2022. "Interpreting OLS Estimands When Treatment Effects Are Heterogeneous: Smaller Groups Get Larger Weights," The Review of Economics and Statistics, MIT Press, vol. 104(3), pages 501-509, May.
    11. Hünermund, Paul & Czarnitzki, Dirk, 2019. "Estimating the causal effect of R&D subsidies in a pan-European program," Research Policy, Elsevier, vol. 48(1), pages 115-124.
    12. Unsal, Omer & Houston, Reza, 2024. "R&D grants and medical innovation," Journal of Economics and Business, Elsevier, vol. 128(C).
    13. Ali Merima & Shifa Abdulaziz B. & Shimeles Abebe & Woldeyes Firew, 2017. "Working Paper 290 - Building Fiscal Capacity The role of ICT," Working Paper Series 2404, African Development Bank.
    14. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    15. Marco Cozzi, 2020. "Public Funding of Research and Grant Proposals in the Social Sciences: Empirical Evidence from Canada," Department Discussion Papers 1809, Department of Economics, University of Victoria.
    16. Bergemann, Dirk & Ottaviani, Marco, 2021. "Information Markets and Nonmarkets," CEPR Discussion Papers 16459, C.E.P.R. Discussion Papers.
    17. McLaughlin, Joanne Song, 2017. "Does Communist party membership pay? Estimating the economic returns to party membership in the labor market in China," Journal of Comparative Economics, Elsevier, vol. 45(4), pages 963-983.
    18. Shooshan Danagoulian, 2018. "Policy of prevention: Medical utilization under a wellness plan," Health Economics, John Wiley & Sons, Ltd., vol. 27(11), pages 1843-1858, November.
    19. Martin Beraja & David Y Yang & Noam Yuchtman, 2023. "Data-intensive Innovation and the State: Evidence from AI Firms in China," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(4), pages 1701-1723.
    20. Nan L. Maxwell & Dana Rotz, 2017. "Potential Assistance for Disadvantaged Workers: Employment Social Enterprises," Journal of Labor Research, Springer, vol. 38(2), pages 145-168, June.

    More about this item

    JEL classification:

    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity
    • O3 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights
    • O38 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Government Policy

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:26841. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.