IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0106474.html
   My bibliography  Save this article

The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies

Author

Listed:
  • Stephen A Gallo
  • Afton S Carpenter
  • David Irwin
  • Caitlin D McPartland
  • Joseph Travis
  • Sofie Reynders
  • Lisa A Thompson
  • Scott R Glisson

Abstract

There is a paucity of data in the literature concerning the validation of the grant application peer review process, which is used to help direct billions of dollars in research funds. Ultimately, this validation will hinge upon empirical data relating the output of funded projects to the predictions implicit in the overall scientific merit scores from the peer review of submitted applications. In an effort to address this need, the American Institute of Biological Sciences (AIBS) conducted a retrospective analysis of peer review data of 2,063 applications submitted to a particular research program and the bibliometric output of the resultant 227 funded projects over an 8-year period. Peer review scores associated with applications were found to be moderately correlated with the total time-adjusted citation output of funded projects, although a high degree of variability existed in the data. Analysis over time revealed that as average annual scores of all applications (both funded and unfunded) submitted to this program improved with time, the average annual citation output per application increased. Citation impact did not correlate with the amount of funds awarded per application or with the total annual programmatic budget. However, the number of funded applications per year was found to correlate well with total annual citation impact, suggesting that improving funding success rates by reducing the size of awards may be an efficient strategy to optimize the scientific impact of research program portfolios. This strategy must be weighed against the need for a balanced research portfolio and the inherent high costs of some areas of research. The relationship observed between peer review scores and bibliometric output lays the groundwork for establishing a model system for future prospective testing of the validity of peer review formats and procedures.

Suggested Citation

  • Stephen A Gallo & Afton S Carpenter & David Irwin & Caitlin D McPartland & Joseph Travis & Sofie Reynders & Lisa A Thompson & Scott R Glisson, 2014. "The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies," PLOS ONE, Public Library of Science, vol. 9(9), pages 1-9, September.
    • Handle: RePEc:plo:pone00:0106474
    DOI: 10.1371/journal.pone.0106474
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0106474
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0106474&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0106474?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kerry Dwan & Douglas G Altman & Juan A Arnaiz & Jill Bloom & An-Wen Chan & Eugenia Cronin & Evelyne Decullier & Philippa J Easterbrook & Erik Von Elm & Carrol Gamble & Davina Ghersi & John P A Ioannid, 2008. "Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias," PLOS ONE, Public Library of Science, vol. 3(8), pages 1-31, August.
    2. Joost C. F. Winter & Amir A. Zadpoor & Dimitra Dodou, 2014. "The expansion of Google Scholar versus Web of Science: a longitudinal study," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1547-1565, February.
    3. John P. A. Ioannidis, 2011. "Fund people not projects," Nature, Nature, vol. 477(7366), pages 529-531, September.
    4. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    5. Stephen A Gallo & Afton S Carpenter & Scott R Glisson, 2013. "Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes," PLOS ONE, Public Library of Science, vol. 8(8), pages 1-9, August.
    6. Markowitz, Harry M, 1991. "Foundations of Portfolio Theory," Journal of Finance, American Finance Association, vol. 46(2), pages 469-477, June.
    7. Richard Van Noorden, 2010. "Metrics: A profusion of measures," Nature, Nature, vol. 465(7300), pages 864-866, June.
    8. L. Erik Clavería & Eliseo Guallar & Jordi Camí & José Conde & Roberto Pastor & José R. Ricoy & Eduardo Rodríguez-Farré & Fernando Ruiz-Palomo & Emilio Muñoz, 2000. "Does Peer Review Predict the Performance of Research Projects in Health Sciences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 47(1), pages 11-23, January.
    9. Meredith Wadman, 2010. "Study says middle sized labs do best," Nature, Nature, vol. 468(7322), pages 356-357, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yu-Wei Chang & Dar-Zen Chen & Mu-Hsuan Huang, 2021. "Do extraordinary science and technology scientists balance their publishing and patenting activities?," PLOS ONE, Public Library of Science, vol. 16(11), pages 1-20, November.
    2. Jürgen Janger & Nicole Schmidt-Padickakudy & Anna Strauss-Kollin, 2019. "International Differences in Basic Research Grant Funding. A Systematic Comparison," WIFO Studies, WIFO, number 61664, August.
    3. Gerald Schweiger & Adrian Barnett & Peter van den Besselaar & Lutz Bornmann & Andreas De Block & John P. A. Ioannidis & Ulf Sandstrom & Stijn Conix, 2024. "The Costs of Competition in Distributing Scarce Research Funds," Papers 2403.16934, arXiv.org.
    4. Ginther, Donna K. & Heggeness, Misty L., 2020. "Administrative discretion in scientific funding: Evidence from a prestigious postdoctoral training program✰," Research Policy, Elsevier, vol. 49(4).
    5. Donna K. Ginther & Misty L. Heggeness, 2020. "Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program," NBER Working Papers 26841, National Bureau of Economic Research, Inc.
    6. Madsen, Emil Bargmann & Andersen, Jens Peter, 2024. "Funding priorities and health outcomes in Danish medical research," Social Science & Medicine, Elsevier, vol. 360(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Muhammad Raheel & Samreen Ayaz & Muhammad Tanvir Afzal, 2018. "Evaluation of h-index, its variants and extensions based on publication age & citation intensity in civil engineering," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1107-1127, March.
    2. Deming Lin & Tianhui Gong & Wenbin Liu & Martin Meyer, 2020. "An entropy-based measure for the evolution of h index research," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2283-2298, December.
    3. Markowitz, Harry, 2014. "Mean–variance approximations to expected utility," European Journal of Operational Research, Elsevier, vol. 234(2), pages 346-355.
    4. Waltman, L. & van Eck, N.J.P., 2009. "A Taxonomy of Bibliometric Performance Indicators Based on the Property of Consistency," ERIM Report Series Research in Management ERS-2009-014-LIS, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    5. Kristjan Liivamägi, 2015. "Investor Education and Portfolio Diversification on the Stock Market," Research in Economics and Business: Central and Eastern Europe, Tallinn School of Economics and Business Administration, Tallinn University of Technology, vol. 7(1).
    6. Takanori Ida & Naomi Fukuzawa, 2013. "Effects of large-scale research funding programs: a Japanese case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1253-1273, March.
    7. Narayan, Paresh Kumar & Sharma, Susan Sunila, 2011. "New evidence on oil price and firm returns," Journal of Banking & Finance, Elsevier, vol. 35(12), pages 3253-3262.
    8. Schaub, Jason & Stander, Willem J. & Montgomery, Paul, 2022. "LGBTQ+ Young People’s Health and Well-being Experiences in Out-of-home Social Care: A scoping review," Children and Youth Services Review, Elsevier, vol. 143(C).
    9. Penikas, Henry, 2010. "Copula-Models in Foreign Exchange Risk-Management of a Bank," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 17(1), pages 62-87.
    10. Muhammad Amjed Iqbal & Muhammad Rizwan & Azhar Abbas & Muhammad Sohail Amjad Makhdum & Rakhshanda Kousar & Muhammad Nazam & Abdus Samie & Nasir Nadeem, 2021. "A Quest for Livelihood Sustainability? Patterns, Motives and Determinants of Non-Farm Income Diversification among Agricultural Households in Punjab, Pakistan," Sustainability, MDPI, vol. 13(16), pages 1-14, August.
    11. Nikolaos Pandis & Padhraig S Fleming & Helen Worthington & Kerry Dwan & Georgia Salanti, 2015. "Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-10, September.
    12. Ricardo Eito-Brun & María Ledesma Rodríguez, 2016. "50 years of space research in Europe: a bibliometric profile of the European Space Agency (ESA)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(1), pages 551-576, October.
    13. Jérôme Héricourt & Clément Nedoncelle, 2015. "Relative Real Exchange-Rate Volatility, Multi-Destination Firms and Trade: Micro Evidence and Aggregate Implications," Working Papers 2015-03, CEPII research center.
    14. Franceschini, Fiorenzo & Maisano, Domenico, 2011. "Structured evaluation of the scientific output of academic research groups by recent h-based indicators," Journal of Informetrics, Elsevier, vol. 5(1), pages 64-74.
    15. Allen, D.E. & McAleer, M.J. & Powell, R.J. & Singh, A.K., 2015. "Down-side Risk Metrics as Portfolio Diversification Strategies across the GFC," Econometric Institute Research Papers EI2015-32, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.
    16. Hui-Zhen Fu & Yuh-Shan Ho, 2013. "Comparison of independent research of China’s top universities using bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 259-276, July.
    17. Cinzia Daraio & Simone Di Leo & Loet Leydesdorff, 2022. "Using the Leiden Rankings as a Heuristics: Evidence from Italian universities in the European landscape," LEM Papers Series 2022/08, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    18. Llopis, Oscar & D'Este, Pablo & McKelvey, Maureen & Yegros, Alfredo, 2022. "Navigating multiple logics: Legitimacy and the quest for societal impact in science," Technovation, Elsevier, vol. 110(C).
    19. Briec, Walter & Kerstens, Kristiaan, 2010. "Portfolio selection in multidimensional general and partial moment space," Journal of Economic Dynamics and Control, Elsevier, vol. 34(4), pages 636-656, April.
    20. Viktoriia Myronchuk & Oleksandr Yatsenko & Dmytro Riznyk & Olena Hurina & Andrii Frolov, 2024. "Financing Sustainable Development: Analysis of Modern Approaches and Practices in the Context of Financial and Credit Activities," International Journal of Economics and Financial Issues, Econjournals, vol. 14(5), pages 317-329, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0106474. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.