IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0106474.html
   My bibliography  Save this article

The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies

Author

Listed:
  • Stephen A Gallo
  • Afton S Carpenter
  • David Irwin
  • Caitlin D McPartland
  • Joseph Travis
  • Sofie Reynders
  • Lisa A Thompson
  • Scott R Glisson

Abstract

There is a paucity of data in the literature concerning the validation of the grant application peer review process, which is used to help direct billions of dollars in research funds. Ultimately, this validation will hinge upon empirical data relating the output of funded projects to the predictions implicit in the overall scientific merit scores from the peer review of submitted applications. In an effort to address this need, the American Institute of Biological Sciences (AIBS) conducted a retrospective analysis of peer review data of 2,063 applications submitted to a particular research program and the bibliometric output of the resultant 227 funded projects over an 8-year period. Peer review scores associated with applications were found to be moderately correlated with the total time-adjusted citation output of funded projects, although a high degree of variability existed in the data. Analysis over time revealed that as average annual scores of all applications (both funded and unfunded) submitted to this program improved with time, the average annual citation output per application increased. Citation impact did not correlate with the amount of funds awarded per application or with the total annual programmatic budget. However, the number of funded applications per year was found to correlate well with total annual citation impact, suggesting that improving funding success rates by reducing the size of awards may be an efficient strategy to optimize the scientific impact of research program portfolios. This strategy must be weighed against the need for a balanced research portfolio and the inherent high costs of some areas of research. The relationship observed between peer review scores and bibliometric output lays the groundwork for establishing a model system for future prospective testing of the validity of peer review formats and procedures.

Suggested Citation

  • Stephen A Gallo & Afton S Carpenter & David Irwin & Caitlin D McPartland & Joseph Travis & Sofie Reynders & Lisa A Thompson & Scott R Glisson, 2014. "The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies," PLOS ONE, Public Library of Science, vol. 9(9), pages 1-9, September.
    • Handle: RePEc:plo:pone00:0106474
    DOI: 10.1371/journal.pone.0106474
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0106474
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0106474&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0106474?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Richard Van Noorden, 2010. "Metrics: A profusion of measures," Nature, Nature, vol. 465(7300), pages 864-866, June.
    2. Kerry Dwan & Douglas G Altman & Juan A Arnaiz & Jill Bloom & An-Wen Chan & Eugenia Cronin & Evelyne Decullier & Philippa J Easterbrook & Erik Von Elm & Carrol Gamble & Davina Ghersi & John P A Ioannid, 2008. "Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias," PLOS ONE, Public Library of Science, vol. 3(8), pages 1-31, August.
    3. Joost C. F. Winter & Amir A. Zadpoor & Dimitra Dodou, 2014. "The expansion of Google Scholar versus Web of Science: a longitudinal study," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1547-1565, February.
    4. John P. A. Ioannidis, 2011. "Fund people not projects," Nature, Nature, vol. 477(7366), pages 529-531, September.
    5. L. Erik Clavería & Eliseo Guallar & Jordi Camí & José Conde & Roberto Pastor & José R. Ricoy & Eduardo Rodríguez-Farré & Fernando Ruiz-Palomo & Emilio Muñoz, 2000. "Does Peer Review Predict the Performance of Research Projects in Health Sciences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 47(1), pages 11-23, January.
    6. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    7. Meredith Wadman, 2010. "Study says middle sized labs do best," Nature, Nature, vol. 468(7322), pages 356-357, November.
    8. Markowitz, Harry M, 1991. "Foundations of Portfolio Theory," Journal of Finance, American Finance Association, vol. 46(2), pages 469-477, June.
    9. Stephen A Gallo & Afton S Carpenter & Scott R Glisson, 2013. "Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes," PLOS ONE, Public Library of Science, vol. 8(8), pages 1-9, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yu-Wei Chang & Dar-Zen Chen & Mu-Hsuan Huang, 2021. "Do extraordinary science and technology scientists balance their publishing and patenting activities?," PLOS ONE, Public Library of Science, vol. 16(11), pages 1-20, November.
    2. Jürgen Janger & Nicole Schmidt-Padickakudy & Anna Strauss-Kollin, 2019. "International Differences in Basic Research Grant Funding. A Systematic Comparison," WIFO Studies, WIFO, number 61664, January.
    3. Gerald Schweiger & Adrian Barnett & Peter van den Besselaar & Lutz Bornmann & Andreas De Block & John P. A. Ioannidis & Ulf Sandstrom & Stijn Conix, 2024. "The Costs of Competition in Distributing Scarce Research Funds," Papers 2403.16934, arXiv.org.
    4. Ginther, Donna K. & Heggeness, Misty L., 2020. "Administrative discretion in scientific funding: Evidence from a prestigious postdoctoral training program✰," Research Policy, Elsevier, vol. 49(4).
    5. Donna K. Ginther & Misty L. Heggeness, 2020. "Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program," NBER Working Papers 26841, National Bureau of Economic Research, Inc.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Muhammad Raheel & Samreen Ayaz & Muhammad Tanvir Afzal, 2018. "Evaluation of h-index, its variants and extensions based on publication age & citation intensity in civil engineering," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1107-1127, March.
    2. Vîiu, Gabriel-Alexandru, 2016. "A theoretical evaluation of Hirsch-type bibliometric indicators confronted with extreme self-citation," Journal of Informetrics, Elsevier, vol. 10(2), pages 552-566.
    3. Deming Lin & Tianhui Gong & Wenbin Liu & Martin Meyer, 2020. "An entropy-based measure for the evolution of h index research," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2283-2298, December.
    4. Markowitz, Harry, 2014. "Mean–variance approximations to expected utility," European Journal of Operational Research, Elsevier, vol. 234(2), pages 346-355.
    5. Kaur, Jasleen & Radicchi, Filippo & Menczer, Filippo, 2013. "Universality of scholarly impact metrics," Journal of Informetrics, Elsevier, vol. 7(4), pages 924-932.
    6. Waltman, L. & van Eck, N.J.P., 2009. "A Taxonomy of Bibliometric Performance Indicators Based on the Property of Consistency," ERIM Report Series Research in Management ERS-2009-014-LIS, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    7. Takanori Ida & Naomi Fukuzawa, 2013. "Effects of large-scale research funding programs: a Japanese case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1253-1273, March.
    8. Hyeonchae Yang & Woo-Sung Jung, 2015. "A strategic management approach for Korean public research institutes based on bibliometric investigation," Quality & Quantity: International Journal of Methodology, Springer, vol. 49(4), pages 1437-1464, July.
    9. Kristjan Liivamägi, 2015. "Investor Education and Portfolio Diversification on the Stock Market," Research in Economics and Business: Central and Eastern Europe, Tallinn School of Economics and Business Administration, Tallinn University of Technology, vol. 7(1).
    10. Elie Matta & Jean McGuire, 2008. "Too Risky to Hold? The Effect of Downside Risk, Accumulated Equity Wealth, and Firm Performance on CEO Equity Reduction," Organization Science, INFORMS, vol. 19(4), pages 567-580, August.
    11. Vivek Kumar Singh & Prashasti Singh & Mousumi Karmakar & Jacqueline Leta & Philipp Mayr, 2021. "The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5113-5142, June.
    12. Narayan, Paresh Kumar & Sharma, Susan Sunila, 2011. "New evidence on oil price and firm returns," Journal of Banking & Finance, Elsevier, vol. 35(12), pages 3253-3262.
    13. Schaub, Jason & Stander, Willem J. & Montgomery, Paul, 2022. "LGBTQ+ Young People’s Health and Well-being Experiences in Out-of-home Social Care: A scoping review," Children and Youth Services Review, Elsevier, vol. 143(C).
    14. JinHyo Joseph Yun & EuiSeob Jeong & Xiaofei Zhao & Sung Deuk Hahm & KyungHun Kim, 2019. "Collective Intelligence: An Emerging World in Open Innovation," Sustainability, MDPI, vol. 11(16), pages 1-15, August.
    15. Penikas, Henry, 2010. "Copula-Models in Foreign Exchange Risk-Management of a Bank," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 17(1), pages 62-87.
    16. Mutz, Rüdiger & Daniel, Hans-Dieter, 2018. "The bibliometric quotient (BQ), or how to measure a researcher’s performance capacity: A Bayesian Poisson Rasch model," Journal of Informetrics, Elsevier, vol. 12(4), pages 1282-1295.
    17. Paolo Crosetto & Antonio Filippin & Janna Heider, 2015. "A Study of Outcome Reporting Bias Using Gender Differences in Risk Attitudes," CESifo Economic Studies, CESifo Group, vol. 61(1), pages 239-262.
    18. Prakash, Arun J. & Chang, Chun-Hao & Pactwa, Therese E., 2003. "Selecting a portfolio with skewness: Recent evidence from US, European, and Latin American equity markets," Journal of Banking & Finance, Elsevier, vol. 27(7), pages 1375-1390, July.
    19. Riikka Sievänen & Hannu Rita & Bert Scholtens, 2017. "European Pension Funds and Sustainable Development: Trade‐Offs between Finance and Responsibility," Business Strategy and the Environment, Wiley Blackwell, vol. 26(7), pages 912-926, November.
    20. Muhammad Amjed Iqbal & Muhammad Rizwan & Azhar Abbas & Muhammad Sohail Amjad Makhdum & Rakhshanda Kousar & Muhammad Nazam & Abdus Samie & Nasir Nadeem, 2021. "A Quest for Livelihood Sustainability? Patterns, Motives and Determinants of Non-Farm Income Diversification among Agricultural Households in Punjab, Pakistan," Sustainability, MDPI, vol. 13(16), pages 1-14, August.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0106474. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.