IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v114y2018i2d10.1007_s11192-017-2609-2.html
   My bibliography  Save this article

Toward predicting research proposal success

Author

Listed:
  • Kevin W. Boyack

    (SciTech Strategies, Inc.)

  • Caleb Smith

    (University of Michigan Medical School)

  • Richard Klavans

    (SciTech Strategies, Inc.)

Abstract

Citation analysis and discourse analysis of 369 R01 NIH proposals are used to discover possible predictors of proposal success. We focused on two issues: the Matthew effect in science—Merton’s claim that eminent scientists have an inherent advantage in the competition for funds—and quality of writing or clarity. Our results suggest that a clearly articulated proposal is more likely to be funded than a proposal with lower quality of discourse. We also find that proposal success is correlated with a high level of topical overlap between the proposal references and the applicant’s prior publications. Implications associated with the analysis of proposal data are discussed.

Suggested Citation

  • Kevin W. Boyack & Caleb Smith & Richard Klavans, 2018. "Toward predicting research proposal success," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 449-461, February.
    • Handle: RePEc:spr:scient:v:114:y:2018:i:2:d:10.1007_s11192-017-2609-2
    DOI: 10.1007/s11192-017-2609-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-017-2609-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-017-2609-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marianne Hörlesberger & Ivana Roche & Dominique Besagni & Thomas Scherngell & Claire François & Pascal Cuxac & Edgar Schiebel & Michel Zitt & Dirk Holste, 2013. "A concept for inferring ‘frontier research’ in grant proposals," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 129-148, November.
    2. Jörg Neufeld & Stefan Hornbostel, 2012. "Funding programmes for young scientists--Do the 'best' apply?," Research Evaluation, Oxford University Press, vol. 21(4), pages 270-279, September.
    3. Rüdiger Mutz & Lutz Bornmann & Hans-Dieter Daniel, 2015. "Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data approach using ex-ante and ex-post peer evaluation data from the Austr," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2321-2339, November.
    4. Viner, Neil & Powell, Philip & Green, Rod, 2004. "Institutionalized biases in the award of research grants: a preliminary analysis revisiting the principle of accumulative advantage," Research Policy, Elsevier, vol. 33(3), pages 443-454, April.
    5. Göran Melin & Rickard Danell, 2006. "The top eight percent: Development of approved and rejected applicants for a prestigious grant in Sweden," Science and Public Policy, Oxford University Press, vol. 33(10), pages 702-712, December.
    6. Klavans, Richard & Boyack, Kevin W., 2017. "Research portfolio analysis and topic prominence," Journal of Informetrics, Elsevier, vol. 11(4), pages 1158-1174.
    7. van den Besselaar, Peter & Sandström, Ulf, 2015. "Early career grants, performance, and careers: A study on predictive validity of grant decisions," Journal of Informetrics, Elsevier, vol. 9(4), pages 826-838.
    8. Martin Reinhart, 2009. "Peer review of grant applications in biology and medicine. Reliability, fairness, and validity," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 789-809, December.
    9. Lutz Bornmann & Gerlind Wallon & Anna Ledin, 2008. "Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes," PLOS ONE, Public Library of Science, vol. 3(10), pages 1-11, October.
    10. Stefan Hornbostel & Susan Böhmer & Bernd Klingsporn & Jörg Neufeld & Markus Ins, 2009. "Funding of young scientist and scientific excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(1), pages 171-190, April.
    11. Peter van den Besselaar & Loet Leydesdorff, 2009. "Past performance, peer review and project selection: a case study in the social and behavioral sciences," Research Evaluation, Oxford University Press, vol. 18(4), pages 273-288, October.
    12. Simen G. Enger & Fulvio Castellacci, 2016. "Who gets Horizon 2020 research grants? Propensity to apply and probability to succeed in a two-step analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1611-1638, December.
    13. Henry Mintzberg & James A. Waters, 1985. "Of strategies, deliberate and emergent," Strategic Management Journal, Wiley Blackwell, vol. 6(3), pages 257-272, July.
    14. Ted von Hippel & Courtney von Hippel, 2015. "To Apply or Not to Apply: A Survey Analysis of Grant Writing Costs and Benefits," PLOS ONE, Public Library of Science, vol. 10(3), pages 1-8, March.
    15. Mark D Lindner & Richard K Nakamura, 2015. "Examining the Predictive Validity of NIH Peer Review Scores," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-12, June.
    16. Bornmann, Lutz & Leydesdorff, Loet & Van den Besselaar, Peter, 2010. "A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications," Journal of Informetrics, Elsevier, vol. 4(3), pages 211-220.
    17. Jacob, Brian A. & Lefgren, Lars, 2011. "The impact of research grant funding on scientific productivity," Journal of Public Economics, Elsevier, vol. 95(9), pages 1168-1177.
    18. Lutz Bornmann & Hans-Dieter Daniel, 2005. "Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(2), pages 297-320, April.
    19. Lutz Bornmann & Hans-Dieter Daniel, 2006. "Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(3), pages 427-440, September.
    20. Thed N. van Leeuwen & Henk F. Moed, 2012. "Funding decisions, peer review, and scientific excellence in physical sciences, chemistry, and geosciences," Research Evaluation, Oxford University Press, vol. 21(3), pages 189-198, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Benjamin M. Knisely & Holly H. Pavliscsak, 2023. "Research proposal content extraction using natural language processing and semi-supervised clustering: A demonstration and comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(5), pages 3197-3224, May.
    2. Jorge A. V. Tohalino & Laura V. C. Quispe & Diego R. Amancio, 2021. "Analyzing the relationship between text features and grants productivity," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(5), pages 4255-4275, May.
    3. Tohalino, Jorge A.V. & Amancio, Diego R., 2022. "On predicting research grants productivity via machine learning," Journal of Informetrics, Elsevier, vol. 16(2).
    4. van den Besselaar, Peter & Mom, Charlie, 2022. "The effect of writing style on success in grant applications," Journal of Informetrics, Elsevier, vol. 16(1).
    5. Győrffy, Balázs & Herman, Péter & Szabó, István, 2020. "Research funding: past performance is a stronger predictor of future scientific output than reviewer scores," Journal of Informetrics, Elsevier, vol. 14(3).
    6. Li, Kai & Yan, Erjia, 2019. "Are NIH-funded publications fulfilling the proposed research? An examination of concept-matchedness between NIH research grants and their supported publications," Journal of Informetrics, Elsevier, vol. 13(1), pages 226-237.
    7. Seeber, Marco & Alon, Ilan & Pina, David G. & Piro, Fredrik Niclas & Seeber, Michele, 2022. "Predictors of applying for and winning an ERC Proof-of-Concept grant: An automated machine learning model," Technological Forecasting and Social Change, Elsevier, vol. 184(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. van den Besselaar, Peter & Sandström, Ulf, 2015. "Early career grants, performance, and careers: A study on predictive validity of grant decisions," Journal of Informetrics, Elsevier, vol. 9(4), pages 826-838.
    2. Andrea Bonaccorsi & Luca Secondi, 2017. "The determinants of research performance in European universities: a large scale multilevel analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1147-1178, September.
    3. Tóth, Tamás & Demeter, Márton & Csuhai, Sándor & Major, Zsolt Balázs, 2024. "When career-boosting is on the line: Equity and inequality in grant evaluation, productivity, and the educational backgrounds of Marie Skłodowska-Curie Actions individual fellows in social sciences an," Journal of Informetrics, Elsevier, vol. 18(2).
    4. Benda, Wim G.G. & Engels, Tim C.E., 2011. "The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science," International Journal of Forecasting, Elsevier, vol. 27(1), pages 166-182.
    5. Benda, Wim G.G. & Engels, Tim C.E., 2011. "The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science," International Journal of Forecasting, Elsevier, vol. 27(1), pages 166-182, January.
    6. Gemma Elizabeth Derrick & Alessandra Zimmermann & Helen Greaves & Jonathan Best & Richard Klavans, 2024. "Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices," Research Evaluation, Oxford University Press, vol. 32(4), pages 648-657.
    7. Gerald Schweiger & Adrian Barnett & Peter van den Besselaar & Lutz Bornmann & Andreas De Block & John P. A. Ioannidis & Ulf Sandstrom & Stijn Conix, 2024. "The Costs of Competition in Distributing Scarce Research Funds," Papers 2403.16934, arXiv.org.
    8. Adriana Bin & Sergio Salles-Filho & Ana Carolina Spatti & Jesús Pascual Mena-Chalco & Fernando Antonio Basile Colugnati, 2022. "How much does a Ph.D. scholarship program impact an emerging economy research performance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 6935-6960, December.
    9. Walker, James & Brewster, Chris & Fontinha, Rita & Haak-Saheem, Washika & Benigni, Stefano & Lamperti, Fabio & Ribaudo, Dalila, 2022. "The unintended consequences of the pandemic on non-pandemic research activities," Research Policy, Elsevier, vol. 51(1).
    10. Marco Cozzi, 2020. "Public Funding of Research and Grant Proposals in the Social Sciences: Empirical Evidence from Canada," Department Discussion Papers 1809, Department of Economics, University of Victoria.
    11. Banal-Estañol, Albert & Macho-Stadler, Inés & Pérez-Castrillo, David, 2019. "Evaluation in research funding agencies: Are structurally diverse teams biased against?," Research Policy, Elsevier, vol. 48(7), pages 1823-1840.
    12. Balázs Győrffy & Andrea Magda Nagy & Péter Herman & Ádám Török, 2018. "Factors influencing the scientific performance of Momentum grant holders: an evaluation of the first 117 research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 409-426, October.
    13. Lawson, Cornelia & Salter, Ammon, 2023. "Exploring the effect of overlapping institutional applications on panel decision-making," Research Policy, Elsevier, vol. 52(9).
    14. Maaike Verbree & Edwin Horlings & Peter Groenewegen & Inge Weijden & Peter Besselaar, 2015. "Organizational factors influencing scholarly performance: a multivariate study of biomedical research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 25-49, January.
    15. Kok, Holmer & Faems, Dries & de Faria, Pedro, 2022. "Pork Barrel or Barrel of Gold? Examining the performance implications of earmarking in public R&D grants," Research Policy, Elsevier, vol. 51(7).
    16. Bornmann, Lutz & Leydesdorff, Loet & Van den Besselaar, Peter, 2010. "A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications," Journal of Informetrics, Elsevier, vol. 4(3), pages 211-220.
    17. Stefan Hornbostel & Susan Böhmer & Bernd Klingsporn & Jörg Neufeld & Markus Ins, 2009. "Funding of young scientist and scientific excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(1), pages 171-190, April.
    18. Peter van den Besselaar & Ulf Sandström & Hélène Schiffbaenker, 2018. "Studying grant decision-making: a linguistic analysis of review reports," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 313-329, October.
    19. Sandström, Ulf & Van den Besselaar, Peter, 2018. "Funding, evaluation, and the performance of national research systems," Journal of Informetrics, Elsevier, vol. 12(1), pages 365-384.
    20. Gill, Chelsea & Mehrotra, Vishal & Moses, Olayinka & Bui, Binh, 2023. "The impact of the pitching research framework on AFAANZ grant applications," Pacific-Basin Finance Journal, Elsevier, vol. 77(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:114:y:2018:i:2:d:10.1007_s11192-017-2609-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.