IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v16y2022i2s1751157722000414.html
   My bibliography  Save this article

What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports

Author

Listed:
  • Hren, Darko
  • Pina, David G.
  • Norman, Christopher R.
  • Marušić, Ana

Abstract

The evaluation of grant proposals is an essential aspect of competitive research funding. Funding bodies and agencies rely in many instances on external peer reviewers for grant assessment. Most of the research available is about quantitative aspects of this assessment, and there is little evidence from qualitative studies. We used a combination of machine learning and qualitative analysis methods to analyse the reviewers' comments in evaluation reports from 3667 grant applications to the Initial Training Networks (ITN) of the Marie Curie Actions under the Seventh Framework Programme (FP7). Our results show that the reviewers' comments for each evaluation criterion were aligned with the Action's prespecified criteria and that the evaluation outcome was more influenced by the proposals’ weaknesses than by their strengths.

Suggested Citation

  • Hren, Darko & Pina, David G. & Norman, Christopher R. & Marušić, Ana, 2022. "What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports," Journal of Informetrics, Elsevier, vol. 16(2).
    • Handle: RePEc:eee:infome:v:16:y:2022:i:2:s1751157722000414
    DOI: 10.1016/j.joi.2022.101289
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157722000414
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2022.101289?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Győrffy, Balázs & Herman, Péter & Szabó, István, 2020. "Research funding: past performance is a stronger predictor of future scientific output than reviewer scores," Journal of Informetrics, Elsevier, vol. 14(3).
    2. Tohalino, Jorge A.V. & Amancio, Diego R., 2022. "On predicting research grants productivity via machine learning," Journal of Informetrics, Elsevier, vol. 16(2).
    3. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    4. David G Pina & Darko Hren & Ana Marušić, 2015. "Peer Review Evaluation Process of Marie Curie Actions under EU’s Seventh Framework Programme for Research," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-15, June.
    5. Marco Seeber & Jef Vlegels & Mattia Cattaneo, 2022. "Conditions that do or do not disadvantage interdisciplinary research proposals in project evaluation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(8), pages 1106-1126, August.
    6. Lutz Bornmann & Markus Wolf & Hans-Dieter Daniel, 2012. "Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 843-856, June.
    7. Dzieżyc, Maciej & Kazienko, Przemysław, 2022. "Effectiveness of research grants funded by European Research Council and Polish National Science Centre," Journal of Informetrics, Elsevier, vol. 16(1).
    8. Peter van den Besselaar & Ulf Sandström & Hélène Schiffbaenker, 2018. "Studying grant decision-making: a linguistic analysis of review reports," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 313-329, October.
    9. Bayindir, Esra Eren & Gurdal, Mehmet Yigit & Saglam, Ismail, 2019. "A Game Theoretic Approach to Peer Review of Grant Proposals," Journal of Informetrics, Elsevier, vol. 13(4).
    10. Ayoubi, Charles & Pezzoni, Michele & Visentin, Fabiana, 2019. "The important thing is not to win, it is to take part: What if scientists benefit from participating in research grant competitions?," Research Policy, Elsevier, vol. 48(1), pages 84-97.
    11. Ron Johnston & Kelvyn Jones & David Manley, 2018. "Confounding and collinearity in regression analysis: a cautionary tale and an alternative procedure, illustrated by studies of British voting behaviour," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(4), pages 1957-1976, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tóth, Tamás & Demeter, Márton & Csuhai, Sándor & Major, Zsolt Balázs, 2024. "When career-boosting is on the line: Equity and inequality in grant evaluation, productivity, and the educational backgrounds of Marie Skłodowska-Curie Actions individual fellows in social sciences an," Journal of Informetrics, Elsevier, vol. 18(2).
    2. Andrijana Perković Paloš & Antonija Mijatović & Ivan Buljan & Daniel Garcia-Costa & Elena Álvarez-García & Francisco Grimaldo & Ana Marušić, 2023. "Linguistic and semantic characteristics of articles and peer review reports in Social Sciences and Medical and Health Sciences: analysis of articles published in Open Research Central," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4707-4729, August.
    3. Gabriel Okasa & Alberto de Le'on & Michaela Strinzel & Anne Jorstad & Katrin Milzow & Matthias Egger & Stefan Muller, 2024. "A Supervised Machine Learning Approach for Assessing Grant Peer Review Reports," Papers 2411.16662, arXiv.org, revised Dec 2024.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Seeber, Marco & Alon, Ilan & Pina, David G. & Piro, Fredrik Niclas & Seeber, Michele, 2022. "Predictors of applying for and winning an ERC Proof-of-Concept grant: An automated machine learning model," Technological Forecasting and Social Change, Elsevier, vol. 184(C).
    2. Kok, Holmer & Faems, Dries & de Faria, Pedro, 2022. "Pork Barrel or Barrel of Gold? Examining the performance implications of earmarking in public R&D grants," Research Policy, Elsevier, vol. 51(7).
    3. Shan Jiang, 2021. "Understanding authors' psychological reactions to peer reviews: a text mining approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6085-6103, July.
    4. Giada Baldessarelli & Nathalie Lazaric & Michele Pezzoni, 2022. "Organizational routines: Evolution in the research landscape of two core communities," Post-Print halshs-03718851, HAL.
    5. Han-Sol Lee & Sergey U. Chernikov & Szabolcs Nagy & Ekaterina A. Degtereva, 2022. "The Impact of National Culture on Innovation: A Comparative Analysis between Developed and Developing Nations during the Pre- and Post-Crisis Period 2007–2021," Social Sciences, MDPI, vol. 11(11), pages 1-14, November.
    6. Fernandez Martinez, Roberto & Lostado Lorza, Ruben & Santos Delgado, Ana Alexandra & Piedra, Nelson, 2021. "Use of classification trees and rule-based models to optimize the funding assignment to research projects: A case study of UTPL," Journal of Informetrics, Elsevier, vol. 15(1).
    7. Marton Demeter & Agnes Jele & Zsolt Balázs Major, 2022. "The model of maximum productivity for research universities SciVal author ranks, productivity, university rankings, and their implications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4335-4361, August.
    8. Tuyen Quang Tran & Huong Van Vu, 2021. "The impact of land fragmentation on food security in the North Central Coast, Vietnam," Asia and the Pacific Policy Studies, Wiley Blackwell, vol. 8(2), pages 327-345, May.
    9. Marco Cozzi, 2020. "Public Funding of Research and Grant Proposals in the Social Sciences: Empirical Evidence from Canada," Department Discussion Papers 1809, Department of Economics, University of Victoria.
    10. Joachim P. Hasebrook & Leonie Michalak & Anna Wessels & Sabine Koenig & Stefan Spierling & Stefan Kirmsse, 2022. "Green Behavior: Factors Influencing Behavioral Intention and Actual Environmental Behavior of Employees in the Financial Service Sector," Sustainability, MDPI, vol. 14(17), pages 1-35, August.
    11. Norbert L. W. Wilson & Lurleen M. Walters & Tara Wade & Kenesha Reynolds, 2024. "The distribution of competitive research grants from the National Institute for Food and Agriculture: A comparison of 1862 land grant universities, 1890 land grant universities, and other institutions," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 46(1), pages 76-94, March.
    12. Flávia dos Santos Barbosa Brito & Emanuela Santos da Costa & Ariane Cristina Thoaldo Romeiro & Debora Martins dos Santos & Alexandre dos Santos Brito & Alessandra Silva Dias de Oliveira & Amanda Rodri, 2024. "Exploring Maternal Socio-Demographic Factors Shaping Children’s Dietary Patterns in Brazil: Results from the 2019 National Health Survey," IJERPH, MDPI, vol. 21(8), pages 1-12, July.
    13. Adam R. Pearson & Guadalupe A. Bacio & Sarah Naiman & Rainer Romero-Canyas & Jonathon P. Schuldt, 2021. "Cultural determinants of climate change opinion: familism predicts climate beliefs and policy support among US Latinos," Climatic Change, Springer, vol. 167(1), pages 1-8, July.
    14. Todd Pezzuti & James M. Leonhardt, 2023. "What’s not to like? Negations in brand messages increase consumer engagement," Journal of the Academy of Marketing Science, Springer, vol. 51(3), pages 675-694, May.
    15. Li, Heyang & Wu, Meijun & Wang, Yougui & Zeng, An, 2022. "Bibliographic coupling networks reveal the advantage of diversification in scientific projects," Journal of Informetrics, Elsevier, vol. 16(3).
    16. Shahd Al-Janabi & Lee Wei Lim & Luca Aquili, 2021. "Development of a tool to accurately predict UK REF funding allocation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8049-8062, September.
    17. Wen Lou & Jiangen He & Lingxin Zhang & Zhijie Zhu & Yongjun Zhu, 2023. "Support behind the scenes: the relationship between acknowledgement, coauthor, and citation in Nobel articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5767-5790, October.
    18. Stephen Morse & Nora McNamara & Nancy Nathan & Shuaibu Adamu & Oluwayemisi Idowu Micah & Muhammed Kabir & Augustine Sunday Onwuaroh & Nathaniel Otene, 2023. "The Leveraging of Support by Faith-Based Social Groups in Rural Villages of the Federal Capital Territory, Nigeria," Sustainability, MDPI, vol. 15(19), pages 1-19, September.
    19. Dorceta E. Taylor & Ashley Bell & Destiny Treloar & Ashia Ajani & Marco Alvarez & Tevin Hamilton & Jayson Velazquez & Pwintphyu Nandar & Lily Fillwalk & Kerry J. Ard, 2024. "Defying the Food Desert, Food Swamp, and Supermarket Redlining Stereotypes in Detroit: Comparing the Distribution of Food Outlets in 2013 and 2023," Sustainability, MDPI, vol. 16(16), pages 1-59, August.
    20. Ginther, Donna K. & Heggeness, Misty L., 2020. "Administrative discretion in scientific funding: Evidence from a prestigious postdoctoral training program✰," Research Policy, Elsevier, vol. 49(4).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:16:y:2022:i:2:s1751157722000414. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.