IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v112y2017i3d10.1007_s11192-017-2417-8.html
   My bibliography  Save this article

How do NIHR peer review panels use bibliometric information to support their decisions?

Author

Listed:
  • Salil Gunashekar

    (RAND Europe)

  • Steven Wooding

    (University of Cambridge)

  • Susan Guthrie

    (RAND Europe)

Abstract

Bibliometrics is widely used as an evaluation tool to assist prospective R&D decision-making. In the UK, for example, the National Institute for Health Research (NIHR) has employed bibliometric analysis alongside wider information in several awarding panels for major funding schemes. In this paper, we examine various aspects of the use of bibliometric information by members of these award selection panels, based on interviews with ten panel members from three NIHR panels, alongside analysis of the information provided to those panels. The aim of the work is to determine what influence bibliometrics has on their decision-making, to see which types of bibliometric measures they find more and less useful, and to identify the challenges they have when using these data. We find that panel members broadly support the use of bibliometrics in panel decision-making, and that the data are primarily used in the initial individual assessment of candidates, playing a smaller role in the selection panel meeting. Panel members felt that the most useful measures of performance are normalised citation scores and the number or proportion of papers in the most highly cited X% (e.g. 5, 10%) for the field. Panel members expressed concerns around the comparability of bibliometrics between fields, but the discussion suggested this largely represents a lack of understanding of bibliometric techniques, confirming that effective background information is important. Based on the evidence around panel behaviour and concerns, we set out guidance around providing bibliometrics to research funding panels.

Suggested Citation

  • Salil Gunashekar & Steven Wooding & Susan Guthrie, 2017. "How do NIHR peer review panels use bibliometric information to support their decisions?," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1813-1835, September.
    • Handle: RePEc:spr:scient:v:112:y:2017:i:3:d:10.1007_s11192-017-2417-8
    DOI: 10.1007/s11192-017-2417-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-017-2417-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-017-2417-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Vincent Larivière & Chaoqun Ni & Yves Gingras & Blaise Cronin & Cassidy R. Sugimoto, 2013. "Bibliometrics: Global gender disparities in science," Nature, Nature, vol. 504(7479), pages 211-213, December.
    2. Bornmann, Lutz & Mutz, Rüdiger & Daniel, Hans-Dieter, 2007. "Gender differences in grant peer review: A meta-analysis," Journal of Informetrics, Elsevier, vol. 1(3), pages 226-238.
    3. Grant Lewison & Robert Cottrell & Diane Dixon, 1999. "Bibliometric indicators to assist the peer review process in grant decisions," Research Evaluation, Oxford University Press, vol. 8(1), pages 47-52, April.
    4. Nederhof, A. J. & van Raan, A. F. J., 1993. "A bibliometric analysis of six economics research groups: A comparison with peer review," Research Policy, Elsevier, vol. 22(4), pages 353-368, August.
    5. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lutz Bornmann & Julian N. Marewski, 2019. "Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 419-459, August.
    2. Cruz-Castro, Laura & Sanz-Menendez, Luis, 2021. "What should be rewarded? Gender and evaluation criteria for tenure and promotion," Journal of Informetrics, Elsevier, vol. 15(3).
    3. Bornmann, Lutz & Tekles, Alexander & Zhang, Helena H. & Ye, Fred Y., 2019. "Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data," Journal of Informetrics, Elsevier, vol. 13(4).
    4. Balázs Győrffy & Andrea Magda Nagy & Péter Herman & Ádám Török, 2018. "Factors influencing the scientific performance of Momentum grant holders: an evaluation of the first 117 research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 409-426, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nakajima, Kazuki & Liu, Ruodan & Shudo, Kazuyuki & Masuda, Naoki, 2023. "Quantifying gender imbalance in East Asian academia: Research career and citation practice," Journal of Informetrics, Elsevier, vol. 17(4).
    2. Ehsan Mohammadi & Mike Thelwall, 2013. "Assessing non-standard article impact using F1000 labels," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 383-395, November.
    3. Kim, Lanu & Smith, Daniel Scott & Hofstra, Bas & McFarland, Daniel A., 2022. "Gendered knowledge in fields and academic careers," Research Policy, Elsevier, vol. 51(1).
    4. Mancuso, Raffaele & Rossi-Lamastra, Cristina & Franzoni, Chiara, 2023. "Topic choice, gendered language, and the under-funding of female scholars in mission-oriented research," Research Policy, Elsevier, vol. 52(6).
    5. Paul Siu Fai Yip & Yunyu Xiao & Clifford Long Hin Wong & Terry Kit Fong Au, 2020. "Is there gender bias in research grant success in social sciences?: Hong Kong as a case study," Palgrave Communications, Palgrave Macmillan, vol. 7(1), pages 1-10, December.
    6. Zhang, Ning & He, Guangye & Shi, Dongbo & Zhao, Zhenyue & Li, Jiang, 2022. "Does a gender-neutral name associate with the research impact of a scientist?," Journal of Informetrics, Elsevier, vol. 16(1).
    7. Clemens B. Fell & Cornelius J. König, 2016. "Is there a gender difference in scientific collaboration? A scientometric examination of co-authorships among industrial–organizational psychologists," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(1), pages 113-141, July.
    8. Emre Özel, 2024. "What is Gender Bias in Grant Peer review?," Working Papers halshs-03862027, HAL.
    9. Goodall, Amanda H., 2009. "Highly cited leaders and the performance of research universities," Research Policy, Elsevier, vol. 38(7), pages 1079-1092, September.
    10. Lin Zhang & Yuanyuan Shang & Ying Huang & Gunnar Sivertsen, 2022. "Gender differences among active reviewers: an investigation based on publons," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 145-179, January.
    11. Hussinger, Katrin & Pellens, Maikel, 2019. "Guilt by association: How scientific misconduct harms prior collaborators," Research Policy, Elsevier, vol. 48(2), pages 516-530.
    12. Wu, Jiang & Ou, Guiyan & Liu, Xiaohui & Dong, Ke, 2022. "How does academic education background affect top researchers’ performance? Evidence from the field of artificial intelligence," Journal of Informetrics, Elsevier, vol. 16(2).
    13. Chaojiang Wu & Erjia Yan & Yongjun Zhu & Kai Li, 2021. "Gender imbalance in the productivity of funded projects: A study of the outputs of National Institutes of Health R01 grants," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 72(11), pages 1386-1399, November.
    14. Martin Ricker, 2017. "Letter to the Editor: About the quality and impact of scientific articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1851-1855, June.
    15. Alberto Anfossi & Alberto Ciolfi & Filippo Costa & Giorgio Parisi & Sergio Benedetto, 2016. "Large-scale assessment of research outputs through a weighted combination of bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 671-683, May.
    16. Day, Theodore Eugene, 2015. "The big consequences of small biases: A simulation of peer review," Research Policy, Elsevier, vol. 44(6), pages 1266-1270.
    17. Kwiek, Marek & Roszka, Wojciech, 2021. "Gender-based homophily in research: A large-scale study of man-woman collaboration," Journal of Informetrics, Elsevier, vol. 15(3).
    18. H. Martinez & A. Jaime & J. Camacho, 2012. "Relative absorptive capacity: a research profiling," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(3), pages 657-674, September.
    19. Kwiek, Marek & Szymula, Łukasz, 2024. "Growth of Science and Women: Methodological Challenges of Using Structured Big Data," SocArXiv w34pr, Center for Open Science.
    20. Zheng Yan & Wenqian Robertson & Yaosheng Lou & Tom W. Robertson & Sung Yong Park, 2021. "Finding leading scholars in mobile phone behavior: a mixed-method analysis of an emerging interdisciplinary field," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9499-9517, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:112:y:2017:i:3:d:10.1007_s11192-017-2417-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.