IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0104244.html
   My bibliography  Save this article

Characterization of the Peer Review Network at the Center for Scientific Review, National Institutes of Health

Author

Listed:
  • Kevin W Boyack
  • Mei-Ching Chen
  • George Chacko

Abstract

The National Institutes of Health (NIH) is the largest source of funding for biomedical research in the world. This funding is largely effected through a competitive grants process. Each year the Center for Scientific Review (CSR) at NIH manages the evaluation, by peer review, of more than 55,000 grant applications. A relevant management question is how this scientific evaluation system, supported by finite resources, could be continuously evaluated and improved for maximal benefit to the scientific community and the taxpaying public. Towards this purpose, we have created the first system-level description of peer review at CSR by applying text analysis, bibliometric, and graph visualization techniques to administrative records. We identify otherwise latent relationships across scientific clusters, which in turn suggest opportunities for structural reorganization of the system based on expert evaluation. Such studies support the creation of monitoring tools and provide transparency and knowledge to stakeholders.

Suggested Citation

  • Kevin W Boyack & Mei-Ching Chen & George Chacko, 2014. "Characterization of the Peer Review Network at the Center for Scientific Review, National Institutes of Health," PLOS ONE, Public Library of Science, vol. 9(8), pages 1-13, August.
    • Handle: RePEc:plo:pone00:0104244
    DOI: 10.1371/journal.pone.0104244
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0104244
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0104244&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0104244?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Johan Bollen & Herbert Van de Sompel & Aric Hagberg & Luis Bettencourt & Ryan Chute & Marko A Rodriguez & Lyudmila Balakireva, 2009. "Clickstream Data Yields High-Resolution Maps of Science," PLOS ONE, Public Library of Science, vol. 4(3), pages 1-11, March.
    2. Richard Klavans & Kevin W. Boyack, 2011. "Using global mapping to create more accurate document-level maps of research fields," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(1), pages 1-18, January.
    3. Boyack, Kevin W. & Klavans, Richard, 2008. "Measuring science–technology interaction using rare inventor–author names," Journal of Informetrics, Elsevier, vol. 2(3), pages 173-182.
    4. Luciano Kay & Nils Newman & Jan Youtie & Alan L. Porter & Ismael Rafols, 2014. "Patent overlay mapping: Visualizing technological distance," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(12), pages 2432-2443, December.
    5. Ismael Rafols & Alan L. Porter & Loet Leydesdorff, 2010. "Science overlay maps: A new tool for research policy and library management," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(9), pages 1871-1887, September.
    6. Kevin W. Boyack & Richard Klavans, 2014. "Creation of a highly detailed, dynamic, global model and map of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 670-685, April.
    7. Ed J. Rinia & Thed N. van Leeuwen & Eppo E. W. Bruins & Hendrik G. van Vuren & Anthony F. J. van Raan, 2002. "Measuring knowledge transfer between fields of science," Scientometrics, Springer;Akadémiai Kiadó, vol. 54(3), pages 347-362, July.
    8. Richard Klavans & Kevin W. Boyack, 2006. "Quantitative evaluation of large maps of science," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(3), pages 475-499, September.
    9. Kevin W. Boyack & Richard Klavans & Katy Börner, 2005. "Mapping the backbone of science," Scientometrics, Springer;Akadémiai Kiadó, vol. 64(3), pages 351-374, August.
    10. Richard Klavans & Kevin W. Boyack, 2011. "Using global mapping to create more accurate document‐level maps of research fields," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(1), pages 1-18, January.
    11. Henry Small, 1973. "Co‐citation in the scientific literature: A new measure of the relationship between two documents," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 24(4), pages 265-269, July.
    12. Kevin W Boyack & David Newman & Russell J Duhon & Richard Klavans & Michael Patek & Joseph R Biberstine & Bob Schijvenaars & André Skupin & Nianli Ma & Katy Börner, 2011. "Clustering More than Two Million Biomedical Publications: Comparing the Accuracies of Nine Text-Based Similarity Approaches," PLOS ONE, Public Library of Science, vol. 6(3), pages 1-11, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sjögårde, Peter & Ahlgren, Per, 2018. "Granularity of algorithmically constructed publication-level classifications of research publications: Identification of topics," Journal of Informetrics, Elsevier, vol. 12(1), pages 133-152.
    2. Ying Huang & Wolfgang Glänzel & Lin Zhang, 2021. "Tracing the development of mapping knowledge domains," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6201-6224, July.
    3. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    4. Yan, Erjia & Ding, Ying & Cronin, Blaise & Leydesdorff, Loet, 2013. "A bird's-eye view of scientific trading: Dependency relations among fields of science," Journal of Informetrics, Elsevier, vol. 7(2), pages 249-264.
    5. Ryo Takahashi & Kenji Kaibe & Kazuyuki Suzuki & Sayaka Takahashi & Kotaro Takeda & Marc Hansen & Michiaki Yumoto, 2023. "New concept of the affinity between research fields using academic journal data in Scopus," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3507-3534, June.
    6. Shuo Xu & Liyuan Hao & Xin An & Hongshen Pang & Ting Li, 2020. "Review on emerging research topics with key-route main path analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 607-624, January.
    7. Miguel R. Guevara & Dominik Hartmann & Manuel Aristarán & Marcelo Mendoza & César A. Hidalgo, 2016. "The research space: using career paths to predict the evolution of the research output of individuals, institutions, and nations," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1695-1709, December.
    8. Ismael Rafols & Alan Porter & Loet Leydesdorff, 2009. "Overlay Maps of Science: a New Tool for Research Policy," SPRU Working Paper Series 179, SPRU - Science Policy Research Unit, University of Sussex Business School.
    9. Kevin W. Boyack, 2017. "Thesaurus-based methods for mapping contents of publication sets," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(2), pages 1141-1155, May.
    10. Rons, Nadine, 2018. "Bibliometric approximation of a scientific specialty by combining key sources, title words, authors and references," Journal of Informetrics, Elsevier, vol. 12(1), pages 113-132.
    11. Yan, Erjia & Ding, Ying & Milojević, Staša & Sugimoto, Cassidy R., 2012. "Topics in dynamic research communities: An exploratory study for the field of information retrieval," Journal of Informetrics, Elsevier, vol. 6(1), pages 140-153.
    12. Pin Li & Guoli Yang & Chuanqi Wang, 2019. "Visual topical analysis of library and information science," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(3), pages 1753-1791, December.
    13. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    14. Zhang, Yi & Shang, Lining & Huang, Lu & Porter, Alan L. & Zhang, Guangquan & Lu, Jie & Zhu, Donghua, 2016. "A hybrid similarity measure method for patent portfolio analysis," Journal of Informetrics, Elsevier, vol. 10(4), pages 1108-1130.
    15. Jochen Gläser & Wolfgang Glänzel & Andrea Scharnhorst, 2017. "Same data—different results? Towards a comparative approach to the identification of thematic structures in science," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(2), pages 981-998, May.
    16. Yanto Chandra, 2018. "Mapping the evolution of entrepreneurship as a field of research (1990–2013): A scientometric analysis," PLOS ONE, Public Library of Science, vol. 13(1), pages 1-24, January.
    17. Nieminen, Paavo & Pölönen, Ilkka & Sipola, Tuomo, 2013. "Research literature clustering using diffusion maps," Journal of Informetrics, Elsevier, vol. 7(4), pages 874-886.
    18. Loet Leydesdorff & Lutz Bornmann, 2012. "Mapping (USPTO) patent data using overlays to Google Maps," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(7), pages 1442-1458, July.
    19. Yuya Kajikawa, 2022. "Reframing evidence in evidence-based policy making and role of bibliometrics: toward transdisciplinary scientometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5571-5585, September.
    20. Boyack, Kevin W. & Klavans, Richard, 2014. "Including cited non-source items in a large-scale map of science: What difference does it make?," Journal of Informetrics, Elsevier, vol. 8(3), pages 569-580.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0104244. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.