IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v93y2012i3d10.1007_s11192-012-0731-8.html
   My bibliography  Save this article

In public peer review of submitted manuscripts, how do reviewer comments differ from comments written by interested members of the scientific community? A content analysis of comments written for Atmospheric Chemistry and Physics

Author

Listed:
  • Lutz Bornmann

    (Max Planck Society, Administrative Headquarters)

  • Hanna Herich

    (Institute of Atmospheric and Climate Science, ETH Zurich
    Swiss Federal Laboratories for Materials Science and Technology, Empa)

  • Hanna Joos

    (Institute of Atmospheric and Climate Science, ETH Zurich)

  • Hans-Dieter Daniel

    (Professorship for Social Psychology and Research on Higher Education, ETH Zurich
    University of Zurich)

Abstract

Whereas in traditional peer review a few selected researchers (peers) are included in the manuscript review process, public peer review includes both invited reviewers (who write ‘reviewer comments’) and interested members of the scientific community who write comments (‘short comments’). Available to us for this investigation are 390 reviewer comments and short comments assessing 119 manuscripts submitted to the journal Atmospheric Chemistry and Physics (ACP). We conducted a content analysis of these comments to determine differences in the main thematic areas considered by the scientists in their assessment comments. The results of the analysis show that in contrast to interested members of the scientific community, reviewers focus mainly on (1) the formal qualities of a manuscript, such as writing style, (2) the conclusions drawn in a manuscript, and (3) the future “gain” that could result from publication of a manuscript. All in all, it appears that ‘reviewer comments’ better than ‘short comments’ by interested members of the scientific community support the two main functions of peer review: selection and improvement of what is published.

Suggested Citation

  • Lutz Bornmann & Hanna Herich & Hanna Joos & Hans-Dieter Daniel, 2012. "In public peer review of submitted manuscripts, how do reviewer comments differ from comments written by interested members of the scientific community? A content analysis of comments written for Atmo," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 915-929, December.
    • Handle: RePEc:spr:scient:v:93:y:2012:i:3:d:10.1007_s11192-012-0731-8
    DOI: 10.1007/s11192-012-0731-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-012-0731-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-012-0731-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bornmann, Lutz & Mutz, Rüdiger & Daniel, Hans-Dieter, 2008. "Latent Markov modeling applied to grant peer review," Journal of Informetrics, Elsevier, vol. 2(3), pages 217-228.
    2. Günter Krampen & Ralf Becker & Ute Wahner & Leo Montada, 2007. "On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 71(2), pages 191-202, May.
    3. Lutz Bornmann & Rüdiger Mutz & Hans-Dieter Daniel, 2009. "The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 407-411, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Niccolò Casnici & Francisco Grimaldo & Nigel Gilbert & Pierpaolo Dondio & Flaminio Squazzoni, 2017. "Assessing peer review by gauging the fate of rejected manuscripts: the case of the Journal of Artificial Societies and Social Simulation," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 533-546, October.
    2. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2017. "Problems with open participation in peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1881-1885, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    2. Ponomariov, Branco & Toivanen, Hannes, 2014. "Knowledge flows and bases in emerging economy innovation systems: Brazilian research 2005–2009," Research Policy, Elsevier, vol. 43(3), pages 588-596.
    3. Wu, Jiang & Ou, Guiyan & Liu, Xiaohui & Dong, Ke, 2022. "How does academic education background affect top researchers’ performance? Evidence from the field of artificial intelligence," Journal of Informetrics, Elsevier, vol. 16(2).
    4. Masaki Eto, 2013. "Evaluations of context-based co-citation searching," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 651-673, February.
    5. Carlo D'Ippoliti, 2021. "“Many‐Citedness”: Citations Measure More Than Just Scientific Quality," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1271-1301, December.
    6. Stremersch, S. & Camacho, N.M.A. & Vanneste, S. & Verniers, I.W.J., 2014. "Unraveling Scientific Impact," ERIM Report Series Research in Management ERS-2014-014-MKT, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    7. F. Bartolucci & A. Farcomeni & F. Pennoni, 2014. "Latent Markov models: a review of a general framework for the analysis of longitudinal data with covariates," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 23(3), pages 433-465, September.
    8. Stremersch, Stefan & Camacho, Nuno & Vanneste, Sofie & Verniers, Isabel, 2015. "Unraveling scientific impact: Citation types in marketing journals," International Journal of Research in Marketing, Elsevier, vol. 32(1), pages 64-77.
    9. José Álvarez-García & Amador Durán-Sánchez & Néstor Montalván-Burbano & María de la Cruz del Río-Rama, 2023. "Spanish Journal of Finance and Accounting (SJFA): Mapping of Knowledge over the Last 25 Years," Publications, MDPI, vol. 11(1), pages 1-19, February.
    10. Chi-Shiou Lin, 2018. "An analysis of citation functions in the humanities and social sciences research from the perspective of problematic citation analysis assumptions," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 797-813, August.
    11. Jessica Petersen & Fabian Hattke & Rick Vogel, 2017. "Editorial governance and journal impact: a study of management and business journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1593-1614, September.
    12. Huang, Chen-Hao & Liu, John S. & Ho, Mei Hsiu-Ching & Chou, Tzu-Chuan, 2022. "Towards more convergent main paths: A relevance-based approach," Journal of Informetrics, Elsevier, vol. 16(3).
    13. Zhang, Chengzhi & Zhou, Qingqing, 2020. "Assessing books’ depth and breadth via multi-level mining on tables of contents," Journal of Informetrics, Elsevier, vol. 14(2).
    14. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    15. Mingliang Yue & Hongbo Tang & Fan Liu & Tingcan Ma, 2021. "Consistency index: measuring the performances of scholar journal reviewers," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7183-7195, August.
    16. Bradford Demarest & Guo Freeman & Cassidy R. Sugimoto, 2014. "The reviewer in the mirror: examining gendered and ethnicized notions of reciprocity in peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 717-735, October.
    17. Andrea Diem & Stefan C. Wolter, 2011. "The Use of Bibliometrics to Measure Research Performance in Education Sciences," Economics of Education Working Paper Series 0066, University of Zurich, Department of Business Administration (IBW), revised May 2013.
    18. Günter Krampen, 2010. "Acceleration of citing behavior after the millennium? Exemplary bibliometric reference analyses for psychology journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(2), pages 507-513, May.
    19. Kousha, Kayvan & Thelwall, Mike, 2018. "Can Microsoft Academic help to assess the citation impact of academic books?," Journal of Informetrics, Elsevier, vol. 12(3), pages 972-984.
    20. Lutz Bornmann & Rüdiger Mutz & Hans-Dieter Daniel, 2009. "The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 407-411, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:93:y:2012:i:3:d:10.1007_s11192-012-0731-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.