IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v126y2021i8d10.1007_s11192-021-04013-x.html
   My bibliography  Save this article

Consistency index: measuring the performances of scholar journal reviewers

Author

Listed:
  • Mingliang Yue

    (Wuhan Library of Chinese Academy of Science
    University of Chinese Academy of Sciences
    Hubei Key Laboratory of Big Data in Science and Technology)

  • Hongbo Tang

    (Wuhan Library of Chinese Academy of Science
    Hubei Key Laboratory of Big Data in Science and Technology)

  • Fan Liu

    (Wuhan University)

  • Tingcan Ma

    (Wuhan Library of Chinese Academy of Science
    University of Chinese Academy of Sciences
    Hubei Key Laboratory of Big Data in Science and Technology)

Abstract

Peer review is one of the most important processes of scientific evaluation. This paper proposes a consistency index to measure the performances of scholar journal reviewers on recognizing papers with higher scientific influences with respect to citations. The index is initially designed based on three consistencies, i.e., expert consistency, editor consistency and academic community consistency, to capture both the knowledge implied in the review data and the bibliometric data. An iterative computation process is then proposed to make the consistency flow among experts based on their historical performances. Further, a quantitative indicator is designed to characterize the ability of an expert of recognizing papers with higher quality during review process. Experiments are conducted to verify the convergence of the iterative computation, and to evaluate the effectiveness of the consistency index. The experimental results show that the performance of the top experts (with regard to consistency index) is always better than the tail experts, and the performance can be maintained over time. We believe that the proposed index can help journal editors get deeper understanding to their reviewers, and provide evidence in the process of reviewer selection.

Suggested Citation

  • Mingliang Yue & Hongbo Tang & Fan Liu & Tingcan Ma, 2021. "Consistency index: measuring the performances of scholar journal reviewers," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7183-7195, August.
    • Handle: RePEc:spr:scient:v:126:y:2021:i:8:d:10.1007_s11192-021-04013-x
    DOI: 10.1007/s11192-021-04013-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-021-04013-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-021-04013-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Omar Sabaj & Valderrama José O. & Carlos González-Vergara & Alvaro Pina-Stranger, 2015. "Relationship between the duration of peer-review, publication decision, and agreement among reviewers in three Chilean journals," Post-Print halshs-01242627, HAL.
    2. Lutz Bornmann & Rüdiger Mutz & Hans-Dieter Daniel, 2009. "The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 407-411, November.
    3. Janine Huisman & Jeroen Smits, 2017. "Duration and quality of the peer review process: the author’s perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 633-650, October.
    4. Lowell L. Hargens & Jerald R. Herting, 2006. "Analyzing the association between referees' recommendations and editors' decisions," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(1), pages 15-26, April.
    5. Ludo Waltman & Nees Jan Eck & Thed N. Leeuwen & Martijn S. Visser & Anthony F. J. Raan, 2011. "On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 1017-1022, September.
    6. Lipworth, Wendy L. & Kerridge, Ian H. & Carter, Stacy M. & Little, Miles, 2011. "Journal peer review in context: A qualitative study of the social and subjective dimensions of manuscript review in biomedical publishing," Social Science & Medicine, Elsevier, vol. 72(7), pages 1056-1063, April.
    7. Lowell L. Hargens & Jerald R. Herting, 2006. "Analyzing the association between referees' recommendations and editors' decisions," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(1), pages 15-26, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zehra Taşkın & Abdülkadir Taşkın & Güleda Doğan & Emanuel Kulczycki, 2022. "Factors affecting time to publication in information science," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7499-7515, December.
    2. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    3. Ivana Drvenica & Giangiacomo Bravo & Lucija Vejmelka & Aleksandar Dekanski & Olgica Nedić, 2018. "Peer Review of Reviewers: The Author’s Perspective," Publications, MDPI, vol. 7(1), pages 1-10, December.
    4. Michael C Grant & Luke Geoghegan & Marc Arbyn & Zakaria Mohammed & Luke McGuinness & Emily L Clarke & Ryckie G Wade, 2020. "The prevalence of symptoms in 24,410 adults infected by the novel coronavirus (SARS-CoV-2; COVID-19): A systematic review and meta-analysis of 148 studies from 9 countries," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-19, June.
    5. Nicholas Yee Liang Hing & Xin Ci Wong & Pei Xuan Kuan & Mohan Dass Pathmanathan & Mohd Aizuddin Abdul Rahman & Kalaiarasu M. Peariasamy, 2022. "Scientific Abstract to Full Paper: Publication Rate over a 3-Year Period in a Malaysian Clinical Research Conference," Publications, MDPI, vol. 10(4), pages 1-13, October.
    6. Zhuanlan Sun & C. Clark Cao & Sheng Liu & Yiwei Li & Chao Ma, 2024. "Behavioral consequences of second-person pronouns in written communications between authors and reviewers of scientific papers," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    7. Louis Mesnard, 2010. "On Hochberg et al.’s “The tragedy of the reviewer commons”," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 903-917, September.
    8. Püttmann, Vitus & Thomsen, Stephan L. & Trunzer, Johannes, 2020. "Zur Relevanz von Ausstattungsunterschieden für Forschungsleistungsvergleiche: Ein Diskussionsbeitrag für die Wirtschaftswissenschaften in Deutschland," Hannover Economic Papers (HEP) dp-679, Leibniz Universität Hannover, Wirtschaftswissenschaftliche Fakultät, revised Mar 2021.
    9. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    10. Katrin Hussinger & Lorenzo Palladini, 2024. "Information accessibility and knowledge creation: the impact of Google’s withdrawal from China on scientific research," Industry and Innovation, Taylor & Francis Journals, vol. 31(6), pages 753-783, July.
    11. Narjes Vara & Mahdieh Mirzabeigi & Hajar Sotudeh & Seyed Mostafa Fakhrahmad, 2022. "Application of k-means clustering algorithm to improve effectiveness of the results recommended by journal recommender system," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3237-3252, June.
    12. Lucas Rodriguez Forti & Luiz A. Solino & Judit K. Szabo, 2021. "Trade-off between urgency and reduced editorial capacity affect publication speed in ecological and medical journals during 2020," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-9, December.
    13. Ehsan Mohammadi & Mike Thelwall, 2013. "Assessing non-standard article impact using F1000 labels," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 383-395, November.
    14. Lutz Bornmann, 2012. "The Hawthorne effect in journal peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 857-862, June.
    15. Rovetta, Alessandro & Castaldo, Lucia, 2022. "Are We Sure We Fully Understand What an Infodemic Is? A Global Perspective on Infodemiological Problems," SocArXiv xw723, Center for Open Science.
    16. Andrijana Perković Paloš & Antonija Mijatović & Ivan Buljan & Daniel Garcia-Costa & Elena Álvarez-García & Francisco Grimaldo & Ana Marušić, 2023. "Linguistic and semantic characteristics of articles and peer review reports in Social Sciences and Medical and Health Sciences: analysis of articles published in Open Research Central," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4707-4729, August.
    17. José Luis Gallego Ortega & Antonio Rodríguez Fuentes & Antonio García Guzmán, 2021. "Application of Mathematical Methods to the Study of Special-Needs Education in Spanish Journals," Mathematics, MDPI, vol. 9(6), pages 1-17, March.
    18. Sven E. Hug & Mirjam Aeschbach, 2020. "Criteria for assessing grant applications: a systematic review," Palgrave Communications, Palgrave Macmillan, vol. 6(1), pages 1-15, December.
    19. Seeber, Marco & Cattaneo, Mattia & Meoli, Michele & Malighetti, Paolo, 2019. "Self-citations as strategic response to the use of metrics for career decisions," Research Policy, Elsevier, vol. 48(2), pages 478-491.
    20. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2021. "The editor-manuscript game," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(5), pages 4277-4295, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:126:y:2021:i:8:d:10.1007_s11192-021-04013-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.