IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v126y2021i8d10.1007_s11192-021-04013-x.html
   My bibliography  Save this article

Consistency index: measuring the performances of scholar journal reviewers

Author

Listed:
  • Mingliang Yue

    (Wuhan Library of Chinese Academy of Science
    University of Chinese Academy of Sciences
    Hubei Key Laboratory of Big Data in Science and Technology)

  • Hongbo Tang

    (Wuhan Library of Chinese Academy of Science
    Hubei Key Laboratory of Big Data in Science and Technology)

  • Fan Liu

    (Wuhan University)

  • Tingcan Ma

    (Wuhan Library of Chinese Academy of Science
    University of Chinese Academy of Sciences
    Hubei Key Laboratory of Big Data in Science and Technology)

Abstract

Peer review is one of the most important processes of scientific evaluation. This paper proposes a consistency index to measure the performances of scholar journal reviewers on recognizing papers with higher scientific influences with respect to citations. The index is initially designed based on three consistencies, i.e., expert consistency, editor consistency and academic community consistency, to capture both the knowledge implied in the review data and the bibliometric data. An iterative computation process is then proposed to make the consistency flow among experts based on their historical performances. Further, a quantitative indicator is designed to characterize the ability of an expert of recognizing papers with higher quality during review process. Experiments are conducted to verify the convergence of the iterative computation, and to evaluate the effectiveness of the consistency index. The experimental results show that the performance of the top experts (with regard to consistency index) is always better than the tail experts, and the performance can be maintained over time. We believe that the proposed index can help journal editors get deeper understanding to their reviewers, and provide evidence in the process of reviewer selection.

Suggested Citation

  • Mingliang Yue & Hongbo Tang & Fan Liu & Tingcan Ma, 2021. "Consistency index: measuring the performances of scholar journal reviewers," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7183-7195, August.
    • Handle: RePEc:spr:scient:v:126:y:2021:i:8:d:10.1007_s11192-021-04013-x
    DOI: 10.1007/s11192-021-04013-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-021-04013-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-021-04013-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Lutz Bornmann & Rüdiger Mutz & Hans-Dieter Daniel, 2009. "The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 407-411, November.
    2. Lowell L. Hargens & Jerald R. Herting, 2006. "Analyzing the association between referees' recommendations and editors' decisions," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(1), pages 15-26, April.
    3. Omar Sabaj & Valderrama José O. & Carlos González-Vergara & Alvaro Pina-Stranger, 2015. "Relationship between the duration of peer-review, publication decision, and agreement among reviewers in three Chilean journals," Post-Print halshs-01242627, HAL.
    4. Ludo Waltman & Nees Jan Eck & Thed N. Leeuwen & Martijn S. Visser & Anthony F. J. Raan, 2011. "On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 1017-1022, September.
    5. Janine Huisman & Jeroen Smits, 2017. "Duration and quality of the peer review process: the author’s perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 633-650, October.
    6. Lipworth, Wendy L. & Kerridge, Ian H. & Carter, Stacy M. & Little, Miles, 2011. "Journal peer review in context: A qualitative study of the social and subjective dimensions of manuscript review in biomedical publishing," Social Science & Medicine, Elsevier, vol. 72(7), pages 1056-1063, April.
    7. Lowell L. Hargens & Jerald R. Herting, 2006. "Analyzing the association between referees' recommendations and editors' decisions," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(1), pages 15-26, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zehra Taşkın & Abdülkadir Taşkın & Güleda Doğan & Emanuel Kulczycki, 2022. "Factors affecting time to publication in information science," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7499-7515, December.
    2. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    3. Hugo S. Gonçalves & Sérgio Moro, 2023. "On the economic impacts of COVID‐19: A text mining literature analysis," Review of Development Economics, Wiley Blackwell, vol. 27(1), pages 375-394, February.
    4. Ivana Drvenica & Giangiacomo Bravo & Lucija Vejmelka & Aleksandar Dekanski & Olgica Nedić, 2018. "Peer Review of Reviewers: The Author’s Perspective," Publications, MDPI, vol. 7(1), pages 1-10, December.
    5. Marcel Knöchelmann, 2019. "Open Science in the Humanities, or: Open Humanities?," Publications, MDPI, vol. 7(4), pages 1-17, November.
    6. Michael C Grant & Luke Geoghegan & Marc Arbyn & Zakaria Mohammed & Luke McGuinness & Emily L Clarke & Ryckie G Wade, 2020. "The prevalence of symptoms in 24,410 adults infected by the novel coronavirus (SARS-CoV-2; COVID-19): A systematic review and meta-analysis of 148 studies from 9 countries," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-19, June.
    7. Fabian Scheidegger & Andre Briviba & Bruno S. Frey, 2023. "Behind the curtains of academic publishing: strategic responses of economists and business scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4765-4790, August.
    8. Gangan Prathap, 2012. "Energy indicators and percentile ranking normalization," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 997-1003, June.
    9. Nicholas Yee Liang Hing & Xin Ci Wong & Pei Xuan Kuan & Mohan Dass Pathmanathan & Mohd Aizuddin Abdul Rahman & Kalaiarasu M. Peariasamy, 2022. "Scientific Abstract to Full Paper: Publication Rate over a 3-Year Period in a Malaysian Clinical Research Conference," Publications, MDPI, vol. 10(4), pages 1-13, October.
    10. Katrin Hussinger & Lorenzo Palladini, 2024. "Information accessibility and knowledge creation: the impact of Google’s withdrawal from China on scientific research," Industry and Innovation, Taylor & Francis Journals, vol. 31(6), pages 753-783, July.
    11. Hendrik P. Dalen, 2021. "How the publish-or-perish principle divides a science: the case of economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1675-1694, February.
    12. Zhuanlan Sun & C. Clark Cao & Sheng Liu & Yiwei Li & Chao Ma, 2024. "Behavioral consequences of second-person pronouns in written communications between authors and reviewers of scientific papers," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    13. Adriana Bin & Sergio Salles-Filho & Luiza Maria Capanema & Fernando Antonio Basile Colugnati, 2015. "What difference does it make? Impact of peer-reviewed scholarships on scientific production," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1167-1188, February.
    14. Louis Mesnard, 2010. "On Hochberg et al.’s “The tragedy of the reviewer commons”," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 903-917, September.
    15. van Dalen, Hendrik Peter, 2021. "How the publish-or-perish principle divides a science: The case of economists," Other publications TiSEM a6a5a855-bb5a-4d52-a841-3, Tilburg University, School of Economics and Management.
    16. Püttmann, Vitus & Thomsen, Stephan L. & Trunzer, Johannes, 2020. "Zur Relevanz von Ausstattungsunterschieden für Forschungsleistungsvergleiche: Ein Diskussionsbeitrag für die Wirtschaftswissenschaften in Deutschland," Hannover Economic Papers (HEP) dp-679, Leibniz Universität Hannover, Wirtschaftswissenschaftliche Fakultät, revised Mar 2021.
    17. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    18. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    19. Daisuke Sakai, 2019. "Who is peer reviewed? Comparing publication patterns of peer-reviewed and non-peer-reviewed papers in Japanese political science," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 65-80, October.
    20. Narjes Vara & Mahdieh Mirzabeigi & Hajar Sotudeh & Seyed Mostafa Fakhrahmad, 2022. "Application of k-means clustering algorithm to improve effectiveness of the results recommended by journal recommender system," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3237-3252, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:126:y:2021:i:8:d:10.1007_s11192-021-04013-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.