IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v17y2023i4s1751157723000743.html
   My bibliography  Save this article

The academic status of reviewers predicts their language use

Author

Listed:
  • Sun, Zhuanlan
  • Clark Cao, C.
  • Ma, Chao
  • Li, Yiwei

Abstract

Peer review plays an essential role in scientific research, but the influence of reviewers' academic status is often overlooked during this process. By accessing peer review reports, in this study we empirically investigate this effect. Specifically, we analyzed 2,580 peer review histories from eLife submissions between 2016 and 2021 to examine the relationship between reviewers’ academic status and their language usage in the first round of peer review. We focused on two types of language features: emotional features (e.g., positivity and subjectivity) and linguistic features (e.g., number of long words and complex words). Our findings revealed no significant reviewer bias of academic status, such that the reviewers’ comments with emotional features were not significantly associated with reviewers’ awareness of being more prestigious than the last corresponding author of the manuscripts. More accomplished reviewers, however, were more likely to use longer and more complex words. Additionally, the results of linguistic features remained robust in the group where the last author served as the last corresponding author. Overall, our findings suggest that the quality of peer review remains the primary consideration for reviewers when evaluating submissions. These results have significant implications for open peer review practices and the fair assessment of the peer review process.

Suggested Citation

  • Sun, Zhuanlan & Clark Cao, C. & Ma, Chao & Li, Yiwei, 2023. "The academic status of reviewers predicts their language use," Journal of Informetrics, Elsevier, vol. 17(4).
    • Handle: RePEc:eee:infome:v:17:y:2023:i:4:s1751157723000743
    DOI: 10.1016/j.joi.2023.101449
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157723000743
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2023.101449?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Oecd, 2008. "DAC Peer Review of the European Community," OECD Journal on Development, OECD Publishing, vol. 8(4), pages 127-261.
    2. Oecd, 2008. "DAC Peer Review of Greece," OECD Journal on Development, OECD Publishing, vol. 7(4), pages 7-93.
    3. Jue Ni & Zhenyue Zhao & Yupo Shao & Shuo Liu & Wanlin Li & Yaoze Zhuang & Junmo Qu & Yu Cao & Nayuan Lian & Jiang Li, 2021. "The influence of opening up peer review on the citations of journal articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9393-9404, December.
    4. Shan Jiang, 2021. "Understanding authors' psychological reactions to peer reviews: a text mining approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6085-6103, July.
    5. Oecd, 2008. "DAC Peer Review of the United Kingdom," OECD Journal on Development, OECD Publishing, vol. 7(3), pages 7-114.
    6. ederico Bianchi & Flaminio Squazzoni, 2022. "Can transparency undermine peer review? A simulation model of scientist behavior under open peer review [Reviewing Peer Review]," Science and Public Policy, Oxford University Press, vol. 49(5), pages 791-800.
    7. Oecd, 2008. "DAC Peer Review of the Netherlands," OECD Journal on Development, OECD Publishing, vol. 7(3), pages 115-214.
    8. Costas, Rodrigo & Bordons, María, 2007. "The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level," Journal of Informetrics, Elsevier, vol. 1(3), pages 193-203.
    9. Pauline Mattsson & Carl Johan Sundberg & Patrice Laget, 2011. "Is correspondence reflected in the author position? A bibliometric study of the relation between corresponding author and byline position," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(1), pages 99-105, April.
    10. Wei Huang, 2015. "DO ABCs GET MORE CITATIONS THAN XYZs?," Economic Inquiry, Western Economic Association International, vol. 53(1), pages 773-789, January.
    11. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    12. Zhou-min Yuan & Mingxin Yao, 2022. "Is academic writing becoming more positive? A large-scale diachronic case study of Science research articles across 25 years," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6191-6207, November.
    13. Yun Wei Zhao & Chi-Hung Chi & Willem-Jan Heuvel, 2015. "Imperfect referees: Reducing the impact of multiple biases in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2340-2356, November.
    14. Sun, Zhuanlan & Liu, Sheng & Li, Yiwei & Ma, Chao, 2023. "Expedited editorial decision in COVID-19 pandemic," Journal of Informetrics, Elsevier, vol. 17(1).
    15. Giangiacomo Bravo & Francisco Grimaldo & Emilia López-Iñesta & Bahar Mehmani & Flaminio Squazzoni, 2019. "The effect of publishing peer review reports on referee behavior in five scholarly journals," Nature Communications, Nature, vol. 10(1), pages 1-8, December.
    16. Oecd, 2008. "DAC Peer Review of Denmark," OECD Journal on Development, OECD Publishing, vol. 8(4), pages 7-125.
    17. Isabelle Régner & Catherine Thinus-Blanc & Agnès Netter & Toni Schmader & Pascal Huguet, 2019. "Committees with implicit biases promote fewer women when they do not believe gender bias exists," Nature Human Behaviour, Nature, vol. 3(11), pages 1171-1179, November.
    18. Lutz Bornmann & Hans-Dieter Daniel, 2005. "Does the h-index for ranking of scientists really work?," Scientometrics, Springer;Akadémiai Kiadó, vol. 65(3), pages 391-392, December.
    19. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    20. Jurgen Huber & Sabiou M. Inoua & Rudolf Kerschbamer & Christian Konig-Kersting & Stefan Palan & Vernon L. Smith, 2022. "Nobel and Novice: Author Prominence Affects Peer Review," Working Papers 22-15, Chapman University, Economic Science Institute.
    21. Oecd, 2008. "DAC Peer Review of Canada," OECD Journal on Development, OECD Publishing, vol. 8(4), pages 263-387.
    22. Zhang, Guangyao & Xu, Shenmeng & Sun, Yao & Jiang, Chunlin & Wang, Xianwen, 2022. "Understanding the peer review endeavor in scientific publishing," Journal of Informetrics, Elsevier, vol. 16(2).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sun, Zhuanlan, 2024. "Textual features of peer review predict top-cited papers: An interpretable machine learning perspective," Journal of Informetrics, Elsevier, vol. 18(2).
    2. Sun, Zhuanlan & He, Dongjin & Li, Yiwei, 2024. "How the readability of manuscript before journal submission advantages peer review process: Evidence from biomedical scientific publications," Journal of Informetrics, Elsevier, vol. 18(3).
    3. Wenqing Wu & Haixu Xi & Chengzhi Zhang, 2024. "Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4109-4135, July.
    4. Buljan, Ivan & Garcia-Costa, Daniel & Grimaldo, Francisco & Klein, Richard A. & Bakker, Marjan & Marušić, Ana, 2024. "Development and application of a comprehensive glossary for the identification of statistical and methodological concepts in peer review reports," Journal of Informetrics, Elsevier, vol. 18(3).
    5. Sun, Zhuanlan & Pang, Ka Lok & Li, Yiwei, 2024. "The fading of status bias during the open peer review process," Journal of Informetrics, Elsevier, vol. 18(3).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. ederico Bianchi & Flaminio Squazzoni, 2022. "Can transparency undermine peer review? A simulation model of scientist behavior under open peer review [Reviewing Peer Review]," Science and Public Policy, Oxford University Press, vol. 49(5), pages 791-800.
    2. Sun, Zhuanlan & Pang, Ka Lok & Li, Yiwei, 2024. "The fading of status bias during the open peer review process," Journal of Informetrics, Elsevier, vol. 18(3).
    3. Sun, Zhuanlan, 2024. "Textual features of peer review predict top-cited papers: An interpretable machine learning perspective," Journal of Informetrics, Elsevier, vol. 18(2).
    4. Maciej J. Mrowinski & Agata Fronczak & Piotr Fronczak & Olgica Nedic & Aleksandar Dekanski, 2020. "The hurdles of academic publishing from the perspective of journal editors: a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 115-133, October.
    5. Zhao, Zhi-Dan & Chen, Jiahao & Lu, Yichuan & Zhao, Na & Jiang, Dazhi & Wang, Bing-Hong, 2021. "Dynamic patterns of open review process," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 582(C).
    6. Besim Bilalli & Rana Faisal Munir & Alberto Abelló, 2021. "A framework for assessing the peer review duration of journals: case study in computer science," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 545-563, January.
    7. Siddarth Srinivasan & Jamie Morgenstern, 2021. "Auctions and Peer Prediction for Academic Peer Review," Papers 2109.00923, arXiv.org, revised May 2023.
    8. Cheng, Xi & Wang, Haoran & Tang, Li & Jiang, Weiyan & Zhou, Maotian & Wang, Guoyan, 2024. "Open peer review correlates with altmetrics but not with citations: Evidence from Nature Communications and PLoS One," Journal of Informetrics, Elsevier, vol. 18(3).
    9. Stephen A Gallo & Afton S Carpenter & Scott R Glisson, 2013. "Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes," PLOS ONE, Public Library of Science, vol. 8(8), pages 1-9, August.
    10. Hou, Li & Wu, Qiang & Xie, Yundong, 2024. "Does open identity of peer reviewers positively relate to citations?," Journal of Informetrics, Elsevier, vol. 18(1).
    11. José Cendejas Bueno & Cecilia Font de Villanueva, 2015. "Convergence of inflation with a common cycle: estimating and modelling Spanish historical inflation from the 16th to the 18th centuries," Empirical Economics, Springer, vol. 48(4), pages 1643-1665, June.
    12. Lucas Rodriguez Forti & Luiz A. Solino & Judit K. Szabo, 2021. "Trade-off between urgency and reduced editorial capacity affect publication speed in ecological and medical journals during 2020," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-9, December.
    13. Sun, Zhuanlan & He, Dongjin & Li, Yiwei, 2024. "How the readability of manuscript before journal submission advantages peer review process: Evidence from biomedical scientific publications," Journal of Informetrics, Elsevier, vol. 18(3).
    14. Jürgen Janger & Nicole Schmidt-Padickakudy & Anna Strauss-Kollin, 2019. "International Differences in Basic Research Grant Funding. A Systematic Comparison," WIFO Studies, WIFO, number 61664.
    15. Louise Bedsworth, 2012. "California’s local health agencies and the state’s climate adaptation strategy," Climatic Change, Springer, vol. 111(1), pages 119-133, March.
    16. Rodríguez Sánchez, Isabel & Makkonen, Teemu & Williams, Allan M., 2019. "Peer review assessment of originality in tourism journals: critical perspective of key gatekeepers," Annals of Tourism Research, Elsevier, vol. 77(C), pages 1-11.
    17. Parul Khurana & Kiran Sharma, 2022. "Impact of h-index on author’s rankings: an improvement to the h-index for lower-ranked authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4483-4498, August.
    18. Zhentao Liang & Jin Mao & Gang Li, 2023. "Bias against scientific novelty: A prepublication perspective," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(1), pages 99-114, January.
    19. Elena Veretennik & Maria Yudkevich, 2023. "Inconsistent quality signals: evidence from the regional journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3675-3701, June.
    20. Meyer, Matthias & Waldkirch, Rüdiger W. & Duscher, Irina & Just, Alexander, 2018. "Drivers of citations: An analysis of publications in “top” accounting journals," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 51(C), pages 24-46.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:17:y:2023:i:4:s1751157723000743. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.