IDEAS home Printed from https://ideas.repec.org/a/pal/palcom/v11y2024i1d10.1057_s41599-024-04088-w.html
   My bibliography  Save this article

Can peer review accolade awards motivate reviewers? A large-scale quasi-natural experiment

Author

Listed:
  • Houqiang Yu

    (Sun Yat-sen University)

  • Yian Liang

    (Sun Yat-sen University)

  • Yinghua Xie

    (Sun Yat-sen University)

Abstract

The utilization of accolade awards to motivate reviewers is widespread, but their effectiveness remains uncertain. We aim to explore how receiving an accolade award affects reviewers’ subsequent number of reviews. In contrast to small-scale, unrepresentative experiments, we perform a large-scale, global, and all-disciplinary analysis based on quasi-natural experiments. By integrating the Publons, ORCID, MAG, SciSciNet, and OpenAlex databases, a large dataset that tracks the reviewers’ annual number of reviews and their bibliometric indicators is compiled, encompassing 179,794 individuals. Among them, 6605 individuals who receive the “Publons Global Peer Review Award” in 2018 are constituted as the experimental group. Those non-winners are then matched to form a control group. Using propensity score matching (PSM), 711 and 762 reviewers are matched as the experimental and control groups respectively. An analysis employing the difference-in-differences (DiD) method is conducted to examine the impact of an accolade award on reviewers’ subsequent number of reviews. It is found that, following the receipt of an accolade award, reviewers, on average, reviews about four fewer manuscripts, with the reduction exhibiting a V-shaped pattern. Additional analyses are conducted to examine how individual differences and socio-economic factors influence the awarding effects. Besides, we analyze the mechanisms underlying the awarding effect and propose strategies to motivate reviewers. In brief, the sharp decrease in marginal utility of accolades, the voluntary nature of peer review, and the unexpected properties of accolade awards are the potential mechanisms that generate the negative effect. The academic community should reassess the existing incentive strategies for reviewers.

Suggested Citation

  • Houqiang Yu & Yian Liang & Yinghua Xie, 2024. "Can peer review accolade awards motivate reviewers? A large-scale quasi-natural experiment," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-13, December.
    • Handle: RePEc:pal:palcom:v:11:y:2024:i:1:d:10.1057_s41599-024-04088-w
    DOI: 10.1057/s41599-024-04088-w
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1057/s41599-024-04088-w
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1057/s41599-024-04088-w?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Squazzoni, Flaminio & Bravo, Giangiacomo & Takács, Károly, 2013. "Does incentive provision increase the quality of peer review? An experimental study," Research Policy, Elsevier, vol. 42(1), pages 287-294.
    2. Clément de Chaisemartin & Xavier D'Haultfœuille, 2020. "Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects," American Economic Review, American Economic Association, vol. 110(9), pages 2964-2996, September.
    3. Neckermann, Susanne & Yang, Xiaolan, 2017. "Understanding the (unexpected) consequences of unexpected recognition," Journal of Economic Behavior & Organization, Elsevier, vol. 135(C), pages 131-142.
    4. Jordan, Peter J., 2020. "Quality standards and training are important in the peer review process, but what about engagement?," Industrial and Organizational Psychology, Cambridge University Press, vol. 13(1), pages 61-63, March.
    5. José Luis Ortega, 2017. "Are peer-review activities related to reviewer bibliometric performance? A scientometric analysis of Publons," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(2), pages 947-962, August.
    6. Li Zhang & Wei Lu & Jinqing Yang, 2023. "LAGOS‐AND: A large gold standard dataset for scholarly author name disambiguation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(2), pages 168-185, February.
    7. Jana Gallus & Bruno S. Frey, 2016. "Awards: A strategic management perspective," Strategic Management Journal, Wiley Blackwell, vol. 37(8), pages 1699-1714, August.
    8. Monica Aniela Zaharie & Marco Seeber, 2018. "Are non-monetary rewards effective in attracting peer reviewers? A natural experiment," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(3), pages 1587-1609, December.
    9. Zhang, Guangyao & Xu, Shenmeng & Sun, Yao & Jiang, Chunlin & Wang, Xianwen, 2022. "Understanding the peer review endeavor in scientific publishing," Journal of Informetrics, Elsevier, vol. 16(2).
    10. Jose A. García & Rosa Rodriguez-Sánchez & Joaquín Fdez-Valdivia, 2015. "The principal-agent problem in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(2), pages 297-308, February.
    11. Raj Chetty & Emmanuel Saez & Laszlo Sandor, 2014. "What Policies Increase Prosocial Behavior? An Experiment with Referees at the Journal of Public Economics," Journal of Economic Perspectives, American Economic Association, vol. 28(3), pages 169-188, Summer.
    12. Shu Zhao & Dong Zhang & Zhen Duan & Jie Chen & Yan-ping Zhang & Jie Tang, 2018. "A novel classification method for paper-reviewer recommendation," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1293-1313, June.
    13. Sergio Copiello, 2018. "On the money value of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 613-620, April.
    14. Bianchi, Federico & Grimaldo, Francisco & Squazzoni, Flaminio, 2019. "The F3-index. Valuing reviewers for scholarly journals," Journal of Informetrics, Elsevier, vol. 13(1), pages 78-86.
    15. Robinson, Carly D. & Gallus, Jana & Lee, Monica G. & Rogers, Todd, 2021. "The demotivating effect (and unintended message) of awards," Organizational Behavior and Human Decision Processes, Elsevier, vol. 163(C), pages 51-64.
    16. Hoogveld, Nicky & Zubanov, Nick, 2017. "The power of (no) recognition: Experimental evidence from the university classroom," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 67(C), pages 75-84.
    17. Carol Nash, 2023. "Roles and Responsibilities for Peer Reviewers of International Journals," Publications, MDPI, vol. 11(2), pages 1-24, June.
    18. Bruno S. Frey & Reto Jegen, 2001. "Motivation Crowding Theory," Journal of Economic Surveys, Wiley Blackwell, vol. 15(5), pages 589-611, December.
    19. Sizo, Amanda & Lino, Adriano & Reis, Luis Paulo & Rocha, Álvaro, 2019. "An overview of assessing the quality of peer review reports of scientific articles," International Journal of Information Management, Elsevier, vol. 46(C), pages 286-293.
    20. Federico Bianchi & Francisco Grimaldo & Giangiacomo Bravo & Flaminio Squazzoni, 2018. "The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1401-1420, September.
    21. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    22. Ho, Daniel & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2011. "MatchIt: Nonparametric Preprocessing for Parametric Causal Inference," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i08).
    23. Gregory B. Northcraft & Ann E. Tenbrunsel, 2011. "Effective Matrices, Decision Frames, and Cooperation in Volunteer Dilemmas: A Theoretical Perspective on Academic Peer Review," Organization Science, INFORMS, vol. 22(5), pages 1277-1285, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Monica Aniela Zaharie & Marco Seeber, 2018. "Are non-monetary rewards effective in attracting peer reviewers? A natural experiment," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(3), pages 1587-1609, December.
    2. García, J.A. & Montero-Parodi, J.J. & Rodriguez-Sánchez, Rosa & Fdez-Valdivia, J., 2023. "How to motivate a reviewer with a present bias to work harder," Journal of Informetrics, Elsevier, vol. 17(4).
    3. Li, Teng & Lu, Runjing, 2022. "Social undermining as a dark side of symbolic awards: Evidence from a regression discontinuity design," Organizational Behavior and Human Decision Processes, Elsevier, vol. 173(C).
    4. Robinson, Carly D. & Gallus, Jana & Lee, Monica G. & Rogers, Todd, 2021. "The demotivating effect (and unintended message) of awards," Organizational Behavior and Human Decision Processes, Elsevier, vol. 163(C), pages 51-64.
    5. Chervier, Colas & Le Velly, Gwenolé & Ezzine-de-Blas, Driss, 2019. "When the Implementation of Payments for Biodiversity Conservation Leads to Motivation Crowding-out: A Case Study From the Cardamoms Forests, Cambodia," Ecological Economics, Elsevier, vol. 156(C), pages 499-510.
    6. Bianchi, Federico & Grimaldo, Francisco & Squazzoni, Flaminio, 2019. "The F3-index. Valuing reviewers for scholarly journals," Journal of Informetrics, Elsevier, vol. 13(1), pages 78-86.
    7. Sergio Copiello, 2018. "On the money value of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 613-620, April.
    8. Rana, Pushpendra & Sills, Erin O., 2024. "Inviting oversight: Effects of forest certification on deforestation in the Brazilian Amazon," World Development, Elsevier, vol. 173(C).
    9. Zaharie, Monica Aniela & Osoian, Codruţa Luminiţa, 2016. "Peer review motivation frames: A qualitative approach," European Management Journal, Elsevier, vol. 34(1), pages 69-79.
    10. Chervier, Colas & Ximenes, Arimatéa C. & Mihigo, Blaise-Pascal Ntirumenyerwa & Doumenge, Charles, 2024. "Impact of industrial logging concession on deforestation and forest degradation in the DRC," World Development, Elsevier, vol. 173(C).
    11. Christiansen, Tobias Gabel, 2024. "Dynamic effects of tax audits and the role of intentions," Journal of Public Economics, Elsevier, vol. 234(C).
    12. Zhang, Guangyao & Xu, Shenmeng & Sun, Yao & Jiang, Chunlin & Wang, Xianwen, 2022. "Understanding the peer review endeavor in scientific publishing," Journal of Informetrics, Elsevier, vol. 16(2).
    13. Luo, Lianfa & Cheng, Zhiming & Ye, Qingqing & Cheng, Yanjun & Smyth, Russell & Yang, Zhiqing & Zhang, Le, 2024. "Nonmonetary awards and innovation: Evidence from winning China's Top Brand Contest," China Economic Review, Elsevier, vol. 86(C).
    14. Mansaray, Alhassan & Coleman, Simeon & Ataullah, Ali & Sirichand, Kavita, 2021. "Residual government ownership in public-private partnership projects," Journal of Government and Economics, Elsevier, vol. 4(C).
    15. Charles Ayoubi & Boris Thurm, 2023. "Knowledge diffusion and morality: Why do we freely share valuable information with Strangers?," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 32(1), pages 75-99, January.
    16. Rigdon, Joseph & Berkowitz, Seth A. & Seligman, Hilary K. & Basu, Sanjay, 2017. "Re-evaluating associations between the Supplemental Nutrition Assistance Program participation and body mass index in the context of unmeasured confounders," Social Science & Medicine, Elsevier, vol. 192(C), pages 112-124.
    17. Stjepan Srhoj & Michael Lapinski & Janette Walde, 2019. "Size matters? Impact evaluation of business development grants on SME performance," Working Papers 2019-14, Faculty of Economics and Statistics, Universität Innsbruck.
    18. Pavithra Harsha & Shivaram Subramanian & Joline Uichanco, 2019. "Dynamic Pricing of Omnichannel Inventories," Service Science, INFORMS, vol. 21(1), pages 47-65, January.
    19. Lenis, David & Ackerman, Benjamin & Stuart, Elizabeth A., 2018. "Measuring model misspecification: Application to propensity score methods with complex survey data," Computational Statistics & Data Analysis, Elsevier, vol. 128(C), pages 48-57.
    20. Raj Chetty & Emmanuel Saez & Laszlo Sandor, 2014. "What Policies Increase Prosocial Behavior? An Experiment with Referees at the Journal of Public Economics," Journal of Economic Perspectives, American Economic Association, vol. 28(3), pages 169-188, Summer.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pal:palcom:v:11:y:2024:i:1:d:10.1057_s41599-024-04088-w. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: https://www.nature.com/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.