IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v49y2020i1s0048733319301519.html
   My bibliography  Save this article

How to avoid borrowed plumes in academia

Author

Listed:
  • Osterloh, Margit
  • Frey, Bruno S.

Abstract

Publications in top journals today have a powerful influence on academic careers although there is much criticism of using journal rankings to evaluate individual articles. We ask why this practice of performance evaluation is still so influential. We suggest this is the case because a majority of authors benefit from the present system due to the extreme skewness of citation distributions. “Performance paradox” effects aggravate the problem. Three extant suggestions for reforming performance management are critically discussed. We advance a new proposal based on the insight that fundamental uncertainty is symptomatic for scholarly work. It suggests focal randomization using a rationally founded and well-orchestrated procedure.

Suggested Citation

  • Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    • Handle: RePEc:eee:respol:v:49:y:2020:i:1:s0048733319301519
    DOI: 10.1016/j.respol.2019.103831
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733319301519
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2019.103831?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paul Nightingale & Alister Scott, 2007. "Peer review and the relevance gap: Ten suggestions for policy-makers," Science and Public Policy, Oxford University Press, vol. 34(8), pages 543-553, October.
    2. Alexandru T. Balaban, 2012. "Positive and negative aspects of citation indices and journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 241-247, August.
    3. V. Ginsburgh & Sheila Weyers, 2014. "Nominees, winners, and losers," Journal of Cultural Economics, Springer;The Association for Cultural Economics International, vol. 38(4), pages 291-313, November.
    4. James J. Heckman & Sidharth Moktan, 2020. "Publishing and promotion in economics - The tyranny of the Top Five," Vox eBook Chapters, in: Sebastian Galliani & Ugo Panizza (ed.), Publishing and Measuring Success in Economics, edition 1, volume 1, chapter 1, pages 23-32, Centre for Economic Policy Research.
    5. Bruno S. Frey & Katja Rost, 2010. "Do rankings reflect research quality?," Journal of Applied Economics, Universidad del CEMA, vol. 13, pages 1-38, May.
    6. Goodall, Amanda H., 2009. "Highly cited leaders and the performance of research universities," Research Policy, Elsevier, vol. 38(7), pages 1079-1092, September.
    7. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision‐Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    8. Frey, Bruno S, 2003. "Publishing as Prostitution?--Choosing between One's Own Ideas and Academic Success," Public Choice, Springer, vol. 116(1-2), pages 205-223, July.
    9. Wang, Jian & Veugelers, Reinhilde & Stephan, Paula, 2017. "Bias against novelty in science: A cautionary tale for users of bibliometric indicators," Research Policy, Elsevier, vol. 46(8), pages 1416-1436.
    10. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    11. Qiang Zhi & Tianguang Meng, 2016. "Funding allocation, inequality, and scientific research output: an empirical study based on the life science sector of Natural Science Foundation of China," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 603-628, February.
    12. Vogel, Rick & Hattke, Fabian & Petersen, Jessica, 2017. "Journal rankings in management and business studies: What rules do we play by?," Research Policy, Elsevier, vol. 46(10), pages 1707-1722.
    13. Stephan, Paula E., 2010. "The Economics of Science," Handbook of the Economics of Innovation, in: Bronwyn H. Hall & Nathan Rosenberg (ed.), Handbook of the Economics of Innovation, edition 1, volume 1, chapter 0, pages 217-273, Elsevier.
    14. Henk F Moed, 2007. "The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review," Science and Public Policy, Oxford University Press, vol. 34(8), pages 575-583, October.
    15. Peter Stolz, 1986. "Parteienwettbewerb, politisches Kartell und Tausch zwischen sozioökonomischen Gruppen," Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 122(IV), pages 657-675, December.
    16. Joshua S. Gans & George B. Shepherd, 1994. "How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists," Journal of Economic Perspectives, American Economic Association, vol. 8(1), pages 165-179, Winter.
    17. Ewen Callaway, 2016. "Beat it, impact factor! Publishing elite turns against controversial metric," Nature, Nature, vol. 535(7611), pages 210-211, July.
    18. Elise S Brezis, 2007. "Focal randomisation: An optimal mechanism for the evaluation of R&D projects," Science and Public Policy, Oxford University Press, vol. 34(10), pages 691-698, December.
    19. Wilde, Christian & Krahnen, Jan Pieter & Ockenfels, Peter, 2014. "Measuring Ambiguity Aversion: A Systematic Experimental Approach," VfS Annual Conference 2014 (Hamburg): Evidence-based Economic Policy 100557, Verein für Socialpolitik / German Economic Association.
    20. Bruno Frey, 2009. "Economists in the PITS?," International Review of Economics, Springer;Happiness Economics and Interpersonal Relations (HEIRS), vol. 56(4), pages 335-346, December.
    21. David N. Laband, 2013. "On the Use and Abuse of Economics Journal Rankings," Economic Journal, Royal Economic Society, vol. 0, pages 223-254, August.
    22. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    23. Partha, Dasgupta & David, Paul A., 1994. "Toward a new economics of science," Research Policy, Elsevier, vol. 23(5), pages 487-521, September.
    24. Daniel S. Hamermesh, 2018. "Citations in Economics: Measurement, Uses, and Impacts," Journal of Economic Literature, American Economic Association, vol. 56(1), pages 115-156, March.
    25. Ichiko Fuyuno & David Cyranoski, 2006. "Cash for papers: putting a premium on publication," Nature, Nature, vol. 441(7095), pages 792-792, June.
    26. William H. Starbuck, 2005. "How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication," Organization Science, INFORMS, vol. 16(2), pages 180-200, April.
    27. J Mingers, 2008. "Exploring the dynamics of journal citations: Modelling with s-curves," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 59(8), pages 1013-1025, August.
    28. Macdonald, Stuart & Kam, Jacqueline, 2010. "Counting footnotes: Citability in management studies," Scandinavian Journal of Management, Elsevier, vol. 26(2), pages 189-203, June.
    29. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    30. Alexander T. Nicolai & Stanislaw Schmal & Charlotte L. Schuster, 2015. "Interrater Reliability of the Peer Review Process in Management Journals," Springer Books, in: Isabell M. Welpe & Jutta Wollersheim & Stefanie Ringelhan & Margit Osterloh (ed.), Incentives and Performance, edition 127, pages 107-119, Springer.
    31. Mingers, John & Xu, Fang, 2010. "The drivers of citations in management science journals," European Journal of Operational Research, Elsevier, vol. 205(2), pages 422-430, September.
    32. David N. Laband & Robert D. Tollison, 2003. "Dry Holes in Economic Research," Kyklos, Wiley Blackwell, vol. 56(2), pages 161-173, May.
    33. Katja Rost & Thorsten Teichert & Alan Pilkington, 2017. "Social network analytics for advanced bibliometrics: referring to actor roles of management journals instead of journal rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1631-1657, September.
    34. Margit Osterloh & Alfred Kieser, 2015. "Double-Blind Peer Review: How to Slaughter a Sacred Cow," Springer Books, in: Isabell M. Welpe & Jutta Wollersheim & Stefanie Ringelhan & Margit Osterloh (ed.), Incentives and Performance, edition 127, pages 307-321, Springer.
    35. Mårtensson, Pär & Fors, Uno & Wallin, Sven-Bertil & Zander, Udo & Nilsson, Gunnar H, 2016. "Evaluating research: A multidisciplinary approach to assessing research practice and quality," Research Policy, Elsevier, vol. 45(3), pages 593-603.
    36. Giovanni Dosi & Franco Malerba & Giovanni B. Ramello & Francesco Silva, 2006. "Information, appropriability, and the generation of innovative knowledge four decades after Arrow and Nelson: an introduction," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 15(6), pages 891-901, December.
    37. Éric Archambault & Vincent Larivière, 2009. "History of the journal impact factor: Contingencies and consequences," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(3), pages 635-649, June.
    38. Stuart Macdonald & Jacqueline Kam, 2007. "Ring a Ring o’ Roses: Quality Journals and Gamesmanship in Management Studies," Journal of Management Studies, Wiley Blackwell, vol. 44(4), pages 640-655, June.
    39. Wallace, Matthew L. & Larivière, Vincent & Gingras, Yves, 2009. "Modeling a century of citation distributions," Journal of Informetrics, Elsevier, vol. 3(4), pages 296-303.
    40. Richard R. Nelson, 2006. "The Market Economy and the Scientific Commons," Chapters, in: Birgitte Andersen (ed.), Intellectual Property Rights, chapter 1, Edward Elgar Publishing.
    41. Anne‐Wil Harzing & Ron van der Wal, 2009. "A Google Scholar h‐index for journals: An alternative metric to measure journal impact in economics and business," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 41-46, January.
    42. Richard R. Nelson, 1959. "The Simple Economics of Basic Scientific Research," Journal of Political Economy, University of Chicago Press, vol. 67(3), pages 297-297.
    43. Mats Alvesson & Jörgen Sandberg, 2013. "Has Management Studies Lost Its Way? Ideas for More Imaginative and Innovative Research," Journal of Management Studies, Wiley Blackwell, vol. 50(1), pages 128-152, January.
    44. William H. Starbuck, 2015. "Issues and Trends in Publishing Behavioral Science: A Quarrelsome Crew Struggling with a Disintegrating Boat on a Stormy Sea," Springer Books, in: Isabell M. Welpe & Jutta Wollersheim & Stefanie Ringelhan & Margit Osterloh (ed.), Incentives and Performance, edition 127, pages 3-18, Springer.
    45. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    46. Juan Miguel Campanario, 2009. "Rejecting and resisting Nobel class discoveries: accounts by Nobel Laureates," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 549-565, November.
    47. Michelle Gittelman & Bruce Kogut, 2003. "Does Good Science Lead to Valuable Knowledge? Biotechnology Firms and the Evolutionary Logic of Citation Patterns," Management Science, INFORMS, vol. 49(4), pages 366-382, April.
    48. Nicholas Graves & Adrian G. Barnett & Philip Clarke, 2011. "Cutting random funding decisions," Nature, Nature, vol. 469(7330), pages 299-299, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. van Dalen, Hendrik Peter, 2021. "How the publish-or-perish principle divides a science: The case of economists," Other publications TiSEM a6a5a855-bb5a-4d52-a841-3, Tilburg University, School of Economics and Management.
    2. Emily Yarrow & Karen Johnston, 2023. "Athena SWAN: “Institutional peacocking” in the neoliberal university," Gender, Work and Organization, Wiley Blackwell, vol. 30(3), pages 757-772, May.
    3. Amanda Goodall & Margit Osterloh & Mandy Fong, 2020. "Women Shy Away From Competition – How To Overcome It," CREMA Working Paper Series 2020-21, Center for Research in Economics, Management and the Arts (CREMA).
    4. Peter Andre & Armin Falk, 2021. "What’s Worth Knowing? Economists’ Opinions about Economics," ECONtribute Discussion Papers Series 102, University of Bonn and University of Cologne, Germany.
    5. Marta Kuc-Czarnecka & Magdalena Olczyk, 2020. "How ethics combine with big data: a bibliometric analysis," Palgrave Communications, Palgrave Macmillan, vol. 7(1), pages 1-9, December.
    6. Thanos Fragkandreas, 2023. "Case study research on innovation systems: paradox, dialectical analysis and resolution," Working Papers 65, Birkbeck Centre for Innovation Management Research, revised 15 May 2023.
    7. Bruno S. Frey & Margit Osterloh & Katja Rost, 2021. "Die Rationalität des qualifizierten Losverfahrens," CREMA Working Paper Series 2021-37, Center for Research in Economics, Management and the Arts (CREMA).
    8. Gabriel-Alexandru Vîiu & Mihai Păunescu, 2021. "The citation impact of articles from which authors gained monetary rewards based on journal metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4941-4974, June.
    9. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    10. Hendrik P. Dalen, 2021. "How the publish-or-perish principle divides a science: the case of economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1675-1694, February.
    11. Bruno S. Frey, 2021. "Backward‐oriented economics," Kyklos, Wiley Blackwell, vol. 74(2), pages 187-195, May.
    12. Fabian Scheidegger & Andre Briviba & Bruno S. Frey, 2023. "Behind the curtains of academic publishing: strategic responses of economists and business scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4765-4790, August.
    13. Elena Veretennik & Maria Yudkevich, 2023. "Inconsistent quality signals: evidence from the regional journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3675-3701, June.
    14. Drivas, Kyriakos & Kremmydas, Dimitris, 2020. "The Matthew effect of a journal's ranking," Research Policy, Elsevier, vol. 49(4).
    15. Schmal, W. Benedikt, 2024. "The X Factor: Open Access, New Journals, and Incumbent Competitors," VfS Annual Conference 2024 (Berlin): Upcoming Labor Market Challenges 302342, Verein für Socialpolitik / German Economic Association.
    16. Auschra, Carolin & Bartosch, Julia & Lohmeyer, Nora, 2022. "Differences in female representation in leading management and organization journals: Establishing a benchmark," Research Policy, Elsevier, vol. 51(3).
    17. Buehling, Kilian, 2021. "Changing research topic trends as an effect of publication rankings – The case of German economists and the Handelsblatt Ranking," Journal of Informetrics, Elsevier, vol. 15(3).
    18. Tran, Trung & Linh, Hoang Khanh & La, Viet-Phuong & Ho, Manh-Toan & Vuong, Quan-Hoang, 2020. "Scrambling for higher metrics in the Journal Impact Factor bubble period: a real-world problem in science management and its implications," OSF Preprints dmsp9, Center for Open Science.
    19. Valdivia, Walter, 2021. "A Lottery for the Republic of Science: Chance, Merit, and Fairness in the Process of Awarding Research Grants," Working Papers 10461, George Mason University, Mercatus Center.
    20. Matteo Migheli & Giovanni B. Ramello, 2021. "The unbearable lightness of scientometric indices," Managerial and Decision Economics, John Wiley & Sons, Ltd., vol. 42(8), pages 1933-1944, December.
    21. Follert, Florian & Naumann, Chantal & Thieme, Lutz, 2020. "Between scientific publication and public perception: Some economic remarks on the allocation of time in science," Working Papers of the European Institute for Socioeconomics 34, European Institute for Socioeconomics (EIS), Saarbrücken.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Drivas, Kyriakos & Kremmydas, Dimitris, 2020. "The Matthew effect of a journal's ranking," Research Policy, Elsevier, vol. 49(4).
    2. Margit Osterloh & Bruno S. Frey, 2010. "Academic rankings and research governance," IEW - Working Papers 482, Institute for Empirical Research in Economics - University of Zurich.
    3. Margit Osterloh & Bruno S. Frey, 2009. "Research Governance in Academia: Are there Alternatives to Academic Rankings?," CREMA Working Paper Series 2009-17, Center for Research in Economics, Management and the Arts (CREMA).
    4. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    5. Matthias Aistleitner & Jakob Kapeller & Stefan Steinerberger, 2018. "Citation Patterns in Economics and Beyond," Working Papers Series 85, Institute for New Economic Thinking.
    6. Carlo D'Ippoliti, 2021. "“Many‐Citedness”: Citations Measure More Than Just Scientific Quality," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1271-1301, December.
    7. van Dalen, Hendrik Peter, 2020. "How the Publish-or-Perish Principle Divides a Science : The Case of Academic Economists," Other publications TiSEM 6fbb6b92-0e06-4271-b6e7-3, Tilburg University, School of Economics and Management.
    8. van Dalen, Hendrik Peter, 2021. "How the publish-or-perish principle divides a science: The case of economists," Other publications TiSEM a6a5a855-bb5a-4d52-a841-3, Tilburg University, School of Economics and Management.
    9. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    10. Buehling, Kilian, 2021. "Changing research topic trends as an effect of publication rankings – The case of German economists and the Handelsblatt Ranking," Journal of Informetrics, Elsevier, vol. 15(3).
    11. Katja Rost & Bruno S. Frey, 2011. "Quantitative and Qualitative Rankings of Scholars," Schmalenbach Business Review (sbr), LMU Munich School of Management, vol. 63(1), pages 63-91, January.
    12. Dell'Anno, Roberto & Caferra, Rocco & Morone, Andrea, 2020. "A “Trojan Horse” in the peer-review process of fee-charging economic journals," Journal of Informetrics, Elsevier, vol. 14(3).
    13. Battiston, Pietro & Sacco, Pier Luigi & Stanca, Luca, 2022. "Cover effects on citations uncovered: Evidence from Nature," Journal of Informetrics, Elsevier, vol. 16(2).
    14. Rodríguez Sánchez, Isabel & Makkonen, Teemu & Williams, Allan M., 2019. "Peer review assessment of originality in tourism journals: critical perspective of key gatekeepers," Annals of Tourism Research, Elsevier, vol. 77(C), pages 1-11.
    15. Gans, Joshua S. & Murray, Fiona E. & Stern, Scott, 2017. "Contracting over the disclosure of scientific knowledge: Intellectual property and academic publication," Research Policy, Elsevier, vol. 46(4), pages 820-835.
    16. Veugelers, Reinhilde & Wang, Jian, 2019. "Scientific novelty and technological impact," Research Policy, Elsevier, vol. 48(6), pages 1362-1372.
    17. repec:hal:cepnwp:hal-01922263 is not listed on IDEAS
    18. Hendrik P. Dalen, 2021. "How the publish-or-perish principle divides a science: the case of economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1675-1694, February.
    19. Joseph Gerald Hirschberg & Jeanette Ngaire Lye, 2020. "Grading Journals In Economics: The Abcs Of The Abdc," Journal of Economic Surveys, Wiley Blackwell, vol. 34(4), pages 876-921, September.
    20. Charles Ayoubi & Michele Pezzoni & Fabiana Visentin, 2021. "Does It Pay to Do Novel Science? The Selectivity Patterns in Science Funding," Science and Public Policy, Oxford University Press, vol. 48(5), pages 635-648.
    21. J. Atsu Amegashie, 2020. "Citations And Incentives In Academic Contests," Economic Inquiry, Western Economic Association International, vol. 58(3), pages 1233-1244, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:49:y:2020:i:1:s0048733319301519. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.