IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v53y2024i10s0048733324001422.html
   My bibliography  Save this article

Beyond declarations: Metrics, rankings and responsible assessment

Author

Listed:
  • Morgan-Thomas, Anna
  • Tsoukas, Serafeim
  • Dudau, Adina
  • Gąska, Paweł

Abstract

Responsible assessment promotes expert judgment and opposes sole reliance on research metrics when assessing research excellence. While many institutions and national research panels declare commitment to responsible assessment practices, we ask: have these declarations affected the outcomes of research evaluation? Using data from the UK's 2021 national research quality exercise and focusing on the business and management discipline, we show that the strong association between journal rankings and expert evaluations has not changed, despite institutional endorsements of DORA (Declaration on Research Assessment). Additionally, we find that this correlation is strongest for the most prestigious journals. The implications of these findings are profound: they enhance understanding of the use of metrics in research evaluations post-DORA and highlight potential constraints in the deployment of responsible assessment.

Suggested Citation

  • Morgan-Thomas, Anna & Tsoukas, Serafeim & Dudau, Adina & Gąska, Paweł, 2024. "Beyond declarations: Metrics, rankings and responsible assessment," Research Policy, Elsevier, vol. 53(10).
    • Handle: RePEc:eee:respol:v:53:y:2024:i:10:s0048733324001422
    DOI: 10.1016/j.respol.2024.105093
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733324001422
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2024.105093?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    2. James J. Heckman & Sidharth Moktan, 2020. "Publishing and promotion in economics - The tyranny of the Top Five," Vox eBook Chapters, in: Sebastian Galliani & Ugo Panizza (ed.), Publishing and Measuring Success in Economics, edition 1, volume 1, chapter 1, pages 23-32, Centre for Economic Policy Research.
    3. Bryce, Cormac & Dowling, Michael & Lucey, Brian, 2020. "The journal quality perception gap," Research Policy, Elsevier, vol. 49(5).
    4. Wang, Jian & Veugelers, Reinhilde & Stephan, Paula, 2017. "Bias against novelty in science: A cautionary tale for users of bibliometric indicators," Research Policy, Elsevier, vol. 46(8), pages 1416-1436.
    5. Vogel, Rick & Hattke, Fabian & Petersen, Jessica, 2017. "Journal rankings in management and business studies: What rules do we play by?," Research Policy, Elsevier, vol. 46(10), pages 1707-1722.
    6. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the research excellence framework using departmental h-index: revisited," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 1013-1017, September.
    7. Duncan A Thomas & Maria Nedeva & Mayra M Tirado & Merle Jacob, 2020. "Changing research on research evaluation: A critical literature review to revisit the agenda," Research Evaluation, Oxford University Press, vol. 29(3), pages 275-288.
    8. Brooks, Chris & Schopohl, Lisa & Walker, James T., 2023. "Comparing perceptions of the impact of journal rankings between fields," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 90(C).
    9. Civera, Alice & Lehmann, Erik E. & Paleari, Stefano & Stockinger, Sarah A.E., 2020. "Higher education policy: Why hope for quality when rewarding quantity?," Research Policy, Elsevier, vol. 49(8).
    10. Arne Risa Hole, 2017. "Ranking Economics Journals Using Data From a National Research Evaluation Exercise," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 79(5), pages 621-636, October.
    11. Daniele Checchi & Alberto Ciolfi & Gianni De Fraja & Irene Mazzotta & Stefano Verzillo, 2021. "Have you Read This? An Empirical Comparison of the British REF Peer Review and the Italian VQR Bibliometric Algorithm," Economica, London School of Economics and Political Science, vol. 88(352), pages 1107-1129, October.
    12. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the Research Excellence Framework using departmental h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2165-2180, March.
    13. Jappelli, Tullio & Nappi, Carmela Anna & Torrini, Roberto, 2017. "Gender effects in research evaluation," Research Policy, Elsevier, vol. 46(5), pages 911-924.
    14. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    15. Ramani, Ravi S. & Aguinis, Herman & Coyle-Shapiro, Jacqueline A.M., 2022. "Defining, measuring, and rewarding scholarly impact: mind the level of analysis," LSE Research Online Documents on Economics 117286, London School of Economics and Political Science, LSE Library.
    16. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    17. Paul Wouters & Cassidy R. Sugimoto & Vincent Larivière & Marie E. McVeigh & Bernd Pulverer & Sarah de Rijcke & Ludo Waltman, 2019. "Rethinking impact factors: better ways to judge a journal," Nature, Nature, vol. 569(7758), pages 621-623, May.
    18. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    19. Ryazanova, Olga & McNamara, Peter & Aguinis, Herman, 2017. "Research performance as a quality signal in international labor markets: Visibility of business schools worldwide through a global research performance system," Journal of World Business, Elsevier, vol. 52(6), pages 831-841.
    20. Banal-Estañol, Albert & Jofre-Bonet, Mireia & Iori, Giulia & Maynou, Laia & Tumminello, Michele & Vassallo, Pietro, 2023. "Performance-based research funding: Evidence from the largest natural experiment worldwide," Research Policy, Elsevier, vol. 52(6).
    21. V. A. Traag & L. Waltman, 2019. "Systematic analysis of agreement between metrics and peer review in the UK REF," Palgrave Communications, Palgrave Macmillan, vol. 5(1), pages 1-12, December.
    22. Brooks, Chris & Fenton, Evelyn & Schopohl, Lisa & Walker, James, 2019. "Why does research in finance have so little impact?," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 58(C), pages 24-52.
    23. Gianni De Fraja & Giovanni Facchini & John Gathergood, 2019. "Academic salaries and public evaluation of university research: Evidence from the UK Research Excellence Framework," Economic Policy, CEPR, CESifo, Sciences Po;CES;MSH, vol. 34(99), pages 523-583.
    24. Drivas, Kyriakos & Kremmydas, Dimitris, 2020. "The Matthew effect of a journal's ranking," Research Policy, Elsevier, vol. 49(4).
    25. Daniel Feenberg & Ina Ganguli & Patrick Gaulé & Jonathan Gruber, 2017. "It’s Good to Be First: Order Bias in Reading and Citing NBER Working Papers," The Review of Economics and Statistics, MIT Press, vol. 99(1), pages 32-39, March.
    26. Kate Williams, 2020. "Playing the fields: Theorizing research impact and its assessment," Research Evaluation, Oxford University Press, vol. 29(2), pages 191-202.
    27. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    28. Linton, O. B. & Xu, E., 2022. "Auditing the Auditors: An evaluation of the REF2021 Output Results," Cambridge Working Papers in Economics 2266, Faculty of Economics, University of Cambridge.
    29. Khoo, Shee-Yee & Perotti, Pietro & Verousis, Thanos & Watermeyer, Richard, 2024. "Vice-chancellor narcissism and university performance," Research Policy, Elsevier, vol. 53(1).
    30. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thelwall, Mike & Kousha, Kayvan & Stuart, Emma & Makita, Meiko & Abdoli, Mahshid & Wilson, Paul & Levitt, Jonathan, 2023. "Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?," Research Policy, Elsevier, vol. 52(8).
    2. Brooks, Chris & Schopohl, Lisa & Walker, James T., 2023. "Comparing perceptions of the impact of journal rankings between fields," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 90(C).
    3. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    4. Drivas, Kyriakos & Kremmydas, Dimitris, 2020. "The Matthew effect of a journal's ranking," Research Policy, Elsevier, vol. 49(4).
    5. Banal-Estañol, Albert & Jofre-Bonet, Mireia & Iori, Giulia & Maynou, Laia & Tumminello, Michele & Vassallo, Pietro, 2023. "Performance-based research funding: Evidence from the largest natural experiment worldwide," Research Policy, Elsevier, vol. 52(6).
    6. Hudson, Robert, 2024. "Responding to incentives or gaming the system? How UK business academics respond to the Academic Journal Guide," Research Policy, Elsevier, vol. 53(9).
    7. Daniele Checchi & Alberto Ciolfi & Gianni De Fraja & Irene Mazzotta & Stefano Verzillo, 2021. "Have you Read This? An Empirical Comparison of the British REF Peer Review and the Italian VQR Bibliometric Algorithm," Economica, London School of Economics and Political Science, vol. 88(352), pages 1107-1129, October.
    8. Auschra, Carolin & Bartosch, Julia & Lohmeyer, Nora, 2022. "Differences in female representation in leading management and organization journals: Establishing a benchmark," Research Policy, Elsevier, vol. 51(3).
    9. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    10. Charemza, Wojciech & Lewandowski, Michał & Woźny, Łukasz, 2024. "On journal rankings and researchers' abilities," Journal of Informetrics, Elsevier, vol. 18(3).
    11. Hendrik P. Dalen, 2021. "How the publish-or-perish principle divides a science: the case of economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1675-1694, February.
    12. van Dalen, Hendrik Peter, 2021. "How the publish-or-perish principle divides a science: The case of economists," Other publications TiSEM a6a5a855-bb5a-4d52-a841-3, Tilburg University, School of Economics and Management.
    13. Kayvan Kousha & Mike Thelwall, 2024. "Factors associating with or predicting more cited or higher quality journal articles: An Annual Review of Information Science and Technology (ARIST) paper," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 75(3), pages 215-244, March.
    14. Yves Fassin, 2021. "Does the Financial Times FT50 journal list select the best management and economics journals?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5911-5943, July.
    15. Corsi, Marcella & D’Ippoliti, Carlo & Zacchia, Giulia, 2019. "Diversity of backgrounds and ideas: The case of research evaluation in economics," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    16. Cruz-Castro, Laura & Sanz-Menendez, Luis, 2021. "What should be rewarded? Gender and evaluation criteria for tenure and promotion," Journal of Informetrics, Elsevier, vol. 15(3).
    17. Piotr Śpiewanowski & Oleksandr Talavera, 2021. "Journal rankings and publication strategy," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3227-3242, April.
    18. Bryce, Cormac & Dowling, Michael & Lucey, Brian, 2020. "The journal quality perception gap," Research Policy, Elsevier, vol. 49(5).
    19. Shahd Al-Janabi & Lee Wei Lim & Luca Aquili, 2021. "Development of a tool to accurately predict UK REF funding allocation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8049-8062, September.
    20. Syed Hasan & Robert Breunig, 2021. "Article length and citation outcomes," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7583-7608, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:53:y:2024:i:10:s0048733324001422. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.