IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v52y2023i6s0048733323000641.html
   My bibliography  Save this article

Performance-based research funding: Evidence from the largest natural experiment worldwide

Author

Listed:
  • Banal-Estañol, Albert
  • Jofre-Bonet, Mireia
  • Iori, Giulia
  • Maynou, Laia
  • Tumminello, Michele
  • Vassallo, Pietro

Abstract

A performance-based research funding system (PRFS) is a nationwide incentive scheme that promotes and rewards university research performance through competition for government funding. The UK’s PRFS, currently the Research Excellence Framework (REF), is considered the oldest, largest and most developed payment-by-results system in academia worldwide. Surprisingly, and despite the strong criticisms, little has been done to quantitatively and casually evaluate the intended and unintended effects of the PRFSs. In this paper, we evaluate the incremental impact of the REF 2014 in the fields of Economics and Business. We use a synthetic control method to compare the performance of UK universities with their artificial counterfactual units constructed using data from US universities. Our analysis shows, on the whole, that the introduction of the REF had a significant and positive impact on the quantity and quality of the scientific research produced at UK universities. However, we do not find a significant effect on the per author measures, suggesting that the REF did not result in an increase in research productivity. We also show that the effects are more heterogeneous across universities than across academic disciplines. We do not find evidence of a shift of research focus from Economics to Business topics, as some feared. But our analysis indicates that the REF 2014 may have contributed to the concentration of research excellence in elite institutions.

Suggested Citation

  • Banal-Estañol, Albert & Jofre-Bonet, Mireia & Iori, Giulia & Maynou, Laia & Tumminello, Michele & Vassallo, Pietro, 2023. "Performance-based research funding: Evidence from the largest natural experiment worldwide," Research Policy, Elsevier, vol. 52(6).
    • Handle: RePEc:eee:respol:v:52:y:2023:i:6:s0048733323000641
    DOI: 10.1016/j.respol.2023.104780
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733323000641
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2023.104780?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Thomas Zacharewicz & Benedetto Lepori & Emanuela Reale & Koen Jonkers, 2019. "Performance-based research funding in EU Member States—a comparative assessment," Science and Public Policy, Oxford University Press, vol. 46(1), pages 105-115.
    2. Auranen, Otto & Nieminen, Mika, 2010. "University research funding and publication performance--An international comparison," Research Policy, Elsevier, vol. 39(6), pages 822-834, July.
    3. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    4. Joonha Jeon & So Young Kim, 2018. "Is the gap widening among universities? On research output inequality and its measurement in the Korean higher education system," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(2), pages 589-606, March.
    5. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    6. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    7. Engelbert Stockhammer & Quirin Dammerer & Sukriti Kapur, 2021. "The Research Excellence Framework 2014, journal ratings and the marginalisation of heterodox economics," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 45(2), pages 243-269.
    8. Basso, Antonella & di Tollo, Giacomo, 2022. "Prediction of UK research excellence framework assessment by the departmental h-index," European Journal of Operational Research, Elsevier, vol. 296(3), pages 1036-1049.
    9. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    10. Noémi Kreif & Richard Grieve & Dominik Hangartner & Alex James Turner & Silviya Nikolova & Matt Sutton, 2016. "Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1514-1528, December.
    11. Geuna, Aldo & Piolatto, Matteo, 2016. "Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while)," Research Policy, Elsevier, vol. 45(1), pages 260-271.
    12. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the research excellence framework using departmental h-index: revisited," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 1013-1017, September.
    13. Koenraad Debackere & Wolfgang Glänzel, 2004. "Using a bibliometric approach to support research policy making: The case of the Flemish BOF-key," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(2), pages 253-276, February.
    14. Frederic S. Lee & Xuan Pham & Gyun Gu, 2013. "The UK Research Assessment Exercise and the narrowing of UK economics," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 37(4), pages 693-717.
    15. Arne Risa Hole, 2017. "Ranking Economics Journals Using Data From a National Research Evaluation Exercise," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 79(5), pages 621-636, October.
    16. Acemoglu, Daron & Johnson, Simon & Kermani, Amir & Kwak, James & Mitton, Todd, 2016. "The value of connections in turbulent times: Evidence from the United States," Journal of Financial Economics, Elsevier, vol. 121(2), pages 368-391.
    17. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the Research Excellence Framework using departmental h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2165-2180, March.
    18. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    19. Bruno S. Frey & Reto Jegen, 2001. "Motivation Crowding Theory," Journal of Economic Surveys, Wiley Blackwell, vol. 15(5), pages 589-611, December.
    20. Daniele Checchi & Irene Mazzotta & Sandro Momigliano & Francesco Olivanti, 2020. "Convergence or polarisation? The impact of research assessment exercises in the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1439-1455, August.
    21. Koen Jonkers & Thomas Zacharewicz, 2016. "Research Performance Based Funding Systems: a Comparative Assessment," JRC Research Reports JRC101043, Joint Research Centre.
    22. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    23. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    24. Björn Hammarfelt & Sarah de Rijcke, 2015. "Accountability in context: effects of research evaluation systems on publication practices, disciplinary norms, and individual working routines in the faculty of Arts at Uppsala University," Research Evaluation, Oxford University Press, vol. 24(1), pages 63-77.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Vasile Cernat, 2024. "The unprincipled principal: how Romania’s inconsistent research reform impacted scientific output," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(9), pages 5557-5575, September.
    2. Oliver, Edward & Kourouklis, Dimitrios & Jofre-Bonet, Mireia, 2024. "Do R&D tax credits impact pharmaceutical innovation? Evidence from a synthetic control approach," Research Policy, Elsevier, vol. 53(8).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ajab Khan & Ali Sina Önder & Sercan Özcan, 2023. "Does Performance-based Public Funding Pay off? UK’s Research Excellence Framework (REF) and Research Productivity," Working Papers in Economics & Finance 2023-08, University of Portsmouth, Portsmouth Business School, Economics and Finance Subject Group.
    2. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    3. Tomasz Serwach, 2022. "The European Union and within-country income inequalities. The case of the New Member States," Working Papers hal-03548416, HAL.
    4. Emery, Thomas & Mélon, Lela & Spruk, Rok, 2023. "Does e-procurement matter for economic growth? Subnational evidence from Australia," The Quarterly Review of Economics and Finance, Elsevier, vol. 89(C), pages 318-334.
    5. Cummins Joseph & Miller Douglas L. & Smith Brock & Simon David, 2024. "Matching on Noise: Finite Sample Bias in the Synthetic Control Estimator," Journal of Econometric Methods, De Gruyter, vol. 13(1), pages 67-95, January.
    6. Samuel Verevis & Murat Üngör, 2021. "What has New Zealand gained from The FTA with China?: Two counterfactual analyses†," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(1), pages 20-50, February.
    7. Tomasz Serwach, 2023. "The European Union and within‐country income inequalities. The case of the new member states," The World Economy, Wiley Blackwell, vol. 46(7), pages 1890-1939, July.
    8. David Gilchrist & Thomas Emery & Nuno Garoupa & Rok Spruk, 2023. "Synthetic Control Method: A tool for comparative case studies in economic history," Journal of Economic Surveys, Wiley Blackwell, vol. 37(2), pages 409-445, April.
    9. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    10. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2021. "Decoupling synthetic control methods to ensure stability, accuracy and meaningfulness," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 12(4), pages 549-584, December.
    11. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    12. Mehmet Pinar, 2023. "Do research performances of universities and disciplines in England converge or diverge? An assessment of the progress between research excellence frameworks in 2014 and 2021," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5731-5766, October.
    13. Olper, Alessandro & Curzi, Daniele & Swinnen, Johan, 2018. "Trade liberalization and child mortality: A Synthetic Control Method," World Development, Elsevier, vol. 110(C), pages 394-410.
    14. Matej Opatrny, 2021. "The impact of the Brexit vote on UK financial markets: a synthetic control method approach," Empirica, Springer;Austrian Institute for Economic Research;Austrian Economic Association, vol. 48(2), pages 559-587, May.
    15. Chan, Ho Fai & Gangl, Katharina & Supriyadi, Mohammad Wangsit & Torgler, Benno, 2023. "The effects of increased monitoring on high wealth individuals: Evidence from a quasi-natural experiment in Indonesia," Journal of Economic Behavior & Organization, Elsevier, vol. 215(C), pages 244-267.
    16. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2020. "Ensuring Stability, Accuracy and Meaningfulness in Synthetic Control Methods: The Regularized SHAP-Distance Method," IREA Working Papers 202005, University of Barcelona, Research Institute of Applied Economics, revised Apr 2020.
    17. Manuel Funke & Moritz Schularick & Christoph Trebesch, 2023. "Populist Leaders and the Economy," American Economic Review, American Economic Association, vol. 113(12), pages 3249-3288, December.
    18. Pekka Malo & Juha Eskelinen & Xun Zhou & Timo Kuosmanen, 2024. "Computing Synthetic Controls Using Bilevel Optimization," Computational Economics, Springer;Society for Computational Economics, vol. 64(2), pages 1113-1136, August.
    19. Silvia Marchesi & Tania Masi, 2019. "Sovereign risk after sovereign restructuring. Private and official default," Working Papers 423, University of Milano-Bicocca, Department of Economics, revised Nov 2019.
    20. Barseghyan, Gayane, 2019. "Sanctions and counter-sanctions : What did they do?," BOFIT Discussion Papers 24/2019, Bank of Finland, Institute for Economies in Transition.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:52:y:2023:i:6:s0048733323000641. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.