IDEAS home Printed from https://ideas.repec.org/p/vuw/vuwcpf/20928.html
   My bibliography  Save this paper

An Evaluation of Metrics Used by the Performance-Based Research Fund Process in New Zealand

Author

Listed:
  • Buckle, Robert A
  • Creedy, John

Abstract

The New Zealand Performance-based Research Fund (PBRF) applies a unique set of metrics to assess researchers and rank disciplines and universities. The process involves giving a ‘raw score’ to individual researchers and then assigning them to one of four Quality Evaluation Categories (QECs), used to derive Average Quality Scores (AQS). This paper evaluates the properties of these metrics and argues that QEC thresholds influence the final distribution of assessments. The paper also demonstrates that the derivation of AQSs depends on the weights assigned to each QEC and the distribution of portfolios. The method used to determine the raw scores also has an independent effect on the distribution of scores. The number of researchers included in evaluations also influences the rankings. The paper compares how research rankings of New Zealand universities would vary if alternative summary measures, based on the raw scores rather than QECs, were used to evaluate peformance.

Suggested Citation

  • Buckle, Robert A & Creedy, John, 2017. "An Evaluation of Metrics Used by the Performance-Based Research Fund Process in New Zealand," Working Paper Series 20928, Victoria University of Wellington, Chair in Public Finance.
    • Handle: RePEc:vuw:vuwcpf:20928
    as

    Download full text from publisher

    File URL: https://ir.wgtn.ac.nz/handle/123456789/20928
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Robert A. Buckle & John Creedy, 2019. "The evolution of research quality in New Zealand universities as measured by the performance-based research fund process," New Zealand Economic Papers, Taylor & Francis Journals, vol. 53(2), pages 144-165, May.
    2. Perc, Matjaž, 2010. "Zipf’s law and log-normal distributions in measures of scientific output across fields and institutions: 40 years of Slovenia’s research as an example," Journal of Informetrics, Elsevier, vol. 4(3), pages 358-364.
    3. James J. Heckman & Michael Sattinger, 2015. "Introduction to The Distribution of Earnings and of Individual Output, by A.D. Roy," Economic Journal, Royal Economic Society, vol. 0(583), pages 378-402, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    2. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    3. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 9447, Victoria University of Wellington, Chair in Public Finance.
    4. Robert A. Buckle & John Creedy & Norman Gemmell, 2020. "Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand," Applied Economics, Taylor & Francis Journals, vol. 52(36), pages 3919-3932, July.
    5. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    6. Robert A. Buckle and John Creedy, 2018. "The Impact on Research Quality of Performance-Based Funding: The Case of New Zealand’s PBRF Scheme," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 24(1), pages 25-48.
    7. Robert A. Buckle and John Creedy, 2018. "The Impact on Research Quality of Performance-Based Funding: The Case of New Zealand’s PBRF Scheme," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 24(1), pages 25-48.
    8. Buckle, Robert A & Creedy, John, 2022. "The Performance Based Research Fund in NZ: Taking Stock and Looking Forward," Working Paper Series 21354, Victoria University of Wellington, Chair in Public Finance.
    9. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    10. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Robert A. Buckle & John Creedy, 2019. "An evaluation of metrics used by the Performance-based Research Fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 53(3), pages 270-287, September.
    2. Marian-Gabriel Hâncean & Matjaž Perc & Lazăr Vlăsceanu, 2014. "Fragmented Romanian Sociology: Growth and Structure of the Collaboration Network," PLOS ONE, Public Library of Science, vol. 9(11), pages 1-9, November.
    3. Andrej Kastrin & Jelena Klisara & Borut Lužar & Janez Povh, 2017. "Analysis of Slovenian research community through bibliographic networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 791-813, February.
    4. Tomer Blumkin & Leif Danziger & Eran Yashiv, 2017. "Optimal unemployment benefit policy and the firm productivity distribution," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 24(1), pages 36-59, February.
    5. Tol, Richard S.J., 2013. "The Matthew effect for cohorts of economists," Journal of Informetrics, Elsevier, vol. 7(2), pages 522-527.
    6. Robert A. Buckle & John Creedy & Norman Gemmell, 2020. "Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand," Applied Economics, Taylor & Francis Journals, vol. 52(36), pages 3919-3932, July.
    7. Lina M. Cortés & Andrés Mora-Valencia & Javier Perote, 2016. "The productivity of top researchers: a semi-nonparametric approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 891-915, November.
    8. Robert A. Buckle & John Creedy, 2020. "The ‘disciplinary effect’ of the performance-based research fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 54(2), pages 107-126, May.
    9. Robert A. Buckle and John Creedy, 2018. "The Impact on Research Quality of Performance-Based Funding: The Case of New Zealand’s PBRF Scheme," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 24(1), pages 25-48.
    10. Zhihui Zhang & Ying Cheng & Nian Cai Liu, 2015. "Improving the normalization effect of mean-based method from the perspective of optimization: optimization-based linear methods and their performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 587-607, January.
    11. Hang Keun Ryu & Daniel J. Slottje, 2020. "Does Political Dominance Impact Economic Inequality?," Advances in Decision Sciences, Asia University, Taiwan, vol. 24(1), pages 121-149, March.
    12. Vîiu, Gabriel-Alexandru, 2018. "The lognormal distribution explains the remarkable pattern documented by characteristic scores and scales in scientometrics," Journal of Informetrics, Elsevier, vol. 12(2), pages 401-415.
    13. Rojko, Katarina & Lužar, Borut, 2022. "Scientific performance across research disciplines: Trends and differences in the case of Slovenia," Journal of Informetrics, Elsevier, vol. 16(2).
    14. Sangwal, Keshra, 2013. "Comparison of different mathematical functions for the analysis of citation distribution of papers of individual authors," Journal of Informetrics, Elsevier, vol. 7(1), pages 36-49.
    15. Perc, Matjaž, 2010. "Growth and structure of Slovenia’s scientific collaboration network," Journal of Informetrics, Elsevier, vol. 4(4), pages 475-482.
    16. Hang Keun Ryu & Daniel J. Slottje, 2020. "Does Political Dominance Impact Economic Inequality?," International Association of Decision Sciences, Asia University, Taiwan, vol. 24(1), pages 121-149, March.
    17. Yong Huang & Yi Bu & Ying Ding & Wei Lu, 2018. "Number versus structure: towards citing cascades," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(3), pages 2177-2193, December.
    18. Clémence Berson & Raphaël Lardeux & Claire Lelarge, 2021. "The Cognitive Load of Financing Constraints: Evidence from Large-Scale Wage Surveys," Working papers 836, Banque de France.
    19. Tao Zhou & Matúš Medo & Giulio Cimini & Zi-Ke Zhang & Yi-Cheng Zhang, 2011. "Emergence of Scale-Free Leadership Structure in Social Recommender Systems," PLOS ONE, Public Library of Science, vol. 6(7), pages 1-6, July.
    20. Zheng Xie & Zonglin Xie & Miao Li & Jianping Li & Dongyun Yi, 2017. "Modeling the coevolution between citations and coauthorship of scientific papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 483-507, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:vuw:vuwcpf:20928. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Library Technology Services (email available below). General contact details of provider: https://edirc.repec.org/data/fcvuwnz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.