IDEAS home Printed from https://ideas.repec.org/a/spr/sankhb/v78y2016i2d10.1007_s13571-016-0124-8.html
   My bibliography  Save this article

Statistical Auditing of Non-transparent Expert Assessments

Author

Listed:
  • Michael Cain

    (Vietnam National University—HCMC)

  • Stuart McLeay

    (University of Wales)

Abstract

A statistical model is developed for an audit of the assessments of a panel of experts when little information is made available beyond a final announcement of the individual assessed ratings given. The application is to the process for the research assessment exercise for UK universities. Based on the proportions of the publications a panel deems to be International standard, National standard or Unclassified, a department’s research output is rated by the panel on a seven point scale. The expert panel’s remit is carefully interpreted and the given ratings are modelled via an underlying trinomial random variable with a bivariate Normal approximation. A likelihood function is developed and maximised in order to obtain fitted ratings for all units of assessment. The model’s fitted values for the given ratings explain outcomes remarkably well and there are few mis-classifications; but there are some surprising outliers that do still require some explanation. The procedure illustrates well how Statisticians, surprisingly, might be able to model and audit for consistency the work of experts even if little or no information is provided, beyond vague prior published guidelines for the assessments and the final ratings given.

Suggested Citation

  • Michael Cain & Stuart McLeay, 2016. "Statistical Auditing of Non-transparent Expert Assessments," Sankhya B: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 78(2), pages 362-385, November.
    • Handle: RePEc:spr:sankhb:v:78:y:2016:i:2:d:10.1007_s13571-016-0124-8
    DOI: 10.1007/s13571-016-0124-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s13571-016-0124-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s13571-016-0124-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Jill Johnes & Jim Taylor & Brian Francis, 1993. "The Research Performance of UK Universities: A Statistical Analysis of the Results of the 1989 Research Selectivity Exercise," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 156(2), pages 271-286, March.
    2. Doyle, J. R. & Arthurs, A. J. & Green, R. H. & McAulay, L. & Pitt, M. R. & Bottomley, P. A. & Evans, W., 1996. "The judge, the model of the judge, and the model of the judged as judge: Analyses of the UK 1992 research assessment exercise data for business and management studies," Omega, Elsevier, vol. 24(1), pages 13-28, February.
    3. Claire Marston & Aishah Ayub, 2000. "Relationship between publications in selected journals and Research Assessment Exercise rankings in 1996 for UK accountancy departments," Accounting Education, Taylor & Francis Journals, vol. 9(1), pages 93-102.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Johnes, Jill, 1996. "Performance assessment in higher education in Britain," European Journal of Operational Research, Elsevier, vol. 89(1), pages 18-33, February.
    2. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    3. Doyle, J. R. & Arthurs, A. J. & Mcaulay, L. & Osborne, P. G., 1996. "Citation as effortful voting: A reply to ones, Brinn and Pendlebury," Omega, Elsevier, vol. 24(5), pages 603-606, October.
    4. Doyle, John R & Arthurs, Alan J, 1998. "Grade inflation in the UK's 1996 research assessment exercise?," Omega, Elsevier, vol. 26(4), pages 461-465, August.
    5. Mitchell, George, 1996. "Judging research quality and journals: A call for debate," Omega, Elsevier, vol. 24(5), pages 613-613, October.
    6. Massimiliano Bratti & Abigail McKnight & Robin Naylor & Jeremy Smith, 2004. "Higher education outcomes, graduate employment and university performance indicators," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 167(3), pages 475-496, August.
    7. Fogarty, Timothy J. & Zimmerman, Aleksandra B. & Richardson, Vernon J., 2016. "What do we mean by accounting program quality? A decomposition of accounting faculty opinions," Journal of Accounting Education, Elsevier, vol. 36(C), pages 16-42.
    8. Gianni De Fraja & Giovanni Facchini & John Gathergood, 2016. "How Much Is That Star in the Window? Professorial Salaries and Research Performance in UK Universities," Discussion Papers 2016-13, University of Nottingham, GEP.
    9. Battistin, Erich & Ovidi, Marco, 2017. "Rising Stars," IZA Discussion Papers 11198, Institute of Labor Economics (IZA).
    10. Fabian Waldinger, 2016. "Bombs, Brains, and Science: The Role of Human and Physical Capital for the Creation of Scientific Knowledge," The Review of Economics and Statistics, MIT Press, vol. 98(5), pages 811-831, December.
    11. Thanassoulis, E. & Sotiros, D. & Koronakos, G. & Despotis, D., 2018. "Assessing the cost-effectiveness of university academic recruitment and promotion policies," European Journal of Operational Research, Elsevier, vol. 264(2), pages 742-755.
    12. Doyle, J. R. & Arthurs, A. J. & Green, R. H. & McAulay, L. & Pitt, M. R. & Bottomley, P. A. & Evans, W., 1996. "The judge, the model of the judge, and the model of the judged as judge: Analyses of the UK 1992 research assessment exercise data for business and management studies," Omega, Elsevier, vol. 24(1), pages 13-28, February.
    13. Catriona Paisey & Nicholas Paisey, 2005. "The research assessment exercise 2001—insights and implications for accounting education research in the UK," Accounting Education, Taylor & Francis Journals, vol. 14(4), pages 411-426.
    14. Colin Glass, J. & McCallion, Gillian & McKillop, Donal G. & Rasaratnam, Syamarlah & Stringer, Karl S., 2006. "Implications of variant efficiency measures for policy evaluations in UK higher education," Socio-Economic Planning Sciences, Elsevier, vol. 40(2), pages 119-142, June.
    15. Rahma Chekkar & Carole Grillet, 2005. "Research on Accounting and Auditing: French researchers' practices of diffusing knowledge," Working Papers 2005-6, Laboratoire Orléanais de Gestion - université d'Orléans.
    16. V. A. Traag & L. Waltman, 2019. "Systematic analysis of agreement between metrics and peer review in the UK REF," Palgrave Communications, Palgrave Macmillan, vol. 5(1), pages 1-12, December.
    17. Michael John Jones & Roydon Roberts, 2005. "International Publishing Patterns: An Investigation of Leading UK and US Accounting and Finance Journals," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 32(5-6), pages 1107-1140.
    18. Johnes, Geraint & Johnes, Jill & Virmani, Swati, 2022. "Performance and efficiency in Indian universities," Socio-Economic Planning Sciences, Elsevier, vol. 81(C).
    19. Johnes, Jill & Johnes, Geraint, 1995. "Research funding and performance in U.K. University Departments of Economics: A frontier analysis," Economics of Education Review, Elsevier, vol. 14(3), pages 301-314, September.
    20. Donohue, Joan M. & Fox, Jeremy B., 2000. "A multi-method evaluation of journals in the decision and management sciences by US academics," Omega, Elsevier, vol. 28(1), pages 17-36, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:sankhb:v:78:y:2016:i:2:d:10.1007_s13571-016-0124-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.