IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/r487u_v1.html
   My bibliography  Save this paper

Comparative judgement without the fancy statistics

Author

Listed:
  • Jones, Ian

Abstract

Comparative judgement methods for assessment are increasingly popular. They involve assessors making comparisons about the ‘quality’ of pairs of students’ work, and the comparisons are statistically modelled to produce scores. Recently Benton and Gallacher (2018, p.25) claimed that “much of the apparent advantage of [comparative judgement] can be explained by its use of fancy statistics”. They evidenced this by applying ‘fancy statistics’ to raw scores from multiple marked essays, and comparing the predictive value of raw scores with fancy statistics outcomes. Here I take the inverse approach and compare raw scores from comparative judgement assessments with fancy statistics outcomes. I reanalysed studies from peer-reviewed outlets where the prominent measure was based on comparative judgement. I report that raw scores reduced the reliability and validity of outcomes relative to fancy statistics in about one fifth of cases. I consider the implications of the findings for using comparative judgement in educational research.

Suggested Citation

  • Jones, Ian, 2025. "Comparative judgement without the fancy statistics," OSF Preprints r487u_v1, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:r487u_v1
    DOI: 10.31219/osf.io/r487u_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/67bd9d577210c9a6fe2e984d/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/r487u_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Firth, David, 2005. "Bradley-Terry Models in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 12(i01).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. McHale, Ian & Morton, Alex, 2011. "A Bradley-Terry type model for forecasting tennis match results," International Journal of Forecasting, Elsevier, vol. 27(2), pages 619-630, April.
    2. Anna Gottard & Giorgio Calzolari, 2014. "Alternative estimating procedures for multiple membership logit models with mixed effects: indirect inference and data cloning," Econometrics Working Papers Archive 2014_07, Universita' degli Studi di Firenze, Dipartimento di Statistica, Informatica, Applicazioni "G. Parenti".
    3. Vicente Rodríguez Montequín & Joaquín Manuel Villanueva Balsera & Marina Díaz Piloñeta & César Álvarez Pérez, 2020. "A Bradley-Terry Model-Based Approach to Prioritize the Balance Scorecard Driving Factors: The Case Study of a Financial Software Factory," Mathematics, MDPI, vol. 8(2), pages 1-15, February.
    4. Wickelmaier, Florian & Strobl, Carolin & Zeileis, Achim, 2012. "Psychoco: Psychometric Computing in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i01).
    5. David G. Hamilton & Martin J. Whiting & Sarah R. Pryke, 2013. "Fiery frills: carotenoid-based coloration predicts contest success in frillneck lizards," Behavioral Ecology, International Society for Behavioral Ecology, vol. 24(5), pages 1138-1149.
    6. Turner, Heather & Firth, David, 2012. "Bradley-Terry Models in R: The BradleyTerry2 Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i09).
    7. repec:jss:jstsof:32:i10 is not listed on IDEAS
    8. McHale, Ian & Morton, Alex, 2011. "A Bradley-Terry type model for forecasting tennis match results," International Journal of Forecasting, Elsevier, vol. 27(2), pages 619-630.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:r487u_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.