IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/c9q3b_v1.html
   My bibliography  Save this paper

Comparative judgement as a research tool: a meta-analysis of application and reliability

Author

Listed:
  • Kinnear, George

    (University of Edinburgh)

  • Jones, Ian
  • Davies, Ben

Abstract

Comparative judgement (CJ) provides methods for constructing measurement scales, by asking assessors to make a series of pairwise comparisons of the artefacts or representations to be scored. Researchers using CJ need to decide how many assessors to recruit and how many comparisons to collect. They also need to gauge the reliability of the resulting measurement scale, with two different estimates in widespread use: Scale Separation Reliability (SSR) and Split-Halves Reliability (SHR). Previous research has offered guidance on these issues, but with either limited empirical support or focused only on education research. In this paper, we offer guidance based on our analysis of 101 CJ datasets that we collated from previous research across a range of disciplines. We present two novel findings, with substantive implications for future CJ research. First, we find that collecting 10 comparisons for every representation is generally sufficient; a more lenient guideline than previously published. Second, we conclude that SSR can serve as a reliable proxy for inter-rater reliability, but recommend that researchers use a higher threshold of .8, rather than the current standard of .7.

Suggested Citation

  • Kinnear, George & Jones, Ian & Davies, Ben, 2025. "Comparative judgement as a research tool: a meta-analysis of application and reliability," OSF Preprints c9q3b_v1, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:c9q3b_v1
    DOI: 10.31219/osf.io/c9q3b_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/67a2325eca5a514b7d0cb18d/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/c9q3b_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:c9q3b_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.