IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v88y2023i2d10.1007_s11336-022-09881-7.html
   My bibliography  Save this article

The Dependence of Chance-Corrected Weighted Agreement Coefficients on the Power Parameter of the Weighting Scheme: Analysis and Measurement

Author

Listed:
  • Rutger Oest

    (BI Norwegian Business School)

Abstract

We consider the dependence of a broad class of chance-corrected weighted agreement coefficients on the weighting scheme that penalizes rater disagreements. The considered class encompasses many existing coefficients with any number of raters, and one real-valued power parameter defines the weighting scheme that includes linear, quadratic, identity, and radical weights. We obtain the first-order and second-order derivatives of the coefficients with respect to the power parameter and decompose them into components corresponding to all pairs of different category distances. Each component compares its two distances in terms of the ratio of observed to expected-by-chance frequency. A larger ratio for the smaller distance than the larger distance contributes to a positive relationship between the power parameter and the coefficient value; the opposite contributes to a negative relationship. We provide necessary and sufficient conditions for the coefficient value to increase or decrease and the relationship to intensify or weaken as the power parameter increases. We use the first-order and second-order derivatives for corresponding measurement. Furthermore, we show how these two derivatives allow other researchers to obtain quite accurate estimates of the coefficient value for unreported values of the power parameter, even without access to the original data.

Suggested Citation

  • Rutger Oest, 2023. "The Dependence of Chance-Corrected Weighted Agreement Coefficients on the Power Parameter of the Weighting Scheme: Analysis and Measurement," Psychometrika, Springer;The Psychometric Society, vol. 88(2), pages 554-579, June.
    • Handle: RePEc:spr:psycho:v:88:y:2023:i:2:d:10.1007_s11336-022-09881-7
    DOI: 10.1007/s11336-022-09881-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-022-09881-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11336-022-09881-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Matthijs J. Warrens, 2014. "Power Weighted Versions of Bennett, Alpert, and Goldstein’s," Journal of Mathematics, Hindawi, vol. 2014, pages 1-9, December.
    2. Matthijs Warrens, 2011. "Cohen’s Linearly Weighted Kappa is a Weighted Average of 2×2 Kappas," Psychometrika, Springer;The Psychometric Society, vol. 76(3), pages 471-486, July.
    3. Alexandra Raadt & Matthijs J. Warrens & Roel J. Bosker & Henk A. L. Kiers, 2021. "A Comparison of Reliability Coefficients for Ordinal Rating Scales," Journal of Classification, Springer;The Classification Society, vol. 38(3), pages 519-543, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tarald O. Kvålseth, 2018. "An Alternative Interpretation of the Linearly Weighted Kappa Coefficients for Ordinal Data," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 618-627, September.
    2. Matthijs Warrens, 2014. "Corrected Zegers-ten Berge Coefficients Are Special Cases of Cohen’s Weighted Kappa," Journal of Classification, Springer;The Classification Society, vol. 31(2), pages 179-193, July.
    3. Warrens, Matthijs J., 2013. "The Cicchetti–Allison weighting matrix is positive definite," Computational Statistics & Data Analysis, Elsevier, vol. 59(C), pages 180-182.
    4. Matthijs J. Warrens, 2014. "New Interpretations of Cohen’s Kappa," Journal of Mathematics, Hindawi, vol. 2014, pages 1-9, September.
    5. Matthijs Warrens, 2013. "Cohen’s weighted kappa with additive weights," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 7(1), pages 41-55, March.
    6. Pingke Li, 2016. "A Note on the Linearly and Quadratically Weighted Kappa Coefficients," Psychometrika, Springer;The Psychometric Society, vol. 81(3), pages 795-801, September.
    7. Matthijs Warrens, 2012. "Some Paradoxical Results for the Quadratically Weighted Kappa," Psychometrika, Springer;The Psychometric Society, vol. 77(2), pages 315-323, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:88:y:2023:i:2:d:10.1007_s11336-022-09881-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.