IDEAS home Printed from https://ideas.repec.org/a/taf/japsta/v35y2008i4p445-464.html
   My bibliography  Save this article

Behavior of agreement measures in the presence of zero cells and biased marginal distributions

Author

Listed:
  • Viswanathan Shankar
  • Shrikant Bangdiwala

Abstract

Kappa and B assess agreement between two observers independently classifying N units into k categories. We study their behavior under zero cells in the contingency table and unbalanced asymmetric marginal distributions. Zero cells arise when a cross-classification is never endorsed by both observers; biased marginal distributions occur when some categories are preferred differently between the observers. Simulations studied the distributions of the unweighted and weighted statistics for k=4, under fixed proportions of diagonal agreement and different patterns off-diagonal, with various sample sizes, and under various zero cell count scenarios. Marginal distributions were first uniform and homogeneous, and then unbalanced asymmetric distributions. Results for unweighted kappa and B statistics were comparable to work of Munoz and Bangdiwala, even with zero cells. A slight increased variation was observed as the sample size decreased. Weighted statistics did show greater variation as the number of zero cells increased, with weighted kappa increasing substantially more than weighted B. Under biased marginal distributions, weighted kappa with Cicchetti weights were higher than with squared weights. Both statistics for observer agreement behaved well under zero cells. The weighted B was less variable than the weighted kappa under similar circumstances and different weights. In general, B's performance and graphical interpretation make it preferable to kappa under the studied scenarios.

Suggested Citation

  • Viswanathan Shankar & Shrikant Bangdiwala, 2008. "Behavior of agreement measures in the presence of zero cells and biased marginal distributions," Journal of Applied Statistics, Taylor & Francis Journals, vol. 35(4), pages 445-464.
  • Handle: RePEc:taf:japsta:v:35:y:2008:i:4:p:445-464
    DOI: 10.1080/02664760701835052
    as

    Download full text from publisher

    File URL: http://www.tandfonline.com/doi/abs/10.1080/02664760701835052
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1080/02664760701835052?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Sergio Munoz & Shrikant Bangdiwala, 1997. "Interpretation of Kappa and B statistics measures of agreement," Journal of Applied Statistics, Taylor & Francis Journals, vol. 24(1), pages 105-112.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Laura Sampietro-Colom & Carla Fernandez-Barcelo & Ismail Abbas & Blanca Valdasquin & Nicolau Rabasseda & Borja García-Lorenzo & Miquel Sanchez & Mireia Sans & Noemi Garcia & Alicia Granados, 2022. "WtsWrng Interim Comparative Effectiveness Evaluation and Description of the Challenges to Develop, Assess, and Introduce This Novel Digital Application in a Traditional Health System," IJERPH, MDPI, vol. 19(21), pages 1-19, October.
    2. Matthijs J. Warrens & Alexandra Raadt, 2019. "Properties of Bangdiwala’s B," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 13(2), pages 481-493, June.
    3. Gökçe, Osman Zeki & Hatipoglu, Emre & Belaïd, Fateh, 2024. "Navigating energy diplomacy in times of recovery and conflict: A study of cross-border energy trade dynamics," Resources Policy, Elsevier, vol. 90(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexandra Raadt & Matthijs J. Warrens & Roel J. Bosker & Henk A. L. Kiers, 2021. "A Comparison of Reliability Coefficients for Ordinal Rating Scales," Journal of Classification, Springer;The Classification Society, vol. 38(3), pages 519-543, October.
    2. Matthijs J. Warrens & Alexandra Raadt, 2019. "Properties of Bangdiwala’s B," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 13(2), pages 481-493, June.
    3. Stephanie K. Sprogis & Judy Currey & Julie Considine, 2019. "Patient acceptability of wearable vital sign monitoring technologies in the acute care setting: A systematic review," Journal of Clinical Nursing, John Wiley & Sons, vol. 28(15-16), pages 2732-2744, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:japsta:v:35:y:2008:i:4:p:445-464. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Longhurst (email available below). General contact details of provider: http://www.tandfonline.com/CJAS20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.