IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v53y2008i2p554-564.html
   My bibliography  Save this article

Comparison of ICC and CCC for assessing agreement for data without and with replications

Author

Listed:
  • Chen, Chia-Cheng
  • Barnhart, Huiman X.

Abstract

The intraclass correlation coefficient (ICC) has been traditionally used for assessing reliability between multiple observers for data with or without replications. Definitions of different versions of ICCs depend on the assumptions of specific ANOVA models. The parameter estimator for the ICC is usually based on the method of moments with the underlying assumed ANOVA model. This estimator is consistent only if the ANOVA model assumptions hold. Often these ANOVA assumptions are not met in practice and researchers may compute these estimates without verifying the assumptions. ICC is biased if the ANOVA assumptions are not met. We compute the expected value of the ICC estimator under a very general model to get a sense of the population parameter that the ICC estimator provides. We compare this expected value to another popular agreement index, concordance correlation coefficient (CCC), which is defined without ANOVA assumptions. The main findings are reported for data without replication and with replications for three types of ICCs defined by one-way ANOVA model, two-way ANOVA model without interaction and two-way ANOVA model with interaction. A blood pressure example is used for illustration. If the ICC is the choice of agreement index, we recommend to use over other ICCs as its estimate is similar to the estimate of CCC regardless whether the ANOVA assumptions are met or not.

Suggested Citation

  • Chen, Chia-Cheng & Barnhart, Huiman X., 2008. "Comparison of ICC and CCC for assessing agreement for data without and with replications," Computational Statistics & Data Analysis, Elsevier, vol. 53(2), pages 554-564, December.
    • Handle: RePEc:eee:csdana:v:53:y:2008:i:2:p:554-564
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167-9473(08)00457-X
    Download Restriction: Full text for ScienceDirect subscribers only.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Lin L. & Hedayat A. S. & Sinha B. & Yang M., 2002. "Statistical Methods in Assessing Agreement: Models, Issues, and Tools," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 257-270, March.
    2. Huiman X. Barnhart & Michael Haber & Jingli Song, 2002. "Overall Concordance Correlation Coefficient for Evaluating Agreement Among Multiple Observers," Biometrics, The International Biometric Society, vol. 58(4), pages 1020-1027, December.
    3. Tony Vangeneugden & Annouschka Laenen & Helena Geys & Didier Renard & Geert Molenberghs, 2005. "Applying Concepts of Generalizability Theory on Clinical Trial Data to Investigate Sources of Variation and Their Impact on Reliability," Biometrics, The International Biometric Society, vol. 61(1), pages 295-304, March.
    4. Josep L. Carrasco & Lluís Jover, 2003. "Estimating the Generalized Concordance Correlation Coefficient through Variance Components," Biometrics, The International Biometric Society, vol. 59(4), pages 849-858, December.
    5. Huiman X. Barnhart & John M. Williamson, 2001. "Modeling Concordance Correlation via GEE to Evaluate Reproducibility," Biometrics, The International Biometric Society, vol. 57(3), pages 931-940, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Matheus Pereira Libório & Lívia Maria Leite Silva & Petr Iakovlevitch Ekel & Letícia Ribeiro Figueiredo & Patrícia Bernardes, 2022. "Consensus-Based Sub-Indicator Weighting Approach: Constructing Composite Indicators Compatible with Expert Opinion," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 164(3), pages 1073-1099, December.
    2. Chen, Chia-Cheng & Barnhart, Huiman X., 2013. "Assessing agreement with intraclass correlation coefficient and concordance correlation coefficient for data with repeated measures," Computational Statistics & Data Analysis, Elsevier, vol. 60(C), pages 132-145.
    3. Candelaria de la Merced Díaz‐González & Milagros de la Rosa‐Hormiga & Josefa M. Ramal‐López & Juan José González‐Henríquez & María Sandra Marrero‐Morales, 2018. "Factors which influence concordance among measurements obtained by different pulse oximeters currently used in some clinical situations," Journal of Clinical Nursing, John Wiley & Sons, vol. 27(3-4), pages 677-683, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Helenowski Irene B & Vonesh Edward F & Demirtas Hakan & Rademaker Alfred W & Ananthanarayanan Vijayalakshmi & Gann Peter H & Jovanovic Borko D, 2011. "Defining Reproducibility Statistics as a Function of the Spatial Covariance Structures in Biomarker Studies," The International Journal of Biostatistics, De Gruyter, vol. 7(1), pages 1-21, January.
    2. Tsai, Miao-Yu, 2015. "Comparison of concordance correlation coefficient via variance components, generalized estimating equations and weighted approaches with model selection," Computational Statistics & Data Analysis, Elsevier, vol. 82(C), pages 47-58.
    3. Tsai, Miao-Yu & Lin, Chao-Chun, 2018. "Concordance correlation coefficients estimated by variance components for longitudinal normal and Poisson data," Computational Statistics & Data Analysis, Elsevier, vol. 121(C), pages 57-70.
    4. Li, Runze & Chow, Mosuk, 2005. "Evaluation of reproducibility for paired functional data," Journal of Multivariate Analysis, Elsevier, vol. 93(1), pages 81-101, March.
    5. Gao, Jingjing & Pan, Yi & Haber, Michael, 2012. "Assessment of observer agreement for matched repeated binary measurements," Computational Statistics & Data Analysis, Elsevier, vol. 56(5), pages 1052-1060.
    6. Ying Guo & Amita K. Manatunga, 2007. "Nonparametric Estimation of the Concordance Correlation Coefficient under Univariate Censoring," Biometrics, The International Biometric Society, vol. 63(1), pages 164-172, March.
    7. Josep L. Carrasco, 2010. "A Generalized Concordance Correlation Coefficient Based on the Variance Components Generalized Linear Mixed Models for Overdispersed Count Data," Biometrics, The International Biometric Society, vol. 66(3), pages 897-904, September.
    8. Thiago de Paula Oliveira & John Hinde & Silvio Sandoval Zocchi, 2018. "Longitudinal Concordance Correlation Function Based on Variance Components: An Application in Fruit Color Analysis," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 23(2), pages 233-254, June.
    9. Liao Jason J. Z. & Capen Robert, 2011. "An Improved Bland-Altman Method for Concordance Assessment," The International Journal of Biostatistics, De Gruyter, vol. 7(1), pages 1-17, January.
    10. Hutson, Alan D., 2010. "A multi-rater nonparametric test of agreement and corresponding agreement plot," Computational Statistics & Data Analysis, Elsevier, vol. 54(1), pages 109-119, January.
    11. Balakrishnan, Narayanaswamy & Ristić, Miroslav M., 2016. "Multivariate families of gamma-generated distributions with finite or infinite support above or below the diagonal," Journal of Multivariate Analysis, Elsevier, vol. 143(C), pages 194-207.
    12. Masha Kocherginsky & Megan Huisingh-Scheetz & William Dale & Diane S Lauderdale & Linda Waite, 2017. "Measuring Physical Activity with Hip Accelerometry among U.S. Older Adults: How Many Days Are Enough?," PLOS ONE, Public Library of Science, vol. 12(1), pages 1-13, January.
    13. Correndo, Adrian A. & Hefley, Trevor J. & Holzworth, Dean P. & Ciampitti, Ignacio A., 2021. "Revisiting linear regression to test agreement in continuous predicted-observed datasets," Agricultural Systems, Elsevier, vol. 192(C).
    14. Jason Wittenberg, 2013. "How similar are they? rethinking electoral congruence," Quality & Quantity: International Journal of Methodology, Springer, vol. 47(3), pages 1687-1701, April.
    15. Geòrgia Escaramís & Josep L. Carrasco & Carlos Ascaso, 2008. "Detection of Significant Disease Risks Using a Spatial Conditional Autoregressive Model," Biometrics, The International Biometric Society, vol. 64(4), pages 1043-1053, December.
    16. Admassu N. Lamu, 2020. "Does linear equating improve prediction in mapping? Crosswalking MacNew onto EQ-5D-5L value sets," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 21(6), pages 903-915, August.
    17. John J. Chen & Guangxiang Zhang & Chen Ji & George F. Steinhardt, 2011. "Simple moment-based inferences of generalized concordance correlation," Journal of Applied Statistics, Taylor & Francis Journals, vol. 38(9), pages 1867-1882, October.
    18. Choudhary Pankaj K, 2010. "A Unified Approach for Nonparametric Evaluation of Agreement in Method Comparison Studies," The International Journal of Biostatistics, De Gruyter, vol. 6(1), pages 1-26, June.
    19. Erkan Cer, 2019. "The Instruction of Writing Strategies: The Effect of the Metacognitive Strategy on the Writing Skills of Pupils in Secondary Education," SAGE Open, , vol. 9(2), pages 21582440198, April.
    20. Candelaria de la Merced Díaz‐González & Milagros de la Rosa‐Hormiga & Josefa M. Ramal‐López & Juan José González‐Henríquez & María Sandra Marrero‐Morales, 2018. "Factors which influence concordance among measurements obtained by different pulse oximeters currently used in some clinical situations," Journal of Clinical Nursing, John Wiley & Sons, vol. 27(3-4), pages 677-683, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:53:y:2008:i:2:p:554-564. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.