IDEAS home Printed from https://ideas.repec.org/a/spr/jclass/v30y2013i2p251-275.html
   My bibliography  Save this article

Measuring the Reliability of Diagnostic Classification Model Examinee Estimates

Author

Listed:
  • Jonathan Templin
  • Laine Bradshaw

Abstract

Over the past decade, diagnostic classification models (DCMs) have become an active area of psychometric research. Despite their use, the reliability of examinee estimates in DCM applications has seldom been reported. In this paper, a reliability measure for the categorical latent variables of DCMs is defined. Using theory-and simulation-based results, we show how DCMs uniformly provide greater examinee estimate reliability than IRT models for tests of the same length, a result that is a consequence of the smaller range of latent variable values examinee estimates can take in DCMs. We demonstrate this result by comparing DCM and IRT reliability for a series of models estimated with data from an end-of-grade test, culminating with a discussion of how DCMs can be used to change the character of large scale testing, either by shortening tests that measure examinees unidimensionally or by providing more reliable multidimensional measurement for tests of the same length. Copyright Springer Science+Business Media New York 2013

Suggested Citation

  • Jonathan Templin & Laine Bradshaw, 2013. "Measuring the Reliability of Diagnostic Classification Model Examinee Estimates," Journal of Classification, Springer;The Classification Society, vol. 30(2), pages 251-275, July.
    • Handle: RePEc:spr:jclass:v:30:y:2013:i:2:p:251-275
    DOI: 10.1007/s00357-013-9129-4
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s00357-013-9129-4
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s00357-013-9129-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Maydeu-Olivares, Albert & Joe, Harry, 2005. "Limited- and Full-Information Estimation and Goodness-of-Fit Testing in 2n Contingency Tables: A Unified Framework," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 1009-1020, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jonathan Templin & Laine Bradshaw, 2014. "Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 317-339, April.
    2. Matthew S. Johnson & Sandip Sinharay, 2020. "The Reliability of the Posterior Probability of Skill Attainment in Diagnostic Classification Models," Journal of Educational and Behavioral Statistics, , vol. 45(1), pages 5-31, February.
    3. Laine Bradshaw & Jonathan Templin, 2014. "Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions," Psychometrika, Springer;The Psychometric Society, vol. 79(3), pages 403-425, July.
    4. Shiyu Wang & Yan Yang & Steven Andrew Culpepper & Jeffrey A. Douglas, 2018. "Tracking Skill Acquisition With Cognitive Diagnosis Models: A Higher-Order, Hidden Markov Model With Covariates," Journal of Educational and Behavioral Statistics, , vol. 43(1), pages 57-87, February.
    5. Matthew J. Madison & Laine P. Bradshaw, 2018. "Assessing Growth in a Diagnostic Classification Model Framework," Psychometrika, Springer;The Psychometric Society, vol. 83(4), pages 963-990, December.
    6. Muhamad Firdaus Mohd Noh & Mohd Effendi Ewan bin Mohd Matore, 2024. "Conundrum and Considerations in Cognitive Diagnostic Assessment for Language Proficiency Evaluation," Information Management and Business Review, AMH International, vol. 16(2), pages 63-72.
    7. Kazuhiro Yamaguchi & Kensuke Okada, 2020. "Variational Bayes Inference Algorithm for the Saturated Diagnostic Classification Model," Psychometrika, Springer;The Psychometric Society, vol. 85(4), pages 973-995, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Laine Bradshaw & Jonathan Templin, 2014. "Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions," Psychometrika, Springer;The Psychometric Society, vol. 79(3), pages 403-425, July.
    2. Ick Hoon Jin & Minjeong Jeon, 2019. "A Doubly Latent Space Joint Model for Local Item and Person Dependence in the Analysis of Item Response Data," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 236-260, March.
    3. Li Cai, 2010. "A Two-Tier Full-Information Item Factor Analysis Model with Applications," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 581-612, December.
    4. Kim, Sung-Ho & Choi, Hyemi & Lee, Sangjin, 2009. "Estimate-based goodness-of-fit test for large sparse multinomial distributions," Computational Statistics & Data Analysis, Elsevier, vol. 53(4), pages 1122-1131, February.
    5. Wu, Jianmin & Bentler, Peter M., 2013. "Limited information estimation in binary factor analysis: A review and extension," Computational Statistics & Data Analysis, Elsevier, vol. 57(1), pages 392-403.
    6. Qian, Zhiguang & Shapiro, Alexander, 2006. "Simulation-based approach to estimation of latent variable models," Computational Statistics & Data Analysis, Elsevier, vol. 51(2), pages 1243-1259, November.
    7. Jonathan Templin & Laine Bradshaw, 2014. "Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 317-339, April.
    8. Tang, Min & Slud, Eric V. & Pfeiffer, Ruth M., 2014. "Goodness of fit tests for linear mixed models," Journal of Multivariate Analysis, Elsevier, vol. 130(C), pages 176-193.
    9. Harry Joe & Alberto Maydeu-Olivares, 2010. "A General Family of Limited Information Goodness-of-Fit Statistics for Multinomial Data," Psychometrika, Springer;The Psychometric Society, vol. 75(3), pages 393-419, September.
    10. Heon-Jae Jeong & Hsun-Hsiang Liao & Su Ha Han & Wui-Chiang Lee, 2020. "An Application of Item Response Theory to Scoring Patient Safety Culture Survey Data," IJERPH, MDPI, vol. 17(3), pages 1-10, January.
    11. Nuo Xi & Michael W. Browne, 2014. "Contributions to the Underlying Bivariate Normal Method for Factor Analyzing Ordinal Data," Journal of Educational and Behavioral Statistics, , vol. 39(6), pages 583-611, December.
    12. Mark Reiser & Silvia Cagnone & Junfei Zhu, 2023. "An Extended GFfit Statistic Defined on Orthogonal Components of Pearson’s Chi-Square," Psychometrika, Springer;The Psychometric Society, vol. 88(1), pages 208-240, March.
    13. Quentin F. Gronau & Eric-Jan Wagenmakers & Daniel W. Heck & Dora Matzke, 2019. "A Simple Method for Comparing Complex Models: Bayesian Model Comparison for Hierarchical Multinomial Processing Tree Models Using Warp-III Bridge Sampling," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 261-284, March.
    14. Silvia cagnone & Stefania Mignani, 2007. "Assessing the goodness of fit of a latent variable model for ordinal data," Metron - International Journal of Statistics, Dipartimento di Statistica, Probabilità e Statistiche Applicate - University of Rome, vol. 0(3), pages 337-361.
    15. Yang Liu & Ji Seung Yang & Alberto Maydeu-Olivares, 2019. "Restricted Recalibration of Item Response Theory Models," Psychometrika, Springer;The Psychometric Society, vol. 84(2), pages 529-553, June.
    16. Shing-On Leung, 2008. "A Three-Dimensional Latent Variable Model for Attitude Scales," Sociological Methods & Research, , vol. 37(1), pages 135-154, August.
    17. Yunxiao Chen & Irini Moustaki & Haoran Zhang, 2020. "A Note on Likelihood Ratio Tests for Models with Latent Variables," Psychometrika, Springer;The Psychometric Society, vol. 85(4), pages 996-1012, December.
    18. Alberto Maydeu-Olivares & Rosa Montaño, 2013. "How Should We Assess the Fit of Rasch-Type Models? Approximating the Power of Goodness-of-Fit Statistics in Categorical Data Analysis," Psychometrika, Springer;The Psychometric Society, vol. 78(1), pages 116-133, January.
    19. Vassilis Vasdekis & Silvia Cagnone & Irini Moustaki, 2012. "A Composite Likelihood Inference in Latent Variable Models for Ordinal Longitudinal Responses," Psychometrika, Springer;The Psychometric Society, vol. 77(3), pages 425-441, July.
    20. Chen, Yunxiao & Moustaki, Irini & Zhang, H, 2020. "A note on likelihood ratio tests for models with latent variables," LSE Research Online Documents on Economics 107490, London School of Economics and Political Science, LSE Library.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jclass:v:30:y:2013:i:2:p:251-275. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.