IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v49y2024i3p311-341.html
   My bibliography  Save this article

Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching

Author

Listed:
  • Sijia Huang

    (Indiana University Bloomington)

  • Li Cai

    (University of California, Los Angeles)

Abstract

The cross-classified data structure is ubiquitous in education, psychology, and health outcome sciences. In these areas, assessment instruments that are made up of multiple items are frequently used to measure latent constructs. The presence of both the cross-classified structure and multivariate categorical outcomes leads to the so-called item-level data with cross-classified structure. An example of such data structure is the routinely collected student evaluation of teaching (SET) data. Motivated by the lack of research on multilevel IRT modeling with crossed random effects and the need of an approach that can properly handle SET data, this study proposed a cross-classified IRT model, which takes into account both the cross-classified data structure and properties of multiple items in an assessment instrument. A new variant of the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm was introduced to address the computational complexities in estimating the proposed model. A preliminary simulation study was conducted to evaluate the performance of the algorithm for fitting the proposed model to data. The results indicated that model parameters were well recovered. The proposed model was also applied to SET data collected at a large public university to answer empirical research questions. Limitations and future research directions were discussed.

Suggested Citation

  • Sijia Huang & Li Cai, 2024. "Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching," Journal of Educational and Behavioral Statistics, , vol. 49(3), pages 311-341, June.
    • Handle: RePEc:sae:jedbes:v:49:y:2024:i:3:p:311-341
    DOI: 10.3102/10769986231193351
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986231193351
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986231193351?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jon Rasbash & George Leckie & Rebecca Pillinger & Jennifer Jenkins, 2010. "Children's educational progress: partitioning family, school and area effects," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(3), pages 657-682, July.
    2. D. Clayton & J. Rasbash, 1999. "Estimation in large cross random‐effect models by data augmentation," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 162(3), pages 425-436.
    3. George Leckie, 2009. "The complexity of school and neighbourhood effects and movements of pupils on school differences in models of educational achievement," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(3), pages 537-554, June.
    4. Cho, S.-J. & Rabe-Hesketh, S., 2011. "Alternating imputation posterior estimation of models with crossed random effects," Computational Statistics & Data Analysis, Elsevier, vol. 55(1), pages 12-25, January.
    5. R. Darrell Bock, 1972. "Estimating item parameters and latent ability when responses are scored in two or more nominal categories," Psychometrika, Springer;The Psychometric Society, vol. 37(1), pages 29-51, March.
    6. Jean-Paul Fox & Cees Glas, 2001. "Bayesian estimation of a multilevel IRT model using gibbs sampling," Psychometrika, Springer;The Psychometric Society, vol. 66(2), pages 271-288, June.
    7. Li Cai, 2010. "High-dimensional Exploratory Item Factor Analysis by A Metropolis–Hastings Robbins–Monro Algorithm," Psychometrika, Springer;The Psychometric Society, vol. 75(1), pages 33-57, March.
    8. Seungwon Chung & Li Cai, 2021. "Cross-Classified Random Effects Modeling for Moderated Item Calibration," Journal of Educational and Behavioral Statistics, , vol. 46(6), pages 651-681, December.
    9. Li Cai, 2010. "Metropolis-Hastings Robbins-Monro Algorithm for Confirmatory Item Factor Analysis," Journal of Educational and Behavioral Statistics, , vol. 35(3), pages 307-335, June.
    10. Paul Boeck, 2008. "Random Item IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 73(4), pages 533-559, December.
    11. Sophia Rabe-Hesketh & Anders Skrondal & Andrew Pickles, 2004. "Generalized multilevel structural equation modeling," Psychometrika, Springer;The Psychometric Society, vol. 69(2), pages 167-190, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sun-Joo Cho & Allan S. Cohen, 2010. "A Multilevel Mixture IRT Model With an Application to DIF," Journal of Educational and Behavioral Statistics, , vol. 35(3), pages 336-370, June.
    2. Hanneke Geerlings & Cees Glas & Wim Linden, 2011. "Modeling Rule-Based Item Generation," Psychometrika, Springer;The Psychometric Society, vol. 76(2), pages 337-359, April.
    3. Sun-Joo Cho & Paul Boeck & Susan Embretson & Sophia Rabe-Hesketh, 2014. "Additive Multilevel Item Structure Models with Random Residuals: Item Modeling for Explanation and Item Generation," Psychometrika, Springer;The Psychometric Society, vol. 79(1), pages 84-104, January.
    4. Minjeong Jeon & Sophia Rabe-Hesketh, 2012. "Profile-Likelihood Approach for Estimating Generalized Linear Mixed Models With Factor Structures," Journal of Educational and Behavioral Statistics, , vol. 37(4), pages 518-542, August.
    5. Anders Skrondal & Sophia Rabe‐Hesketh, 2007. "Latent Variable Modelling: A Survey," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 34(4), pages 712-745, December.
    6. Andersson, Björn & Jin, Shaobo & Zhang, Maoxin, 2023. "Fast estimation of multiple group generalized linear latent variable models for categorical observed variables," Computational Statistics & Data Analysis, Elsevier, vol. 182(C).
    7. Ji Seung Yang & Li Cai, 2014. "Estimation of Contextual Effects Through Nonlinear Multilevel Latent Variable Modeling With a Metropolis–Hastings Robbins–Monro Algorithm," Journal of Educational and Behavioral Statistics, , vol. 39(6), pages 550-582, December.
    8. Sun-Joo Cho & Sarah Brown-Schmidt & Woo-yeol Lee, 2018. "Autoregressive Generalized Linear Mixed Effect Models with Crossed Random Effects: An Application to Intensive Binary Time Series Eye-Tracking Data," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 751-771, September.
    9. Mariagiulia Matteucci & Bernard Veldkamp, 2015. "The approach of power priors for ability estimation in IRT models," Quality & Quantity: International Journal of Methodology, Springer, vol. 49(3), pages 917-926, May.
    10. Björn Andersson & Tao Xin, 2021. "Estimation of Latent Regression Item Response Theory Models Using a Second-Order Laplace Approximation," Journal of Educational and Behavioral Statistics, , vol. 46(2), pages 244-265, April.
    11. Yoav Bergner & Peter Halpin & Jill-Jênn Vie, 2022. "Multidimensional Item Response Theory in the Style of Collaborative Filtering," Psychometrika, Springer;The Psychometric Society, vol. 87(1), pages 266-288, March.
    12. Gregory Camilli & Jean-Paul Fox, 2015. "An Aggregate IRT Procedure for Exploratory Factor Analysis," Journal of Educational and Behavioral Statistics, , vol. 40(4), pages 377-401, August.
    13. Peida Zhan & Hong Jiao & Dandan Liao & Feiming Li, 2019. "A Longitudinal Higher-Order Diagnostic Classification Model," Journal of Educational and Behavioral Statistics, , vol. 44(3), pages 251-281, June.
    14. Alexander Robitzsch, 2020. "L p Loss Functions in Invariance Alignment and Haberman Linking with Few or Many Groups," Stats, MDPI, vol. 3(3), pages 1-38, August.
    15. Yang Liu & Jan Hannig, 2017. "Generalized Fiducial Inference for Logistic Graded Response Models," Psychometrika, Springer;The Psychometric Society, vol. 82(4), pages 1097-1125, December.
    16. Isabella Sulis & Mariano Porcu, 2015. "Assessing Divergences in Mathematics and Reading Achievement in Italian Primary Schools: A Proposal of Adjusted Indicators of School Effectiveness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 122(2), pages 607-634, June.
    17. Mike Smet & Barbara Janssens, 2014. "Determinants of the choice for professional teacher education programs: A multinomial multilevel approach," Investigaciones de Economía de la Educación volume 9, in: Adela García Aracil & Isabel Neira Gómez (ed.), Investigaciones de Economía de la Educación 9, edition 1, volume 9, chapter 41, pages 797-815, Asociación de Economía de la Educación.
    18. Juan Merlo & Philippe Wagner & Nermin Ghith & George Leckie, 2016. "An Original Stepwise Multilevel Logistic Regression Analysis of Discriminatory Accuracy: The Case of Neighbourhoods and Health," PLOS ONE, Public Library of Science, vol. 11(4), pages 1-31, April.
    19. Dylan Molenaar, 2015. "Heteroscedastic Latent Trait Models for Dichotomous Data," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 625-644, September.
    20. Yang Liu & Weimeng Wang, 2022. "Semiparametric Factor Analysis for Item-Level Response Time Data," Psychometrika, Springer;The Psychometric Society, vol. 87(2), pages 666-692, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:49:y:2024:i:3:p:311-341. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.