IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v46y2021i6p651-681.html
   My bibliography  Save this article

Cross-Classified Random Effects Modeling for Moderated Item Calibration

Author

Listed:
  • Seungwon Chung

    (University of Minnesota)

  • Li Cai

    (University of California, Los Angeles)

Abstract

In the research reported here, we propose a new method for scale alignment and test scoring in the context of supporting students with disabilities. In educational assessment, students from these special populations take modified tests because of a demonstrated disability that requires more assistance than standard testing accommodation. Updated federal education legislation and guidance require that these students be assessed and included in state education accountability systems, and their achievement reported with respect to the same rigorous content and achievement standards that the state adopted. Routine item calibration and linking methods are not feasible because the size of these special populations tends to be small. We develop a unified cross-classified random effects model that utilizes item response data from the general population as well as judge-provided data from subject matter experts in order to obtain revised item parameter estimates for use in scoring modified tests. We extend the Metropolis–Hastings Robbins–Monro algorithm to estimate the parameters of this model. The proposed method is applied to Braille test forms in a large operational multistate English language proficiency assessment program. Our work not only allows a broader range of modifications that is routinely considered in large-scale educational assessments but also directly incorporates the input from subject matter experts who work directly with the students needing support. Their structured and informed feedback deserves more attention from the psychometric community.

Suggested Citation

  • Seungwon Chung & Li Cai, 2021. "Cross-Classified Random Effects Modeling for Moderated Item Calibration," Journal of Educational and Behavioral Statistics, , vol. 46(6), pages 651-681, December.
    • Handle: RePEc:sae:jedbes:v:46:y:2021:i:6:p:651-681
    DOI: 10.3102/1076998620983908
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998620983908
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998620983908?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Minjeong Jeon & Frank Rijmen & Sophia Rabe-Hesketh, 2017. "A Variational Maximization–Maximization Algorithm for Generalized Linear Mixed Models with Crossed Random Effects," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 693-716, September.
    2. Cho, S.-J. & Rabe-Hesketh, S., 2011. "Alternating imputation posterior estimation of models with crossed random effects," Computational Statistics & Data Analysis, Elsevier, vol. 55(1), pages 12-25, January.
    3. Li Cai, 2010. "High-dimensional Exploratory Item Factor Analysis by A Metropolis–Hastings Robbins–Monro Algorithm," Psychometrika, Springer;The Psychometric Society, vol. 75(1), pages 33-57, March.
    4. Joe, Harry, 2008. "Accuracy of Laplace approximation for discrete response mixed models," Computational Statistics & Data Analysis, Elsevier, vol. 52(12), pages 5066-5074, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sijia Huang & Li Cai, 2024. "Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching," Journal of Educational and Behavioral Statistics, , vol. 49(3), pages 311-341, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sun-Joo Cho & Sarah Brown-Schmidt & Woo-yeol Lee, 2018. "Autoregressive Generalized Linear Mixed Effect Models with Crossed Random Effects: An Application to Intensive Binary Time Series Eye-Tracking Data," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 751-771, September.
    2. Sijia Huang & Li Cai, 2024. "Cross-Classified Item Response Theory Modeling With an Application to Student Evaluation of Teaching," Journal of Educational and Behavioral Statistics, , vol. 49(3), pages 311-341, June.
    3. Björn Andersson & Tao Xin, 2021. "Estimation of Latent Regression Item Response Theory Models Using a Second-Order Laplace Approximation," Journal of Educational and Behavioral Statistics, , vol. 46(2), pages 244-265, April.
    4. Sun-Joo Cho & Paul Boeck & Susan Embretson & Sophia Rabe-Hesketh, 2014. "Additive Multilevel Item Structure Models with Random Residuals: Item Modeling for Explanation and Item Generation," Psychometrika, Springer;The Psychometric Society, vol. 79(1), pages 84-104, January.
    5. Yoav Bergner & Peter Halpin & Jill-Jênn Vie, 2022. "Multidimensional Item Response Theory in the Style of Collaborative Filtering," Psychometrika, Springer;The Psychometric Society, vol. 87(1), pages 266-288, March.
    6. Minjeong Jeon & Sophia Rabe-Hesketh, 2012. "Profile-Likelihood Approach for Estimating Generalized Linear Mixed Models With Factor Structures," Journal of Educational and Behavioral Statistics, , vol. 37(4), pages 518-542, August.
    7. Motonori Oka & Kensuke Okada, 2023. "Scalable Bayesian Approach for the Dina Q-Matrix Estimation Combining Stochastic Optimization and Variational Inference," Psychometrika, Springer;The Psychometric Society, vol. 88(1), pages 302-331, March.
    8. Minjeong Jeon & Frank Rijmen & Sophia Rabe-Hesketh, 2017. "A Variational Maximization–Maximization Algorithm for Generalized Linear Mixed Models with Crossed Random Effects," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 693-716, September.
    9. Merkle, Edgar C. & Steyvers, Mark & Mellers, Barbara & Tetlock, Philip E., 2017. "A neglected dimension of good forecasting judgment: The questions we choose also matter," International Journal of Forecasting, Elsevier, vol. 33(4), pages 817-832.
    10. Luo, Nanyu & Ji, Feng & Han, Yuting & He, Jinbo & Zhang, Xiaoya, 2024. "Fitting item response theory models using deep learning computational frameworks," OSF Preprints tjxab, Center for Open Science.
    11. Chun Wang, 2015. "On Latent Trait Estimation in Multidimensional Compensatory Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 428-449, June.
    12. Karl, Andrew T. & Yang, Yan & Lohr, Sharon L., 2014. "Computation of maximum likelihood estimates for multiresponse generalized linear mixed models with non-nested, correlated random effects," Computational Statistics & Data Analysis, Elsevier, vol. 73(C), pages 146-162.
    13. Hanneke Geerlings & Cees Glas & Wim Linden, 2011. "Modeling Rule-Based Item Generation," Psychometrika, Springer;The Psychometric Society, vol. 76(2), pages 337-359, April.
    14. Sun-Joo Cho & Allan Cohen & Brian Bottge, 2013. "Detecting Intervention Effects Using a Multilevel Latent Transition Analysis with a Mixture IRT Model," Psychometrika, Springer;The Psychometric Society, vol. 78(3), pages 576-600, July.
    15. Kazuhiro Yamaguchi & Kensuke Okada, 2020. "Variational Bayes Inference for the DINA Model," Journal of Educational and Behavioral Statistics, , vol. 45(5), pages 569-597, October.
    16. Yang Liu, 2020. "A Riemannian Optimization Algorithm for Joint Maximum Likelihood Estimation of High-Dimensional Exploratory Item Factor Analysis," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 439-468, June.
    17. Sijia Huang & Li Cai, 2021. "Lord–Wingersky Algorithm Version 2.5 with Applications," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 973-993, December.
    18. Jiming Jiang & Matt P. Wand & Aishwarya Bhaskaran, 2022. "Usable and precise asymptotics for generalized linear mixed model analysis and design," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(1), pages 55-82, February.
    19. Gregory Camilli & Jean-Paul Fox, 2015. "An Aggregate IRT Procedure for Exploratory Factor Analysis," Journal of Educational and Behavioral Statistics, , vol. 40(4), pages 377-401, August.
    20. Li Cai, 2010. "A Two-Tier Full-Information Item Factor Analysis Model with Applications," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 581-612, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:46:y:2021:i:6:p:651-681. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.