IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v45y2020i3p251-273.html
   My bibliography  Save this article

A Bias-Corrected RMSD Item Fit Statistic: An Evaluation and Comparison to Alternatives

Author

Listed:
  • Carmen Köhler

    (DIPF | Leibniz Institute for Research and Information in Education)

  • Alexander Robitzsch

    (Leibniz Institute for Science and Mathematics Education (IPN)
    Centre for International Student Assessment (ZIB))

  • Johannes Hartig

    (DIPF | Leibniz Institute for Research and Information in Education)

Abstract

Testing whether items fit the assumptions of an item response theory model is an important step in evaluating a test. In the literature, numerous item fit statistics exist, many of which show severe limitations. The current study investigates the root mean squared deviation (RMSD) item fit statistic, which is used for evaluating item fit in various large-scale assessment studies. The three research questions of this study are (1) whether the empirical RMSD is an unbiased estimator of the population RMSD; (2) if this is not the case, whether this bias can be corrected; and (3) whether the test statistic provides an adequate significance test to detect misfitting items. Using simulation studies, it was found that the empirical RMSD is not an unbiased estimator of the population RMSD, and nonparametric bootstrapping falls short of entirely eliminating this bias. Using parametric bootstrapping, however, the RMSD can be used as a test statistic that outperforms the other approaches—infit and outfit, S − X 2 —with respect to both Type I error rate and power. The empirical application showed that parametric bootstrapping of the RMSD results in rather conservative item fit decisions, which suggests more lenient cut-off criteria.

Suggested Citation

  • Carmen Köhler & Alexander Robitzsch & Johannes Hartig, 2020. "A Bias-Corrected RMSD Item Fit Statistic: An Evaluation and Comparison to Alternatives," Journal of Educational and Behavioral Statistics, , vol. 45(3), pages 251-273, June.
    • Handle: RePEc:sae:jedbes:v:45:y:2020:i:3:p:251-273
    DOI: 10.3102/1076998619890566
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998619890566
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998619890566?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. R. Darrell Bock, 1972. "Estimating item parameters and latent ability when responses are scored in two or more nominal categories," Psychometrika, Springer;The Psychometric Society, vol. 37(1), pages 29-51, March.
    2. White, Halbert, 1982. "Maximum Likelihood Estimation of Misspecified Models," Econometrica, Econometric Society, vol. 50(1), pages 1-25, January.
    3. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    4. Thomas Warm, 1989. "Weighted likelihood estimation of ability in item response theory," Psychometrika, Springer;The Psychometric Society, vol. 54(3), pages 427-450, September.
    5. J.J. Heckman & E.E. Leamer (ed.), 2001. "Handbook of Econometrics," Handbook of Econometrics, Elsevier, edition 1, volume 5, number 5.
    6. Maydeu-Olivares, Albert & Joe, Harry, 2005. "Limited- and Full-Information Estimation and Goodness-of-Fit Testing in 2n Contingency Tables: A Unified Framework," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 1009-1020, September.
    7. Alberto Maydeu-Olivares, 2017. "Assessing the Size of Model Misfit in Structural Equation Models," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 533-558, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sean Joo & Montserrat Valdivia & Dubravka Svetina Valdivia & Leslie Rutkowski, 2024. "Alternatives to Weighted Item Fit Statistics for Establishing Measurement Invariance in Many Groups," Journal of Educational and Behavioral Statistics, , vol. 49(3), pages 465-493, June.
    2. Scott Monroe, 2021. "Testing Latent Variable Distribution Fit in IRT Using Posterior Residuals," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 374-398, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jochen Ranger & Kay Brauer, 2022. "On the Generalized S − X 2 –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization," Journal of Educational and Behavioral Statistics, , vol. 47(2), pages 202-230, April.
    2. Björn Andersson & Tao Xin, 2021. "Estimation of Latent Regression Item Response Theory Models Using a Second-Order Laplace Approximation," Journal of Educational and Behavioral Statistics, , vol. 46(2), pages 244-265, April.
    3. Daphna Harel & Russell J. Steele, 2018. "An Information Matrix Test for the Collapsing of Categories Under the Partial Credit Model," Journal of Educational and Behavioral Statistics, , vol. 43(6), pages 721-750, December.
    4. Laine Bradshaw & Jonathan Templin, 2014. "Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions," Psychometrika, Springer;The Psychometric Society, vol. 79(3), pages 403-425, July.
    5. Elina Tsigeman & Sebastian Silas & Klaus Frieler & Maxim Likhanov & Rebecca Gelding & Yulia Kovas & Daniel Müllensiefen, 2022. "The Jack and Jill Adaptive Working Memory Task: Construction, Calibration and Validation," PLOS ONE, Public Library of Science, vol. 17(1), pages 1-29, January.
    6. Ick Hoon Jin & Minjeong Jeon, 2019. "A Doubly Latent Space Joint Model for Local Item and Person Dependence in the Analysis of Item Response Data," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 236-260, March.
    7. Alexander Robitzsch, 2021. "A Comprehensive Simulation Study of Estimation Methods for the Rasch Model," Stats, MDPI, vol. 4(4), pages 1-23, October.
    8. Yingbin Zhang & Zhaoxi Yang & Yehui Wang, 2022. "The Impact of Extreme Response Style on the Mean Comparison of Two Independent Samples," SAGE Open, , vol. 12(2), pages 21582440221, June.
    9. Harry Joe & Alberto Maydeu-Olivares, 2010. "A General Family of Limited Information Goodness-of-Fit Statistics for Multinomial Data," Psychometrika, Springer;The Psychometric Society, vol. 75(3), pages 393-419, September.
    10. Yang Liu & Weimeng Wang, 2022. "Semiparametric Factor Analysis for Item-Level Response Time Data," Psychometrika, Springer;The Psychometric Society, vol. 87(2), pages 666-692, June.
    11. Joel L. Horowitz, 2018. "Bootstrap methods in econometrics," CeMMAP working papers CWP53/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    12. David Magis, 2015. "A Note on Weighted Likelihood and Jeffreys Modal Estimation of Proficiency Levels in Polytomous Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 200-204, March.
    13. Mark Reiser & Silvia Cagnone & Junfei Zhu, 2023. "An Extended GFfit Statistic Defined on Orthogonal Components of Pearson’s Chi-Square," Psychometrika, Springer;The Psychometric Society, vol. 88(1), pages 208-240, March.
    14. Brzezińska Justyna, 2018. "Item Response Theory Models in the Measurement Theory with the Use of ltm Package in R," Econometrics. Advances in Applied Data Analysis, Sciendo, vol. 22(1), pages 11-25, March.
    15. David Magis & Norman Verhelst, 2017. "On the Finiteness of the Weighted Likelihood Estimator of Ability," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 637-647, September.
    16. Maxwell Hong & Lizhen Lin & Ying Cheng, 2021. "Asymptotically Corrected Person Fit Statistics for Multidimensional Constructs with Simple Structure and Mixed Item Types," Psychometrika, Springer;The Psychometric Society, vol. 86(2), pages 464-488, June.
    17. King, Maxwell L. & Zhang, Xibin & Akram, Muhammad, 2020. "Hypothesis testing based on a vector of statistics," Journal of Econometrics, Elsevier, vol. 219(2), pages 425-455.
    18. Chun Wang, 2015. "On Latent Trait Estimation in Multidimensional Compensatory Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 428-449, June.
    19. Shaobo Jin & Fan Yang-Wallentin, 2017. "Asymptotic Robustness Study of the Polychoric Correlation Estimation," Psychometrika, Springer;The Psychometric Society, vol. 82(1), pages 67-85, March.
    20. Yang Liu & Jan Hannig & Abhishek Pal Majumder, 2019. "Second-Order Probability Matching Priors for the Person Parameter in Unidimensional IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 701-718, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:45:y:2020:i:3:p:251-273. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.