IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v49y2024i5p753-779.html
   My bibliography  Save this article

Analyzing Polytomous Test Data: A Comparison Between an Information-Based IRT Model and the Generalized Partial Credit Model

Author

Listed:
  • Joakim Wallmark

    (Umeå University)

  • James O. Ramsay

    (McGill University)

  • Juan Li

    (Ottawa Hospital Research Institute)

  • Marie Wiberg

    (Umeå University)

Abstract

Item response theory (IRT) models the relationship between the possible scores on a test item against a test taker’s attainment of the latent trait that the item is intended to measure. In this study, we compare two models for tests with polytomously scored items: the optimal scoring (OS) model, a nonparametric IRT model based on the principles of information theory, and the generalized partial credit (GPC) model, a widely used parametric alternative. We evaluate these models using both simulated and real test data. In the real data examples, the OS model demonstrates superior model fit compared to the GPC model across all analyzed datasets. In our simulation study, the OS model outperforms the GPC model in terms of bias, but at the cost of larger standard errors for the probabilities along the estimated item response functions. Furthermore, we illustrate how surprisal arc length, an IRT scale invariant measure of ability with metric properties, can be used to put scores from vastly different types of IRT models on a common scale. We also demonstrate how arc length can be a viable alternative to sum scores for scoring test takers.

Suggested Citation

  • Joakim Wallmark & James O. Ramsay & Juan Li & Marie Wiberg, 2024. "Analyzing Polytomous Test Data: A Comparison Between an Information-Based IRT Model and the Generalized Partial Credit Model," Journal of Educational and Behavioral Statistics, , vol. 49(5), pages 753-779, October.
  • Handle: RePEc:sae:jedbes:v:49:y:2024:i:5:p:753-779
    DOI: 10.3102/10769986231207879
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986231207879
    Download Restriction: no

    File URL: https://libkey.io/10.3102/10769986231207879?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Geoff Masters, 1982. "A rasch model for partial credit scoring," Psychometrika, Springer;The Psychometric Society, vol. 47(2), pages 149-174, June.
    2. Gu, Chong, 2014. "Smoothing Spline ANOVA Models: R Package gss," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 58(i05).
    3. Marie Wiberg & James O. Ramsay & Juan Li, 2019. "Optimal Scores: An Alternative to Parametric Item Response Theory and Sum Scores," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 310-322, March.
    4. J. Ramsay, 1991. "Kernel smoothing approaches to nonparametric item characteristic curve estimation," Psychometrika, Springer;The Psychometric Society, vol. 56(4), pages 611-630, December.
    5. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    6. Carol Woods & David Thissen, 2006. "Item Response Theory with Estimation of the Latent Population Distribution Using Spline-Based Densities," Psychometrika, Springer;The Psychometric Society, vol. 71(2), pages 281-301, June.
    7. Carol M. Woods & David Thissen, 2006. "Item Response Theory with Estimation of the Latent Population Distribution Using Spline-Based Densities," Psychometrika, Springer;The Psychometric Society, vol. 71(2), pages 281-301, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christopher J. Urban & Daniel J. Bauer, 2021. "A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 1-29, March.
    2. James O. Ramsay & Marie Wiberg, 2017. "A Strategy for Replacing Sum Scoring," Journal of Educational and Behavioral Statistics, , vol. 42(3), pages 282-307, June.
    3. Marie Wiberg & James O. Ramsay & Juan Li, 2019. "Optimal Scores: An Alternative to Parametric Item Response Theory and Sum Scores," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 310-322, March.
    4. Yang Liu & Ji Seung Yang, 2018. "Bootstrap-Calibrated Interval Estimates for Latent Variable Scores in Item Response Theory," Psychometrika, Springer;The Psychometric Society, vol. 83(2), pages 333-354, June.
    5. Steven P. Reise & Han Du & Emily F. Wong & Anne S. Hubbard & Mark G. Haviland, 2021. "Matching IRT Models to Patient-Reported Outcomes Constructs: The Graded Response and Log-Logistic Models for Scaling Depression," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 800-824, September.
    6. James Ramsay & Marie Wiberg & Juan Li, 2020. "Full Information Optimal Scoring," Journal of Educational and Behavioral Statistics, , vol. 45(3), pages 297-315, June.
    7. Longjuan Liang & Michael W. Browne, 2015. "A Quasi-Parametric Method for Fitting Flexible Item Response Functions," Journal of Educational and Behavioral Statistics, , vol. 40(1), pages 5-34, February.
    8. Xi Wang & Yang Liu, 2020. "Detecting Compromised Items Using Information From Secure Items," Journal of Educational and Behavioral Statistics, , vol. 45(6), pages 667-689, December.
    9. Luo, Nanyu & Ji, Feng & Han, Yuting & He, Jinbo & Zhang, Xiaoya, 2024. "Fitting item response theory models using deep learning computational frameworks," OSF Preprints tjxab, Center for Open Science.
    10. Yang Liu, 2020. "A Riemannian Optimization Algorithm for Joint Maximum Likelihood Estimation of High-Dimensional Exploratory Item Factor Analysis," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 439-468, June.
    11. Li Cai, 2010. "A Two-Tier Full-Information Item Factor Analysis Model with Applications," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 581-612, December.
    12. J. R. Lockwood & Katherine E. Castellano & Benjamin R. Shear, 2018. "Flexible Bayesian Models for Inferences From Coarsened, Group-Level Achievement Data," Journal of Educational and Behavioral Statistics, , vol. 43(6), pages 663-692, December.
    13. Ke-Hai Yuan & Ying Cheng & Jeff Patton, 2014. "Information Matrices and Standard Errors for MLEs of Item Parameters in IRT," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 232-254, April.
    14. Fumiko Samejima, 1997. "Departure from normal assumptions: A promise for future psychometrics with substantive mathematical modeling," Psychometrika, Springer;The Psychometric Society, vol. 62(4), pages 471-493, December.
    15. Laura Maldonado-Murciano & Halley M. Pontes & Mark D. Griffiths & Maite Barrios & Juana Gómez-Benito & Georgina Guilera, 2020. "The Spanish Version of the Internet Gaming Disorder Scale-Short Form (IGDS9-SF): Further Examination Using Item Response Theory," IJERPH, MDPI, vol. 17(19), pages 1-14, September.
    16. Rikkert M. van der Lans & Ridwan Maulana & Michelle Helms-Lorenz & Carmen-María Fernández-García & Seyeoung Chun & Thelma de Jager & Yulia Irnidayanti & Mercedes Inda-Caro & Okhwa Lee & Thys Coetze, 2021. "Student Perceptions of Teaching Quality in Five Countries: A Partial Credit Model Approach to Assess Measurement Invariance," SAGE Open, , vol. 11(3), pages 21582440211, August.
    17. Ying Cheng & Cheng Liu & John Behrens, 2015. "Standard Error of Ability Estimates and the Classification Accuracy and Consistency of Binary Decisions," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 645-664, September.
    18. Sara Fernandes & Guillaume Fond & Xavier Zendjidjian & Pierre Michel & Karine Baumstarck & Christophe Lançon & Ludovic Samalin & Pierre-Michel Llorca & Magali Coldefy & Pascal Auquier & Laurent Boyer , 2022. "Development and Calibration of the PREMIUM Item Bank for Measuring Respect and Dignity for Patients with Severe Mental Illness," Post-Print hal-03649277, HAL.
    19. Felix Zimmer & Clemens Draxler & Rudolf Debelak, 2023. "Power Analysis for the Wald, LR, Score, and Gradient Tests in a Marginal Maximum Likelihood Framework: Applications in IRT," Psychometrika, Springer;The Psychometric Society, vol. 88(4), pages 1249-1298, December.
    20. Marius Leckelt & Eunike Wetzel & Tanja M. Gerlach & Robert A. Ackerman & Joshua D. Miller & William J. Chopik & Lars Penke & Katharina Geukes & Albrecht C. P. Küfner & Roos Hutteman & David Richter & , 2016. "Validation of the Narcissistic Admiration and Rivalry Questionnaire Short Scale (NARQ-S) in Convenience and Representative Samples," SOEPpapers on Multidisciplinary Panel Data Research 884, DIW Berlin, The German Socio-Economic Panel (SOEP).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:49:y:2024:i:5:p:753-779. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.