IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v85y2020i1d10.1007_s11336-019-09687-0.html
   My bibliography  Save this article

Optimal Online Calibration Designs for Item Replenishment in Adaptive Testing

Author

Listed:
  • Yinhong He

    (Nanjing University of Information Science and Technology
    Beijing Normal University)

  • Ping Chen

    (Beijing Normal University)

Abstract

The maintenance of item bank is essential for continuously implementing adaptive tests. Calibration of new items online provides an opportunity to efficiently replenish items for the operational item bank. In this study, a new optimal design for online calibration (referred to as D-c) is proposed by incorporating the idea of original D-optimal design into the reformed D-optimal design proposed by van der Linden and Ren (Psychometrika 80:263–288, 2015) (denoted as D-VR design). To deal with the dependence of design criteria on the unknown item parameters of new items, Bayesian versions of the locally optimal designs (e.g., D-c and D-VR) are put forward by adding prior information to the new items. In the simulation implementation of the locally optimal designs, five calibration sample sizes were used to obtain different levels of estimation precision for the initial item parameters, and two approaches were used to obtain the prior distributions in Bayesian optimal designs. Results showed that the D-c design performed well and retired smaller number of new items than the D-VR design at almost all levels of examinee sample size; the Bayesian version of D-c using the prior obtained from the operational items worked better than that using the default priors in BILOG-MG and PARSCALE; and Bayesian optimal designs generally outperformed locally optimal designs when the initial item parameters of the new items were poorly estimated.

Suggested Citation

  • Yinhong He & Ping Chen, 2020. "Optimal Online Calibration Designs for Item Replenishment in Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 85(1), pages 35-55, March.
  • Handle: RePEc:spr:psycho:v:85:y:2020:i:1:d:10.1007_s11336-019-09687-0
    DOI: 10.1007/s11336-019-09687-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-019-09687-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11336-019-09687-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hung-Yi Lu, 2014. "Application of Optimal Designs to Item Calibration," PLOS ONE, Public Library of Science, vol. 9(9), pages 1-8, September.
    2. Robert Tsutakawa & Jane Johnson, 1990. "The effect of uncertainty of item parameter estimation on ability estimates," Psychometrika, Springer;The Psychometric Society, vol. 55(2), pages 371-390, June.
    3. Martijn Berger, 1992. "Sequential sampling designs for the two-parameter item response theory model," Psychometrika, Springer;The Psychometric Society, vol. 57(4), pages 521-538, December.
    4. Ping Chen & Tao Xin & Chun Wang & Hua-Hua Chang, 2012. "Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT," Psychometrika, Springer;The Psychometric Society, vol. 77(2), pages 201-222, April.
    5. Martha Stocking, 1990. "Specifying optimum examinees for item parameter estimation in item response theory," Psychometrika, Springer;The Psychometric Society, vol. 55(3), pages 461-475, September.
    6. Wim Linden & Hao Ren, 2015. "Optimal Bayesian Adaptive Design for Test-Item Calibration," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 263-288, June.
    7. Ping Chen & Chun Wang, 2016. "A New Online Calibration Method for Multidimensional Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 81(3), pages 674-701, September.
    8. Hao Ren & Wim J. van der Linden & Qi Diao, 2017. "Continuous Online Item Calibration: Parameter Recovery and Item Utilization," Psychometrika, Springer;The Psychometric Society, vol. 82(2), pages 498-522, June.
    9. Martijn Berger & C. Joy King & Weng Wong, 2000. "Minimax d-optimal designs for item response theory models," Psychometrika, Springer;The Psychometric Society, vol. 65(3), pages 377-390, September.
    10. Douglas Jones & Zhiying Jin, 1994. "Optimal sequential designs for on-line item estimation," Psychometrika, Springer;The Psychometric Society, vol. 59(1), pages 59-75, March.
    11. Ying Cheng & Ke-Hai Yuan, 2010. "The Impact of Fallible Item Parameter Estimates on Latent Trait Recovery," Psychometrika, Springer;The Psychometric Society, vol. 75(2), pages 280-291, June.
    12. Yuan-chin Chang & Hung-Yi Lu, 2010. "Online Calibration Via Variable Length Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 75(1), pages 140-157, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mahmood Ul Hassan & Frank Miller, 2019. "Optimal Item Calibration for Computerized Achievement Tests," Psychometrika, Springer;The Psychometric Society, vol. 84(4), pages 1101-1128, December.
    2. Ul Hassan, Mahmood & Miller, Frank, 2021. "An exchange algorithm for optimal calibration of items in computerized achievement tests," Computational Statistics & Data Analysis, Elsevier, vol. 157(C).
    3. Hyeon-Ah Kang & Yi Zheng & Hua-Hua Chang, 2020. "Online Calibration of a Joint Model of Item Responses and Response Times in Computerized Adaptive Testing," Journal of Educational and Behavioral Statistics, , vol. 45(2), pages 175-208, April.
    4. Wim Linden & Hao Ren, 2015. "Optimal Bayesian Adaptive Design for Test-Item Calibration," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 263-288, June.
    5. Ping Chen, 2017. "A Comparative Study of Online Item Calibration Methods in Multidimensional Computerized Adaptive Testing," Journal of Educational and Behavioral Statistics, , vol. 42(5), pages 559-590, October.
    6. Hao Ren & Wim J. van der Linden & Qi Diao, 2017. "Continuous Online Item Calibration: Parameter Recovery and Item Utilization," Psychometrika, Springer;The Psychometric Society, vol. 82(2), pages 498-522, June.
    7. Yuan-chin Chang & Hung-Yi Lu, 2010. "Online Calibration Via Variable Length Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 75(1), pages 140-157, March.
    8. Ping Chen & Chun Wang, 2016. "A New Online Calibration Method for Multidimensional Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 81(3), pages 674-701, September.
    9. Wim J. van der Linden & Bingnan Jiang, 2020. "A Shadow-Test Approach to Adaptive Item Calibration," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 301-321, June.
    10. Ogasawara, Haruhiko, 2013. "Asymptotic cumulants of ability estimators using fallible item parameters," Journal of Multivariate Analysis, Elsevier, vol. 119(C), pages 144-162.
    11. Yang Liu & Ji Seung Yang, 2018. "Interval Estimation of Latent Variable Scores in Item Response Theory," Journal of Educational and Behavioral Statistics, , vol. 43(3), pages 259-285, June.
    12. Wim J. van der Linden & Hao Ren, 2020. "A Fast and Simple Algorithm for Bayesian Adaptive Testing," Journal of Educational and Behavioral Statistics, , vol. 45(1), pages 58-85, February.
    13. Ying Cheng & Ke-Hai Yuan, 2010. "The Impact of Fallible Item Parameter Estimates on Latent Trait Recovery," Psychometrika, Springer;The Psychometric Society, vol. 75(2), pages 280-291, June.
    14. Giada Spaccapanico Proietti & Mariagiulia Matteucci & Stefania Mignani & Bernard P. Veldkamp, 2024. "Chance-Constrained Automated Test Assembly," Journal of Educational and Behavioral Statistics, , vol. 49(1), pages 92-120, February.
    15. C. Glas & Anna Dagohoy, 2007. "A Person Fit Test For Irt Models For Polytomous Items," Psychometrika, Springer;The Psychometric Society, vol. 72(2), pages 159-180, June.
    16. Qingrong Tan & Yan Cai & Fen Luo & Dongbo Tu, 2023. "Development of a High-Accuracy and Effective Online Calibration Method in CD-CAT Based on Gini Index," Journal of Educational and Behavioral Statistics, , vol. 48(1), pages 103-141, February.
    17. Yang Liu & Jan Hannig & Abhishek Pal Majumder, 2019. "Second-Order Probability Matching Priors for the Person Parameter in Unidimensional IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 701-718, September.
    18. Yuan-chin Chang, 2011. "Sequential estimation in generalized linear models when covariates are subject to errors," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 73(1), pages 93-120, January.
    19. Chiara Tommasi & Juan M. Rodríguez-Díaz & Jesús F. López-Fidalgo, 2023. "An equivalence theorem for design optimality with respect to a multi-objective criterion," Statistical Papers, Springer, vol. 64(4), pages 1041-1056, August.
    20. Douglas Jones & Zhiying Jin, 1994. "Optimal sequential designs for on-line item estimation," Psychometrika, Springer;The Psychometric Society, vol. 59(1), pages 59-75, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:85:y:2020:i:1:d:10.1007_s11336-019-09687-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.