IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v48y2023i4p521-542.html
   My bibliography  Save this article

Detecting Item Preknowledge Using Revisits With Speed and Accuracy

Author

Listed:
  • Onur Demirkaya

    (University of Illinois at Urbana-Champaign)

  • Ummugul Bezirhan

    (Boston College)

  • Jinming Zhang

    (University of Illinois at Urbana-Champaign)

Abstract

Examinees with item preknowledge tend to obtain inflated test scores that undermine test score validity. With the availability of process data collected in computer-based assessments, the research on detecting item preknowledge has progressed on using both item scores and response times. Item revisit patterns of examinees can also be utilized as an additional source of information. This study proposes a new statistic for detecting item preknowledge when compromised items are known by utilizing the hierarchical speed–accuracy revisits model. By simultaneously evaluating abnormal changes in the latent abilities, speeds, and revisit propensities of examinees, the procedure was found to provide greater statistical power and stronger substantive evidence that an examinee had indeed benefited from item preknowledge.

Suggested Citation

  • Onur Demirkaya & Ummugul Bezirhan & Jinming Zhang, 2023. "Detecting Item Preknowledge Using Revisits With Speed and Accuracy," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 521-542, August.
    • Handle: RePEc:sae:jedbes:v:48:y:2023:i:4:p:521-542
    DOI: 10.3102/10769986231153403
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986231153403
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986231153403?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Edison M. Choe & Jinming Zhang & Hua-Hua Chang, 2018. "Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 650-673, September.
    2. Hua-Hua Chang & Jinming Zhang, 2002. "Hypergeometric family and item overlap rates in computerized adaptive testing," Psychometrika, Springer;The Psychometric Society, vol. 67(3), pages 387-398, September.
    3. Wim Linden & Fanmin Guo, 2008. "Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 73(3), pages 365-384, September.
    4. Wim van der Linden, 2007. "A Hierarchical Framework for Modeling Speed and Accuracy on Test Items," Psychometrika, Springer;The Psychometric Society, vol. 72(3), pages 287-308, September.
    5. Chun Wang & Yi Zheng & Hua-Hua Chang, 2014. "Does Standard Deviation Matter? Using “Standard Deviation” to Quantify Security of Multistage Testing," Psychometrika, Springer;The Psychometric Society, vol. 79(1), pages 154-174, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hyeon-Ah Kang, 2023. "Sequential Generalized Likelihood Ratio Tests for Online Item Monitoring," Psychometrika, Springer;The Psychometric Society, vol. 88(2), pages 672-696, June.
    2. Hongyue Zhu & Hong Jiao & Wei Gao & Xiangbin Meng, 2023. "Bayesian Change-Point Analysis Approach to Detecting Aberrant Test-Taking Behavior Using Response Times," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 490-520, August.
    3. Edison M. Choe & Jinming Zhang & Hua-Hua Chang, 2018. "Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 650-673, September.
    4. Maria Bolsinova & Paul Boeck & Jesper Tijmstra, 2017. "Modelling Conditional Dependence Between Response Time and Accuracy," Psychometrika, Springer;The Psychometric Society, vol. 82(4), pages 1126-1148, December.
    5. Sandip Sinharay & Peter W. van Rijn, 2020. "Assessing Fit of the Lognormal Model for Response Times," Journal of Educational and Behavioral Statistics, , vol. 45(5), pages 534-568, October.
    6. Chun Wang & Gongjun Xu & Zhuoran Shang, 2018. "A Two-Stage Approach to Differentiating Normal and Aberrant Behavior in Computer Based Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 223-254, March.
    7. Renske E. Kuijpers & Ingmar Visser & Dylan Molenaar, 2021. "Testing the Within-State Distribution in Mixture Models for Responses and Response Times," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 348-373, June.
    8. Wim J. van der Linden, 2009. "A Bivariate Lognormal Response-Time Model for the Detection of Collusion Between Test Takers," Journal of Educational and Behavioral Statistics, , vol. 34(3), pages 378-394, September.
    9. Steffi Pohl & Esther Ulitzsch & Matthias Davier, 2019. "Using Response Times to Model Not-Reached Items due to Time Limits," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 892-920, September.
    10. Dylan Molenaar & Paul Boeck, 2018. "Response Mixture Modeling: Accounting for Heterogeneity in Item Characteristics across Response Times," Psychometrika, Springer;The Psychometric Society, vol. 83(2), pages 279-297, June.
    11. Yue Liu & Hongyun Liu, 2021. "Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process," Journal of Educational and Behavioral Statistics, , vol. 46(6), pages 717-752, December.
    12. Hyeon-Ah Kang & Yi Zheng & Hua-Hua Chang, 2020. "Online Calibration of a Joint Model of Item Responses and Response Times in Computerized Adaptive Testing," Journal of Educational and Behavioral Statistics, , vol. 45(2), pages 175-208, April.
    13. Matthias von Davier & Lale Khorramdel & Qiwei He & Hyo Jeong Shin & Haiwen Chen, 2019. "Developments in Psychometric Population Models for Technology-Based Large-Scale Assessments: An Overview of Challenges and Opportunities," Journal of Educational and Behavioral Statistics, , vol. 44(6), pages 671-705, December.
    14. Edison M. Choe & Hua-Hua Chang, 2019. "The Asymptotic Distribution of Average Test Overlap Rate in Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 84(4), pages 1129-1151, December.
    15. Steven Andrew Culpepper & James Joseph Balamuta, 2017. "A Hierarchical Model for Accuracy and Choice on Standardized Tests," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 820-845, September.
    16. Sora Lee & Daniel M. Bolt, 2018. "Asymmetric Item Characteristic Curves and Item Complexity: Insights from Simulation and Real Data Analyses," Psychometrika, Springer;The Psychometric Society, vol. 83(2), pages 453-475, June.
    17. Yi-Hsuan Lee & Zhiliang Ying, 2015. "A Mixture Cure-Rate Model for Responses and Response Times in Time-Limit Tests," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 748-775, September.
    18. Frederik Coomans & Abe Hofman & Matthieu Brinkhuis & Han L J van der Maas & Gunter Maris, 2016. "Distinguishing Fast and Slow Processes in Accuracy - Response Time Data," PLOS ONE, Public Library of Science, vol. 11(5), pages 1-19, May.
    19. Susu Zhang & Zhi Wang & Jitong Qi & Jingchen Liu & Zhiliang Ying, 2023. "Accurate Assessment via Process Data," Psychometrika, Springer;The Psychometric Society, vol. 88(1), pages 76-97, March.
    20. Shiyu Wang & Yinghan Chen, 2020. "Using Response Times and Response Accuracy to Measure Fluency Within Cognitive Diagnosis Models," Psychometrika, Springer;The Psychometric Society, vol. 85(3), pages 600-629, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:48:y:2023:i:4:p:521-542. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.