IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v43y2018i4p469-501.html
   My bibliography  Save this article

Detecting Aberrant Behavior and Item Preknowledge: A Comparison of Mixture Modeling Method and Residual Method

Author

Listed:
  • Chun Wang

    (University of Minnesota)

  • Gongjun Xu

    (University of Michigan)

  • Zhuoran Shang
  • Nathan Kuncel

    (University of Minnesota)

Abstract

The modern web-based technology greatly popularizes computer-administered testing, also known as online testing. When these online tests are administered continuously within a certain “testing window,†many items are likely to be exposed and compromised, posing a type of test security concern. In addition, if the testing time is limited, another recognized aberrant behavior is rapid guessing, which refers to quickly answering an item without processing its meaning. Both cheating behavior and rapid guessing result in extremely short response times. This article introduces a mixture hierarchical item response theory model, using both response accuracy and response time information, to help differentiate aberrant behavior from normal behavior. The model-based approach is compared to the Bayesian residual-based fit statistic in both simulation study and two real data examples. Results show that the mixture model approach consistently outperforms the residual method in terms of correct detection rate and false positive error rate, in particular when the proportion of aberrance is high. Moreover, the model-based approach is also able to correctly identify compromised items better than residual method.

Suggested Citation

  • Chun Wang & Gongjun Xu & Zhuoran Shang & Nathan Kuncel, 2018. "Detecting Aberrant Behavior and Item Preknowledge: A Comparison of Mixture Modeling Method and Residual Method," Journal of Educational and Behavioral Statistics, , vol. 43(4), pages 469-501, August.
    • Handle: RePEc:sae:jedbes:v:43:y:2018:i:4:p:469-501
    DOI: 10.3102/1076998618767123
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998618767123
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998618767123?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Yu-Wei Chang & Rung-Ching Tsai & Nan-Jung Hsu, 2014. "A Speeded Item Response Model: Leave the Harder till Later," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 255-274, April.
    2. Jeffrey Rouder & Dongchu Sun & Paul Speckman & Jun Lu & Duo Zhou, 2003. "A hierarchical bayesian statistical framework for response time distributions," Psychometrika, Springer;The Psychometric Society, vol. 68(4), pages 589-606, December.
    3. Chun Wang & Yi Zheng & Hua-Hua Chang, 2014. "Does Standard Deviation Matter? Using “Standard Deviation” to Quantify Security of Multistage Testing," Psychometrika, Springer;The Psychometric Society, vol. 79(1), pages 154-174, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chun Wang & Gongjun Xu & Zhuoran Shang, 2018. "A Two-Stage Approach to Differentiating Normal and Aberrant Behavior in Computer Based Testing," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 223-254, March.
    2. Merkle, Edgar C. & Steyvers, Mark & Mellers, Barbara & Tetlock, Philip E., 2017. "A neglected dimension of good forecasting judgment: The questions we choose also matter," International Journal of Forecasting, Elsevier, vol. 33(4), pages 817-832.
    3. Gerard Breukelen, 2005. "Psychometrics, Psychonomics, Psychonometrics: Rejoinder To Rouder And Wenger," Psychometrika, Springer;The Psychometric Society, vol. 70(2), pages 389-391, June.
    4. Jeffrey Rouder & Jordan Province & Richard Morey & Pablo Gomez & Andrew Heathcote, 2015. "The Lognormal Race: A Cognitive-Process Model of Choice and Latency with Desirable Psychometric Properties," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 491-513, June.
    5. Michael D. Lee, 2018. "Bayesian methods for analyzing true-and-error models," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(6), pages 622-635, November.
    6. Peter W. Rijn & Usama S. Ali, 2018. "A Generalized Speed–Accuracy Response Model for Dichotomous Items," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 109-131, March.
    7. Yi-Hsuan Lee & Zhiliang Ying, 2015. "A Mixture Cure-Rate Model for Responses and Response Times in Time-Limit Tests," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 748-775, September.
    8. Peter Craigmile & Mario Peruggia & Trisha Van Zandt, 2010. "Hierarchical Bayes Models for Response Time Data," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 613-632, December.
    9. Dora Matzke & Conor Dolan & William Batchelder & Eric-Jan Wagenmakers, 2015. "Bayesian Estimation of Multinomial Processing Tree Models with Heterogeneity in Participants and Items," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 205-235, March.
    10. Yu-Wei Chang & Jyun-Ye Tu, 2022. "Bayesian estimation for an item response tree model for nonresponse modeling," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 85(8), pages 1023-1047, November.
    11. Edison M. Choe & Justin L. Kern & Hua-Hua Chang, 2018. "Optimizing the Use of Response Times for Item Selection in Computerized Adaptive Testing," Journal of Educational and Behavioral Statistics, , vol. 43(2), pages 135-158, April.
    12. Onur Demirkaya & Ummugul Bezirhan & Jinming Zhang, 2023. "Detecting Item Preknowledge Using Revisits With Speed and Accuracy," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 521-542, August.
    13. Hua-Hua Chang, 2015. "Psychometrics Behind Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 1-20, March.
    14. repec:cup:judgdm:v:13:y:2018:i:6:p:622-635 is not listed on IDEAS
    15. Brandon Turner & Trisha Zandt, 2014. "Hierarchical Approximate Bayesian Computation," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 185-209, April.
    16. T. Loeys & Y. Rosseel & K. Baten, 2011. "A Joint Modeling Approach for Reaction Time and Accuracy in Psycholinguistic Experiments," Psychometrika, Springer;The Psychometric Society, vol. 76(3), pages 487-503, July.
    17. Maria Bolsinova & Jesper Tijmstra, 2019. "Modeling Differences Between Response Times of Correct and Incorrect Responses," Psychometrika, Springer;The Psychometric Society, vol. 84(4), pages 1018-1046, December.
    18. Chun Wang & Zhewen Fan & Hua-Hua Chang & Jeffrey A. Douglas, 2013. "A Semiparametric Model for Jointly Analyzing Response Times and Accuracy in Computerized Testing," Journal of Educational and Behavioral Statistics, , vol. 38(4), pages 381-417, August.
    19. Wim van der Linden, 2007. "A Hierarchical Framework for Modeling Speed and Accuracy on Test Items," Psychometrika, Springer;The Psychometric Society, vol. 72(3), pages 287-308, September.
    20. Jeffrey Rouder, 2005. "The applicability of deadline models: Comment on Glickman, Gray, and Morales (2005)," Psychometrika, Springer;The Psychometric Society, vol. 70(3), pages 427-430, September.
    21. Tong-Yu Lu & Wai-Yin Poon & Siu Cheung, 2014. "A Unified Framework for the Comparison of Treatments with Ordinal Responses," Psychometrika, Springer;The Psychometric Society, vol. 79(4), pages 605-620, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:43:y:2018:i:4:p:469-501. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.