IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v44y2019i3p309-341.html
   My bibliography  Save this article

Detection and Treatment of Careless Responses to Improve Item Parameter Estimation

Author

Listed:
  • Jeffrey M. Patton

    (Financial Industry Regulatory Authority (FINRA))

  • Ying Cheng
  • Maxwell Hong

    (University of Notre Dame)

  • Qi Diao

    (Educational Testing Service)

Abstract

In psychological and survey research, the prevalence and serious consequences of careless responses from unmotivated participants are well known. In this study, we propose to iteratively detect careless responders and cleanse the data by removing their responses. The careless responders are detected using person-fit statistics. In two simulation studies, the iterative procedure leads to nearly perfect power in detecting extremely careless responders and much higher power than the noniterative procedure in detecting moderately careless responders. Meanwhile, the false-positive error rate is close to the nominal level. In addition, item parameter estimation is much improved by iteratively cleansing the calibration sample. The bias in item discrimination and location parameter estimates is substantially reduced. The standard error estimates, which are spuriously small in the presence of careless responses, are corrected by the iterative cleansing procedure. An empirical example is also presented to illustrate the proposed procedure. These results suggest that the proposed procedure is a promising way to improve item parameter estimation for tests of 20 items or longer when data are contaminated by careless responses.

Suggested Citation

  • Jeffrey M. Patton & Ying Cheng & Maxwell Hong & Qi Diao, 2019. "Detection and Treatment of Careless Responses to Improve Item Parameter Estimation," Journal of Educational and Behavioral Statistics, , vol. 44(3), pages 309-341, June.
    • Handle: RePEc:sae:jedbes:v:44:y:2019:i:3:p:309-341
    DOI: 10.3102/1076998618825116
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998618825116
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998618825116?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(4), pages 691-705, August.
    2. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(5), pages 879-883, October.
    3. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    4. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(6), pages 1195-1198, December.
    5. Thomas Warm, 1989. "Weighted likelihood estimation of ability in item response theory," Psychometrika, Springer;The Psychometric Society, vol. 54(3), pages 427-450, September.
    6. Tom Snijders, 2001. "Asymptotic null distribution of person fit statistics with estimated person parameter," Psychometrika, Springer;The Psychometric Society, vol. 66(3), pages 331-342, September.
    7. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(1), pages 225-228, February.
    8. ,, 2003. "Problems And Solutions," Econometric Theory, Cambridge University Press, vol. 19(2), pages 411-413, April.
    9. Can Shao & Jun Li & Ying Cheng, 2016. "Detection of Test Speededness Using Change-Point Analysis," Psychometrika, Springer;The Psychometric Society, vol. 81(4), pages 1118-1141, December.
    10. C. Glas & Anna Dagohoy, 2007. "A Person Fit Test For Irt Models For Polytomous Items," Psychometrika, Springer;The Psychometric Society, vol. 72(2), pages 159-180, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yue Liu & Hongyun Liu, 2021. "Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process," Journal of Educational and Behavioral Statistics, , vol. 46(6), pages 717-752, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yakut, Oguz, 2021. "Implementation of hydraulically driven barrel shooting control by utilizing artificial neural networks," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 190(C), pages 1206-1223.
    2. X. Qin & G. Huang, 2009. "An Inexact Chance-constrained Quadratic Programming Model for Stream Water Quality Management," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 23(4), pages 661-695, March.
    3. Md. Yousuf Gazi & Khandakar Tahmida Tafhim, 2019. "Investigation of Heavy-mineral Deposits Using Multispectral Satellite Imagery in the Eastern Coastal Margin of Bangladesh," Earth Sciences Malaysia (ESMY), Zibeline International Publishing, vol. 3(2), pages 16-22, October.
    4. Minghe Sun, 2005. "Warm-Start Routines for Solving Augmented Weighted Tchebycheff Network Programs in Multiple-Objective Network Programming," INFORMS Journal on Computing, INFORMS, vol. 17(4), pages 422-437, November.
    5. François Clautiaux & Cláudio Alves & José Valério de Carvalho & Jürgen Rietz, 2011. "New Stabilization Procedures for the Cutting Stock Problem," INFORMS Journal on Computing, INFORMS, vol. 23(4), pages 530-545, November.
    6. Tansel, Aysit & Karao?lan, Deniz, 2016. "The Causal Effect of Education on Health Behaviors: Evidence from Turkey," IZA Discussion Papers 10020, Institute of Labor Economics (IZA).
    7. Timothy K.M. Beatty & Erling Røed Larsen & Dag Einar Sommervoll, 2005. "Measuring the Price of Housing Consumption for Owners in the CPI," Discussion Papers 427, Statistics Norway, Research Department.
    8. Melega, Gislaine Mara & de Araujo, Silvio Alexandre & Jans, Raf, 2018. "Classification and literature review of integrated lot-sizing and cutting stock problems," European Journal of Operational Research, Elsevier, vol. 271(1), pages 1-19.
    9. Roth, Alvin E. & Sonmez, Tayfun & Utku Unver, M., 2005. "Pairwise kidney exchange," Journal of Economic Theory, Elsevier, vol. 125(2), pages 151-188, December.
    10. repec:dau:papers:123456789/5389 is not listed on IDEAS
    11. Wong, Patricia J.Y., 2015. "Eigenvalues of a general class of boundary value problem with derivative-dependent nonlinearity," Applied Mathematics and Computation, Elsevier, vol. 259(C), pages 908-930.
    12. A. Bensoussan & K. Sung & S. Yam, 2013. "Linear–Quadratic Time-Inconsistent Mean Field Games," Dynamic Games and Applications, Springer, vol. 3(4), pages 537-552, December.
    13. Kojima, Fuhito, 2013. "Efficient resource allocation under multi-unit demand," Games and Economic Behavior, Elsevier, vol. 82(C), pages 1-14.
    14. Chein-Shan Liu & Zhuojia Fu & Chung-Lun Kuo, 2017. "Directional Method of Fundamental Solutions for Three-dimensional Laplace Equation," Journal of Mathematics Research, Canadian Center of Science and Education, vol. 9(6), pages 112-123, December.
    15. Alberto Cabada & Om Kalthoum Wanassi, 2020. "Existence Results for Nonlinear Fractional Problems with Non-Homogeneous Integral Boundary Conditions," Mathematics, MDPI, vol. 8(2), pages 1-13, February.
    16. Odysseas Kosmas & Pieter Boom & Andrey P. Jivkov, 2021. "On the Geometric Description of Nonlinear Elasticity via an Energy Approach Using Barycentric Coordinates," Mathematics, MDPI, vol. 9(14), pages 1-16, July.
    17. Hossein Karshenas & Concha Bielza & Pedro Larrañaga, 2015. "Interval-based ranking in noisy evolutionary multi-objective optimization," Computational Optimization and Applications, Springer, vol. 61(2), pages 517-555, June.
    18. B. S. C. Campello & C. T. L. S. Ghidini & A. O. C. Ayres & W. A. Oliveira, 2022. "A residual recombination heuristic for one-dimensional cutting stock problems," TOP: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 30(1), pages 194-220, April.
    19. Beddoe, Gareth R. & Petrovic, Sanja, 2006. "Selecting and weighting features using a genetic algorithm in a case-based reasoning approach to personnel rostering," European Journal of Operational Research, Elsevier, vol. 175(2), pages 649-671, December.
    20. Hans Wiklund, 2011. "Why High Participatory Ideals Fail In Practice: A Bottom-Up Approach To Public Nonparticipation In Eia," Journal of Environmental Assessment Policy and Management (JEAPM), World Scientific Publishing Co. Pte. Ltd., vol. 13(02), pages 159-178.
    21. Vítor João Pereira Domingues Martinho, 2020. "Exploring the Topics of Soil Pollution and Agricultural Economics: Highlighting Good Practices," Agriculture, MDPI, vol. 10(1), pages 1-19, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:44:y:2019:i:3:p:309-341. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.