IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v84y2019i1d10.1007_s11336-018-09652-3.html
   My bibliography  Save this article

Effects of Discontinue Rules on Psychometric Properties of Test Scores

Author

Listed:
  • Matthias von Davier

    (National Board of Medical Examiners)

  • Youngmi Cho

    (American Institutes for Research)

  • Tianshu Pan

    (Pearson)

Abstract

This paper provides results on a form of adaptive testing that is used frequently in intelligence testing. In these tests, items are presented in order of increasing difficulty. The presentation of items is adaptive in the sense that a session is discontinued once a test taker produces a certain number of incorrect responses in sequence, with subsequent (not observed) responses commonly scored as wrong. The Stanford-Binet Intelligence Scales (SB5; Riverside Publishing Company, 2003) and the Kaufman Assessment Battery for Children (KABC-II; Kaufman and Kaufman, 2004), the Kaufman Adolescent and Adult Intelligence Test (Kaufman and Kaufman 2014) and the Universal Nonverbal Intelligence Test (2nd ed.) (Bracken and McCallum 2015) are some of the many examples using this rule. He and Wolfe (Educ Psychol Meas 72(5):808–826, 2012. https://doi.org/10.1177/0013164412441937 ) compared different ability estimation methods in a simulation study for this discontinue rule adaptation of test length. However, there has been no study, to our knowledge, of the underlying distributional properties based on analytic arguments drawing on probability theory, of what these authors call stochastic censoring of responses. The study results obtained by He and Wolfe (Educ Psychol Meas 72(5):808–826, 2012. https://doi.org/10.1177/0013164412441937 ) agree with results presented by DeAyala et al. (J Educ Meas 38:213–234, 2001) as well as Rose et al. (Modeling non-ignorable missing data with item response theory (IRT; ETS RR-10-11), Educational Testing Service, Princeton, 2010) and Rose et al. (Psychometrika 82:795–819, 2017. https://doi.org/10.1007/s11336-016-9544-7 ) in that ability estimates are biased most when scoring the not observed responses as wrong. This scoring is used operationally, so more research is needed in order to improve practice in this field. The paper extends existing research on adaptivity by discontinue rules in intelligence tests in multiple ways: First, an analytical study of the distributional properties of discontinue rule scored items is presented. Second, a simulation is presented that includes additional scoring rules and uses ability estimators that may be suitable to reduce bias for discontinue rule scored intelligence tests.

Suggested Citation

  • Matthias von Davier & Youngmi Cho & Tianshu Pan, 2019. "Effects of Discontinue Rules on Psychometric Properties of Test Scores," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 147-163, March.
    • Handle: RePEc:spr:psycho:v:84:y:2019:i:1:d:10.1007_s11336-018-09652-3
    DOI: 10.1007/s11336-018-09652-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-018-09652-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11336-018-09652-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Norman Rose & Matthias Davier & Benjamin Nagengast, 2017. "Modeling Omitted and Not-Reached Items in IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 795-819, September.
    2. Little, Roderick J A, 1988. "Missing-Data Adjustments in Large Surveys: Reply," Journal of Business & Economic Statistics, American Statistical Association, vol. 6(3), pages 300-301, July.
    3. Thomas Warm, 1989. "Weighted likelihood estimation of ability in item response theory," Psychometrika, Springer;The Psychometric Society, vol. 54(3), pages 427-450, September.
    4. Rubin, Donald B, 1986. "Statistical Matching Using File Concatenation with Adjusted Weights and Multiple Imputations," Journal of Business & Economic Statistics, American Statistical Association, vol. 4(1), pages 87-94, January.
    5. Little, Roderick J A, 1988. "Missing-Data Adjustments in Large Surveys," Journal of Business & Economic Statistics, American Statistical Association, vol. 6(3), pages 287-296, July.
    6. Roderick J. Little & Donald B. Rubin & Sahar Z. Zangeneh, 2017. "Conditions for Ignoring the Missing-Data Mechanism in Likelihood Inferences for Parameter Subsets," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(517), pages 314-320, January.
    7. Roderick J. Little & Nanhua Zhang, 2011. "Subsample ignorable likelihood for regression analysis with missing data," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 60(4), pages 591-605, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joost Ginkel & Pieter Kroonenberg, 2014. "Using Generalized Procrustes Analysis for Multiple Imputation in Principal Component Analysis," Journal of Classification, Springer;The Classification Society, vol. 31(2), pages 242-269, July.
    2. Brownstone, David, 1997. "Multiple Imputation Methodology for Missing Data, Non-Random Response, and Panel Attrition," University of California Transportation Center, Working Papers qt2zd6w6hh, University of California Transportation Center.
    3. Westermeier, Christian & Grabka, Markus M., 2016. "Longitudinal Wealth Data and Multiple Imputation: An Evaluation Study," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 10(3), pages 237-252.
    4. Arif Mamun & David Wittenburg & Noelle Denny-Brown & Michael Levere & David Mann & Rebecca Coughlin & Sarah Croake & Heather Gordon & Denise Hoffman & Rachel Holzwart & Rosalind Keith & Brittany McGil, "undated". "Promoting Opportunity Demonstration: Interim Evaluation Report," Mathematica Policy Research Reports caa99d38a8b14f968ea3438e5, Mathematica Policy Research.
    5. Baltussen, Guido & Swinkels, Laurens & Van Vliet, Pim, 2021. "Global factor premiums," Journal of Financial Economics, Elsevier, vol. 142(3), pages 1128-1154.
    6. Sean Mc Auliffe & Georg U. Thunecke & Georg Wamser, 2023. "The Tax-Elasticity of Tangible Fixed Assets: Evidence from Novel Corporate Tax Data," CESifo Working Paper Series 10628, CESifo.
    7. Leonie C. Steckermeier & Jan Delhey, 2019. "Better for Everyone? Egalitarian Culture and Social Wellbeing in Europe," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 143(3), pages 1075-1108, June.
    8. Saeideh Kamgar & Florian Meinfelder & Ralf Münnich & Hamidreza Navvabpour, 2020. "Estimation within the new integrated system of household surveys in Germany," Statistical Papers, Springer, vol. 61(5), pages 2091-2117, October.
    9. Filippo Battistoni & Marco Martinez, 2022. "Rome and the Polis: Tradition and Change in the Financial Accounts of Tauromenion, 1st Century B.C," Annals of the Fondazione Luigi Einaudi. An Interdisciplinary Journal of Economics, History and Political Science, Fondazione Luigi Einaudi, Torino (Italy), vol. 56(1), pages 149-176, June.
    10. Roderick J. A. Little & Donald B. Rubin, 1989. "The Analysis of Social Science Data with Missing Values," Sociological Methods & Research, , vol. 18(2-3), pages 292-326, November.
    11. Jana Emmenegger & Ralf Münnich & Jannik Schaller, 2022. "Evaluating Data Fusion Methods to Improve Income Modelling," Research Papers in Economics 2022-03, University of Trier, Department of Economics.
    12. Rebecca R. Andridge & Roderick J. A. Little, 2010. "A Review of Hot Deck Imputation for Survey Non‐response," International Statistical Review, International Statistical Institute, vol. 78(1), pages 40-64, April.
    13. Chenyang Gu & Roee Gutman, 2017. "Combining item response theory with multiple imputation to equate health assessment questionnaires," Biometrics, The International Biometric Society, vol. 73(3), pages 990-998, September.
    14. Chia-Ning Wang & Roderick Little & Bin Nan & Siobán D. Harlow, 2011. "A Hot-Deck Multiple Imputation Procedure for Gaps in Longitudinal Recurrent Event Histories," Biometrics, The International Biometric Society, vol. 67(4), pages 1573-1582, December.
    15. Morris A. Davis & William D. Larson & Stephen D. Oliner & Benjamin Smith, 2019. "Mortgage Risk Since 1990," FHFA Staff Working Papers 19-02, Federal Housing Finance Agency.
    16. Fernandes, Mario & Hilber, Simon & Sturm, Jan-Egbert & Walter, Andreas, 2023. "Closing the gender gap in academia? Evidence from an affirmative action program," Research Policy, Elsevier, vol. 52(9).
    17. Sean Mc Auliffe & Georg U. Thunecke & Georg Wamser, 2023. "The Tax-Elasticity of Tangible Fixed Assets: Heterogeneous Effects of Homogeneous Tax Policy Changes," Working Papers tax-mpg-rps-2023-25, Max Planck Institute for Tax Law and Public Finance.
    18. Patrick M. Joyce & Donald Malec & Roderick J. A. Little & Aaron Gilary & Alfredo Navarro & Mark E. Asiala, 2014. "Statistical Modeling Methodology for the Voting Rights Act Section 203 Language Assistance Determinations," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(505), pages 36-47, March.
    19. Eduardo Lora & Miguel Benítez & Diego Gutiérrez, 2024. "Annualizing labor market, inequality, and poverty indicators," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 22(1), pages 131-164, March.
    20. Mingyang Cai & Gerko Vink, 2022. "A note on imputing squares via polynomial combination approach," Computational Statistics, Springer, vol. 37(5), pages 2185-2201, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:84:y:2019:i:1:d:10.1007_s11336-018-09652-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.