IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v46y2021i4p430-465.html
   My bibliography  Save this article

On the Treatment of Missing Data in Background Questionnaires in Educational Large-Scale Assessments: An Evaluation of Different Procedures

Author

Listed:
  • Simon Grund
  • Oliver Lüdtke
  • Alexander Robitzsch

    (28393Leibniz Institute for Science and Mathematics Education, Kiel, Germany
    Centre for International Student Assessment, Germany)

Abstract

Large-scale assessments (LSAs) use Mislevy’s “plausible value†(PV) approach to relate student proficiency to noncognitive variables administered in a background questionnaire. This method requires background variables to be completely observed, a requirement that is seldom fulfilled. In this article, we evaluate and compare the properties of methods used in current practice for dealing with missing data in background variables in educational LSAs, which rely on the missing indicator method (MIM), with other methods based on multiple imputation. In this context, we present a fully conditional specification (FCS) approach that allows for a joint treatment of PVs and missing data. Using theoretical arguments and two simulation studies, we illustrate under what conditions the MIM provides biased or unbiased estimates of population parameters and provide evidence that methods such as FCS can provide an effective alternative to the MIM. We discuss the strengths and weaknesses of the approaches and outline potential consequences for operational practice in educational LSAs. An illustration is provided using data from the PISA 2015 study.

Suggested Citation

  • Simon Grund & Oliver Lüdtke & Alexander Robitzsch, 2021. "On the Treatment of Missing Data in Background Questionnaires in Educational Large-Scale Assessments: An Evaluation of Different Procedures," Journal of Educational and Behavioral Statistics, , vol. 46(4), pages 430-465, August.
    • Handle: RePEc:sae:jedbes:v:46:y:2021:i:4:p:430-465
    DOI: 10.3102/1076998620959058
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998620959058
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998620959058?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Donald B. Rubin, 2003. "Nested multiple imputation of NMES via partially incompatible MCMC," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 57(1), pages 3-18, February.
    2. Norman Rose & Matthias Davier & Benjamin Nagengast, 2017. "Modeling Omitted and Not-Reached Items in IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 795-819, September.
    3. Joseph G. Ibrahim & Ming-Hui Chen & Stuart R. Lipsitz & Amy H. Herring, 2005. "Missing-Data Methods for Generalized Linear Models: A Comparative Review," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 332-346, March.
    4. van Buuren, Stef & Groothuis-Oudshoorn, Karin, 2011. "mice: Multivariate Imputation by Chained Equations in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 45(i03).
    5. R. Bock & Murray Aitkin, 1981. "Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm," Psychometrika, Springer;The Psychometric Society, vol. 46(4), pages 443-459, December.
    6. Matthew Blackwell & James Honaker & Gary King, 2017. "A Unified Approach to Measurement Error and Missing Data: Details and Extensions," Sociological Methods & Research, , vol. 46(3), pages 342-369, August.
    7. Irini Moustaki & Martin Knott, 2000. "Weighting for item non‐response in attitude scales by using latent variable models with covariates," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 163(3), pages 445-459.
    8. King, Gary & Honaker, James & Joseph, Anne & Scheve, Kenneth, 2001. "Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation," American Political Science Review, Cambridge University Press, vol. 95(1), pages 49-69, March.
    9. C. O'Muircheartaigh & I. Moustaki, 1999. "Symmetric pattern models: a latent variable approach to item non‐response in attitude scales," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 162(2), pages 177-194.
    10. Robert Mislevy, 1991. "Randomization-based inference about latent variables from complex samples," Psychometrika, Springer;The Psychometric Society, vol. 56(2), pages 177-196, June.
    11. Matthew Blackwell & James Honaker & Gary King, 2017. "A Unified Approach to Measurement Error and Missing Data: Overview and Applications," Sociological Methods & Research, , vol. 46(3), pages 303-341, August.
    12. Frederic Lord, 1983. "Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability," Psychometrika, Springer;The Psychometric Society, vol. 48(2), pages 233-245, June.
    13. Deping Li & Andreas Oranje & Yanlin Jiang, 2009. "On the Estimation of Hierarchical Latent Regression Models for Large-Scale Assessments," Journal of Educational and Behavioral Statistics, , vol. 34(4), pages 433-463, December.
    14. Lynne Schofield & Brian Junker & Lowell Taylor & Dan Black, 2015. "Predictive Inference Using Latent Variables with Covariates," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 727-747, September.
    15. Frank Rijmen & Minjeong Jeon & Matthias von Davier & Sophia Rabe-Hesketh, 2014. "A Third-Order Item Response Theory Model for Modeling the Effects of Domains and Subdomains in Large-Scale Educational Assessment Surveys," Journal of Educational and Behavioral Statistics, , vol. 39(4), pages 235-256, August.
    16. Thomas Warm, 1989. "Weighted likelihood estimation of ability in item response theory," Psychometrika, Springer;The Psychometric Society, vol. 54(3), pages 427-450, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Robitzsch, Alexander & Lüdtke, Oliver, 2022. "Comparing Different Trend Estimation Approaches in International Large-Scale Assessment Studies," OSF Preprints u8kf5, Center for Open Science.
    2. Xie, Zilong & Chen, Yunxiao & von Davier, Matthias & Weng, Haolei, 2023. "Variable selection in latent variable models via knockoffs: an application to international large-scale assessment in education," LSE Research Online Documents on Economics 120812, London School of Economics and Political Science, LSE Library.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Simon Grund & Oliver Lüdtke & Alexander Robitzsch, 2018. "Multiple Imputation of Missing Data at Level 2: A Comparison of Fully Conditional and Joint Modeling in Multilevel Designs," Journal of Educational and Behavioral Statistics, , vol. 43(3), pages 316-353, June.
    2. Matthew Blackwell & James Honaker & Gary King, 2017. "A Unified Approach to Measurement Error and Missing Data: Overview and Applications," Sociological Methods & Research, , vol. 46(3), pages 303-341, August.
    3. Robitzsch, Alexander, 2020. "About Still Nonignorable Consequences of (Partially) Ignoring Missing Item Responses in Large-scale Assessment," OSF Preprints hmy45, Center for Open Science.
    4. Ogasawara, Haruhiko, 2013. "Asymptotic cumulants of ability estimators using fallible item parameters," Journal of Multivariate Analysis, Elsevier, vol. 119(C), pages 144-162.
    5. Lee, Min Cherng & Mitra, Robin, 2016. "Multiply imputing missing values in data sets with mixed measurement scales using a sequence of generalised linear models," Computational Statistics & Data Analysis, Elsevier, vol. 95(C), pages 24-38.
    6. Norman Rose & Matthias Davier & Benjamin Nagengast, 2017. "Modeling Omitted and Not-Reached Items in IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 795-819, September.
    7. Ogasawara, Haruhiko, 2013. "Asymptotic properties of the Bayes and pseudo Bayes estimators of ability in item response theory," Journal of Multivariate Analysis, Elsevier, vol. 114(C), pages 359-377.
    8. Hammon, Angelina & Zinn, Sabine, 2020. "Multiple imputation of binary multilevel missing not at random data," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 69(3), pages 547-564.
    9. Seonghoon Kim, 2012. "A Note on the Reliability Coefficients for Item Response Model-Based Ability Estimates," Psychometrika, Springer;The Psychometric Society, vol. 77(1), pages 153-162, January.
    10. Jinxin Guo & Xin Xu & Zhiliang Ying & Susu Zhang, 2022. "Modeling Not-Reached Items in Timed Tests: A Response Time Censoring Approach," Psychometrika, Springer;The Psychometric Society, vol. 87(3), pages 835-867, September.
    11. Sandip Sinharay, 2022. "Reporting Proficiency Levels for Examinees With Incomplete Data," Journal of Educational and Behavioral Statistics, , vol. 47(3), pages 263-296, June.
    12. Magis, David & Raîche, Gilles, 2012. "Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package catR," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i08).
    13. Angelina Hammon & Sabine Zinn, 2020. "Multiple imputation of binary multilevel missing not at random data," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(3), pages 547-564, June.
    14. Maarten Marsman & Gunter Maris & Timo Bechger & Cees Glas, 2016. "What can we learn from Plausible Values?," Psychometrika, Springer;The Psychometric Society, vol. 81(2), pages 274-289, June.
    15. Jiwei Zhang & Zhaoyuan Zhang & Jian Tao, 2021. "A Bayesian algorithm based on auxiliary variables for estimating GRM with non-ignorable missing data," Computational Statistics, Springer, vol. 36(4), pages 2643-2669, December.
    16. Li Cai & Carrie R. Houts, 2021. "Longitudinal Analysis of Patient-Reported Outcomes in Clinical Trials: Applications of Multilevel and Multidimensional Item Response Theory," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 754-777, September.
    17. Meyer, Bruce D. & Mittag, Nikolas, 2019. "Combining Administrative and Survey Data to Improve Income Measurement," IZA Discussion Papers 12266, Institute of Labor Economics (IZA).
    18. Sebastian Barfort & Nikolaj Harmon & Frederik Hjorth & Asmus Leth Olsen, 2015. "Dishonesty and Selection into Public Service in Denmark: Who Runs the World’s Least Corrupt Public Sector?," Discussion Papers 15-12, University of Copenhagen. Department of Economics.
    19. Michela Battauz & Ruggero Bellio, 2011. "Structural Modeling of Measurement Error in Generalized Linear Models with Rasch Measures as Covariates," Psychometrika, Springer;The Psychometric Society, vol. 76(1), pages 40-56, January.
    20. Xiang Liu & James Yang & Hui Soo Chae & Gary Natriello, 2020. "Power Divergence Family of Statistics for Person Parameters in IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 502-525, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:46:y:2021:i:4:p:430-465. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.