IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v39y2014i1p22-52.html
   My bibliography  Save this article

Correcting for Test Score Measurement Error in ANCOVA Models for Estimating Treatment Effects

Author

Listed:
  • J. R. Lockwood
  • Daniel F. McCaffrey

Abstract

A common strategy for estimating treatment effects in observational studies using individual student-level data is analysis of covariance (ANCOVA) or hierarchical variants of it, in which outcomes (often standardized test scores) are regressed on pretreatment test scores, other student characteristics, and treatment group indicators. Measurement error in the prior test scores, which typically is both large and heteroscedastic, is regularly overlooked in empirical analyses and may erode the ability of regression models to adjust for student factors and may result in biased treatment effect estimates. We develop extensions of method-of-moments, Simulation-Extrapolation, and latent regression approaches to correcting for measurement error using the conditional standard errors of measure of test scores, and demonstrate their effectiveness relative to simpler alternatives using both simulation and a case study of teacher value-added effect estimation using longitudinal data from a large suburban school district.

Suggested Citation

  • J. R. Lockwood & Daniel F. McCaffrey, 2014. "Correcting for Test Score Measurement Error in ANCOVA Models for Estimating Treatment Effects," Journal of Educational and Behavioral Statistics, , vol. 39(1), pages 22-52, February.
    • Handle: RePEc:sae:jedbes:v:39:y:2014:i:1:p:22-52
    DOI: 10.3102/1076998613509405
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998613509405
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998613509405?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jesse Rothstein, 2009. "Student Sorting and Bias in Value-Added Estimation: Selection on Observables and Unobservables," Education Finance and Policy, MIT Press, vol. 4(4), pages 537-571, October.
    2. Sophia Rabe-Hesketh & Anders Skrondal & Andrew Pickles, 2004. "GLLAMM Manual," U.C. Berkeley Division of Biostatistics Working Paper Series 1160, Berkeley Electronic Press.
    3. Brian Junker & Lynne Schofield & Lowell Taylor, 2012. "The use of cognitive ability measures as explanatory variables in regression analysis," IZA Journal of Labor Economics, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 1(1), pages 1-19, December.
    4. Dale Ballou, 2009. "Test Scaling and Value-Added Measurement," Education Finance and Policy, MIT Press, vol. 4(4), pages 351-383, October.
    5. White, Halbert, 1982. "Maximum Likelihood Estimation of Misspecified Models," Econometrica, Econometric Society, vol. 50(1), pages 1-25, January.
    6. Devanarayan, Viswanath & Stefanski, Leonard A., 2002. "Empirical simulation extrapolation for measurement error models with replicate measurements," Statistics & Probability Letters, Elsevier, vol. 59(3), pages 219-225, October.
    7. Daniel F. McCaffrey & J. R. Lockwood & Claude M. Setodji, 2013. "Inverse probability weighting with error-prone covariates," Biometrika, Biometrika Trust, vol. 100(3), pages 671-680.
    8. Cory Koedel & Julian Betts, 2007. "Re-Examining the Role of Teacher Quality In the Educational Production Function," Working Papers 0708, Department of Economics, University of Missouri.
    9. Kata Mihaly & Daniel F. McCaffrey & J. R. Lockwood & Tim R. Sass, 2010. "Centering and reference groups for estimates of fixed effects: Modifications to felsdvreg," Stata Journal, StataCorp LP, vol. 10(1), pages 82-103, March.
    10. Donald Boyd & Hamilton Lankford & Susanna Loeb & James Wyckoff, 2012. "Measuring Test Measurement Error: A General Approach," NBER Working Papers 18010, National Bureau of Economic Research, Inc.
    11. Koedel Cory & Leatherman Rebecca & Parsons Eric, 2012. "Test Measurement Error and Inference from Value-Added Models," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 12(1), pages 1-37, November.
    12. Roger Millsap, 2007. "Invariance in Measurement and Prediction Revisited," Psychometrika, Springer;The Psychometric Society, vol. 72(4), pages 461-473, December.
    13. Derek C. Briggs & Jonathan P. Weeks, 2009. "The Sensitivity of Value-Added Modeling to the Creation of a Vertical Score Scale," Education Finance and Policy, MIT Press, vol. 4(4), pages 384-414, October.
    14. Jean-Paul Fox & Cees Glas, 2003. "Bayesian modeling of measurement error in predictor variables using item response theory," Psychometrika, Springer;The Psychometric Society, vol. 68(2), pages 169-191, June.
    15. Silvia Bianconcini & Silvia Cagnone, 2012. "A General Multivariate Latent Growth Model With Applications to Student Achievement," Journal of Educational and Behavioral Statistics, , vol. 37(2), pages 339-364, April.
    16. Kilchan Choi & Michael Seltzer, 2010. "Modeling Heterogeneity in Relationships Between Initial Status and Rates of Change: Treating Latent Variable Regression Coefficients as Random Coefficients in a Three-Level Hierarchical Model," Journal of Educational and Behavioral Statistics, , vol. 35(1), pages 54-91, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stacy, Brian & Guarino, Cassandra & Wooldridge, Jeffrey, 2018. "Does the precision and stability of value-added estimates of teacher performance depend on the types of students they serve?," Economics of Education Review, Elsevier, vol. 64(C), pages 50-74.
    2. Hwanhee Hong & Kara E. Rudolph & Elizabeth A. Stuart, 2017. "Bayesian Approach for Addressing Differential Covariate Measurement Error in Propensity Score Methods," Psychometrika, Springer;The Psychometric Society, vol. 82(4), pages 1078-1096, December.
    3. Backes, Ben & Cowan, James & Goldhaber, Dan & Koedel, Cory & Miller, Luke C. & Xu, Zeyu, 2018. "The common core conundrum: To what extent should we worry that changes to assessments will affect test-based measures of teacher performance?," Economics of Education Review, Elsevier, vol. 62(C), pages 48-65.
    4. Koedel, Cory & Mihaly, Kata & Rockoff, Jonah E., 2015. "Value-added modeling: A review," Economics of Education Review, Elsevier, vol. 47(C), pages 180-195.
    5. J. R. Lockwood & Daniel F. McCaffrey, 2017. "Simulation-Extrapolation with Latent Heteroskedastic Error Variance," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 717-736, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. J. R. Lockwood & D. McCaffrey, 2020. "Using hidden information and performance level boundaries to study student–teacher assignments: implications for estimating teacher causal effects," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(4), pages 1333-1362, October.
    2. J. R. Lockwood & Daniel F. McCaffrey, 2017. "Simulation-Extrapolation with Latent Heteroskedastic Error Variance," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 717-736, September.
    3. Chun Wang & Gongjun Xu & Xue Zhang, 2019. "Correction for Item Response Theory Latent Trait Measurement Error in Linear Mixed Effects Models," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 673-700, September.
    4. Cory Koedel & Mark Ehlert & Eric Parsons & Michael Podgursky, 2012. "Selecting Growth Measures for School and Teacher Evaluations," Working Papers 1210, Department of Economics, University of Missouri.
    5. Cory Koedel & Jiaxi Li, 2016. "The Efficiency Implications Of Using Proportional Evaluations To Shape The Teaching Workforce," Contemporary Economic Policy, Western Economic Association International, vol. 34(1), pages 47-62, January.
    6. Koedel, Cory & Mihaly, Kata & Rockoff, Jonah E., 2015. "Value-added modeling: A review," Economics of Education Review, Elsevier, vol. 47(C), pages 180-195.
    7. Koedel Cory & Leatherman Rebecca & Parsons Eric, 2012. "Test Measurement Error and Inference from Value-Added Models," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 12(1), pages 1-37, November.
    8. Figlio, D. & Karbownik, K. & Salvanes, K.G., 2016. "Education Research and Administrative Data," Handbook of the Economics of Education,, Elsevier.
    9. Lindsay Fox, 2016. "Playing to Teachers’ Strengths: Using Multiple Measures of Teacher Effectiveness to Improve Teacher Assignments," Education Finance and Policy, MIT Press, vol. 11(1), pages 70-96, Winter.
    10. Goldhaber, Dan & Cowan, James & Walch, Joe, 2013. "Is a good elementary teacher always good? Assessing teacher performance estimates across subjects," Economics of Education Review, Elsevier, vol. 36(C), pages 216-228.
    11. Cory Koedel & Eric Parsons & Michael Podgursky & Mark Ehlert, 2015. "Teacher Preparation Programs and Teacher Quality: Are There Real Differences Across Programs?," Education Finance and Policy, MIT Press, vol. 10(4), pages 508-534, October.
    12. Jason A. Grissom & Demetra Kalogrides & Susanna Loeb, 2012. "Using Student Test Scores to Measure Principal Performance," NBER Working Papers 18568, National Bureau of Economic Research, Inc.
    13. Jesse Rothstein, 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 175-214.
    14. J. R. Lockwood & Daniel F. McCaffrey, 2019. "Impact Evaluation Using Analysis of Covariance With Error-Prone Covariates That Violate Surrogacy," Evaluation Review, , vol. 43(6), pages 335-369, December.
    15. Timothy N. Bond & Kevin Lang, 2018. "The Black–White Education Scaled Test-Score Gap in Grades K-7," Journal of Human Resources, University of Wisconsin Press, vol. 53(4), pages 891-917.
    16. Jeffrey Max & Steven Glazerman, 2014. "Do Disadvantaged Students Get Less Effective Teaching? Key Findings from Recent Institute of Education Sciences Studies (Evaluation Brief)," Mathematica Policy Research Reports a7da30900bb047038d31acd56, Mathematica Policy Research.
    17. repec:mpr:mprres:8000 is not listed on IDEAS
    18. Goldhaber, Dan & Liddle, Stephanie & Theobald, Roddy, 2013. "The gateway to the profession: Assessing teacher preparation programs based on student achievement," Economics of Education Review, Elsevier, vol. 34(C), pages 29-44.
    19. Dan Goldhaber & Michael Hansen, 2013. "Is it Just a Bad Class? Assessing the Long-term Stability of Estimated Teacher Performance," Economica, London School of Economics and Political Science, vol. 80(319), pages 589-612, July.
    20. repec:umc:wpaper:1308 is not listed on IDEAS
    21. Roberto V. Penaloza & Mark Berends, 2022. "The Mechanics of Treatment-effect Estimate Bias for Nonexperimental Data," Sociological Methods & Research, , vol. 51(1), pages 165-202, February.
    22. Seth Gershenson & Diane Whitmore Schanzenbach, 2016. "Linking Teacher Quality, Student Attendance, and Student Achievement," Education Finance and Policy, MIT Press, vol. 11(2), pages 125-149, Spring.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:39:y:2014:i:1:p:22-52. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.