IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp12151.html
   My bibliography  Save this paper

An Empirical Total Survey Error Decomposition Using Data Combination

Author

Listed:
  • Meyer, Bruce D.

    (University of Chicago)

  • Mittag, Nikolas

    (CERGE-EI)

Abstract

Survey error is known to be pervasive and to bias even simple, but important estimates of means, rates, and totals, such as poverty statistics and the unemployment rate. To summarize and analyze the extent, sources, and consequences of survey error, we define empirical counterparts of key components of the Total Survey Error Framework that can be estimated using data combination. Specifically, we estimate total survey error and decompose it into three high level sources of error: representation error, item non-response error and measurement error. We further decompose these sources into lower level sources such as a failure to report a positive amount and errors in amounts conditional on reporting a positive value. For error in dollars paid by two large government transfer programs, we use administrative records on the universe of program payments in New York State linked to three major household surveys to estimate the error components we define. We find that total survey error is large and varies in its size and composition, but measurement error is always by far the largest source of error. Our application shows that data combination makes it possible to routinely measure total survey error and its components. The results allow survey producers to assess error reduction strategies and survey users to mitigate the consequences of survey errors or gauge the reliability of their conclusions.

Suggested Citation

  • Meyer, Bruce D. & Mittag, Nikolas, 2019. "An Empirical Total Survey Error Decomposition Using Data Combination," IZA Discussion Papers 12151, Institute of Labor Economics (IZA).
  • Handle: RePEc:iza:izadps:dp12151
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp12151.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bruce Meyer & Robert Goerge, 2011. "Errors in Survey Reporting and Imputation and Their Effects on Estimates of Food Stamp Program Participation," Working Papers 11-14, Center for Economic Studies, U.S. Census Bureau.
    2. Deborah Wagner & Mary Lane, 2014. "The Person Identification Validation System (PVS): Applying the Center for Administrative Records Research and Applications’ (CARRA) Record Linkage Software," CARRA Working Papers 2014-01, Center for Economic Studies, U.S. Census Bureau.
    3. Claudia Goldin & Lawrence F. Katz, 2017. "Introduction to "Women Working Longer: Increased Employment at Older Ages"," NBER Chapters, in: Women Working Longer: Increased Employment at Older Ages, pages 1-8, National Bureau of Economic Research, Inc.
    4. Barry T. Hirsch & Edward J. Schumacher, 2004. "Match Bias in Wage Gap Estimates Due to Earnings Imputation," Journal of Labor Economics, University of Chicago Press, vol. 22(3), pages 689-722, July.
    5. C. Adam Bee & Joshua Mitchell, 2017. "The Hidden Resources of Women Working Longer: Evidence from Linked Survey-Administrative Data," NBER Chapters, in: Women Working Longer: Increased Employment at Older Ages, pages 269-296, National Bureau of Economic Research, Inc.
    6. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
    7. Bollinger, Christopher R & David, Martin H, 2001. "Estimation with Response Error and Nonresponse: Food-Stamp Participation in the SIPP," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(2), pages 129-141, April.
    8. Meyer, Bruce D. & Mittag, Nikolas & Goerge, Robert M., 2018. "Errors in Survey Reporting and Imputation and Their Effects on Estimates of Food Stamp Program Participation," IZA Discussion Papers 11776, Institute of Labor Economics (IZA).
    9. Bruce D. Meyer & James X. Sullivan, 2008. "Changes in the Consumption, Income, and Well-Being of Single Mother Headed Families," American Economic Review, American Economic Association, vol. 98(5), pages 2221-2241, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bruce D. Meyer & Nikolas Mittag, 2019. "An Empirical Total Survey Error Decomposition Using Data Combination," NBER Working Papers 25737, National Bureau of Economic Research, Inc.
    2. Meyer, Bruce D. & Mittag, Nikolas, 2019. "Combining Administrative and Survey Data to Improve Income Measurement," IZA Discussion Papers 12266, Institute of Labor Economics (IZA).
    3. Bruce D. Meyer & Nikolas Mittag, 2019. "Combining Administrative and Survey Data to Improve Income Measurement," NBER Working Papers 25738, National Bureau of Economic Research, Inc.
    4. Meyer, Bruce D. & Mittag, Nikolas, 2021. "An empirical total survey error decomposition using data combination," Journal of Econometrics, Elsevier, vol. 224(2), pages 286-305.
    5. Celhay, Pablo & Meyer, Bruce D. & Mittag, Nikolas, 2024. "What leads to measurement errors? Evidence from reports of program participation in three surveys," Journal of Econometrics, Elsevier, vol. 238(2).
    6. Meyer, Bruce D. & Mittag, Nikolas, 2017. "Using Linked Survey and Administrative Data to Better Measure Income: Implications for Poverty, Program Effectiveness and Holes in the Safety Net," IZA Discussion Papers 10943, Institute of Labor Economics (IZA).
    7. Bruce Meyer & Nikolas Mittag, 2017. "Using Linked Survey and Administrative Data to Better Measure Income: Implications for Poverty, Program Effectiveness and Holes in the Safety Net," Working Papers 2017-075, Human Capital and Economic Opportunity Working Group.
    8. Zheng, Yuqing & (Jason) Zhao, Jianqiang & Buck, Steven & Burney, Shaheer & Kaiser, Harry M. & Wilson, Norbert L., 2021. "Putting grocery food taxes on the table: Evidence for food security policy-makers," Food Policy, Elsevier, vol. 101(C).
    9. Sansone, Dario, 2019. "Pink work: Same-sex marriage, employment and discrimination," Journal of Public Economics, Elsevier, vol. 180(C).
    10. Charles Courtemanche & Augustine Denteh & Rusty Tchernis, 2019. "Estimating the Associations between SNAP and Food Insecurity, Obesity, and Food Purchases with Imperfect Administrative Measures of Participation," Southern Economic Journal, John Wiley & Sons, vol. 86(1), pages 202-228, July.
    11. Martijn van Hasselt & Christopher R. Bollinger & Jeremy W. Bray, 2022. "A Bayesian approach to account for misclassification in prevalence and trend estimation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(2), pages 351-367, March.
    12. Colleen Heflin & Michah W. Rothbart & Mattie Mackenzie-Liu, 2022. "Below the Tip of the Iceberg: Examining Early Childhood Participation in SNAP and TANF from Birth to Age Six," Population Research and Policy Review, Springer;Southern Demographic Association (SDA), vol. 41(2), pages 729-755, April.
    13. Bruce D. Meyer & Derek Wu & Victoria R. Mooers & Carla Medalia, 2019. "The Use and Misuse of Income Data and Extreme Poverty in the United States," NBER Working Papers 25907, National Bureau of Economic Research, Inc.
    14. Danielle Sandler & Nichole Szembrot, 2019. "Maternal Labor Dynamics: Participation, Earnings, and Employer Changes," Working Papers 19-33, Center for Economic Studies, U.S. Census Bureau.
    15. Lee, Ji Yong & Nayga Jr, Rodolfo M. & Jo, Young & Restrepo, Brandon J., 2022. "Time use and eating patterns of SNAP participants over the benefit month," Food Policy, Elsevier, vol. 106(C).
    16. Kasahara, Hiroyuki & Shimotsu, Katsumi, 2022. "Identification Of Regression Models With A Misclassified And Endogenous Binary Regressor," Econometric Theory, Cambridge University Press, vol. 38(6), pages 1117-1139, December.
    17. Burkhauser, Richard V. & Corinth, Kevin & Elwell, James & Larrimore, Jeff, 2019. "Evaluating the Success of President Johnson's War on Poverty: Revisiting the Historical Record Using a Full-Income Poverty Measure," IZA Discussion Papers 12855, Institute of Labor Economics (IZA).
    18. Atamanov, Aziz & Tandon, Sharad & Lopez-Acevedo, Gladys & Vergara Bahena, Mexico Alberto, 2020. "Measuring Monetary Poverty in the Middle East and North Africa (MENA) Region: Data Gaps and Different Options to Address Them," IZA Discussion Papers 13363, Institute of Labor Economics (IZA).
    19. Pourya Valizadeh & Travis A Smith, 2020. "How Did The American Recovery and Reinvestment Act Affect the Material Well‐Being of SNAP Participants? A Distributional Approach," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 42(3), pages 455-476, September.
    20. James P. Ziliak & Charles Hokayem & Christopher R. Bollinger, 2022. "Trends in Earnings Volatility Using Linked Administrative and Survey Data," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 41(1), pages 12-19, December.

    More about this item

    Keywords

    data combination; linked data; measurement error; administrative data; total survey error;
    All these keywords.

    JEL classification:

    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • D31 - Microeconomics - - Distribution - - - Personal Income and Wealth Distribution
    • I38 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Government Programs; Provision and Effects of Welfare Programs

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp12151. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.