IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v43y2019i5p231-265.html
   My bibliography  Save this article

Special Issue Editors’ Essay

Author

Listed:
  • Burt S. Barnow
  • David H. Greenberg

Abstract

Background: Impact evaluations draw their data from two sources, namely, surveys conducted for the evaluation or administrative data collected for other purposes. Both types of data have been used to estimate program impacts. This is an introductory essay to a Special Issue entitled “Do the Estimated Effects of Social Programs Depend on the Source of Data Used to Measure Them? Survey Data Versus Administrative Data.†In addition to this essay, the Special Issue contains six articles, which appear in Volume 42, Issue 5–6 (October–December 2018) and in this issue (Volume 43, Issue 5 (October 2019)) of Evaluation Review. Objective: To describe and summarize each of the six papers and draw lessons from them. The papers investigate the relative strengths and weaknesses of survey and administrative data for estimating the impacts of policy interventions. Results: This essay first describes a simple model of the mechanisms that can cause impacts estimated with survey data to differ from those estimated with administrative data. It then describes and summarizes each of the papers appearing in this Special Issue and uses the model described to interpret the findings when it is applicable. The final section draws general lessons from the papers. Conclusions: The decision on whether to use survey or administrative data to estimate program impacts can be highly consequential because the estimates can differ considerably. All the papers in this Special Issue point to the importance of using both survey data and administrative data whenever possible.

Suggested Citation

  • Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.
    • Handle: RePEc:sae:evarev:v:43:y:2019:i:5:p:231-265
    DOI: 10.1177/0193841X19865076
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X19865076
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X19865076?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Burt S. Barnow & David Greenberg, 2015. "Do Estimated Impacts on Earnings Depend on the Source of the Data Used to Measure Them? Evidence From Previous Social Experiments," Evaluation Review, , vol. 39(2), pages 179-228, April.
    2. Blakemore, Arthur E & Burgess, Paul L & Low, Stuart A & St Louis, Robert D, 1996. "Employer Tax Evasion in the Unemployment Insurance Program," Journal of Labor Economics, University of Chicago Press, vol. 14(2), pages 210-230, April.
    3. Katharine G. Abraham & John Haltiwanger & Kristin Sandusky & James R. Spletzer, 2013. "Exploring Differences in Employment between Household and Establishment Data," Journal of Labor Economics, University of Chicago Press, vol. 31(S1), pages 129-172.
    4. Bruce D. Meyer & Derek Wu, 2018. "The Poverty Reduction of Social Security and Means-Tested Transfers," NBER Working Papers 24567, National Bureau of Economic Research, Inc.
    5. Annalisa Mastri & Dana Rotz & Elias S. Hanno, "undated". "Comparing Job Training Impact Estimates Using Survey and Administrative Data," Mathematica Policy Research Reports 157778d936f848ddb0b4e8e32, Mathematica Policy Research.
    6. Bruce D. Meyer & Derek Wu, 2018. "The Poverty Reduction of Social Security and Means-Tested Transfers," ILR Review, Cornell University, ILR School, vol. 71(5), pages 1106-1153, October.
    7. V. J. Hotz & J. K. Scholz, "undated". "Measuring Employment and Income for Low-Income Populations with Administrative and Survey Data," Institute for Research on Poverty Discussion Papers 1224-01, University of Wisconsin Institute for Research on Poverty.
    8. Geoffrey L. Wallace & Robert Haveman, 2007. "The implications of differences between employer and worker employment|earnings reports for policy evaluation," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 26(4), pages 737-754.
    9. Reuben Ford & Douwêrê Grékou & Isaac Kwakye & Taylor Shek-wai Hui, 2018. "The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment," Evaluation Review, , vol. 42(5-6), pages 575-615, October.
    10. W. Norton Grubb, 1993. "The Varied Economic Returns to Postsecondary Education: New Evidence from the Class of 1972," Journal of Human Resources, University of Wisconsin Press, vol. 28(2), pages 365-382.
    11. Judith Scott-Clayton & Qiao Wen, 2019. "Estimating Returns to College Attainment: Comparing Survey and State Administrative Data–Based Estimates," Evaluation Review, , vol. 43(5), pages 266-306, October.
    12. Richard Hendra & Aaron Hill, 2019. "Rethinking Response Rates: New Evidence of Little Relationship Between Survey Response Rates and Nonresponse Bias," Evaluation Review, , vol. 43(5), pages 307-330, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Quinn Moore & Irma Perez-Johnson & Robert Santillano, 2018. "Decomposing Differences in Impacts on Survey- and Administrative-Measured Earnings From a Job Training Voucher Experiment," Evaluation Review, , vol. 42(5-6), pages 515-549, October.
    2. Burt S. Barnow & David Greenberg, 2015. "Do Estimated Impacts on Earnings Depend on the Source of the Data Used to Measure Them? Evidence From Previous Social Experiments," Evaluation Review, , vol. 39(2), pages 179-228, April.
    3. Annalisa Mastri & Dana Rotz & Elias S. Hanno, "undated". "Comparing Job Training Impact Estimates Using Survey and Administrative Data," Mathematica Policy Research Reports 157778d936f848ddb0b4e8e32, Mathematica Policy Research.
    4. Lachowska, Marta & Mas, Alexandre & Woodbury, Stephen A., 2022. "How reliable are administrative reports of paid work hours?," Labour Economics, Elsevier, vol. 75(C).
    5. Sheena McConnell & Peter Z. Schochet & Dana Rotz & Ken Fortson & Paul Burkander & Annalisa Mastri, 2021. "The Effects of Employment Counseling on Labor Market Outcomes for Adults and Dislocated Workers: Evidence from a Nationally Representative Experiment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 40(4), pages 1249-1287, September.
    6. Bruce D. Meyer & Derek Wu & Victoria R. Mooers & Carla Medalia, 2019. "The Use and Misuse of Income Data and Extreme Poverty in the United States," NBER Working Papers 25907, National Bureau of Economic Research, Inc.
    7. Hofmarcher, Thomas, 2021. "The effect of education on poverty: A European perspective," Economics of Education Review, Elsevier, vol. 83(C).
    8. Fredrik Andersson & Harry J. Holzer & Julia I. Lane & David Rosenblum & Jeffrey Smith, 2024. "Does Federally Funded Job Training Work? Nonexperimental Estimates of WIA Training Impacts Using Longitudinal Data on Workers and Firms," Journal of Human Resources, University of Wisconsin Press, vol. 59(4), pages 1244-1283.
    9. Gabrielle Pepin, 2022. "The effects of welfare time limits on access to financial resources: Evidence from the 2010s," Southern Economic Journal, John Wiley & Sons, vol. 88(4), pages 1343-1372, April.
    10. Katharine G. Abraham & John Haltiwanger & Kristin Sandusky & James R. Spletzer, 2013. "Exploring Differences in Employment between Household and Establishment Data," Journal of Labor Economics, University of Chicago Press, vol. 31(S1), pages 129-172.
    11. Diane Whitmore Schanzenbach & Michael R. Strain, 2021. "Employment Effects of the Earned Income Tax Credit: Taking the Long View," Tax Policy and the Economy, University of Chicago Press, vol. 35(1), pages 87-129.
    12. Judith Scott-Clayton & Qiao Wen, 2019. "Estimating Returns to College Attainment: Comparing Survey and State Administrative Data–Based Estimates," Evaluation Review, , vol. 43(5), pages 266-306, October.
    13. Burt S. Barnow & Jeffrey Smith, 2015. "Employment and Training Programs," NBER Chapters, in: Economics of Means-Tested Transfer Programs in the United States, Volume 2, pages 127-234, National Bureau of Economic Research, Inc.
    14. Reuben Ford & Douwêrê Grékou & Isaac Kwakye & Taylor Shek-wai Hui, 2018. "The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment," Evaluation Review, , vol. 42(5-6), pages 575-615, October.
    15. Parolin, Zachary & Brady, David, 2018. "Extreme Child Poverty and the Role of Social Policy in the United States," SocArXiv u5ecn, Center for Open Science.
    16. Celhay, Pablo & Meyer, Bruce D. & Mittag, Nikolas, 2024. "What leads to measurement errors? Evidence from reports of program participation in three surveys," Journal of Econometrics, Elsevier, vol. 238(2).
    17. Chiara Mussida & Dario Sciulli, 2022. "The dynamics of poverty in Europe: what has changed after the great recession?," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 20(4), pages 915-937, December.
    18. Koen Caminada & Kees Goudswaard & Chen Wang & Jinxian Wang, 2021. "Antipoverty Effects of Various Social Transfers and Income Taxes Across Countries," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 154(3), pages 1055-1076, April.
    19. Sarah K. Bruch & Janet C. Gornick & Joseph van der Naald, 2020. "Geographic Inequality in Social Provision: Variation across the US States," NBER Chapters, in: Measuring Distribution and Mobility of Income and Wealth, pages 499-527, National Bureau of Economic Research, Inc.
    20. Yan Xin & Dongchuan Wang & Lihui Zhang & Yingyi Ma & Xing Chen & Haiqing Wang & Hongyi Wang & Kangjian Wang & Hui Long & Hua Chai & Jianshe Gao, 2022. "Cooperative analysis of infrastructure perfection and residents’ living standards in poverty-stricken counties in Qinghai Province," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 24(3), pages 3687-3703, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:43:y:2019:i:5:p:231-265. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.