IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v42y2018i5-6p550-574.html
   My bibliography  Save this article

The Importance of Using Multiple Data Sources in Policy Assessments: Lessons From Two Conditional Cash Transfer Programs in New York City

Author

Listed:
  • Edith Yang
  • Richard Hendra

Abstract

Background: The high costs of implementing surveys are increasingly leading research teams to either cut back on surveys or to rely on administrative records. Yet no policy should be based on a single set of estimates, and every approach has its weaknesses. A mixture of approaches, each with its own biases, should provide the analyst with a better understanding of the underlying phenomenon. This claim is illustrated with a comparison of employment effect estimates of two conditional cash transfer programs in New York City using survey and administrative unemployment insurance (UI) data. Objectives: This article explores whether using administrative data and survey data produce different impact estimates and investigates the source of differential effects between data sources. Research design: The results of a survey nonresponse bias analysis and an analysis of characteristics of non-UI-covered job characteristics using data collected on 6,000 families who enrolled in either the Family Rewards or Work Rewards evaluation are presented. Results: In both evaluations, survey data showed positive employment effects, while administrative data showed no statistically significant employment effects. Family Rewards increased employment mostly in non-UI-covered jobs, while the positive survey impact estimates in Work Rewards were partially due to survey nonresponse bias. Conclusions: Despite cost pressures leading researchers to collect and analyze only administrative records, the results suggest that survey and administrative records data both suffer from different kinds of sample attrition, and researchers may need to triangulate data sources to draw accurate conclusions about program effects. Developing more economical data collection practices is a major priority.

Suggested Citation

  • Edith Yang & Richard Hendra, 2018. "The Importance of Using Multiple Data Sources in Policy Assessments: Lessons From Two Conditional Cash Transfer Programs in New York City," Evaluation Review, , vol. 42(5-6), pages 550-574, October.
    • Handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:550-574
    DOI: 10.1177/0193841X18799820
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X18799820
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X18799820?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Katharine G. Abraham & John Haltiwanger & Kristin Sandusky & James R. Spletzer, 2013. "Exploring Differences in Employment between Household and Establishment Data," Journal of Labor Economics, University of Chicago Press, vol. 31(S1), pages 129-172.
    2. Reuben Ford & Douwêrê Grékou & Isaac Kwakye & Taylor Shek-wai Hui, 2018. "The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment," Evaluation Review, , vol. 42(5-6), pages 575-615, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.
    2. Dang, Hai-Anh H. & Serajuddin, Umar, 2020. "Tracking the sustainable development goals: Emerging measurement challenges and further reflections," World Development, Elsevier, vol. 127(C).
    3. Dang, Hai-Anh H & Carletto, Calogero, 2022. "Recall Bias Revisited: Measure Farm Labor Using Mixed-Mode Surveys and Multiple Imputation," IZA Discussion Papers 14997, Institute of Labor Economics (IZA).
    4. Chinhui Juhn & Kristin McCue, 2012. "Workplace Characteristics and Employment of Older Workers," Working Papers 12-31, Center for Economic Studies, U.S. Census Bureau.
    5. Fredrik Andersson & Mónica García-Pérez & John Haltiwanger & Kristin McCue & Seth Sanders, 2014. "Workplace Concentration of Immigrants," Demography, Springer;Population Association of America (PAA), vol. 51(6), pages 2281-2306, December.
    6. Quinn Moore & Irma Perez-Johnson & Robert Santillano, 2018. "Decomposing Differences in Impacts on Survey- and Administrative-Measured Earnings From a Job Training Voucher Experiment," Evaluation Review, , vol. 42(5-6), pages 515-549, October.
    7. Bruce D. Meyer & Derek Wu & Victoria R. Mooers & Carla Medalia, 2019. "The Use and Misuse of Income Data and Extreme Poverty in the United States," NBER Working Papers 25907, National Bureau of Economic Research, Inc.
    8. Aaron Flaaen & Matthew D. Shapiro & Isaac Sorkin, 2019. "Reconsidering the Consequences of Worker Displacements: Firm versus Worker Perspective," American Economic Journal: Macroeconomics, American Economic Association, vol. 11(2), pages 193-227, April.
    9. Jonathan Meer & Jeremy West, 2016. "Effects of the Minimum Wage on Employment Dynamics," Journal of Human Resources, University of Wisconsin Press, vol. 51(2), pages 500-522.
    10. Liu, Ding & Millimet, Daniel L., 2020. "Bounding the Joint Distribution of Disability and Employment with Contaminated Data," IZA Discussion Papers 13020, Institute of Labor Economics (IZA).
    11. Dmitri Koustas, 2020. "Insights from New Tax-Based Measures of Gig Work in the United States," CESifo Forum, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 21(03), pages 5-9, September.
    12. Kevin F. Hallock, 2013. "Data Improvement and Labor Economics," Journal of Labor Economics, University of Chicago Press, vol. 31(S1), pages 1-16.
    13. Bracha, Anat & Burke, Mary A., 2021. "How Big is the Gig? The Extensive Margin, The Intensive Margin, and The Hidden Margin," Labour Economics, Elsevier, vol. 69(C).
    14. Lachowska, Marta & Mas, Alexandre & Woodbury, Stephen A., 2022. "How reliable are administrative reports of paid work hours?," Labour Economics, Elsevier, vol. 75(C).
    15. Jihye Min & Jerome Agrusa & Joseph Lema & Harold Lee, 2020. "The Tourism Sector and U.S. Regional Macroeconomic Stability: A Network Approach," Sustainability, MDPI, vol. 12(18), pages 1-12, September.
    16. Emily Isenberg & Liana Christin Landivar & Esther Mezey, 2013. "A Comparison Of Person-Reported Industry To Employer-Reported Industry In Survey And Administrative Data," Working Papers 13-47, Center for Economic Studies, U.S. Census Bureau.
    17. Sylvia Allegretto & Arindrajit Dube & Michael Reich & Ben Zipperer, 2017. "Credible Research Designs for Minimum Wage Studies," ILR Review, Cornell University, ILR School, vol. 70(3), pages 559-592, May.
    18. Compton, Andrew, 2019. "A Search Theoretic Model of Part-Time Employment and Multiple Job Holdings," MPRA Paper 97003, University Library of Munich, Germany.
    19. Burt S. Barnow & David Greenberg, 2015. "Do Estimated Impacts on Earnings Depend on the Source of the Data Used to Measure Them? Evidence From Previous Social Experiments," Evaluation Review, , vol. 39(2), pages 179-228, April.
    20. Bruce Fallick & John Haltiwanger & Erika McEntarfer, 2012. "Job-to-job flows and the consequences of job separations," Finance and Economics Discussion Series 2012-73, Board of Governors of the Federal Reserve System (U.S.).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:550-574. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.