IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v42y2018i5-6p575-615.html
   My bibliography  Save this article

The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment

Author

Listed:
  • Reuben Ford
  • Douwêrê Grékou
  • Isaac Kwakye
  • Taylor Shek-wai Hui

Abstract

Background: This article reports on the Future to Discover Project—a Canadian randomized controlled trial of two high school interventions—where data on key postsecondary enrollment outcomes were collected for two phases. During the initial phase, outcomes were recorded from administrative data and follow-up surveys. During the later phase, data came from administrative records only. Objectives: The article provides analyses that are informative about the consequences of a change from administrative-only data to survey-only data (and vice versa) for the estimation of impacts. Results: The change from administrative-only to survey-only data tended to produce apparent drops in postsecondary enrollment rates that varied by subgroup and education outcome. Nonetheless, levels and significance of impact with respect to postsecondary enrollment remained relatively stable. Conclusions: The findings of the article provide evidence that estimating education program impacts in the context of a randomized experiment can be relatively robust to the data sources chosen. They suggest that internal validity and conclusions for policy need not be affected by changing data sources even when the change produces marked changes in levels of the outcome of interest observed.

Suggested Citation

  • Reuben Ford & Douwêrê Grékou & Isaac Kwakye & Taylor Shek-wai Hui, 2018. "The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment," Evaluation Review, , vol. 42(5-6), pages 575-615, October.
    • Handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:575-615
    DOI: 10.1177/0193841X18799093
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X18799093
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X18799093?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Burt S. Barnow & David Greenberg, 2015. "Do Estimated Impacts on Earnings Depend on the Source of the Data Used to Measure Them? Evidence From Previous Social Experiments," Evaluation Review, , vol. 39(2), pages 179-228, April.
    2. Richard Dorsett & Richard Hendra & Philip K. Robins, 2018. "Using Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration," Evaluation Review, , vol. 42(5-6), pages 491-514, October.
    3. W. Norton Grubb, 1993. "The Varied Economic Returns to Postsecondary Education: New Evidence from the Class of 1972," Journal of Human Resources, University of Wisconsin Press, vol. 28(2), pages 365-382.
    4. Frenette, Marc, 2007. "Why Are Youth from Lower-income Families Less Likely to Attend University? Evidence from Academic Abilities, Parental Influences, and Financial Constraints," Analytical Studies Branch Research Paper Series 2007295e, Statistics Canada, Analytical Studies Branch.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Edith Yang & Richard Hendra, 2018. "The Importance of Using Multiple Data Sources in Policy Assessments: Lessons From Two Conditional Cash Transfer Programs in New York City," Evaluation Review, , vol. 42(5-6), pages 550-574, October.
    2. Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Quinn Moore & Irma Perez-Johnson & Robert Santillano, 2018. "Decomposing Differences in Impacts on Survey- and Administrative-Measured Earnings From a Job Training Voucher Experiment," Evaluation Review, , vol. 42(5-6), pages 515-549, October.
    2. Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.
    3. D. Flannery & J. Cullinan, 2014. "Where they go, what they do and why it matters: the importance of geographic accessibility and social class for decisions relating to higher education institution type, degree level and field of study," Applied Economics, Taylor & Francis Journals, vol. 46(24), pages 2952-2965, August.
    4. Bhaumik, Sumon Kumar & Gang, Ira N. & Yun, Myeong-Su, 2006. "Ethnic conflict and economic disparity: Serbians and Albanians in Kosovo," Journal of Comparative Economics, Elsevier, vol. 34(4), pages 754-773, December.
    5. Sumon Kumar Bhaumik & Ira N. Gang & Myeong-Su Yun, 2008. "Gender and Ethnicity in Post-Conflict Kosovo," WIDER Working Paper Series RP2008-43, World Institute for Development Economic Research (UNU-WIDER).
    6. Laura A. Boyd, 1995. "Analyzing Postsecondary Returns: Does Educational Loan Default Play A Role?," Contemporary Economic Policy, Western Economic Association International, vol. 13(4), pages 80-92, October.
    7. Watson, Barry & Kong, Nancy & Phipps, Shelley, 2022. "Dreaming of a Brighter Future? The Impact of Economic Vulnerability on University Aspirations," IZA Discussion Papers 15539, Institute of Labor Economics (IZA).
    8. Nicholas Trachter, 2015. "Stepping stone and option value in a model of postsecondary education," Quantitative Economics, Econometric Society, vol. 6(1), pages 223-256, March.
    9. Fredrik Andersson & Harry J. Holzer & Julia I. Lane & David Rosenblum & Jeffrey Smith, 2024. "Does Federally Funded Job Training Work? Nonexperimental Estimates of WIA Training Impacts Using Longitudinal Data on Workers and Firms," Journal of Human Resources, University of Wisconsin Press, vol. 59(4), pages 1244-1283.
    10. Philippe Belley & Marc Frenette & Lance Lochner, 2010. "Post-Secondary Attendance by Parental Income: Comparing the U.S. and Canada," University of Western Ontario, Centre for Human Capital and Productivity (CHCP) Working Papers 20103, University of Western Ontario, Centre for Human Capital and Productivity (CHCP).
    11. Cellini, Stephanie Riegg & Chaudhary, Latika, 2014. "The labor market returns to a for-profit college education," Economics of Education Review, Elsevier, vol. 43(C), pages 125-140.
    12. Kelly Chen & Lars Osberg & Shelley Phipps, 2019. "Unequal opportunities and public policy: The impact of parental disability benefits on child postsecondary attendance," Canadian Journal of Economics, Canadian Economics Association, vol. 52(4), pages 1401-1432, November.
    13. Cellini, Stephanie Riegg, 2012. "For-Profit Higher Education: An Assessment of Costs and Benefits," National Tax Journal, National Tax Association;National Tax Journal, vol. 65(1), pages 153-179, March.
    14. Christopher Dougherty, 2005. "Why Are the Returns to Schooling Higher for Women than for Men?," Journal of Human Resources, University of Wisconsin Press, vol. 40(4), pages 969-988.
    15. Emanuelle Bourbeau & Pierre Lefebvre & Philip Merrigan, 2011. "Provincial Returns to Education for 21 to 35 year-olds: Results from the 1991-2006 Canadian Analytic Censuses Files," Cahiers de recherche 1106, CIRPEE.
    16. Arkes, Jeremy, 1999. "What Do Educational Credentials Signal and Why Do Employers Value Credentials?," Economics of Education Review, Elsevier, vol. 18(1), pages 133-141, February.
    17. Nicholas Trachter, 2011. "Option Value and Transitions in a Model of Postsecondary Education," EIEF Working Papers Series 1103, Einaudi Institute for Economics and Finance (EIEF), revised Jan 2011.
    18. Judith Scott-Clayton & Qiao Wen, 2019. "Estimating Returns to College Attainment: Comparing Survey and State Administrative Data–Based Estimates," Evaluation Review, , vol. 43(5), pages 266-306, October.
    19. Iván Bornacelly, 2013. "Educación técnica y tecnológica para la reducción de la desigualdad salarial y la pobreza," Revista Desarrollo y Sociedad, Universidad de los Andes,Facultad de Economía, CEDE, June.
    20. Burt S. Barnow & Jeffrey Smith, 2015. "Employment and Training Programs," NBER Chapters, in: Economics of Means-Tested Transfer Programs in the United States, Volume 2, pages 127-234, National Bureau of Economic Research, Inc.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:575-615. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.