IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v42y2018i5-6p575-615.html
   My bibliography  Save this article

The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment

Author

Listed:
  • Reuben Ford
  • Douwêrê Grékou
  • Isaac Kwakye
  • Taylor Shek-wai Hui

Abstract

Background: This article reports on the Future to Discover Project—a Canadian randomized controlled trial of two high school interventions—where data on key postsecondary enrollment outcomes were collected for two phases. During the initial phase, outcomes were recorded from administrative data and follow-up surveys. During the later phase, data came from administrative records only. Objectives: The article provides analyses that are informative about the consequences of a change from administrative-only data to survey-only data (and vice versa) for the estimation of impacts. Results: The change from administrative-only to survey-only data tended to produce apparent drops in postsecondary enrollment rates that varied by subgroup and education outcome. Nonetheless, levels and significance of impact with respect to postsecondary enrollment remained relatively stable. Conclusions: The findings of the article provide evidence that estimating education program impacts in the context of a randomized experiment can be relatively robust to the data sources chosen. They suggest that internal validity and conclusions for policy need not be affected by changing data sources even when the change produces marked changes in levels of the outcome of interest observed.

Suggested Citation

  • Reuben Ford & Douwêrê Grékou & Isaac Kwakye & Taylor Shek-wai Hui, 2018. "The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment," Evaluation Review, , vol. 42(5-6), pages 575-615, October.
    • Handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:575-615
    DOI: 10.1177/0193841X18799093
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X18799093
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X18799093?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Richard Dorsett & Richard Hendra & Philip K. Robins, 2018. "Using Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration," Evaluation Review, , vol. 42(5-6), pages 491-514, October.
    2. Burt S. Barnow & David Greenberg, 2015. "Do Estimated Impacts on Earnings Depend on the Source of the Data Used to Measure Them? Evidence From Previous Social Experiments," Evaluation Review, , vol. 39(2), pages 179-228, April.
    3. W. Norton Grubb, 1993. "The Varied Economic Returns to Postsecondary Education: New Evidence from the Class of 1972," Journal of Human Resources, University of Wisconsin Press, vol. 28(2), pages 365-382.
    4. Frenette, Marc, 2007. "Why Are Youth from Lower-income Families Less Likely to Attend University? Evidence from Academic Abilities, Parental Influences, and Financial Constraints," Analytical Studies Branch Research Paper Series 2007295e, Statistics Canada, Analytical Studies Branch.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.
    2. Edith Yang & Richard Hendra, 2018. "The Importance of Using Multiple Data Sources in Policy Assessments: Lessons From Two Conditional Cash Transfer Programs in New York City," Evaluation Review, , vol. 42(5-6), pages 550-574, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Quinn Moore & Irma Perez-Johnson & Robert Santillano, 2018. "Decomposing Differences in Impacts on Survey- and Administrative-Measured Earnings From a Job Training Voucher Experiment," Evaluation Review, , vol. 42(5-6), pages 515-549, October.
    2. Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.
    3. Burt S. Barnow & Jeffrey Smith, 2015. "Employment and Training Programs," NBER Chapters, in: Economics of Means-Tested Transfer Programs in the United States, Volume 2, pages 127-234, National Bureau of Economic Research, Inc.
    4. D. Flannery & J. Cullinan, 2014. "Where they go, what they do and why it matters: the importance of geographic accessibility and social class for decisions relating to higher education institution type, degree level and field of study," Applied Economics, Taylor & Francis Journals, vol. 46(24), pages 2952-2965, August.
    5. Bhaumik, Sumon Kumar & Gang, Ira N. & Yun, Myeong-Su, 2006. "Ethnic conflict and economic disparity: Serbians and Albanians in Kosovo," Journal of Comparative Economics, Elsevier, vol. 34(4), pages 754-773, December.
    6. Sumon Kumar Bhaumik & Ira N. Gang & Myeong-Su Yun, 2008. "Gender and Ethnicity in Post-Conflict Kosovo," WIDER Working Paper Series RP2008-43, World Institute for Development Economic Research (UNU-WIDER).
    7. Colin Busby, 2010. "Manitoba’s Demographic Challenge: Why Improving Aboriginal Education Outcomes Is Vital for Economic Prosperity," e-briefs 99, C.D. Howe Institute.
    8. Darwin Miller, 2007. "Isolating the Causal Impact of Community College Enrollment on Educational Attainment and Labor Market Outcomes in Texas," Discussion Papers 06-033, Stanford Institute for Economic Policy Research.
    9. Rebecca Walcott & Isabelle Cohen & Denise Ferris, 2024. "When Who Matters: Interviewer Effects and Survey Modality," Evaluation Review, , vol. 48(6), pages 1024-1049, December.
    10. Norton Grubb, W., 1997. "The returns to education in the sub-baccalaureate labor market, 1984-1990," Economics of Education Review, Elsevier, vol. 16(3), pages 231-245, June.
    11. Berhman, J-R & Kletzer, L & Constantine, J & McPherson, M & Schapiro, M-O, 1996. "The Impact of College Quality on Wages : Are There Differences Among Demographic Groups?," Williams Project on the Economics of Higher Education DP-38, Department of Economics, Williams College.
    12. Laura A. Boyd, 1995. "Analyzing Postsecondary Returns: Does Educational Loan Default Play A Role?," Contemporary Economic Policy, Western Economic Association International, vol. 13(4), pages 80-92, October.
    13. Eide, Eric & Waehrer, Geetha, 1998. "The Role of the Option Value of College Attendance in College Major Choice," Economics of Education Review, Elsevier, vol. 17(1), pages 73-82, February.
    14. Ishikawa, Mamoru & Ryan, Daniel, 2002. "Schooling, basic skills and economic outcomes," Economics of Education Review, Elsevier, vol. 21(3), pages 231-243, June.
    15. Watson, Barry & Kong, Nancy & Phipps, Shelley, 2022. "Dreaming of a Brighter Future? The Impact of Economic Vulnerability on University Aspirations," IZA Discussion Papers 15539, Institute of Labor Economics (IZA).
    16. Kelly, Elish & O'Connell, Philip J. & Smyth, Emer, 2010. "The economic returns to field of study and competencies among higher education graduates in Ireland," Economics of Education Review, Elsevier, vol. 29(4), pages 650-657, August.
    17. Steven Brint, 2003. "Few Remaining Dreams: Community Colleges Since 1985," The ANNALS of the American Academy of Political and Social Science, , vol. 586(1), pages 16-37, March.
    18. Nicholas Trachter, 2015. "Stepping stone and option value in a model of postsecondary education," Quantitative Economics, Econometric Society, vol. 6(1), pages 223-256, March.
    19. Fredrik Andersson & Harry J. Holzer & Julia I. Lane & David Rosenblum & Jeffrey Smith, 2024. "Does Federally Funded Job Training Work? Nonexperimental Estimates of WIA Training Impacts Using Longitudinal Data on Workers and Firms," Journal of Human Resources, University of Wisconsin Press, vol. 59(4), pages 1244-1283.
    20. Bailey, Thomas & Jeong, Dong Wook & Cho, Sung-Woo, 2010. "Referral, enrollment, and completion in developmental education sequences in community colleges," Economics of Education Review, Elsevier, vol. 29(2), pages 255-270, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:575-615. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.