IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v41y2017i6p593-619.html
   My bibliography  Save this article

Using Synthetic Controls to Evaluate the Effect of Unique Interventions: The Case of Say Yes to Education

Author

Listed:
  • Robert Bifulco
  • Ross Rubenstein
  • Hosung Sohn

Abstract

Background: “Place-based†scholarships seek to improve student outcomes in urban school districts and promote urban revitalization in economically challenged cities. Say Yes to Education is a unique district-wide school reform effort adopted in Syracuse, NY, in 2008. It includes full-tuition scholarships for public and private universities, coupled with extensive wraparound support services in schools. Objectives: This study uses synthetic control methods to evaluate the effect of Say Yes on district enrollment and graduation rates. It also introduces the synthetic control method and provides guidance for its use in evaluating single-site interventions. Method: Combining school district-level data from the National Center for Education Statistics’ Common Core of Data and New York State School Report Cards, this article uses synthetic control methods to construct a synthetic comparison district to estimate counterfactual enrollment and graduation trends for Syracuse. Results: We find that Say Yes to Education was associated with enrollment increases in the Syracuse City School District, a district that had previously experienced decades of sustained enrollment declines. We do not find consistent evidence of changes in graduation rates following adoption of the program. Conclusions: Graduation rate analyses demonstrate that estimates of treatment effects can be sensitive to choices that the researcher has to make in applying synthetic control methods, particularly when pretreatment outcome measures appear to have considerable amounts of noise.

Suggested Citation

  • Robert Bifulco & Ross Rubenstein & Hosung Sohn, 2017. "Using Synthetic Controls to Evaluate the Effect of Unique Interventions: The Case of Say Yes to Education," Evaluation Review, , vol. 41(6), pages 593-619, December.
    • Handle: RePEc:sae:evarev:v:41:y:2017:i:6:p:593-619
    DOI: 10.1177/0193841X17742233
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X17742233
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X17742233?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. LeGower, Michael & Walsh, Randall, 2017. "Promise scholarship programs as place-making policy: Evidence from school enrollment and housing prices," Journal of Urban Economics, Elsevier, vol. 101(C), pages 74-89.
    2. Sarah Bohn & Magnus Lofstrom & Steven Raphael, 2014. "Did the 2007 Legal Arizona Workers Act Reduce the State's Unauthorized Immigrant Population?," The Review of Economics and Statistics, MIT Press, vol. 96(2), pages 258-269, May.
    3. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    4. Campos, Nauro F. & Coricelli, Fabrizio & Moretti, Luigi, 2014. "Economic Growth and Political Integration: Estimating the Benefits from Membership in the European Union Using the Synthetic Counterfactuals Method," IZA Discussion Papers 8162, Institute of Labor Economics (IZA).
    5. Andreas Billmeier & Tommaso Nannicini, 2013. "Assessing Economic Liberalization Episodes: A Synthetic Control Approach," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 983-1001, July.
    6. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    7. Timothy J. Bartik & Brad J. Hershbein & Marta Lachowska, 2015. "The Effects of the Kalamazoo Promise Scholarship on College Enrollment, Persistence, and Completion," Upjohn Working Papers 15-228, W.E. Upjohn Institute for Employment Research.
    8. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    9. Mircea Trandafir, 2014. "The Effect of Same-Sex Marriage Laws on Different-Sex Marriage: Evidence From the Netherlands," Demography, Springer;Population Association of America (PAA), vol. 51(1), pages 317-340, February.
    10. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    11. Timothy J. Bartik & Nathan Sotherland, 2015. "Migration and Housing Price Effects of Place-Based College Scholarships," Upjohn Working Papers 15-245, W.E. Upjohn Institute for Employment Research.
    12. Timothy J. Bartik & Marta Lachowska, "undated". "The Kalamazoo Promise Scholarship," Upjohn Working Papers tjbml14, W.E. Upjohn Institute for Employment Research.
    13. Brad J. Hershbein, 2013. "A Second Look at Enrollment Changes after the Kalamazoo Promise," Upjohn Working Papers 13-200, W.E. Upjohn Institute for Employment Research.
    14. Coricelli, Fabrizio & Campos, Nauro & Moretti, Luigi, 2014. "Economic Growth and Political Integration: Estimating the Benefits from Membership in the European Union Using the Synthetic Co," CEPR Discussion Papers 9968, C.E.P.R. Discussion Papers.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Brett Parker, 2021. "Death Penalty Statutes and Murder Rates: Evidence From Synthetic Controls," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 18(3), pages 488-533, September.
    2. Guillaume Allaire Pouliot & Zhen Xie, 2022. "Degrees of Freedom and Information Criteria for the Synthetic Control Method," Papers 2207.02943, arXiv.org.
    3. Elaine W. Leigh & Manuel S. González Canché, 2021. "The College Promise in Communities: Do Place-based Scholarships Affect Residential Mobility Patterns?," Research in Higher Education, Springer;Association for Institutional Research, vol. 62(3), pages 259-308, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Essers, Dennis & Ide, Stefaan, 2019. "The IMF and precautionary lending: An empirical evaluation of the selectivity and effectiveness of the Flexible Credit Line," Journal of International Money and Finance, Elsevier, vol. 92(C), pages 25-61.
    2. Katarzyna Metelska-Szaniawska, 2016. "Reassessing the Economic Effects of Post-Socialist Constitutions Using the Synthetic Control Method," Working Papers 2016-18, Faculty of Economic Sciences, University of Warsaw.
    3. Dennis Essers & Stefaan Ide, 2017. "The IMF and precautionary lending : An empirical evaluation of the selectivity and effectiveness of the flexible credit line," Working Paper Research 323, National Bank of Belgium.
    4. Erin O Sills & Diego Herrera & A Justin Kirkpatrick & Amintas Brandão Jr. & Rebecca Dickson & Simon Hall & Subhrendu Pattanayak & David Shoch & Mariana Vedoveto & Luisa Young & Alexander Pfaff, 2015. "Estimating the Impacts of Local Policy Innovation: The Synthetic Control Method Applied to Tropical Deforestation," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-15, July.
    5. Barati Mehdi, 2019. "Punishment Severity and Crime: The Case of Arkansas," Review of Law & Economics, De Gruyter, vol. 15(1), pages 1-23, March.
    6. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2020. "Ensuring Stability, Accuracy and Meaningfulness in Synthetic Control Methods: The Regularized SHAP-Distance Method," IREA Working Papers 202005, University of Barcelona, Research Institute of Applied Economics, revised Apr 2020.
    7. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    8. Maximiliano Marzetti & Rok Spruk, 2023. "Long-Term Economic Effects of Populist Legal Reforms: Evidence from Argentina," Comparative Economic Studies, Palgrave Macmillan;Association for Comparative Economic Studies, vol. 65(1), pages 60-95, March.
    9. Niklas Potrafke & Fabian Ruthardt & Kaspar Wuthrich, 2020. "Protectionism and economic growth: Causal evidence from the first era of globalization," Papers 2010.02378, arXiv.org, revised Mar 2022.
    10. Johanna Catherine Maclean & Brendan Saloner, 2018. "Substance Use Treatment Provider Behavior and Healthcare Reform: Evidence from Massachusetts," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 76-101, January.
    11. Robbert Maseland & Rok Spruk, 2023. "The benefits of US statehood: an analysis of the growth effects of joining the USA," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 17(1), pages 49-89, January.
    12. Nauro Campos & Fabrizio Coricelli & Luigi Moretti, 2015. "Norwegian Rhapsody? The Political Economy Benefits of Regional Integration," Working Papers halshs-01267252, HAL.
    13. Makram El-Shagi & Axel Lindner & Gregor von Schweinitz, 2016. "Real Effective Exchange Rate Misalignment in the Euro Area: A Counterfactual Analysis," Review of International Economics, Wiley Blackwell, vol. 24(1), pages 37-66, February.
    14. Pier Basaglia & Sophie Behr & Moritz A. Drupp, 2023. "De-Fueling Externalities: How Tax Salience and Fuel Substitution Mediate Climate and Health Benefits," Discussion Papers of DIW Berlin 2041, DIW Berlin, German Institute for Economic Research.
    15. Bibek Adhikari & James Alm, 2016. "Evaluating the Economic Effects of Flat Tax Reforms Using Synthetic Control Methods," Southern Economic Journal, John Wiley & Sons, vol. 83(2), pages 437-463, October.
    16. Dube, Arindrajit & Zipperer, Ben, 2015. "Pooling Multiple Case Studies Using Synthetic Controls: An Application to Minimum Wage Policies," IZA Discussion Papers 8944, Institute of Labor Economics (IZA).
    17. Samer Matta & Simon Appleton & Michael Bleaney, 2019. "The Impact of the Arab Spring on the Tunisian Economy," The World Bank Economic Review, World Bank, vol. 33(1), pages 231-258.
    18. Yi‐Ting Chen, 2020. "A distributional synthetic control method for policy evaluation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(5), pages 505-525, August.
    19. Alena Bachleitner, 2017. "Abolishing the Wealth Tax. A Case Study for Germany," WIFO Working Papers 545, WIFO.
    20. Björn Falkenhall & Jonas Månsson & Sofia Tano, 2020. "Impact of VAT Reform on Swedish Restaurants: A Synthetic Control Group Approach," Scandinavian Journal of Economics, Wiley Blackwell, vol. 122(2), pages 824-850, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:41:y:2017:i:6:p:593-619. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.