IDEAS home Printed from https://ideas.repec.org/a/wly/hlthec/v25y2016i12p1514-1528.html
   My bibliography  Save this article

Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units

Author

Listed:
  • Noémi Kreif
  • Richard Grieve
  • Dominik Hangartner
  • Alex James Turner
  • Silviya Nikolova
  • Matt Sutton

Abstract

This paper examines the synthetic control method in contrast to commonly used difference‐in‐differences (DiD) estimation, in the context of a re‐evaluation of a pay‐for‐performance (P4P) initiative, the Advancing Quality scheme. The synthetic control method aims to estimate treatment effects by constructing a weighted combination of control units, which represents what the treated group would have experienced in the absence of receiving the treatment. While DiD estimation assumes that the effects of unobserved confounders are constant over time, the synthetic control method allows for these effects to change over time, by re‐weighting the control group so that it has similar pre‐intervention characteristics to the treated group. We extend the synthetic control approach to a setting of evaluation of a health policy where there are multiple treated units. We re‐analyse a recent study evaluating the effects of a hospital P4P scheme on risk‐adjusted hospital mortality. In contrast to the original DiD analysis, the synthetic control method reports that, for the incentivised conditions, the P4P scheme did not significantly reduce mortality and that there is a statistically significant increase in mortality for non‐incentivised conditions. This result was robust to alternative specifications of the synthetic control method. © 2015 The Authors. Health Economics published by John Wiley & Sons Ltd.

Suggested Citation

  • Noémi Kreif & Richard Grieve & Dominik Hangartner & Alex James Turner & Silviya Nikolova & Matt Sutton, 2016. "Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1514-1528, December.
    • Handle: RePEc:wly:hlthec:v:25:y:2016:i:12:p:1514-1528
    DOI: 10.1002/hec.3258
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/hec.3258
    Download Restriction: no
    File URL: https://libkey.io/10.1002/hec.3258?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Coffman, Makena & Noy, Ilan, 2012. "Hurricane Iniki: measuring the long-term economic impact of a natural disaster using synthetic control," Environment and Development Economics, Cambridge University Press, vol. 17(2), pages 187-205, April.
    2. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    3. Rachel Meacock & Søren Rud Kristensen & Matt Sutton, 2014. "The Cost‐Effectiveness Of Using Financial Incentives To Improve Provider Quality: A Framework And Application," Health Economics, John Wiley & Sons, Ltd., vol. 23(1), pages 1-13, January.
    4. Échevin, Damien & Fortin, Bernard, 2014. "Physician payment mechanisms, hospital length of stay and risk of readmission: Evidence from a natural experiment," Journal of Health Economics, Elsevier, vol. 36(C), pages 112-124.
    5. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    6. Charles J. Courtemanche & Daniela Zapata, 2014. "Does Universal Coverage Improve Health? The Massachusetts Experience," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(1), pages 36-69, January.
    7. Currie, Janet & Stabile, Mark & Jones, Lauren, 2014. "Do stimulant medications improve educational and behavioral outcomes for children with ADHD?," Journal of Health Economics, Elsevier, vol. 37(C), pages 58-69.
    8. Andreas Billmeier & Tommaso Nannicini, 2013. "Assessing Economic Liberalization Episodes: A Synthetic Control Approach," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 983-1001, July.
    9. Martin Emmert & Frank Eijkenaar & Heike Kemter & Adelheid Esslinger & Oliver Schöffski, 2012. "Economic evaluation of pay-for-performance in health care: a systematic review," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 13(6), pages 755-767, December.
    10. Cameron, A. Colin & Gelbach, Jonah B. & Miller, Douglas L., 2011. "Robust Inference With Multiway Clustering," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(2), pages 238-249.
    11. Brewer Mike & Crossley Thomas F. & Joyce Robert, 2018. "Inference with Difference-in-Differences Revisited," Journal of Econometric Methods, De Gruyter, vol. 7(1), pages 1-16, January.
    12. Ozkan Eren & I. Serkan Ozbeklik, 2011. "Right-to-Work Laws and State-Level Economic Outcomes: Evidence from the Case Studies of Idaho and Oklahoma Using Synthetic Control Method," Working Papers 1101 Classification-JEL J, University of Nevada, Las Vegas , Department of Economics.
    13. Acemoglu, Daron & Johnson, Simon & Kermani, Amir & Kwak, James & Mitton, Todd, 2016. "The value of connections in turbulent times: Evidence from the United States," Journal of Financial Economics, Elsevier, vol. 121(2), pages 368-391.
    14. David Dranove & Ginger Zhe Jin, 2010. "Quality Disclosure and Certification: Theory and Practice," Journal of Economic Literature, American Economic Association, vol. 48(4), pages 935-963, December.
    15. Alberto Abadie, 2005. "Semiparametric Difference-in-Differences Estimators," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(1), pages 1-19.
    16. Roland G. Fryer, 2011. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools," NBER Working Papers 16850, National Bureau of Economic Research, Inc.
    17. Susan Athey & Guido W. Imbens, 2006. "Identification and Inference in Nonlinear Difference-in-Differences Models," Econometrica, Econometric Society, vol. 74(2), pages 431-497, March.
    18. Peter Hinrichs, 2012. "The Effects of Affirmative Action Bans on College Enrollment, Educational Attainment, and the Demographic Composition of Universities," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 712-722, August.
    19. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    20. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    21. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    22. Frank Eijkenaar, 2013. "Key issues in the design of pay for performance programs," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 14(1), pages 117-131, February.
    23. Hugh Gravelle & Matt Sutton & Ada Ma, 2010. "Doctor Behaviour under a Pay for Performance Contract: Treating, Cheating and Case Finding?," Economic Journal, Royal Economic Society, vol. 120(542), pages 129-156, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. David Gilchrist & Thomas Emery & Nuno Garoupa & Rok Spruk, 2023. "Synthetic Control Method: A tool for comparative case studies in economic history," Journal of Economic Surveys, Wiley Blackwell, vol. 37(2), pages 409-445, April.
    2. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    3. Samuel Verevis & Murat Üngör, 2021. "What has New Zealand gained from The FTA with China?: Two counterfactual analyses†," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(1), pages 20-50, February.
    4. Giulio Grossi & Marco Mariani & Alessandra Mattei & Patrizia Lattarulo & Ozge Oner, 2020. "Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets. A synthetic-control approach," Papers 2004.05027, arXiv.org, revised Nov 2023.
    5. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    6. Kleis, Mischa & Moessinger, Marc-Daniel, 2016. "The long-run effect of fiscal consolidation on economic growth: Evidence from quantitative case studies," ZEW Discussion Papers 16-047, ZEW - Leibniz Centre for European Economic Research, revised 2016.
    7. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    8. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    9. Dietrichson, Jens & Ellegård, Lina Maria, 2015. "Assist or desist? Conditional bailouts and fiscal discipline in local governments," European Journal of Political Economy, Elsevier, vol. 38(C), pages 153-168.
    10. Ben Deaner & Hyejin Ku, 2024. "Causal Duration Analysis with Diff-in-Diff," Papers 2405.05220, arXiv.org.
    11. Cummins Joseph & Miller Douglas L. & Smith Brock & Simon David, 2024. "Matching on Noise: Finite Sample Bias in the Synthetic Control Estimator," Journal of Econometric Methods, De Gruyter, vol. 13(1), pages 67-95, January.
    12. Erlend E. Bø & Joel Slemrod & Thor O. Thoresen, 2015. "Taxes on the Internet: Deterrence Effects of Public Disclosure," American Economic Journal: Economic Policy, American Economic Association, vol. 7(1), pages 36-62, February.
    13. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," VfS Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    14. Olper, Alessandro & Curzi, Daniele & Swinnen, Johan, 2018. "Trade liberalization and child mortality: A Synthetic Control Method," World Development, Elsevier, vol. 110(C), pages 394-410.
    15. Myoung-jae Lee & Yasuyuki Sawada, 2020. "Review on Difference in Differences," Korean Economic Review, Korean Economic Association, vol. 36, pages 135-173.
    16. Cruz A. Echevarría & Javier García-Enríquez, 2020. "The economic cost of the Arab Spring: the case of the Egyptian revolution," Empirical Economics, Springer, vol. 59(3), pages 1453-1477, September.
    17. Rok Spruk & Mitja Kovac, 2020. "Does a ban on trans fats improve public health: synthetic control evidence from Denmark," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 156(1), pages 1-32, December.
    18. Barseghyan, Gayane, 2019. "Sanctions and counter-sanctions : What did they do?," BOFIT Discussion Papers 24/2019, Bank of Finland, Institute for Economies in Transition.
    19. Nauro Campos & Fabrizio Coricelli & Luigi Moretti,, 2020. "Choosing Institutional over Economic Integration: Are There Growth Effects?," Working Papers hal-03028143, HAL.
    20. Victor Chernozhukov & Kaspar Wüthrich & Yinchu Zhu, 2021. "An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1849-1864, October.

    More about this item

    JEL classification:

    • N0 - Economic History - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:hlthec:v:25:y:2016:i:12:p:1514-1528. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www3.interscience.wiley.com/cgi-bin/jhome/5749 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.