IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1909.04706.html
   My bibliography  Save this paper

Regression to the Mean's Impact on the Synthetic Control Method: Bias and Sensitivity Analysis

Author

Listed:
  • Nicholas Illenberger
  • Dylan S. Small
  • Pamela A. Shaw

Abstract

To make informed policy recommendations from observational data, we must be able to discern true treatment effects from random noise and effects due to confounding. Difference-in-Difference techniques which match treated units to control units based on pre-treatment outcomes, such as the synthetic control approach have been presented as principled methods to account for confounding. However, we show that use of synthetic controls or other matching procedures can introduce regression to the mean (RTM) bias into estimates of the average treatment effect on the treated. Through simulations, we show RTM bias can lead to inflated type I error rates as well as decreased power in typical policy evaluation settings. Further, we provide a novel correction for RTM bias which can reduce bias and attain appropriate type I error rates. This correction can be used to perform a sensitivity analysis which determines how results may be affected by RTM. We use our proposed correction and sensitivity analysis to reanalyze data concerning the effects of California's Proposition 99, a large-scale tobacco control program, on statewide smoking rates.

Suggested Citation

  • Nicholas Illenberger & Dylan S. Small & Pamela A. Shaw, 2019. "Regression to the Mean's Impact on the Synthetic Control Method: Bias and Sensitivity Analysis," Papers 1909.04706, arXiv.org.
    • Handle: RePEc:arx:papers:1909.04706
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1909.04706
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Hendryx, Michael & Holland, Benjamin, 2016. "Unintended consequences of the Clean Air Act: Mortality rates in Appalachian coal mining communities," Environmental Science & Policy, Elsevier, vol. 63(C), pages 1-6.
    2. Esther Duflo, 2001. "Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment," American Economic Review, American Economic Association, vol. 91(4), pages 795-813, September.
    3. Alberto Abadie, 2005. "Semiparametric Difference-in-Differences Estimators," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(1), pages 1-19.
    4. Alan B. Krueger & David Card, 2000. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania: Reply," American Economic Review, American Economic Association, vol. 90(5), pages 1397-1420, December.
    5. Susan M. Dynarski, 2003. "Does Aid Matter? Measuring the Effect of Student Aid on College Attendance and Completion," American Economic Review, American Economic Association, vol. 93(1), pages 279-288, March.
    6. Noémi Kreif & Richard Grieve & Dominik Hangartner & Alex James Turner & Silviya Nikolova & Matt Sutton, 2016. "Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1514-1528, December.
    7. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    8. Donald B. Rubin, 2005. "Causal Inference Using Potential Outcomes: Design, Modeling, Decisions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 322-331, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Eli Ben‐Michael & Avi Feller & Jesse Rothstein, 2022. "Synthetic controls with staggered adoption," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(2), pages 351-381, April.
    3. Christian Aleman & Christopher Busch & Alexander Ludwig & Raul Santaeulalia-Llopis, 2022. "A Stage-Based Identification of Policy Effects," PIER Working Paper Archive 22-026, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania.
    4. Taylor K. Odle, 2022. "Free to Spend? Institutional Autonomy and Expenditures on Executive Compensation, Faculty Salaries, and Research Activities," Research in Higher Education, Springer;Association for Institutional Research, vol. 63(1), pages 1-32, February.
    5. Dietrichson, Jens & Ellegård, Lina Maria, 2015. "Assist or desist? Conditional bailouts and fiscal discipline in local governments," European Journal of Political Economy, Elsevier, vol. 38(C), pages 153-168.
    6. Myoung-jae Lee & Yasuyuki Sawada, 2020. "Review on Difference in Differences," Korean Economic Review, Korean Economic Association, vol. 36, pages 135-173.
    7. Robbiano, Simone, 2021. "The innovative impact of public research institutes: evidence from Italy," MPRA Paper 106386, University Library of Munich, Germany.
    8. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2020. "Ensuring Stability, Accuracy and Meaningfulness in Synthetic Control Methods: The Regularized SHAP-Distance Method," IREA Working Papers 202005, University of Barcelona, Research Institute of Applied Economics, revised Apr 2020.
    9. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    10. Irene Botosaru & Bruno Ferman, 2019. "On the role of covariates in the synthetic control method," The Econometrics Journal, Royal Economic Society, vol. 22(2), pages 117-130.
    11. Camilla Beck Olsen & Hans Olav Melberg, 2018. "Did adolescents in Norway respond to the elimination of copayments for general practitioner services?," Health Economics, John Wiley & Sons, Ltd., vol. 27(7), pages 1120-1130, July.
    12. Qi Li & Wei Long, 2018. "Do parole abolition and Truth-in-Sentencing deter violent crimes in Virginia?," Empirical Economics, Springer, vol. 55(4), pages 2027-2045, December.
    13. López-Cazar, Ibeth & Papyrakis, Elissaios & Pellegrini, Lorenzo, 2021. "The Extractive Industries Transparency Initiative (EITI) and corruption in Latin America: Evidence from Colombia, Guatemala, Honduras, Peru, and Trinidad and Tobago," Resources Policy, Elsevier, vol. 70(C).
    14. Alagöz, Nazli, 2024. "Promotion and technological change in the music industry," Other publications TiSEM 511ceba0-62a0-4c60-a76c-f, Tilburg University, School of Economics and Management.
    15. Cattan, Sarah & Lereya, Suzet Tanya & Yoon, Yeosun & Gilbert, Ruth & Deighton, Jessica, 2023. "The impact of area level mental health interventions on outcomes for secondary school pupils: Evidence from the HeadStart programme in England," Economics of Education Review, Elsevier, vol. 96(C).
    16. Havnes, Tarjei & Mogstad, Magne, 2015. "Is universal child care leveling the playing field?," Journal of Public Economics, Elsevier, vol. 127(C), pages 100-114.
    17. Cummins Joseph & Miller Douglas L. & Smith Brock & Simon David, 2024. "Matching on Noise: Finite Sample Bias in the Synthetic Control Estimator," Journal of Econometric Methods, De Gruyter, vol. 13(1), pages 67-95, January.
    18. Tomasz Serwach, 2023. "The European Union and within‐country income inequalities. The case of the new member states," The World Economy, Wiley Blackwell, vol. 46(7), pages 1890-1939, July.
    19. Timo Mitze & Reinhold Kosfeld & Johannes Rode & Klaus Wälde, 2020. "Face Masks Considerably Reduce Covid-19 Cases in Germany - A Synthetic Control Method Approach," CESifo Working Paper Series 8479, CESifo.
    20. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1909.04706. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.