IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/78213.html
   My bibliography  Save this paper

Cherry Picking with Synthetic Controls

Author

Listed:
  • Ferman, Bruno
  • Pinto, Cristine
  • Possebom, Vitor

Abstract

We show that a lack of guidance on how to choose the matching variables used in the Synthetic Control (SC) estimator creates specification-searching opportunities in SC applications. This undermines one of the potential advantages of the method, which is providing a transparent way of choosing comparison units and, therefore, being less susceptible to specification searching than alternative methods. To address this problem, we provide recommendations to limit the possibilities for specification searching in the SC method. Finally, we analyze the possibilities for specification searching and our recommendations in two empirical applications.

Suggested Citation

  • Ferman, Bruno & Pinto, Cristine & Possebom, Vitor, 2017. "Cherry Picking with Synthetic Controls," MPRA Paper 78213, University Library of Munich, Germany.
    • Handle: RePEc:pra:mprapa:78213
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/78213/1/MPRA_paper_78213.pdf
    File Function: original version
    Download Restriction: no
    File URL: https://mpra.ub.uni-muenchen.de/80970/1/MPRA_paper_80970.pdf
    File Function: revised version
    Download Restriction: no
    File URL: https://mpra.ub.uni-muenchen.de/85138/1/MPRA_paper_85138.pdf
    File Function: revised version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Kirkpatrick, A. Justin & Bennear, Lori S., 2014. "Promoting clean energy investment: An empirical analysis of property assessed clean energy," Journal of Environmental Economics and Management, Elsevier, vol. 68(2), pages 357-375.
    2. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    3. Christian Dustmann & Uta Schönberg & Jan Stuhler, 2017. "Labor Supply Shocks, Native Wages, and the Adjustment of Local Employment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(1), pages 435-483.
    4. Sarah Bohn & Magnus Lofstrom & Steven Raphael, 2014. "Did the 2007 Legal Arizona Workers Act Reduce the State's Unauthorized Immigrant Population?," The Review of Economics and Statistics, MIT Press, vol. 96(2), pages 258-269, May.
    5. Scott Cunningham & Manisha Shah, 2018. "Decriminalizing Indoor Prostitution: Implications for Sexual Violence and Public Health," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(3), pages 1683-1715.
    6. Nikolay Doudchenko & Guido W. Imbens, 2016. "Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis," NBER Working Papers 22791, National Bureau of Economic Research, Inc.
    7. Andreas Billmeier & Tommaso Nannicini, 2013. "Assessing Economic Liberalization Episodes: A Synthetic Control Approach," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 983-1001, July.
    8. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    9. Ben Zou, 2018. "The Local Economic Impacts of Military Personnel," Journal of Labor Economics, University of Chicago Press, vol. 36(3), pages 589-621.
    10. Andreas Billmeier & Tommaso Nannicini, 2009. "Trade Openness and Growth: Pursuing Empirical Glasnost," IMF Staff Papers, Palgrave Macmillan, vol. 56(3), pages 447-475, August.
    11. Gregory DeAngelo & Benjamin Hansen, 2014. "Life and Death in the Fast Lane: Police Enforcement and Traffic Fatalities," American Economic Journal: Economic Policy, American Economic Association, vol. 6(2), pages 231-257, May.
    12. Jinjarak, Yothin & Noy, Ilan & Zheng, Huanhuan, 2013. "Capital controls in Brazil – Stemming a tide with a signal?," Journal of Banking & Finance, Elsevier, vol. 37(8), pages 2938-2952.
    13. Acemoglu, Daron & Johnson, Simon & Kermani, Amir & Kwak, James & Mitton, Todd, 2016. "The value of connections in turbulent times: Evidence from the United States," Journal of Financial Economics, Elsevier, vol. 121(2), pages 368-391.
    14. Laurent Gobillon & Thierry Magnac, 2016. "Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 535-551, July.
    15. Shimeng Liu, 2015. "Spillovers from Universities: Evidence from the Land-Grant Program," Working Paper 9410, USC Lusk Center for Real Estate.
    16. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    17. Peter Hinrichs, 2012. "The Effects of Affirmative Action Bans on College Enrollment, Educational Attainment, and the Demographic Composition of Universities," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 712-722, August.
    18. Ann P. Bartel & Maya Rossin†Slater & Christopher J. Ruhm & Jenna Stearns & Jane Waldfogel, 2018. "Paid Family Leave, Fathers’ Leave†Taking, and Leave†Sharing in Dual†Earner Households," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 37(1), pages 10-37, January.
    19. Eduardo Cavallo & Sebastian Galiani & Ilan Noy & Juan Pantano, 2013. "Catastrophic Natural Disasters and Economic Growth," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1549-1561, December.
    20. Lucas C. Coffman & Muriel Niederle, 2015. "Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 81-98, Summer.
    21. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    22. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    23. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    24. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    25. William duPont IV & Ilan Noy, 2015. "What Happened to Kobe? A Reassessment of the Impact of the 1995 Earthquake in Japan," Economic Development and Cultural Change, University of Chicago Press, vol. 63(4), pages 777-812.
    26. Ferman, Bruno & Pinto, Cristine, 2016. "Revisiting the Synthetic Control Estimator," MPRA Paper 73982, University Library of Munich, Germany.
    27. Fafchamps, Marcel & Labonne, Julien, 2017. "Using Split Samples to Improve Inference on Causal Effects," Political Analysis, Cambridge University Press, vol. 25(4), pages 465-482, October.
    28. Calderón Gabriela, 2014. "The Effects of Child Care Provision in Mexico," Working Papers 2014-07, Banco de México.
    29. José G. Montalvo, 2011. "Voting after the Bombings: A Natural Experiment on the Effect of Terrorist Attacks on Democratic Elections," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1146-1154, November.
    30. Ozkan Eren & Serkan Ozbeklik, 2016. "What Do Right‐to‐Work Laws Do? Evidence from a Synthetic Control Method Analysis," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 35(1), pages 173-194, January.
    31. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    32. Ando, Michihito, 2015. "Dreams of urbanization: Quantitative case studies on the local impacts of nuclear power facilities using the synthetic control method," Journal of Urban Economics, Elsevier, vol. 85(C), pages 68-85.
    33. Ferman, Bruno & Pinto, Cristine, 2017. "Placebo Tests for Synthetic Controls," MPRA Paper 78079, University Library of Munich, Germany.
    34. William duPont IV & Ilan Noy, 2015. "What Happened to Kobe? A Reassessment of the Impact of the 1995 Earthquake in Japan," Economic Development and Cultural Change, University of Chicago Press, vol. 63(4), pages 777-812.
    35. Noémi Kreif & Richard Grieve & Dominik Hangartner & Alex James Turner & Silviya Nikolova & Matt Sutton, 2016. "Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1514-1528, December.
    36. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    37. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    38. Amr Hosny, 2012. "Algeria’s Trade with GAFTA Countries: A Synthetic Control Approach," Transition Studies Review, Springer;Central Eastern European University Network (CEEUN), vol. 19(1), pages 35-42, September.
    39. Barone, Guglielmo & Mocetti, Sauro, 2014. "Natural disasters, growth and institutions: A tale of two earthquakes," Journal of Urban Economics, Elsevier, vol. 84(C), pages 52-66.
    40. Firpo Sergio & Possebom Vitor, 2018. "Synthetic Control Method: Inference, Sensitivity Analysis and Confidence Sets," Journal of Causal Inference, De Gruyter, vol. 6(2), pages 1-26, September.
    41. Henrik Jacobsen Kleven & Camille Landais & Emmanuel Saez, 2013. "Taxation and International Migration of Superstars: Evidence from the European Football Market," American Economic Review, American Economic Association, vol. 103(5), pages 1892-1924, August.
    42. Mideksa, Torben K., 2013. "The economic impact of natural resources," Journal of Environmental Economics and Management, Elsevier, vol. 65(2), pages 277-289.
    43. Smith, Brock, 2015. "The resource curse exorcised: Evidence from a panel of countries," Journal of Development Economics, Elsevier, vol. 116(C), pages 57-73.
    44. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    45. Isabela Ferreira Duarte & João Manoel Pinho de Mello & Vinicius Nascimento Carrasco, 2014. "A Década Perdida: 2003 – 2012," Textos para discussão 626, Department of Economics PUC-Rio (Brazil).
    46. Mich�le Belot & Vincent Vandenberghe, 2014. "Evaluating the 'threat' effects of grade repetition: exploiting the 2001 reform by the French-Speaking Community of Belgium," Education Economics, Taylor & Francis Journals, vol. 22(1), pages 73-89, February.
    47. Vincenzo Bove & Leandro Eliay & Ron P Smith, 2014. "The relationship between panel and synthetic control estimators of the effect of civil war," BCAM Working Papers 1406, Birkbeck Centre for Applied Macroeconomics.
    48. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    49. Bibek Adhikari & James Alm, 2016. "Evaluating the Economic Effects of Flat Tax Reforms Using Synthetic Control Methods," Southern Economic Journal, John Wiley & Sons, vol. 83(2), pages 437-463, October.
    50. Jinjarak, Yothin & Noy, Ilan & Zheng, Huanhuan, 2013. "Capital controls in Brazil – Stemming a tide with a signal?," Journal of Banking & Finance, Elsevier, vol. 37(8), pages 2938-2952.
    51. Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
    52. Dube, Arindrajit & Zipperer, Ben, 2015. "Pooling Multiple Case Studies Using Synthetic Controls: An Application to Minimum Wage Policies," IZA Discussion Papers 8944, Institute of Labor Economics (IZA).
    53. Marcos Sanso‐Navarro, 2011. "The Effects on American Foreign Direct Investment in the United Kingdom from Not Adopting the Euro," Journal of Common Market Studies, Wiley Blackwell, vol. 49(2), pages 463-483, March.
    54. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2011. "Synth: An R Package for Synthetic Control Methods in Comparative Case Studies," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i13).
    55. Paolo Pinotti, 2012. "The Economic Costs of Organized Crime: Evidence from Southern Italy," Working Papers 054, "Carlo F. Dondena" Centre for Research on Social Dynamics (DONDENA), Università Commerciale Luigi Bocconi.
    56. repec:wly:soecon:v:83:2:y:2016:p:437-463 is not listed on IDEAS
    57. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    58. Severnini, Edson R., 2014. "The Power of Hydroelectric Dams: Agglomeration Spillovers," IZA Discussion Papers 8082, Institute of Labor Economics (IZA).
    59. Erin O Sills & Diego Herrera & A Justin Kirkpatrick & Amintas Brandão Jr. & Rebecca Dickson & Simon Hall & Subhrendu Pattanayak & David Shoch & Mariana Vedoveto & Luisa Young & Alexander Pfaff, 2015. "Estimating the Impacts of Local Policy Innovation: The Synthetic Control Method Applied to Tropical Deforestation," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-15, July.
    60. Jason M. Lindo & Analisa Packham, 2017. "How Much Can Expanding Access to Long-Acting Reversible Contraceptives Reduce Teen Birth Rates?," American Economic Journal: Economic Policy, American Economic Association, vol. 9(3), pages 348-376, August.
    61. Stefan Klößner & Ashok Kaul & Gregor Pfeifer & Manuel Schieler, 2018. "Comparative politics and the synthetic control method revisited: a note on Abadie et al. (2015)," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 154(1), pages 1-11, December.
    62. Leonardo Baccini & Quan Li & Irina Mirkina, 2014. "Corporate Tax Cuts And Foreign Direct Investment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(4), pages 977-1006, September.
    63. Dhungana, Sandesh, 2011. "Identifying and evaluating large scale policy interventions : what questions can we answer ?," Policy Research Working Paper Series 5918, The World Bank.
    64. Chan, Ho Fai & Frey, Bruno S. & Gallus, Jana & Torgler, Benno, 2014. "Academic honors and performance," Labour Economics, Elsevier, vol. 31(C), pages 188-204.
    65. Bauhoff, Sebastian, 2014. "The effect of school district nutrition policies on dietary intake and overweight: A synthetic control approach," Economics & Human Biology, Elsevier, vol. 12(C), pages 45-55.
    66. Liu, Shimeng, 2015. "Spillovers from universities: Evidence from the land-grant program," Journal of Urban Economics, Elsevier, vol. 87(C), pages 25-41.
    67. Jinyong Hahn & Ruoyao Shi, 2017. "Synthetic Control and Inference," Econometrics, MDPI, vol. 5(4), pages 1-12, November.
    68. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ferman, Bruno & Pinto, Cristine, 2016. "Revisiting the Synthetic Control Estimator," MPRA Paper 73982, University Library of Munich, Germany.
    2. Giulio Grossi & Marco Mariani & Alessandra Mattei & Patrizia Lattarulo & Ozge Oner, 2020. "Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets. A synthetic-control approach," Papers 2004.05027, arXiv.org, revised Nov 2023.
    3. David Gilchrist & Thomas Emery & Nuno Garoupa & Rok Spruk, 2023. "Synthetic Control Method: A tool for comparative case studies in economic history," Journal of Economic Surveys, Wiley Blackwell, vol. 37(2), pages 409-445, April.
    4. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    5. Samuel Verevis & Murat Üngör, 2021. "What has New Zealand gained from The FTA with China?: Two counterfactual analyses†," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(1), pages 20-50, February.
    6. Pier Basaglia & Sophie M. Behr & Moritz A. Drupp, 2023. "De-Fueling Externalities: How Tax Salience and Fuel Substitution Mediate Climate and Health Benefits," Discussion Papers of DIW Berlin 2041, DIW Berlin, German Institute for Economic Research.
    7. Pier Basaglia & Sophie M. Behr & Moritz A. Drupp, 2023. "De-Fueling Externalities: Causal Effects of Fuel Taxation and Mediating Mechanisms for Reducing Climate and Pollution Costs," CESifo Working Paper Series 10508, CESifo.
    8. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," VfS Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    9. Kleis, Mischa & Moessinger, Marc-Daniel, 2016. "The long-run effect of fiscal consolidation on economic growth: Evidence from quantitative case studies," ZEW Discussion Papers 16-047, ZEW - Leibniz Centre for European Economic Research, revised 2016.
    10. Emery, Thomas & Mélon, Lela & Spruk, Rok, 2023. "Does e-procurement matter for economic growth? Subnational evidence from Australia," The Quarterly Review of Economics and Finance, Elsevier, vol. 89(C), pages 318-334.
    11. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2021. "Decoupling synthetic control methods to ensure stability, accuracy and meaningfulness," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 12(4), pages 549-584, December.
    12. Irene Botosaru & Bruno Ferman, 2019. "On the role of covariates in the synthetic control method," The Econometrics Journal, Royal Economic Society, vol. 22(2), pages 117-130.
    13. Maïmouna DIAKITE & Jean-François BRUN & Souleymane DIARRA & Nasser ARY TANIMOUNE, 2017. "The effects of tax coordination on the tax revenue mobilization in West African Economic and Monetary Union (WAEMU)," Working Papers 201712, CERDI.
    14. Matej Opatrny, 2021. "The impact of the Brexit vote on UK financial markets: a synthetic control method approach," Empirica, Springer;Austrian Institute for Economic Research;Austrian Economic Association, vol. 48(2), pages 559-587, May.
    15. David Powell, 2016. "Synthetic Control Estimation Beyond Case Studies Does the Minimum Wage Reduce Employment?," Working Papers WR-1142, RAND Corporation.
    16. Bibek Adhikari & Romain Duval & Bingjie Hu & Prakash Loungani, 2018. "Can Reform Waves Turn the Tide? Some Case Studies using the Synthetic Control Method," Open Economies Review, Springer, vol. 29(4), pages 879-910, September.
    17. Kuosmanen, Timo & Zhou, Xun & Eskelinen, Juha & Malo, Pekka, 2021. "Design Flaw of the Synthetic Control Method," MPRA Paper 106328, University Library of Munich, Germany.
    18. Rodríguez-Puello, Gabriel, 2024. "Digging for Trouble? Uncovering the Link Between Mining Booms and Crime," OSF Preprints s8ayp, Center for Open Science.
    19. Cummins Joseph & Miller Douglas L. & Smith Brock & Simon David, 2024. "Matching on Noise: Finite Sample Bias in the Synthetic Control Estimator," Journal of Econometric Methods, De Gruyter, vol. 13(1), pages 67-95, January.
    20. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2020. "Ensuring Stability, Accuracy and Meaningfulness in Synthetic Control Methods: The Regularized SHAP-Distance Method," IREA Working Papers 202005, University of Barcelona, Research Institute of Applied Economics, revised Apr 2020.

    More about this item

    Keywords

    inference; synthetic control; p-hacking; specification searching;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C33 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Models with Panel Data; Spatio-temporal Models

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:78213. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.