IDEAS home Printed from https://ideas.repec.org/a/eee/ecmode/v107y2022ics0264999321003187.html
   My bibliography  Save this article

No such thing as the perfect match: Bayesian Model Averaging for treatment evaluation

Author

Listed:
  • Lucchetti, Riccardo
  • Pedini, Luca
  • Pigini, Claudia

Abstract

Propensity Score Matching is a popular approach to evaluate treatment effects in observational studies. Regrettably, practitioners often overlook the issue of model uncertainty and its consequences when building the propensity score model. We tackle this problem by Bayesian Model Averaging (BMA) with an application to the 2014 Italian tax credit reform (the so-called “Renzi bonus”). Model uncertainty has a dramatic impact on the estimated treatment effects: using standard model selection procedures may lead to choosing equally defensible models that, however, produce substantially heterogeneous results. By using BMA-based estimates, we find a much more coherent picture: significant effect of the rebate on food consumption only for liquidity constrained households, in line with most recent literature.

Suggested Citation

  • Lucchetti, Riccardo & Pedini, Luca & Pigini, Claudia, 2022. "No such thing as the perfect match: Bayesian Model Averaging for treatment evaluation," Economic Modelling, Elsevier, vol. 107(C).
  • Handle: RePEc:eee:ecmode:v:107:y:2022:i:c:s0264999321003187
    DOI: 10.1016/j.econmod.2021.105729
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0264999321003187
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.econmod.2021.105729?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Horvath, Roman, 2011. "Research & development and growth: A Bayesian model averaging analysis," Economic Modelling, Elsevier, vol. 28(6), pages 2669-2673.
    2. Kitagawa, Toru & Muris, Chris, 2016. "Model averaging in semiparametric estimation of treatment effects," Journal of Econometrics, Elsevier, vol. 193(1), pages 271-289.
    3. William A. Brock & Steven N. Durlauf, 2015. "On Sturdy Policy Evaluation," The Journal of Legal Studies, University of Chicago Press, vol. 44(S2), pages 447-473.
    4. Tullio Jappelli & Luigi Pistaferri, 2010. "The Consumption Response to Income Changes," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 479-506, September.
    5. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018. "Double/debiased machine learning for treatment and structural parameters," Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
    6. Nicholas S. Souleles & Jonathan A. Parker & David S. Johnson, 2006. "Household Expenditure and the Income Tax Rebates of 2001," American Economic Review, American Economic Association, vol. 96(5), pages 1589-1610, December.
    7. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    8. David Kaplan & Jianshen Chen, 2012. "A Two-Step Bayesian Approach for Propensity Score Analysis: Simulations and Case Study," Psychometrika, Springer;The Psychometric Society, vol. 77(3), pages 581-609, July.
    9. Wilcox, David W, 1989. "Social Security Benefits, Consumption Expenditure, and the Life Cycle Hypothesis," Journal of Political Economy, University of Chicago Press, vol. 97(2), pages 288-304, April.
    10. Heim, Bradley T., 2007. "The Effect of Tax Rebates on Consumption Expenditures: Evidence From State Tax Rebates," National Tax Journal, National Tax Association;National Tax Journal, vol. 60(4), pages 685-710, December.
    11. Mark F. J. Steel, 2020. "Model Averaging and Its Use in Economics," Journal of Economic Literature, American Economic Association, vol. 58(3), pages 644-719, September.
    12. Carmen Fernandez & Eduardo Ley & Mark F. J. Steel, 2001. "Model uncertainty in cross-country growth regressions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 16(5), pages 563-576.
    13. William A. Brock & Steven N. Durlauf & Kenneth D. West, 2003. "Policy Evaluation in Uncertain Economic Environments," Brookings Papers on Economic Activity, Economic Studies Program, The Brookings Institution, vol. 34(1), pages 235-322.
    14. Sumit Agarwal & Chunlin Liu & Nicholas S. Souleles, 2007. "The Reaction of Consumer Spending and Debt to Tax Rebates-Evidence from Consumer Credit Data," Journal of Political Economy, University of Chicago Press, vol. 115(6), pages 986-1019, December.
    15. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    16. Chris Chatfield, 1995. "Model Uncertainty, Data Mining and Statistical Inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(3), pages 419-444, May.
    17. Fernandez, Carmen & Ley, Eduardo & Steel, Mark F. J., 2001. "Benchmark priors for Bayesian model averaging," Journal of Econometrics, Elsevier, vol. 100(2), pages 381-427, February.
    18. Jonathan A. Parker, 1999. "The Reaction of Household Consumption to Predictable Changes in Social Security Taxes," American Economic Review, American Economic Association, vol. 89(4), pages 959-973, September.
    19. Millimet, Daniel L. & Tchernis, Rusty, 2009. "On the Specification of Propensity Scores, With Applications to the Analysis of Trade Policies," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(3), pages 397-415.
    20. Jonathan A. Parker & Nicholas S. Souleles & David S. Johnson & Robert McClelland, 2013. "Consumer Spending and the Economic Stimulus Payments of 2008," American Economic Review, American Economic Association, vol. 103(6), pages 2530-2553, October.
    21. Matthew D. Shapiro & Joel Slemrod, 2003. "Did the 2001 Tax Rebate Stimulate Spending? Evidence from Taxpayer Surveys," NBER Chapters, in: Tax Policy and the Economy, Volume 17, pages 83-110, National Bureau of Economic Research, Inc.
    22. Roman Horváth, 2011. "Research & Development and Long-Term Economic Growth: A Bayesian Model Averaging Analysis," Working Papers IES 2011/19, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Jun 2011.
    23. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2014. "Inference on Treatment Effects after Selection among High-Dimensional Controlsâ€," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(2), pages 608-650.
    24. Broda, Christian & Parker, Jonathan A., 2014. "The Economic Stimulus Payments of 2008 and the aggregate demand for consumption," Journal of Monetary Economics, Elsevier, vol. 68(S), pages 20-36.
    25. Greg Kaplan & Giovanni L. Violante, 2014. "A Model of the Consumption Response to Fiscal Stimulus Payments," Econometrica, Econometric Society, vol. 82(4), pages 1199-1239, July.
    26. David Kaplan & Jianshen Chen, 2012. "Erratum to: A Two-Step Bayesian Approach for Propensity Score Analysis: Simulations and Case Study," Psychometrika, Springer;The Psychometric Society, vol. 77(3), pages 610-610, July.
    27. Souleles, Nicholas S., 2002. "Consumer response to the Reagan tax cuts," Journal of Public Economics, Elsevier, vol. 85(1), pages 99-120, July.
    28. Eriṣ, Mehmet N. & Ulaṣan, Bülent, 2013. "Trade openness and economic growth: Bayesian model averaging estimate of cross-country growth regressions," Economic Modelling, Elsevier, vol. 33(C), pages 867-883.
    29. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    30. Matthew D. Shapiro & Joel Slemrod, 2009. "Did the 2008 Tax Rebates Stimulate Spending?," American Economic Review, American Economic Association, vol. 99(2), pages 374-379, May.
    31. Bryson, Alex & Dorsett, Richard & Purdon, Susan, 2002. "The use of propensity score matching in the evaluation of active labour market policies," LSE Research Online Documents on Economics 4993, London School of Economics and Political Science, LSE Library.
    32. A. Belloni & V. Chernozhukov & I. Fernández‐Val & C. Hansen, 2017. "Program Evaluation and Causal Inference With High‐Dimensional Data," Econometrica, Econometric Society, vol. 85, pages 233-298, January.
    33. Xavier Sala-I-Martin & Gernot Doppelhofer & Ronald I. Miller, 2004. "Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach," American Economic Review, American Economic Association, vol. 94(4), pages 813-835, September.
    34. Andini, Monica & Ciani, Emanuele & de Blasio, Guido & D'Ignazio, Alessio & Salvestrini, Viola, 2018. "Targeting with machine learning: An application to a tax rebate program in Italy," Journal of Economic Behavior & Organization, Elsevier, vol. 156(C), pages 86-102.
    35. Matthew D. Shapiro & Joel Slemrod, 2003. "Consumer Response to Tax Rebates," American Economic Review, American Economic Association, vol. 93(1), pages 381-396, March.
    36. Jappelli, Tullio & Pistaferri, Luigi, 2017. "The Economics of Consumption: Theory and Evidence," OUP Catalogue, Oxford University Press, number 9780199383153.
    37. Matthew Cefalu & Francesca Dominici & Nils Arvold & Giovanni Parmigiani, 2017. "Model averaged double robust estimation," Biometrics, The International Biometric Society, vol. 73(2), pages 410-421, June.
    38. Nicholas S. Souleles, 1999. "The Response of Household Consumption to Income Tax Refunds," American Economic Review, American Economic Association, vol. 89(4), pages 947-958, September.
    39. Corwin Matthew Zigler & Francesca Dominici, 2014. "Uncertainty in Propensity Score Estimation: Bayesian Methods for Variable Selection and Model-Averaged Causal Effects," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(505), pages 95-107, March.
    40. Man, Georg, 2015. "Competition and the growth of nations: International evidence from Bayesian model averaging," Economic Modelling, Elsevier, vol. 51(C), pages 491-501.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Boonman, Tjeerd M., 2023. "Portfolio capital flows before and after the Global Financial Crisis," Economic Modelling, Elsevier, vol. 127(C).
    2. Boonman, Tjeerd, 2023. "Have drivers of portfolio capital flows changed since the Global Financial Crisis?," MPRA Paper 116507, University Library of Munich, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Riccardo Lucchetti & Luca Pedini & Claudia Pigini, 2021. "Bayesian Model Averaging For Propensity Score Matching In Tax Rebate," Working Papers 457, Universita' Politecnica delle Marche (I), Dipartimento di Scienze Economiche e Sociali.
    2. Fuchs-Schündeln, N. & Hassan, T.A., 2016. "Natural Experiments in Macroeconomics," Handbook of Macroeconomics, in: J. B. Taylor & Harald Uhlig (ed.), Handbook of Macroeconomics, edition 1, volume 2, chapter 0, pages 923-1012, Elsevier.
    3. Campbell, J.R. & Hercowitz, Zvi, 2018. "Liquidity Constraints of the Middle Class (revision of CentER DP 2015-009)," Discussion Paper 2018-039, Tilburg University, Center for Economic Research.
    4. Broda, Christian & Parker, Jonathan A., 2014. "The Economic Stimulus Payments of 2008 and the aggregate demand for consumption," Journal of Monetary Economics, Elsevier, vol. 68(S), pages 20-36.
    5. María Pía Olivero & Mikheil Dvalishvili, 2023. "What do fiscal stimulus packages mean for household debt?," Contemporary Economic Policy, Western Economic Association International, vol. 41(2), pages 282-299, April.
    6. Radermacher, Jan W., 2023. "Mamma Mia! Revealing hidden heterogeneity by PCA-biplot: MPC puzzle for Italy's elderly poor," SAFE Working Paper Series 382, Leibniz Institute for Financial Research SAFE.
    7. Anna Sokolova, 2023. "Marginal Propensity to Consume and Unemployment: a Meta-analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 813-846, December.
    8. Jonathan A. Parker & Nicholas S. Souleles & David S. Johnson & Robert McClelland, 2013. "Consumer Spending and the Economic Stimulus Payments of 2008," American Economic Review, American Economic Association, vol. 103(6), pages 2530-2553, October.
    9. Arna Vardardottir & Michaela Pagel, 2016. "The Liquid Hand-to-Mouth: Evidence from a Personal Finance Management Software," 2016 Meeting Papers 789, Society for Economic Dynamics.
    10. David Cashin & Jamie Lenney & Byron Lutz & William Peterman, 2018. "Fiscal policy and aggregate demand in the USA before, during, and following the Great Recession," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 25(6), pages 1519-1558, December.
    11. Gelman, Michael & Kariv, Shachar & Shapiro, Matthew D. & Silverman, Dan & Tadelis, Steven, 2020. "How individuals respond to a liquidity shock: Evidence from the 2013 government shutdown," Journal of Public Economics, Elsevier, vol. 189(C).
    12. Grant Graziani & Wilbert Van der Klaauw & Basit Zafar, 2013. "A boost in the paycheck: survey evidence on workers’ response to the 2011 payroll tax cuts," Staff Reports 592, Federal Reserve Bank of New York.
    13. Tullio Jappelli & Luigi Pistaferri, 2014. "Fiscal Policy and MPC Heterogeneity," American Economic Journal: Macroeconomics, American Economic Association, vol. 6(4), pages 107-136, October.
    14. Kanishka Misra & Vishal Singh & Qianyun (Poppy) Zhang, 2022. "Frontiers: Impact of Stay-at-Home-Orders and Cost-of-Living on Stimulus Response: Evidence from the CARES Act," Marketing Science, INFORMS, vol. 41(2), pages 211-229, March.
    15. Grant Graziani & Wilbert van der Klaauw & Basit Zafar, 2016. "Workers' Spending Response to the 2011 Payroll Tax Cuts," American Economic Journal: Economic Policy, American Economic Association, vol. 8(4), pages 124-159, November.
    16. Alan J. Auerbach & William G. Gale, 2009. "Activist fiscal policy to stabilize economic activity," Proceedings - Economic Policy Symposium - Jackson Hole, Federal Reserve Bank of Kansas City, pages 327-374.
    17. Frederik Plesner Lyngse, 2020. "Liquidity Constraints and Demand for Healthcare: Evidence from Danish Welfare Recipients," Papers 2010.14651, arXiv.org.
    18. Frederik Plesner Lyngse, 2020. "Liquidity Constraints and Demand for Healthcare: Evidence from Danish Welfare Recipients," CEBI working paper series 20-28, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    19. Bernard, René, 2023. "Mental accounting and the marginal propensity to consume," Discussion Papers 13/2023, Deutsche Bundesbank.
    20. Mark F. J. Steel, 2020. "Model Averaging and Its Use in Economics," Journal of Economic Literature, American Economic Association, vol. 58(3), pages 644-719, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecmode:v:107:y:2022:i:c:s0264999321003187. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/30411 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.