IDEAS home Printed from https://ideas.repec.org/a/bpj/causin/v11y2023i1p15n1028.html
   My bibliography  Save this article

All models are wrong, but which are useful? Comparing parametric and nonparametric estimation of causal effects in finite samples

Author

Listed:
  • Rudolph Kara E.

    (Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, USA)

  • Williams Nicholas T.

    (Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, USA)

  • Miles Caleb H.

    (Department of Biostatistics, Mailman School of Public Health, Columbia University, New York, USA)

  • Antonelli Joseph

    (Department of Statistics, University of Florida, Gainesville, USA)

  • Diaz Ivan

    (Division of Biostatistics, Department of Population Health, New York University Grossman School of Medicine, New York, USA)

Abstract

There is a long-standing debate in the statistical, epidemiological, and econometric fields as to whether nonparametric estimation that uses machine learning in model fitting confers any meaningful advantage over simpler, parametric approaches in finite sample estimation of causal effects. We address the question: when estimating the effect of a treatment on an outcome, how much does the choice of nonparametric vs parametric estimation matter? Instead of answering this question with simulations that reflect a few chosen data scenarios, we propose a novel approach to compare estimators across a large number of data-generating mechanisms drawn from nonparametric models with semi-informative priors. We apply this proposed approach and compare the performance of two nonparametric estimators (Bayesian adaptive regression tree and a targeted minimum loss-based estimator) to two parametric estimators (a logistic regression-based plug-in estimator and a propensity score estimator) in terms of estimating the average treatment effect across thousands of data-generating mechanisms. We summarize performance in terms of bias, confidence interval coverage, and mean squared error. We find that the two nonparametric estimators can substantially reduce bias as compared to the two parametric estimators in large-sample settings characterized by interactions and nonlinearities while compromising very little in terms of performance even in simple, small-sample settings.

Suggested Citation

  • Rudolph Kara E. & Williams Nicholas T. & Miles Caleb H. & Antonelli Joseph & Diaz Ivan, 2023. "All models are wrong, but which are useful? Comparing parametric and nonparametric estimation of causal effects in finite samples," Journal of Causal Inference, De Gruyter, vol. 11(1), pages 1-15.
  • Handle: RePEc:bpj:causin:v:11:y:2023:i:1:p:15:n:1028
    DOI: 10.1515/jci-2023-0022
    as

    Download full text from publisher

    File URL: https://doi.org/10.1515/jci-2023-0022
    Download Restriction: no

    File URL: https://libkey.io/10.1515/jci-2023-0022?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. van de Walle, Dominique & Mu, Ren, 2007. "Fungibility and the flypaper effect of project aid: Micro-evidence for Vietnam," Journal of Development Economics, Elsevier, vol. 84(2), pages 667-685, November.
    2. Lechner, Michael, 2018. "Modified Causal Forests for Estimating Heterogeneous Causal Effects," IZA Discussion Papers 12040, Institute of Labor Economics (IZA).
    3. Alexandre Belloni & Victor Chernozhukov & Denis Chetverikov & Christian Hansen & Kengo Kato, 2018. "High-dimensional econometrics and regularized GMM," CeMMAP working papers CWP35/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    4. Turner, Alex J. & Fichera, Eleonora & Sutton, Matt, 2021. "The effects of in-utero exposure to influenza on mental health and mortality risk throughout the life-course," Economics & Human Biology, Elsevier, vol. 43(C).
    5. Roxana Elena Manea, 2021. "School Feeding Programmes, Education and Food Security in Rural Malawi," CIES Research Paper series 63-2020, Centre for International Environmental Studies, The Graduate Institute.
    6. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    7. José de Sousa & Guillaume Hollard, 2021. "From Micro to Macro Gender Differences: Evidence from Field Tournaments," Post-Print hal-03389151, HAL.
    8. repec:ags:jrapmc:122316 is not listed on IDEAS
    9. Gunther Bensch & Jörg Peters, 2013. "Alleviating Deforestation Pressures? Impacts of Improved Stove Dissemination on Charcoal Consumption in Urban Senegal," Land Economics, University of Wisconsin Press, vol. 89(4), pages 676-698.
    10. G. Miller & Yuriy Pylypchuk, 2014. "Marital Status, Spousal Characteristics, and the Use of Preventive Care," Journal of Family and Economic Issues, Springer, vol. 35(3), pages 323-338, September.
    11. Kitagawa, Toru & Muris, Chris, 2016. "Model averaging in semiparametric estimation of treatment effects," Journal of Econometrics, Elsevier, vol. 193(1), pages 271-289.
    12. Michael Lechner & Ruth Miquel & Conny Wunsch, 2011. "Long‐Run Effects Of Public Sector Sponsored Training In West Germany," Journal of the European Economic Association, European Economic Association, vol. 9(4), pages 742-784, August.
    13. Li, Linjie & Liu, Xiaming & Yuan, Dong & Yu, Miaojie, 2017. "Does outward FDI generate higher productivity for emerging economy MNEs? – Micro-level evidence from Chinese manufacturing firms," International Business Review, Elsevier, vol. 26(5), pages 839-854.
    14. Olivier Dagnelie & Philippe Lemay‐Boucher, 2012. "Rosca Participation in Benin: A Commitment Issue," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 74(2), pages 235-252, April.
    15. Frölich, Markus & Lechner, Michael, 2010. "Exploiting Regional Treatment Intensity for the Evaluation of Labor Market Policies," Journal of the American Statistical Association, American Statistical Association, vol. 105(491), pages 1014-1029.
    16. Ralf Becker & Maggy Fostier, 2015. "Evaluating non-compulsory educational interventions - the case of peer assisted study groups," Economics Discussion Paper Series 1509, Economics, The University of Manchester.
    17. Katie Meara & Francesco Pastore & Allan Webster, 2020. "The gender pay gap in the USA: a matching study," Journal of Population Economics, Springer;European Society for Population Economics, vol. 33(1), pages 271-305, January.
    18. Ronald Mincy & Jennifer Hill & Marilyn Sinkewicz, 2009. "Marriage: Cause or mere indicator of future earnings growth?," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 28(3), pages 417-439.
    19. Lou, Jiehong & Shen, Xingchi & Niemeier, Deb, 2020. "Are stay-at-home orders more difficult to follow for low-income groups?," Journal of Transport Geography, Elsevier, vol. 89(C).
    20. Gernandt, Johannes & Maier, Michael & Pfeiffer, Friedhelm & Rat-Wirtzler, Julie, 2006. "Distributional effects of the high school degree in Germany," ZEW Discussion Papers 06-088, ZEW - Leibniz Centre for European Economic Research.
    21. Grau, Nicolas & Hojman, Daniel & Mizala, Alejandra, 2018. "School closure and educational attainment: Evidence from a market-based system," Economics of Education Review, Elsevier, vol. 65(C), pages 1-17.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:causin:v:11:y:2023:i:1:p:15:n:1028. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.degruyter.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.