IDEAS home Printed from https://ideas.repec.org/a/bpj/statpp/v14y2023i2p205-326n6.html
   My bibliography  Save this article

Evaluating Different Covariate Balancing Methods: A Monte Carlo Simulation

Author

Listed:
  • Fukui Hideki

    (Faculty of Law and Letters, Ehime University, 3 Bunkyo-Cho, Matsuyama, Ehime 790-8577, Japan)

Abstract

We investigate the effectiveness of five weighting and matching techniques, including propensity score matching (PSM), in improving covariate balance and reducing bias when estimating treatment effects in finite-sample situations through Monte Carlo simulations. King and Nielsen (2019. “Why Propensity Scores Should Not Be Used for Matching.” Political Analysis 27 (4): 435–54) argue that pruning observations based on PSM with 1-to-1 greedy matching can worsen rather than improve covariate balance and can increase the bias in the estimates of treatment effects. In our simulations, we observed this phenomenon not only in PSM with 1-to-1 greedy matching but also in other covariate balancing techniques that King and Nielsen (2019. “Why Propensity Scores Should Not Be Used for Matching.” Political Analysis 27 (4): 435–54) recommend as better matching methods, i.e. Mahalanobis distance matching (MDM) and coarsened exact matching (CEM). Regardless of the weighting/matching techniques and the data generation processes in this study, our findings indicate that matching and weighting under extreme caliper or cut-point settings does not improve covariate balance. In addition, once a substantially improved covariate balance is achieved in a given sample, the estimated bias tends to worsen slightly as the covariate balance continues to improve. Moreover, our simulation results suggest that OLS with proper covariates reduces selection bias as well as other weighting and matching methods. The results suggest that when analyzing observational data, it is important to avoid looking for a one-size-fits-all estimator and to identify the appropriate nonexperimental estimator carefully for the sample by thoroughly investigating the available data’s characteristics.

Suggested Citation

  • Fukui Hideki, 2023. "Evaluating Different Covariate Balancing Methods: A Monte Carlo Simulation," Statistics, Politics and Policy, De Gruyter, vol. 14(2), pages 205-326, June.
    • Handle: RePEc:bpj:statpp:v:14:y:2023:i:2:p:205-326:n:6
    DOI: 10.1515/spp-2022-0019
    as

    Download full text from publisher

    File URL: https://doi.org/10.1515/spp-2022-0019
    Download Restriction: For access to full text, subscription to the journal or payment for the individual article is required.
    File URL: https://libkey.io/10.1515/spp-2022-0019?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    2. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    3. Hainmueller, Jens, 2012. "Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies," Political Analysis, Cambridge University Press, vol. 20(1), pages 25-46, January.
    4. Steven Lehrer & Gregory Kordas, 2013. "Matching using semiparametric propensity scores," Empirical Economics, Springer, vol. 44(1), pages 13-45, February.
    5. Markus Frlich, 2004. "Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 77-90, February.
    6. King, Gary & Nielsen, Richard, 2019. "Why Propensity Scores Should Not Be Used for Matching," Political Analysis, Cambridge University Press, vol. 27(4), pages 435-454, October.
    7. Hainmueller, Jens & Xu, Yiqing, 2013. "ebalance: A Stata Package for Entropy Balancing," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 54(i07).
    8. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    9. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    10. Jeffrey Smith, 2000. "A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies," Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 136(III), pages 247-268, September.
    11. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    12. Heckman, James J. & Robb, Richard Jr., 1985. "Alternative methods for evaluating the impact of interventions : An overview," Journal of Econometrics, Elsevier, vol. 30(1-2), pages 239-267.
    13. Rosenbaum, Paul R., 2010. "Design Sensitivity and Efficiency in Observational Studies," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 692-702.
    14. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 65(2), pages 261-294.
    15. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, January.
    16. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    17. Wooldridge, Jeffrey M., 2016. "Should instrumental variables be used as matching variables?," Research in Economics, Elsevier, vol. 70(2), pages 232-237.
    18. Arceneaux, Kevin & Gerber, Alan S. & Green, Donald P., 2006. "Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment," Political Analysis, Cambridge University Press, vol. 14(1), pages 37-62, January.
    19. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Seonho Shin, 2022. "Evaluating the Effect of the Matching Grant Program for Refugees: An Observational Study Using Matching, Weighting, and the Mantel-Haenszel Test," Journal of Labor Research, Springer, vol. 43(1), pages 103-133, March.
    2. Zhexiao Lin & Fang Han, 2022. "On regression-adjusted imputation estimators of the average treatment effect," Papers 2212.05424, arXiv.org, revised Jan 2023.
    3. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    4. Riccardo Turati, 2025. "Networks abroad and culture: global individual-level evidence," Journal of Population Economics, Springer;European Society for Population Economics, vol. 38(1), pages 1-42, March.
    5. Eliasson, Kent, 2006. "How Robust is the Evidence on the Returns to College Choice? Results Using Swedish Administrative Data," Umeå Economic Studies 692, Umeå University, Department of Economics.
    6. Christian Volpe Martincus & Jerónimo Carballo, 2010. "Entering new country and product markets: does export promotion help?," Review of World Economics (Weltwirtschaftliches Archiv), Springer;Institut für Weltwirtschaft (Kiel Institute for the World Economy), vol. 146(3), pages 437-467, September.
    7. Jan Stede, 2019. "Do Energy Efficiency Networks Save Energy? Evidence from German Plant-Level Data," Discussion Papers of DIW Berlin 1813, DIW Berlin, German Institute for Economic Research.
    8. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," CESifo Working Paper Series 7268, CESifo.
    9. Ferraro, Paul J. & Miranda, Juan José, 2014. "The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 344-365.
    10. Steven Lehrer & Gregory Kordas, 2013. "Matching using semiparametric propensity scores," Empirical Economics, Springer, vol. 44(1), pages 13-45, February.
    11. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    12. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," SOEPpapers on Multidisciplinary Panel Data Research 1004, DIW Berlin, The German Socio-Economic Panel (SOEP).
    13. Riccardo D'Alberto & Francesco Pagliacci & Matteo Zavalloni, 2023. "A socioeconomic impact assessment of three Italian national parks," Journal of Regional Science, Wiley Blackwell, vol. 63(1), pages 114-147, January.
    14. Verena Lauber & Johanna Storck, 2016. "Helping with the Kids? How Family-Friendly Workplaces Affect Parental Well-Being and Behavior," Discussion Papers of DIW Berlin 1630, DIW Berlin, German Institute for Economic Research.
    15. Verena Lauber & Johanna Storck, 2016. "Helping with the Kids? How Family-Friendly Workplaces Affect Parental Well-Being and Behavior," SOEPpapers on Multidisciplinary Panel Data Research 883, DIW Berlin, The German Socio-Economic Panel (SOEP).
    16. Franz R. Hahn & Werner Hölzl & Claudia Kwapil, 2016. "The Credit Channel and the Role of Monetary Policy Before, During and After the Global Financial Crisis. A Micro Data Approach to the Analysis of Bank-firm Relationships," WIFO Studies, WIFO, number 59233, August.
    17. Eliasson, Kent, 2006. "College Choice And Earnings Among University Graduates In Sweden," Umeå Economic Studies 693, Umeå University, Department of Economics.
    18. Ruhose, Jens & Thomsen, Stephan L. & Weilage, Insa, 2019. "The benefits of adult learning: Work-related training, social capital, and earnings," Economics of Education Review, Elsevier, vol. 72(C), pages 166-186.
    19. Uehleke, Reinhard & Petrick, Martin & Hüttel, Silke, 2022. "Evaluations of agri-environmental schemes based on observational farm data: The importance of covariate selection," Land Use Policy, Elsevier, vol. 114(C).
    20. Christian Volpe Martincus & Jerónimo Carballo, 2010. "Is Export Promotion Effective in Developing Countries? Firm-Level Evidence on the Intensive and Extensive Margins of Exports," IDB Publications (Working Papers) 36763, Inter-American Development Bank.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:statpp:v:14:y:2023:i:2:p:205-326:n:6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.degruyter.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.