IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v39y2014i6p612-636.html
   My bibliography  Save this article

A Two-Stage Propensity Score Matching Strategy for Treatment Effect Estimation in a Multisite Observational Study

Author

Listed:
  • Jordan H. Rickles

    (American Institutes for Research)

  • Michael Seltzer

    (University of California, Los Angeles)

Abstract

When nonrandom treatments occur across sites, within-site matching (WM) is often desirable. This approach, however, can significantly reduce treatment group sample size and exclude substantively important subgroups. To limit these drawbacks, we extend a matching approach developed by Stuart and Rubin to a multisite study. We demonstrate the proposed method through a multisite analysis of algebra enrollment effects in 50 middle schools, where within each school students are assigned to algebra or pre-algebra and test the utility of the proposed method with a simulation study. The results document the method’s conceptual appeal and indicate that two-stage matching is a viable alternative to strict WM or matching that ignores the nested data structure (pooled matching).

Suggested Citation

  • Jordan H. Rickles & Michael Seltzer, 2014. "A Two-Stage Propensity Score Matching Strategy for Treatment Effect Estimation in a Multisite Observational Study," Journal of Educational and Behavioral Statistics, , vol. 39(6), pages 612-636, December.
    • Handle: RePEc:sae:jedbes:v:39:y:2014:i:6:p:612-636
    DOI: 10.3102/1076998614559748
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998614559748
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998614559748?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Arpino, Bruno & Mealli, Fabrizia, 2011. "The specification of the propensity score in multilevel observational studies," Computational Statistics & Data Analysis, Elsevier, vol. 55(4), pages 1770-1780, April.
    2. Shadish, William R. & Clark, M. H. & Steiner, Peter M., 2008. "Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1334-1344.
    3. Donald B. Rubin, 2005. "Causal Inference Using Potential Outcomes: Design, Modeling, Decisions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 322-331, March.
    4. Thomas D. Cook & William R. Shadish & Vivian C. Wong, 2008. "Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 27(4), pages 724-750.
    5. Sean F. Reardon & Stephen W. Raudenbush, 2009. "Assumptions of Value-Added Models for Estimating School Effects," Education Finance and Policy, MIT Press, vol. 4(4), pages 492-519, October.
    6. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cousineau, Martin & Verter, Vedat & Murphy, Susan A. & Pineau, Joelle, 2023. "Estimating causal effects with optimization-based methods: A review and empirical comparison," European Journal of Operational Research, Elsevier, vol. 304(2), pages 367-380.
    2. Martin Cousineau & Vedat Verter & Susan A. Murphy & Joelle Pineau, 2022. "Estimating causal effects with optimization-based methods: A review and empirical comparison," Papers 2203.00097, arXiv.org.
    3. Peter M. Steiner, 2011. "Propensity Score Methods for Causal Inference: On the Relative Importance of Covariate Selection, Reliable Measurement, and Choice of Propensity Score Technique," Working Papers 09, AlmaLaurea Inter-University Consortium.
    4. Vivian C. Wong & Peter M. Steiner & Kylie L. Anglin, 2018. "What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?," Evaluation Review, , vol. 42(2), pages 147-175, April.
    5. Katherine Baicker & Theodore Svoronos, 2019. "Testing the Validity of the Single Interrupted Time Series Design," NBER Working Papers 26080, National Bureau of Economic Research, Inc.
    6. Ragni, Alessandra & Ippolito, Daniel & Masci, Chiara, 2024. "Assessing the impact of hybrid teaching on students’ academic performance via multilevel propensity score-based techniques," Socio-Economic Planning Sciences, Elsevier, vol. 92(C).
    7. Ron Zimmer & Brian Gill & Jonathon Attridge & Kaitlin Obenauf, 2014. "Charter School Authorizers and Student Achievement," Education Finance and Policy, MIT Press, vol. 9(1), pages 59-85, January.
    8. Robin Jacob & Marie-Andree Somers & Pei Zhu & Howard Bloom, 2016. "The Validity of the Comparative Interrupted Time Series Design for Evaluating the Effect of School-Level Interventions," Evaluation Review, , vol. 40(3), pages 167-198, June.
    9. Terri J. Sabol & Robert C. Pianta, 2014. "Do Standard Measures of Preschool Quality Used in Statewide Policy Predict School Readiness?," Education Finance and Policy, MIT Press, vol. 9(2), pages 116-164, March.
    10. Gary Henry & Roderick Rose & Doug Lauen, 2014. "Are value-added models good enough for teacher evaluations? Assessing commonly used models with simulated and actual data," Investigaciones de Economía de la Educación volume 9, in: Adela García Aracil & Isabel Neira Gómez (ed.), Investigaciones de Economía de la Educación 9, edition 1, volume 9, chapter 20, pages 383-405, Asociación de Economía de la Educación.
    11. Steven M. Smith, 2019. "The Relative Economic Merits of Alternative Water Rights," Working Papers 2019-08, Colorado School of Mines, Division of Economics and Business.
    12. Zhexiao Lin & Peng Ding & Fang Han, 2023. "Estimation Based on Nearest Neighbor Matching: From Density Ratio to Average Treatment Effect," Econometrica, Econometric Society, vol. 91(6), pages 2187-2217, November.
    13. K. C. Culver & Nathaniel Bray & John Braxton, 2024. "On My Honor: A Quasi-Experimental Analysis of Honors Students’ Perceptions of Workload and Cognitive Challenge," Research in Higher Education, Springer;Association for Institutional Research, vol. 65(4), pages 679-704, June.
    14. Ferraro, Paul J. & Miranda, Juan José, 2014. "The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 344-365.
    15. Hoang, Anh Tuan & Sandro Nižetić, & Olcer, Aykut I. & Ong, Hwai Chyuan & Chen, Wei-Hsin & Chong, Cheng Tung & Thomas, Sabu & Bandh, Suhaib A. & Nguyen, Xuan Phuong, 2021. "Impacts of COVID-19 pandemic on the global energy system and the shift progress to renewable energy: Opportunities, challenges, and policy implications," Energy Policy, Elsevier, vol. 154(C).
    16. Anna Haskins, 2013. "Mass Imprisonment and the Intergenerational Transmission of Disadvantage: Paternal Incarceration and Children’s Cognitive Skill Development," Working Papers wp13-15-ff, Princeton University, School of Public and International Affairs, Center for Research on Child Wellbeing..
    17. Ben Weidmann & Luke Miratrix, 2021. "Lurking Inferential Monsters? Quantifying Selection Bias In Evaluations Of School Programs," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 40(3), pages 964-986, June.
    18. Miquel-Àngel Garcia-López & Albert Solé-Ollé & Elisabet Viladecans Marsal, 2014. "Do Land Use Policies Follow Road Construction," CESifo Working Paper Series 4672, CESifo Group Munich.
    19. Brittany Tarufelli & Ben Gilbert, 2019. "Leakage in Regional Climate Policy? Implications of Electricity Market Design," Working Papers 2019-07, Colorado School of Mines, Division of Economics and Business, revised Dec 2021.
    20. Haotian Zhong & Wei Li & Marlon G Boarnet, 2021. "A two-dimensional propensity score matching method for longitudinal quasi-experimental studies: A focus on travel behavior and the built environment," Environment and Planning B, , vol. 48(7), pages 2110-2122, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:39:y:2014:i:6:p:612-636. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.