IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v39y2014i6p612-636.html
   My bibliography  Save this article

A Two-Stage Propensity Score Matching Strategy for Treatment Effect Estimation in a Multisite Observational Study

Author

Listed:
  • Jordan H. Rickles

    (American Institutes for Research)

  • Michael Seltzer

    (University of California, Los Angeles)

Abstract

When nonrandom treatments occur across sites, within-site matching (WM) is often desirable. This approach, however, can significantly reduce treatment group sample size and exclude substantively important subgroups. To limit these drawbacks, we extend a matching approach developed by Stuart and Rubin to a multisite study. We demonstrate the proposed method through a multisite analysis of algebra enrollment effects in 50 middle schools, where within each school students are assigned to algebra or pre-algebra and test the utility of the proposed method with a simulation study. The results document the method’s conceptual appeal and indicate that two-stage matching is a viable alternative to strict WM or matching that ignores the nested data structure (pooled matching).

Suggested Citation

  • Jordan H. Rickles & Michael Seltzer, 2014. "A Two-Stage Propensity Score Matching Strategy for Treatment Effect Estimation in a Multisite Observational Study," Journal of Educational and Behavioral Statistics, , vol. 39(6), pages 612-636, December.
    • Handle: RePEc:sae:jedbes:v:39:y:2014:i:6:p:612-636
    DOI: 10.3102/1076998614559748
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998614559748
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998614559748?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Shadish, William R. & Clark, M. H. & Steiner, Peter M., 2008. "Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1334-1344.
    2. Arpino, Bruno & Mealli, Fabrizia, 2011. "The specification of the propensity score in multilevel observational studies," Computational Statistics & Data Analysis, Elsevier, vol. 55(4), pages 1770-1780, April.
    3. Thomas D. Cook & William R. Shadish & Vivian C. Wong, 2008. "Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 27(4), pages 724-750.
    4. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    5. Donald B. Rubin, 2005. "Causal Inference Using Potential Outcomes: Design, Modeling, Decisions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 322-331, March.
    6. Sean F. Reardon & Stephen W. Raudenbush, 2009. "Assumptions of Value-Added Models for Estimating School Effects," Education Finance and Policy, MIT Press, vol. 4(4), pages 492-519, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cousineau, Martin & Verter, Vedat & Murphy, Susan A. & Pineau, Joelle, 2023. "Estimating causal effects with optimization-based methods: A review and empirical comparison," European Journal of Operational Research, Elsevier, vol. 304(2), pages 367-380.
    2. Martin Cousineau & Vedat Verter & Susan A. Murphy & Joelle Pineau, 2022. "Estimating causal effects with optimization-based methods: A review and empirical comparison," Papers 2203.00097, arXiv.org.
    3. Peter M. Steiner, 2011. "Propensity Score Methods for Causal Inference: On the Relative Importance of Covariate Selection, Reliable Measurement, and Choice of Propensity Score Technique," Working Papers 09, AlmaLaurea Inter-University Consortium.
    4. Vivian C. Wong & Peter M. Steiner & Kylie L. Anglin, 2018. "What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?," Evaluation Review, , vol. 42(2), pages 147-175, April.
    5. Ron Zimmer & Brian Gill & Jonathon Attridge & Kaitlin Obenauf, 2014. "Charter School Authorizers and Student Achievement," Education Finance and Policy, MIT Press, vol. 9(1), pages 59-85, January.
    6. Gary Henry & Roderick Rose & Doug Lauen, 2014. "Are value-added models good enough for teacher evaluations? Assessing commonly used models with simulated and actual data," Investigaciones de Economía de la Educación volume 9, in: Adela García Aracil & Isabel Neira Gómez (ed.), Investigaciones de Economía de la Educación 9, edition 1, volume 9, chapter 20, pages 383-405, Asociación de Economía de la Educación.
    7. Steven M. Smith, 2019. "The Relative Economic Merits of Alternative Water Rights," Working Papers 2019-08, Colorado School of Mines, Division of Economics and Business.
    8. Ferraro, Paul J. & Miranda, Juan José, 2014. "The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 344-365.
    9. Anna Haskins, 2013. "Mass Imprisonment and the Intergenerational Transmission of Disadvantage: Paternal Incarceration and Children’s Cognitive Skill Development," Working Papers wp13-15-ff, Princeton University, School of Public and International Affairs, Center for Research on Child Wellbeing..
    10. Miquel-Àngel Garcia-López & Albert Solé-Ollé & Elisabet Viladecans Marsal, 2014. "Do Land Use Policies Follow Road Construction," CESifo Working Paper Series 4672, CESifo Group Munich.
    11. Brittany Tarufelli & Ben Gilbert, 2019. "Leakage in Regional Climate Policy? Implications of Electricity Market Design," Working Papers 2019-07, Colorado School of Mines, Division of Economics and Business, revised Dec 2021.
    12. Haotian Zhong & Wei Li & Marlon G Boarnet, 2021. "A two-dimensional propensity score matching method for longitudinal quasi-experimental studies: A focus on travel behavior and the built environment," Environment and Planning B, , vol. 48(7), pages 2110-2122, September.
    13. Bunte, Jonas B. & Desai, Harsh & Gbala, Kanio & Parks, Bradley & Runfola, Daniel Miller, 2018. "Natural resource sector FDI, government policy, and economic growth: Quasi-experimental evidence from Liberia," World Development, Elsevier, vol. 107(C), pages 151-162.
    14. Christoph Wunder & Johannes Schwarze, 2014. "Is Posner Right? An Empirical Test of the Posner Argument for Transferring Health Spending from Old Women to Old Men," Journal of Happiness Studies, Springer, vol. 15(6), pages 1239-1257, December.
    15. Yonatan Eyal, 2020. "Self-Assessment Variables as a Source of Information in the Evaluation of Intervention Programs: A Theoretical and Methodological Framework," SAGE Open, , vol. 10(1), pages 21582440198, January.
    16. Jordan H. Rickles, 2011. "Using Interviews to Understand the Assignment Mechanism in a Nonexperimental Study," Evaluation Review, , vol. 35(5), pages 490-522, October.
    17. Priulla, Andrea & Vittorietti, Martina & Attanasio, Massimo, 2023. "Does taking additional Maths classes in high school affect academic outcomes?," Socio-Economic Planning Sciences, Elsevier, vol. 89(C).
    18. Fortson, Kenneth & Gleason, Philip & Kopa, Emma & Verbitsky-Savitz, Natalya, 2015. "Horseshoes, hand grenades, and treatment effects? Reassessing whether nonexperimental estimators are biased," Economics of Education Review, Elsevier, vol. 44(C), pages 100-113.
    19. Miquel-Àngel Garcia-López & Albert Solé-Ollé & Elisabet Viladecans Marsal, 2014. "Do Land Use Policies Follow Road Construction," CESifo Working Paper Series 4672, CESifo.
    20. Katherine Baicker & Theodore Svoronos, 2019. "Testing the Validity of the Single Interrupted Time Series Design," CID Working Papers 364, Center for International Development at Harvard University.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:39:y:2014:i:6:p:612-636. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.