IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v38y2013i3p239-266.html
   My bibliography  Save this article

Improving Generalizations From Experiments Using Propensity Score Subclassification

Author

Listed:
  • Elizabeth Tipton

Abstract

As a result of the use of random assignment to treatment, randomized experiments typically have high internal validity. However, units are very rarely randomly selected from a well-defined population of interest into an experiment; this results in low external validity. Under nonrandom sampling, this means that the estimate of the sample average treatment effect calculated in the experiment can be a biased estimate of the population average treatment effect. This article explores the use of the propensity score subclassification estimator as a means for improving generalizations from experiments. It first lays out the assumptions necessary for generalizations, then investigates the amount of bias reduction and average variance inflation that is likely when compared to a conventional estimator. It concludes with a discussion of issues that arise when the population of interest is not well represented by the experiment, and an example.

Suggested Citation

  • Elizabeth Tipton, 2013. "Improving Generalizations From Experiments Using Propensity Score Subclassification," Journal of Educational and Behavioral Statistics, , vol. 38(3), pages 239-266, June.
    • Handle: RePEc:sae:jedbes:v:38:y:2013:i:3:p:239-266
    DOI: 10.3102/1076998612441947
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998612441947
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998612441947?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Hong, Guanglei & Raudenbush, Stephen W., 2006. "Evaluating Kindergarten Retention Policy: A Case Study of Causal Inference for Multilevel Observational Data," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 901-910, September.
    2. Evelyn Kitagawa, 1964. "Standardized comparisons in population research," Demography, Springer;Population Association of America (PAA), vol. 1(1), pages 296-315, March.
    3. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    4. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
    5. Nadarajah, Saralees & Kotz, Samuel, 2006. "R Programs for Truncated Distributions," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 16(c02).
    6. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Peter Z. Schochet, "undated". "Statistical Theory for the RCT-YES Software: Design-Based Causal Inference for RCTs," Mathematica Policy Research Reports a0c005c003c242308a92c02dc, Mathematica Policy Research.
    2. Ashley L. Buchanan & Michael G. Hudgens & Stephen R. Cole & Katie R. Mollan & Paul E. Sax & Eric S. Daar & Adaora A. Adimora & Joseph J. Eron & Michael J. Mugavero, 2018. "Generalizing evidence from randomized trials using inverse probability of sampling weights," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(4), pages 1193-1209, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Simone Bertoli & Francesca Marchetta, 2014. "Migration, Remittances and Poverty in Ecuador," Journal of Development Studies, Taylor & Francis Journals, vol. 50(8), pages 1067-1089, August.
    2. Patrick Christian Feihle & Jochen Lawrenz, 2017. "The Issuance of German SME Bonds and its Impact on Operating Performance," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 18(3), pages 227-259, August.
    3. Solomon Asfaw & Silvio Daidone & Benjamin Davis & Josh Dewbre & Alessandro Romeo & Paul Winters & Katia Covarrubias & Habiba Djebbari, 2012. "Analytical Framework for Evaluating the Productive Impact of Cash Transfer Programmes on Household Behaviour – Methodological Guidelines for the From Protection to Production Project," Working Papers 101, International Policy Centre for Inclusive Growth.
    4. Kube, Roland & von Graevenitz, Kathrine & Löschel, Andreas & Massier, Philipp, 2019. "Do voluntary environmental programs reduce emissions? EMAS in the German manufacturing sector," Energy Economics, Elsevier, vol. 84(S1).
    5. K. Poehlmann & R. Helm & O. Mauroner & J. Auburger, 2021. "Corporate spin-offs’ success factors: management lessons from a comparative empirical analysis with research-based spin-offs," Review of Managerial Science, Springer, vol. 15(6), pages 1767-1796, August.
    6. Elaine M. Wolf & Douglas A. Wolf, 2008. "Mixed Results in a Transitional Planning Program for Alternative School Students," Evaluation Review, , vol. 32(2), pages 187-215, April.
    7. Jasjeet Singh Sekhon & Richard D. Grieve, 2012. "A matching method for improving covariate balance in cost‐effectiveness analyses," Health Economics, John Wiley & Sons, Ltd., vol. 21(6), pages 695-714, June.
    8. Shen, Chung-Hua & Wu, Meng-Wen & Chen, Ting-Hsuan & Fang, Hao, 2016. "To engage or not to engage in corporate social responsibility: Empirical evidence from global banking sector," Economic Modelling, Elsevier, vol. 55(C), pages 207-225.
    9. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," Working Papers hal-03455978, HAL.
    10. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    11. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    12. Kathrine Graevenitz & Daniel Römer & Alexander Rohlf, 2018. "The Effect of Emission Information on Housing Prices: Quasi-Experimental Evidence from the European Pollutant Release and Transfer Register," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 69(1), pages 23-74, January.
    13. Ferraro, Paul J. & Miranda, Juan José, 2014. "The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 344-365.
    14. Goller, Daniel & Lechner, Michael & Moczall, Andreas & Wolff, Joachim, 2020. "Does the estimation of the propensity score by machine learning improve matching estimation? The case of Germany's programmes for long term unemployed," Labour Economics, Elsevier, vol. 65(C).
    15. Silvio Daidone & Sudhanshu Handa & Benjamin Davis & Mike Park & Robert D. Osei & Isaac Osei-Akoto, 2015. "Social Networks and Risk Management in Ghana’s Livelihood Empowerment Against Poverty Programme," Papers inwopa781, Innocenti Working Papers.
    16. Gerhard Krug, 2017. "Augmenting propensity score equations to avoid misspecification bias – Evidence from a Monte Carlo simulation [Erweiterung der Propensity Score Gleichung zur Vermeidung von Fehlspezifikationen? Ein," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 11(3), pages 205-231, December.
    17. Canavire-Bacarreza, Gustavo & Hanauer, Merlin M., 2013. "Estimating the Impacts of Bolivia’s Protected Areas on Poverty," World Development, Elsevier, vol. 41(C), pages 265-285.
    18. Haotian Zhong & Wei Li & Marlon G Boarnet, 2021. "A two-dimensional propensity score matching method for longitudinal quasi-experimental studies: A focus on travel behavior and the built environment," Environment and Planning B, , vol. 48(7), pages 2110-2122, September.
    19. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    20. Ravi Bapna & Alok Gupta & Gautam Ray & Shweta Singh, 2016. "Research Note —IT Outsourcing and the Impact of Advisors on Clients and Vendors," Information Systems Research, INFORMS, vol. 27(3), pages 636-647.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:38:y:2013:i:3:p:239-266. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.