IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2401.12050.html
   My bibliography  Save this paper

A Bracketing Relationship for Long-Term Policy Evaluation with Combined Experimental and Observational Data

Author

Listed:
  • Yechan Park
  • Yuya Sasaki

Abstract

Combining short-term experimental data with observational data enables credible long-term policy evaluation. The literature offers two key but non-nested assumptions, namely the latent unconfoundedness (LU; Athey et al., 2020) and equi-confounding bias (ECB; Ghassami et al., 2022) conditions, to correct observational selection. Committing to the wrong assumption leads to biased estimation. To mitigate such risks, we provide a novel bracketing relationship (cf. Angrist and Pischke, 2009) repurposed for the setting with data combination: the LU-based estimand and the ECB-based estimand serve as the lower and upper bounds, respectively, with the true causal effect lying in between if either assumption holds. For researchers further seeking point estimates, our Lalonde-style exercise suggests the conservatively more robust LU-based lower bounds align closely with the hold-out experimental estimates for educational policy evaluation. We investigate the economic substantives of these findings through the lens of a nonparametric class of selection mechanisms and sensitivity analysis. We uncover as key the sub-martingale property and sufficient-statistics role (Chetty, 2009) of the potential outcomes of student test scores (Chetty et al., 2011, 2014).

Suggested Citation

  • Yechan Park & Yuya Sasaki, 2024. "A Bracketing Relationship for Long-Term Policy Evaluation with Combined Experimental and Observational Data," Papers 2401.12050, arXiv.org.
  • Handle: RePEc:arx:papers:2401.12050
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2401.12050
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1593-1660.
    2. Susan Athey & Raj Chetty & Guido W. Imbens & Hyunseung Kang, 2019. "The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely," NBER Working Papers 26463, National Bureau of Economic Research, Inc.
    3. Matthieu Crozet & Emmanuel Milet, 2017. "Should everybody be in services? The effect of servitization on manufacturing firm performance," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 26(4), pages 820-841, December.
    4. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates," American Economic Review, American Economic Association, vol. 104(9), pages 2593-2632, September.
    5. Card, David & Krueger, Alan B, 1992. "Does School Quality Matter? Returns to Education and the Characteristics of Public Schools in the United States," Journal of Political Economy, University of Chicago Press, vol. 100(1), pages 1-40, February.
    6. Raj Chetty, 2009. "Sufficient Statistics for Welfare Analysis: A Bridge Between Structural and Reduced-Form Methods," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 451-488, May.
    7. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    8. Susan Athey & Guido W. Imbens, 2006. "Identification and Inference in Nonlinear Difference-in-Differences Models," Econometrica, Econometric Society, vol. 74(2), pages 431-497, March.
    9. Dalia Ghanem & Pedro H. C. Sant'Anna & Kaspar Wüthrich, 2022. "Selection and Parallel Trends," CESifo Working Paper Series 9910, CESifo.
    10. Orley Ashenfelter & David E. Card, 1984. "Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs," Working Papers 554, Princeton University, Department of Economics, Industrial Relations Section..
    11. Ding, Peng & Li, Fan, 2019. "A Bracketing Relationship between Difference-in-Differences and Lagged-Dependent-Variable Adjustment," Political Analysis, Cambridge University Press, vol. 27(4), pages 605-615, October.
    12. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    13. Susan Athey & Raj Chetty & Guido Imbens, 2020. "Combining Experimental and Observational Data to Estimate Treatment Effects on Long Term Outcomes," Papers 2006.09676, arXiv.org.
    14. Douglas O. Staiger & Jonah E. Rockoff, 2010. "Searching for Effective Teachers with Imperfect Information," Journal of Economic Perspectives, American Economic Association, vol. 24(3), pages 97-118, Summer.
    15. Ashenfelter, Orley & Card, David, 1985. "Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs," The Review of Economics and Statistics, MIT Press, vol. 67(4), pages 648-660, November.
    16. Adam N. Glynn & Konstantin Kashin, 2017. "Front‐Door Difference‐in‐Differences Estimators," American Journal of Political Science, John Wiley & Sons, vol. 61(4), pages 989-1002, October.
    17. Tyler J. VanderWeele, 2013. "Surrogate Measures and Consistent Surrogates," Biometrics, The International Biometric Society, vol. 69(3), pages 561-565, September.
    18. A. D. Roy, 1951. "Some Thoughts On The Distribution Of Earnings," Oxford Economic Papers, Oxford University Press, vol. 3(2), pages 135-146.
    19. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    2. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    3. Peter Hull & Michal Kolesár & Christopher Walters, 2022. "Labor by design: contributions of David Card, Joshua Angrist, and Guido Imbens," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(3), pages 603-645, July.
    4. van der Klaauw, Bas, 2014. "From micro data to causality: Forty years of empirical labor economics," Labour Economics, Elsevier, vol. 30(C), pages 88-97.
    5. Committee, Nobel Prize, 2021. "Answering causal questions using observational data," Nobel Prize in Economics documents 2021-2, Nobel Prize Committee.
    6. Yechan Park & Yuya Sasaki, 2024. "Matching $\leq$ Hybrid $\leq$ Difference in Differences," Papers 2411.07952, arXiv.org, revised Feb 2025.
    7. Susan Athey & Raj Chetty & Guido Imbens, 2020. "Combining Experimental and Observational Data to Estimate Treatment Effects on Long Term Outcomes," Papers 2006.09676, arXiv.org.
    8. Jason Poulos & Andrea Albanese & Andrea Mercatanti & Fan Li, 2021. "Retrospective causal inference via matrix completion, with an evaluation of the effect of European integration on cross-border employment," Papers 2106.00788, arXiv.org.
    9. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    10. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    11. Johnston, Andrew C., 2021. "Preferences, Selection, and the Structure of Teacher Pay," IZA Discussion Papers 14831, Institute of Labor Economics (IZA).
    12. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    13. Jens Ruhose, 2015. "Microeconometric Analyses on Economic Consequences of Selective Migration," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 61, May.
    14. Paul Hunermund & Elias Bareinboim, 2019. "Causal Inference and Data Fusion in Econometrics," Papers 1912.09104, arXiv.org, revised Mar 2023.
    15. Patrick Kline & Christopher R. Walters, 2016. "Evaluating Public Programs with Close Substitutes: The Case of HeadStart," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(4), pages 1795-1848.
    16. Sergio Destefanis & Matteo Fragetta & Emanuel Gasteiger, 2024. "Does one size fit all in the Euro Area? Some counterfactual evidence," Empirical Economics, Springer, vol. 67(4), pages 1615-1647, October.
    17. Nikolay Doudchenko & Guido W. Imbens, 2016. "Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis," NBER Working Papers 22791, National Bureau of Economic Research, Inc.
    18. Guido Imbens & Nathan Kallus & Xiaojie Mao & Yuhao Wang, 2022. "Long-term Causal Inference Under Persistent Confounding via Data Combination," Papers 2202.07234, arXiv.org, revised Aug 2024.
    19. Alex Bell & Raj Chetty & Xavier Jaravel & Neviana Petkova & John Van Reenen, 2019. "Who Becomes an Inventor in America? The Importance of Exposure to Innovation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(2), pages 647-713.
    20. Bonilla-Mejía, Leonardo & Galvis-Aponte, Luis Armando, 2014. "Profesionalización docente y calidad de la educación escolar en Colombia," Chapters, in: Sánchez Jabba, Andrés & Otero Cortés, Andrea (ed.), Educación y desarrollo regional en Colombia, chapter 5, pages 161-209, Banco de la Republica de Colombia.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2401.12050. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.