IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/22468.html
   My bibliography  Save this paper

External and Internal Validity of a Geographic Quasi-Experiment Embedded in Cluster-Randomized Experiment

Author

Listed:
  • Sebastian Galiani
  • Patrick J. McEwan
  • Brian Quistorff

Abstract

This paper analyzes a geographic quasi-experiment embedded in a cluster-randomized experiment in Honduras. In the experiment, average treatment effects on school enrollment and child labor were large—especially in the poorest blocks—and could be generalized to a policy-relevant population given the original sample selection criteria. In contrast, the geographic quasi-experiment yielded point estimates that, for two of three dependent variables, were attenuated. A judicious policy analyst without access to the experimental results might have provided misleading advice based on the magnitude of point estimates. We assessed two main explanations for the difference in point estimates, related to external and internal validity.

Suggested Citation

  • Sebastian Galiani & Patrick J. McEwan & Brian Quistorff, 2016. "External and Internal Validity of a Geographic Quasi-Experiment Embedded in Cluster-Randomized Experiment," NBER Working Papers 22468, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:22468
    Note: DEV
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w22468.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Galiani, Sebastian & McEwan, Patrick J., 2013. "The heterogeneous impact of conditional cash transfers," Journal of Public Economics, Elsevier, vol. 103(C), pages 85-96.
    2. Sandra E. Black, 1999. "Do Better Schools Matter? Parental Valuation of Elementary Education," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(2), pages 577-599.
    3. Sebastian Calonico & Matias D. Cattaneo & Rocio Titiunik, 2014. "Robust Nonparametric Confidence Intervals for Regression‐Discontinuity Designs," Econometrica, Econometric Society, vol. 82, pages 2295-2326, November.
    4. Oosterbeek, Hessel & Ponce, Juan & Schady, Norbert, 2008. "The impact of cash transfers on school enrollment : evidence from Ecuador," Policy Research Working Paper Series 4645, The World Bank.
    5. Keele, Luke & Titiunik, Rocío, 2016. "Natural Experiments Based on Geography," Political Science Research and Methods, Cambridge University Press, vol. 4(1), pages 65-95, January.
    6. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    7. Cameron, A. Colin & Gelbach, Jonah B. & Miller, Douglas L., 2011. "Robust Inference With Multiway Clustering," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(2), pages 238-249.
    8. Yingying Dong & Arthur Lewbel, 2015. "Identifying the Effect of Changing the Policy Threshold in Regression Discontinuity Models," The Review of Economics and Statistics, MIT Press, vol. 97(5), pages 1081-1092, December.
    9. Keele, Luke J. & Titiunik, Rocío, 2015. "Geographic Boundaries as Regression Discontinuities," Political Analysis, Cambridge University Press, vol. 23(1), pages 127-155, January.
    10. Seán M. Muller, 2015. "Causal Interaction and External Validity: Obstacles to the Policy Relevance of Randomized Evaluations," The World Bank Economic Review, World Bank, vol. 29(suppl_1), pages 217-225.
    11. Hahn, Jinyong & Todd, Petra & Van der Klaauw, Wilbert, 2001. "Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design," Econometrica, Econometric Society, vol. 69(1), pages 201-209, January.
    12. Matias Cattaneo & Sebastian Calonico & Rocio Titiunik, 2015. "Robust Inference in Regression-Discontinuity Designs," 2015 Stata Conference 16, Stata Users Group.
    13. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    14. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 281-355, June.
    15. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(3), pages 1117-1165.
    16. María Alzúa & Guillermo Cruces & Laura Ripani, 2013. "Welfare programs and labor supply in developing countries: experimental evidence from Latin America," Journal of Population Economics, Springer;European Society for Population Economics, vol. 26(4), pages 1255-1284, October.
    17. Ariel Fiszbein & Norbert Schady & Francisco H.G. Ferreira & Margaret Grosh & Niall Keleher & Pedro Olinto & Emmanuel Skoufias, 2009. "Conditional Cash Transfers : Reducing Present and Future Poverty," World Bank Publications - Books, The World Bank Group, number 2597.
    18. Lee, David S., 2008. "Randomized experiments from non-random selection in U.S. House elections," Journal of Econometrics, Elsevier, vol. 142(2), pages 675-697, February.
    19. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    20. repec:oup:qjecon:v:129:y:2013:i:1:p:435-476 is not listed on IDEAS
    21. Joshua D. Angrist & Miikka Rokkanen, 2015. "Wanna Get Away? Regression Discontinuity Estimation of Exam School Effects Away From the Cutoff," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1331-1344, December.
    22. Buddelmeyer, Hielke & Skoufias, Emmanuel, 2003. "An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA," IZA Discussion Papers 827, Institute of Labor Economics (IZA).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tarek Azzam & Michael Bates & David Fairris, 2019. "Do Learning Communities Increase First Year College Retention? Testing Sample Selection and External Validity of Randomized Control Trials," Working Papers 202002, University of California at Riverside, Department of Economics.
    2. Matias D. Cattaneo & Rocío Titiunik, 2022. "Regression Discontinuity Designs," Annual Review of Economics, Annual Reviews, vol. 14(1), pages 821-851, August.
    3. Azzam, Tarek & Bates, Michael D. & Fairris, David, 2022. "Do learning communities increase first year college retention? Evidence from a randomized control trial," Economics of Education Review, Elsevier, vol. 89(C).
    4. Aepli, Manuel & Kuhn, Andreas & Schweri, Jürg, 2021. "Culture, norms, and the provision of training by employers: Evidence from the Swiss language border," Labour Economics, Elsevier, vol. 73(C).
    5. Molina Millán, Teresa & Macours, Karen & Maluccio, John A. & Tejerina, Luis, 2020. "Experimental long-term effects of early-childhood and school-age exposure to a conditional cash transfer program," Journal of Development Economics, Elsevier, vol. 143(C).
    6. Molina-Millan, Teresa & Macours, Karen & Maluccio, John A. & Tejerina, Luis, 2019. "The Long-Term Impacts of Honduras’ CCT Program: Higher Education and International Migration," IDB Publications (Working Papers) 9600, Inter-American Development Bank.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    2. Blaise Melly & Rafael Lalive, 2020. "Estimation, Inference, and Interpretation in the Regression Discontinuity Design," Diskussionsschriften dp2016, Universitaet Bern, Departement Volkswirtschaft.
    3. Yang He & Otávio Bartalotti, 2020. "Wild bootstrap for fuzzy regression discontinuity designs: obtaining robust bias-corrected confidence intervals," The Econometrics Journal, Royal Economic Society, vol. 23(2), pages 211-231.
    4. Matias D. Cattaneo & Luke Keele & Rocío Titiunik & Gonzalo Vazquez-Bare, 2021. "Extrapolating Treatment Effects in Multi-Cutoff Regression Discontinuity Designs," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1941-1952, October.
    5. Matias D. Cattaneo & Rocío Titiunik, 2022. "Regression Discontinuity Designs," Annual Review of Economics, Annual Reviews, vol. 14(1), pages 821-851, August.
    6. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    7. Quoc-Anh Do & Yen-Teik Lee & Bang Dang Nguyen, 2016. "Directors as Connectors: The Impact of the External Networks of Directors on Firms," Working Papers hal-03393196, HAL.
    8. Mauricio Villamizar‐Villegas & Freddy A. Pinzon‐Puerto & Maria Alejandra Ruiz‐Sanchez, 2022. "A comprehensive history of regression discontinuity designs: An empirical survey of the last 60 years," Journal of Economic Surveys, Wiley Blackwell, vol. 36(4), pages 1130-1178, September.
    9. Jin-young Choi & Myoung-jae Lee, 2017. "Regression discontinuity: review with extensions," Statistical Papers, Springer, vol. 58(4), pages 1217-1246, December.
    10. Quoc-Anh Do & Yen-Teik Lee & Bang Dang Nguyen, 2016. "Directors as Connectors: The Impact of the External Networks of Directors on Firms," Post-Print hal-03393196, HAL.
    11. Yu‐Chin Hsu & Shu Shen, 2021. "Testing monotonicity of conditional treatment effects under regression discontinuity designs," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(3), pages 346-366, April.
    12. Babii, Andrii & Kumar, Rohit, 2023. "Isotonic regression discontinuity designs," Journal of Econometrics, Elsevier, vol. 234(2), pages 371-393.
    13. Yoichi Arai & Yu‐Chin Hsu & Toru Kitagawa & Ismael Mourifié & Yuanyuan Wan, 2022. "Testing identifying assumptions in fuzzy regression discontinuity designs," Quantitative Economics, Econometric Society, vol. 13(1), pages 1-28, January.
    14. Deng, Taotao & Hu, Yukun & Ma, Mulan, 2019. "Regional policy and tourism: A quasi-natural experiment," Annals of Tourism Research, Elsevier, vol. 74(C), pages 1-16.
    15. Yiqi Liu & Yuan Qi, 2023. "Using Forests in Multivariate Regression Discontinuity Designs," Papers 2303.11721, arXiv.org, revised Jul 2024.
    16. repec:hal:spmain:info:hdl:2441/5q8d3q8agf8hdbs42laqdfujkb is not listed on IDEAS
    17. Hızıroğlu Aygün, Aysun & Kırdar, Murat Güray & Koyuncu, Murat & Stoeffler, Quentin, 2024. "Keeping refugee children in school and out of work: Evidence from the world's largest humanitarian cash transfer program," Journal of Development Economics, Elsevier, vol. 168(C).
    18. Berger, Melissa & Fellner-Röhling, Gerlinde & Sausgruber, Rupert & Traxler, Christian, 2016. "Higher taxes, more evasion? Evidence from border differentials in TV license fees," Journal of Public Economics, Elsevier, vol. 135(C), pages 74-86.
    19. Hsu, Yu-Chin & Shen, Shu, 2019. "Testing treatment effect heterogeneity in regression discontinuity designs," Journal of Econometrics, Elsevier, vol. 208(2), pages 468-486.
    20. Bauer, Thomas K. & Bender, Stefan & Paloyo, Alfredo R. & Schmidt, Christoph M., 2012. "Evaluating the labor-market effects of compulsory military service," European Economic Review, Elsevier, vol. 56(4), pages 814-829.
    21. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.

    More about this item

    JEL classification:

    • O22 - Economic Development, Innovation, Technological Change, and Growth - - Development Planning and Policy - - - Project Analysis

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:22468. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.