IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/22167.html
   My bibliography  Save this paper

Decision Theoretic Approaches to Experiment Design and External Validity

Author

Listed:
  • Abhijit Banerjee
  • Sylvain Chassang
  • Erik Snowberg

Abstract

A modern, decision-theoretic framework can help clarify important practical questions of experimental design. Building on our recent work, this chapter begins by summarizing our framework for understanding the goals of experimenters, and applying this to re-randomization. We then use this framework to shed light on questions related to experimental registries, pre-analysis plans, and most importantly, external validity. Our framework implies that even when large samples can be collected, external decision-making remains inherently subjective. We embrace this conclusion, and argue that in order to improve external validity, experimental research needs to create a space for structured speculation.

Suggested Citation

  • Abhijit Banerjee & Sylvain Chassang & Erik Snowberg, 2016. "Decision Theoretic Approaches to Experiment Design and External Validity," NBER Working Papers 22167, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:22167
    Note: DEV ED EH LS PE POL
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w22167.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bruno Crépon & Esther Duflo & Marc Gurgand & Roland Rathelot & Philippe Zamora, 2013. "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(2), pages 531-580.
    2. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    3. Philippe Aghion & Patrick Bolton & Christopher Harris & Bruno Jullien, 1991. "Optimal Learning by Experimentation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 58(4), pages 621-654.
    4. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    5. James Andreoni & Michael Callen & Karrar Hussain & Muhammad Yasir Khan & Charles Sprenger, 2023. "Using Preference Estimates to Customize Incentives: An Application to Polio Vaccination Drives in Pakistan," Journal of the European Economic Association, European Economic Association, vol. 21(4), pages 1428-1477.
    6. Dean Karlan, Ryan Knight, and Christopher Udry, 2012. "Hoping to Win, Expected to Lose: Theory and Lessons on Microenterprise Development," Working Papers 312, Center for Global Development.
    7. James J. Heckman & Edward Vytlacil, 2005. "Structural Equations, Treatment Effects, and Econometric Policy Evaluation," Econometrica, Econometric Society, vol. 73(3), pages 669-738, May.
    8. repec:hal:pseose:halshs-00840901 is not listed on IDEAS
    9. Faruk Gul & Wolfgang Pesendorfer, 2001. "Temptation and Self-Control," Econometrica, Econometric Society, vol. 69(6), pages 1403-1435, November.
    10. Humphreys, Macartan & Sanchez de la Sierra, Raul & van der Windt, Peter, 2013. "Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration," Political Analysis, Cambridge University Press, vol. 21(1), pages 1-20, January.
    11. Grossman, Sanford J & Stiglitz, Joseph E, 1980. "On the Impossibility of Informationally Efficient Markets," American Economic Review, American Economic Association, vol. 70(3), pages 393-408, June.
    12. James Berry & Greg Fischer & Raymond Guiteras, 2020. "Eliciting and Utilizing Willingness to Pay: Evidence from Field Trials in Northern Ghana," Journal of Political Economy, University of Chicago Press, vol. 128(4), pages 1436-1473.
    13. Milgrom, Paul R, 1981. "Rational Expectations, Information Acquisition, and Competitive Bidding," Econometrica, Econometric Society, vol. 49(4), pages 921-943, June.
    14. Sarah Baird & Aislinn Bohren & Berk Ozler & Craig McIntosh, 2014. "Designing Experiments to Measure Spillover Effects," Working Papers 2014-11, The George Washington University, Institute for International Economic Policy.
    15. Philippe Aghion & Jean Tirole, 1994. "The Management of Innovation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 109(4), pages 1185-1209.
    16. Vivi Alatas & Abhijit Banerjee & Rema Hanna & Benjamin A. Olken & Julia Tobias, 2012. "Targeting the Poor: Evidence from a Field Experiment in Indonesia," American Economic Review, American Economic Association, vol. 102(4), pages 1206-1240, June.
    17. Bergemann, Dirk & Valimaki, Juuso, 1996. "Learning and Strategic Pricing," Econometrica, Econometric Society, vol. 64(5), pages 1125-1149, September.
    18. Kasy, Maximilian, 2016. "Why Experimenters Might Not Always Want to Randomize, and What They Could Do Instead," Political Analysis, Cambridge University Press, vol. 24(3), pages 324-338, July.
    19. Rothschild, Michael, 1974. "A two-armed bandit theory of market pricing," Journal of Economic Theory, Elsevier, vol. 9(2), pages 185-202, October.
    20. Machina, Mark J, 1989. "Dynamic Consistency and Non-expected Utility Models of Choice under Uncertainty," Journal of Economic Literature, American Economic Association, vol. 27(4), pages 1622-1668, December.
    21. Sylvain Chassang & Gerard Padro I Miquel & Erik Snowberg, 2012. "Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments," American Economic Review, American Economic Association, vol. 102(4), pages 1279-1309, June.
    22. Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
    23. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    24. Pascaline Dupas, 2014. "Short‐Run Subsidies and Long‐Run Adoption of New Health Products: Evidence From a Field Experiment," Econometrica, Econometric Society, vol. 82(1), pages 197-228, January.
    25. B. Kelsey Jack, 2013. "Private Information and the Allocation of Land Use Subsidies in Malawi," American Economic Journal: Applied Economics, American Economic Association, vol. 5(3), pages 113-135, July.
    26. Dean Karlan & Jonathan Zinman, 2009. "Observing Unobservables: Identifying Information Asymmetries With a Consumer Credit Field Experiment," Econometrica, Econometric Society, vol. 77(6), pages 1993-2008, November.
    27. Gilboa, Itzhak & Postlewaite, Andrew & Schmeidler, David, 2009. "Is It Always Rational To Satisfy Savage'S Axioms?," Economics and Philosophy, Cambridge University Press, vol. 25(3), pages 285-296, November.
    28. Sylvain Chassang & Erik Snowberg & Ben Seymour & Cayley Bowles, 2015. "Accounting for Behavior in Treatment Effects: New Applications for Blind Trials," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-13, June.
    29. Gilboa, Itzhak & Schmeidler, David, 1989. "Maxmin expected utility with non-unique prior," Journal of Mathematical Economics, Elsevier, vol. 18(2), pages 141-153, April.
    30. Jacobs,Donald P. & Kalai,Ehud & Kamien,Morton I. & Schwartz,Nancy L. (ed.), 1998. "Frontiers of Research in Economic Theory," Cambridge Books, Cambridge University Press, number 9780521635387, November.
    31. Abhijit V. Banerjee, 1992. "A Simple Model of Herd Behavior," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 107(3), pages 797-817.
    32. Daniel Ellsberg, 1961. "Risk, Ambiguity, and the Savage Axioms," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 75(4), pages 643-669.
    33. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    34. Sylvain Chassang, 2013. "Calibrated Incentive Contracts," Econometrica, Econometric Society, vol. 81(5), pages 1935-1971, September.
    35. Peter Klibanoff & Massimo Marinacci & Sujoy Mukerji, 2005. "A Smooth Model of Decision Making under Ambiguity," Econometrica, Econometric Society, vol. 73(6), pages 1849-1892, November.
    36. Nicola Persico, 2000. "Information Acquisition in Auctions," Econometrica, Econometric Society, vol. 68(1), pages 135-148, January.
    37. Nava Ashraf & James Berry & Jesse M. Shapiro, 2010. "Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia," American Economic Review, American Economic Association, vol. 100(5), pages 2383-2413, December.
    38. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    39. Schmeidler, David, 1989. "Subjective Probability and Expected Utility without Additivity," Econometrica, Econometric Society, vol. 57(3), pages 571-587, May.
    40. Jessica Cohen & Pascaline Dupas, 2010. "Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 1-45.
    41. Wojciech Olszewski & Marcin Pęski, 2011. "The Principal-Agent Approach to Testing Experts," American Economic Journal: Microeconomics, American Economic Association, vol. 3(2), pages 89-113, May.
    42. Kreps, David M & Porteus, Evan L, 1978. "Temporal Resolution of Uncertainty and Dynamic Choice Theory," Econometrica, Econometric Society, vol. 46(1), pages 185-200, January.
    43. Erwin Bulte & Gonne Beekman & Salvatore Di Falco & Joseph Hella & Pan Lei, 2014. "Behavioral Responses and the Impact of New Agricultural Technologies: Evidence from a Double-blind Field Experiment in Tanzania," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 96(3), pages 813-830.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    2. James Andreoni & Michael Callen & Karrar Hussain & Muhammad Yasir Khan & Charles Sprenger, 2023. "Using Preference Estimates to Customize Incentives: An Application to Polio Vaccination Drives in Pakistan," Journal of the European Economic Association, European Economic Association, vol. 21(4), pages 1428-1477.
    3. Marcelo Arbex & Justin M. Carre & Shawn N. Geniole & Enlinson Mattos, 2018. "Testosterone, personality traits and tax evasion," Working Papers 1801, University of Windsor, Department of Economics.
    4. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    5. Justman, Moshe, 2018. "Randomized controlled trials informing public policy: Lessons from project STAR and class size reduction," European Journal of Political Economy, Elsevier, vol. 54(C), pages 167-174.
    6. James Andreoni & Michael Callen & Muhammad Karrar Hussain & Muhammad Yasir Khan & Charles Sprenger, 2017. "Creating Investment Scheme with State Space Modeling ," CIRJE F-Series CIRJE-F-1039, CIRJE, Faculty of Economics, University of Tokyo.
    7. Mark Rosenzweig & Christopher Udry, 2016. "External Validity in a Stochastic World," Working Papers id:11134, eSocialSciences.
    8. Troost, Christian & Huber, Robert & Bell, Andrew R. & van Delden, Hedwig & Filatova, Tatiana & Le, Quang Bao & Lippe, Melvin & Niamir, Leila & Polhill, J. Gareth & Sun, Zhanli & Berger, Thomas, 2023. "How to keep it adequate: A protocol for ensuring validity in agent-based simulation," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 159, pages 1-21.
    9. de Souza Leão, Luciana & Eyal, Gil, 2020. "Searching under the streetlight: A historical perspective on the rise of randomistas," World Development, Elsevier, vol. 127(C).
    10. Moshe Justman, 2016. "Economic Research and Education Policy: Project STAR and Class Size Reduction," Melbourne Institute Working Paper Series wp2016n37, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.
    11. Arbex, Marcelo Aarestru & Carré, Justin M. & Geniole, Shawn N. & Mattos, Enlinson, 2018. "Tax evasion, testosterone and personality traits," Textos para discussão 466, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
    12. Kannan Perumal, 2021. "Rent seeking, supervisor approvals and conventional corruption control approach—an Indian experience," Journal of Social and Economic Development, Springer;Institute for Social and Economic Change, vol. 23(2), pages 357-376, December.
    13. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    14. Heiland, Frank & Korenman, Sanders & Smith, Rachel A., 2019. "Estimating the educational consequences of teenage childbearing: Identification, heterogeneous effects and the value of biological relationship information," Economics & Human Biology, Elsevier, vol. 33(C), pages 15-28.
    15. Aufenanger, Tobias, 2018. "Treatment allocation for linear models," FAU Discussion Papers in Economics 14/2017, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics, revised 2018.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abhijit Banerjee & Sylvain Chassang & Sergio Montero & Erik Snowberg, 2017. "A Theory of Experimenters," CESifo Working Paper Series 6678, CESifo.
    2. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    3. Sylvain Chassang & Gerard Padro I Miquel & Erik Snowberg, 2012. "Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments," American Economic Review, American Economic Association, vol. 102(4), pages 1279-1309, June.
    4. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    5. Peters, Jörg & Langbein, Jörg & Roberts, Gareth, 2016. "Policy evaluation, randomized controlled trials, and external validity—A systematic review," Economics Letters, Elsevier, vol. 147(C), pages 51-54.
    6. Paulina Oliva & B. Kelsey Jack & Samuel Bell & Elizabeth Mettetal & Christopher Severen, 2020. "Technology Adoption under Uncertainty: Take-Up and Subsequent Investment in Zambia," The Review of Economics and Statistics, MIT Press, vol. 102(3), pages 617-632, July.
    7. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    8. Yusuke Narita, 2018. "Experiment-as-Market: Incorporating Welfare into Randomized Controlled Trials," Cowles Foundation Discussion Papers 2127r, Cowles Foundation for Research in Economics, Yale University, revised May 2019.
    9. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    10. Karthik Muralidharan & Mauricio Romero & Kaspar Wüthrich, 2019. "Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments," NBER Working Papers 26562, National Bureau of Economic Research, Inc.
    11. Eddie Dekel & Barton L. Lipman, 2010. "How (Not) to Do Decision Theory," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 257-282, September.
    12. Burchardi, Konrad B. & de Quidt, Jonathan & Gulesci, Selim & Lerva, Benedetta & Tripodi, Stefano, 2021. "Testing willingness to pay elicitation mechanisms in the field: Evidence from Uganda," Journal of Development Economics, Elsevier, vol. 152(C).
    13. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    14. Karni, Edi & Maccheroni, Fabio & Marinacci, Massimo, 2015. "Ambiguity and Nonexpected Utility," Handbook of Game Theory with Economic Applications,, Elsevier.
    15. Pascaline Dupas & Edward Miguel, 2016. "Impacts and Determinants of Health Levels in Low-Income Countries," NBER Working Papers 22235, National Bureau of Economic Research, Inc.
    16. Antony Millner & Simon Dietz & Geoffrey Heal, 2013. "Scientific Ambiguity and Climate Policy," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 55(1), pages 21-46, May.
    17. Li, Jian, 2020. "Preferences for partial information and ambiguity," Theoretical Economics, Econometric Society, vol. 15(3), July.
    18. Baillon, Aurélien & Driesen, Bram & Wakker, Peter P., 2012. "Relative concave utility for risk and ambiguity," Games and Economic Behavior, Elsevier, vol. 75(2), pages 481-489.
    19. Hoffmann, Bridget, 2018. "Do non-monetary prices target the poor? Evidence from a field experiment in India," Journal of Development Economics, Elsevier, vol. 133(C), pages 15-32.
    20. Basieva, Irina & Khrennikova, Polina & Pothos, Emmanuel M. & Asano, Masanari & Khrennikov, Andrei, 2018. "Quantum-like model of subjective expected utility," Journal of Mathematical Economics, Elsevier, vol. 78(C), pages 150-162.

    More about this item

    JEL classification:

    • B23 - Schools of Economic Thought and Methodology - - History of Economic Thought since 1925 - - - Econometrics; Quantitative and Mathematical Studies
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • O1 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:22167. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.