IDEAS home Printed from https://ideas.repec.org/a/aea/aejapp/v1y2009i4p200-232.html
   My bibliography  Save this article

In Pursuit of Balance: Randomization in Practice in Development Field Experiments

Author

Listed:
  • Miriam Bruhn
  • David McKenzie

Abstract

We present new evidence on the randomization methods used in existing experiments, and new simulations comparing these methods. We find that many papers do not describe the randomization in detail, implying that better reporting is needed. Our simulations suggest that in samples of 300 or more, the different methods perform similarly. However, for very persistent outcome variables, and in smaller samples, pair-wise matching and stratification perform best and appear to dominate the rerandomization methods commonly used in practice. The simulations also point to specific recommendations for which variables to balance on, and for which controls to include in the ex post analysis. (JEL C83, C93, O12)

Suggested Citation

  • Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    • Handle: RePEc:aea:aejapp:v:1:y:2009:i:4:p:200-232
    Note: DOI: 10.1257/app.1.4.200
    as

    Download full text from publisher

    File URL: http://www.aeaweb.org/articles.php?doi=10.1257/app.1.4.200
    Download Restriction: no
    File URL: http://www.aeaweb.org/aej/app/data/2008-0182_data.zip
    Download Restriction: no
    File URL: http://www.aeaweb.org/aej/app/app/2008-0182_app.pdf
    Download Restriction: Access to full text is restricted to AEA members and institutional subscribers.
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Esther Duflo, 2005. "Monitoring Works: Getting Teachers to Come to School," Working Papers id:301, eSocialSciences.
    2. Pascaline Dupas, 2006. "Relative risks and the market for sex: Teenagers, sugar daddies, and hiv in kenya," Natural Field Experiments 00235, The Field Experiments Website.
    3. Gustavo J. Bobonis & Edward Miguel & Charu Puri-Sharma, 2006. "Anemia and School Participation," Journal of Human Resources, University of Wisconsin Press, vol. 41(4).
    4. Abhijit Vinayak Banerjee & Alice H. Amsden & Robert H. Bates & Jagdish Bhagwati & Angus Deaton & Nicholas Stern, 2007. "Making Aid Work," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262026155, April.
    5. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    6. Tahir Andrabi & Jishnu Das & Asim Ijaz Khwaja & Tristan Zajonc, 2011. "Do Value-Added Estimates Add Value? Accounting for Learning Dynamics," American Economic Journal: Applied Economics, American Economic Association, vol. 3(3), pages 29-54, July.
    7. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    8. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    9. Dean Karlan & Martin Valdivia, 2011. "Teaching Entrepreneurship: Impact of Business Training on Microfinance Clients and Institutions," The Review of Economics and Statistics, MIT Press, vol. 93(2), pages 510-527, May.
    10. Benjamin A. Olken, 2007. "Monitoring Corruption: Evidence from a Field Experiment in Indonesia," Journal of Political Economy, University of Chicago Press, vol. 115(2), pages 200-249.
    11. Claudio Ferraz & Frederico Finan, 2008. "Exposing Corrupt Politicians: The Effects of Brazil's Publicly Released Audits on Electoral Outcomes," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 123(2), pages 703-745.
    12. Glewwe, Paul & Kremer, Michael & Moulin, Sylvie & Zitzewitz, Eric, 2004. "Retrospective vs. prospective analyses of school inputs: the case of flip charts in Kenya," Journal of Development Economics, Elsevier, vol. 74(1), pages 251-268, June.
    13. Glewwe, Paul & Park, Albert & Zhao, Meng, 2006. "The Impact of Eyeglasses on the Academic Performance of Primary School Students: Evidence from a Randomized Trial in Rural China," Conference Papers 6644, University of Minnesota, Center for International Food and Agricultural Policy.
    14. Editors The, 2008. "From the Editors," Basic Income Studies, De Gruyter, vol. 3(1), pages 1-1, July.
    15. Erica Field & Rohini Pande, 2008. "Repayment Frequency and Default in Microfinance: Evidence From India," Journal of the European Economic Association, MIT Press, vol. 6(2-3), pages 501-509, 04-05.
    16. Ashraf Nava & Karlan Dean & Yin Wesley, 2006. "Deposit Collectors," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 6(2), pages 1-24, March.
    17. Nava Ashraf & Dean Karlan & Wesley Yin, 2006. "Tying Odysseus to the Mast: Evidence From a Commitment Savings Product in the Philippines," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 121(2), pages 635-672.
    18. Nava Ashraf & James Berry & Jesse M. Shapiro, 2010. "Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia," American Economic Review, American Economic Association, vol. 100(5), pages 2383-2413, December.
    19. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    20. repec:feb:artefa:0087 is not listed on IDEAS
    21. Michael Kremer, 2003. "Randomized Evaluations of Educational Programs in Developing Countries: Some Lessons," American Economic Review, American Economic Association, vol. 93(2), pages 102-106, May.
    22. Bjorkman, Martina & Svensson, Jakob, 2007. "Power to the people : evidence from a randomized field experiment of a community-based monitoring project in Uganda," Policy Research Working Paper Series 4268, The World Bank.
    23. Suresh de Mel & David McKenzie & Christopher Woodruff, 2009. "Returns to Capital in Microenterprises: Evidence from a Field Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 124(1), pages 423-423.
    24. Editors The, 2008. "From the Editors," Basic Income Studies, De Gruyter, vol. 2(2), pages 1-3, January.
    25. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    26. Martina Björkman & Jakob Svensson, 2009. "Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 124(2), pages 735-769.
    27. Skoufias, Emmanuel, 2005. "PROGRESA and its impacts on the welfare of rural households in Mexico:," Research reports 139, International Food Policy Research Institute (IFPRI).
    28. Marianne Bertrand & Simeon Djankov & Rema Hanna & Sendhil Mullainathan, 2007. "Obtaining a Driver's License in India: An Experimental Approach to Studying Corruption," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(4), pages 1639-1676.
    29. Gary Burtless, 1995. "The Case for Randomized Field Trials in Economic and Policy Research," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 63-84, Spring.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    2. Benjamin A. Olken, 2020. "Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 122(3), pages 853-878, July.
    3. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    4. William Easterly, 2009. "Can the West Save Africa?," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 373-447, June.
    5. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    6. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    7. G�nther Fink & Margaret McConnell & Sebastian Vollmer, 2014. "Testing for heterogeneous treatment effects in experimental data: false discovery risks and correction procedures," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(1), pages 44-57, January.
    8. McKenzie, David, 2012. "Beyond baseline and follow-up: The case for more T in experiments," Journal of Development Economics, Elsevier, vol. 99(2), pages 210-221.
    9. Angus Deaton, 2009. "Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development," Working Papers 1128, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Health and Wellbeing..
    10. David K. Evans & Arkadipta Ghosh, 2008. "Prioritizing Educational Investments in Children in the Developing World," Working Papers WR-587, RAND Corporation.
    11. David K. Evans & Arkadipta Ghosh, 2008. "Prioritizing Educational Investments in Children in the Developing World," Working Papers 587, RAND Corporation.
    12. Abhijit V. Banerjee & Esther Duflo, 2010. "Giving Credit Where It Is Due," Journal of Economic Perspectives, American Economic Association, vol. 24(3), pages 61-80, Summer.
    13. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    14. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    15. Gisselquist, Rachel & Niño-Zarazúa, Miguel, 2013. "What can experiments tell us about how to improve governance?," MPRA Paper 49300, University Library of Munich, Germany.
    16. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    17. Larrú, José María, 2012. "Evaluar para aprender. Eficiencia en salud y desarrollo [Evaluating for learning. Efficiency in health and development]," MPRA Paper 38788, University Library of Munich, Germany.
    18. Committee, Nobel Prize, 2019. "Understanding development and poverty alleviation," Nobel Prize in Economics documents 2019-2, Nobel Prize Committee.
    19. Temple, Jonathan R.W., 2010. "Aid and Conditionality," Handbook of Development Economics, in: Dani Rodrik & Mark Rosenzweig (ed.), Handbook of Development Economics, edition 1, volume 5, chapter 0, pages 4415-4523, Elsevier.
    20. Gisselquist, Rachel & Niño-Zarazúa, Miguel, 2013. "What can experiments tell us about how to improve governance?," MPRA Paper 49300, University Library of Munich, Germany.

    More about this item

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • O12 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Microeconomic Analyses of Economic Development

    Lists

    This item is featured on the following reading lists, Wikipedia, or ReplicationWiki pages:
    1. In Pursuit of Balance: Randomization in Practice in Development Field Experiments (American Economic Journal: Applied Economics 2009) in ReplicationWiki

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:aejapp:v:1:y:2009:i:4:p:200-232. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michael P. Albert (email available below). General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.