IDEAS home Printed from https://ideas.repec.org/p/zbw/rwirep/589.html
   My bibliography  Save this paper

Policy evaluation, randomized controlled trials, and external validity: A systematic review

Author

Listed:
  • Peters, Jörg
  • Langbein, Jörg
  • Roberts, Gareth

Abstract

When properly implemented, Randomized Controlled Trials (RCT) can achieve a high degree of internal validity. Yet, if an RCT is to inform policy interventions that extend beyond the experimental population, it is critical to establish external validity. In this paper, we first present a theoretical framework of external validity and identify the potential hazards that compromise generalizing results beyond the studied population, namely Hawthorne effects, general equilibrium effects, specific sample problems, and special care in the provision of the randomized treatment. Second, we reviewed all RCTs published in leading economic journals between 2009 and 2014 and scrutinized the way they deal with external validity. Based on a set of objective indicators, we find that many published RCTs do not discuss hazards to external validity and do not provide the information that is necessary to assess potential problems. Apparently, external validity is not an important matter of concern during the peer review process. To conclude, we call for a more systematic approach to report the results of RCTs, including external validity dimensions.

Suggested Citation

  • Peters, Jörg & Langbein, Jörg & Roberts, Gareth, 2015. "Policy evaluation, randomized controlled trials, and external validity: A systematic review," Ruhr Economic Papers 589, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    • Handle: RePEc:zbw:rwirep:589
    DOI: 10.4419/86788684
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/123694/1/841211906.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.4419/86788684?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. repec:mpr:mprres:7375 is not listed on IDEAS
    2. Pedro C. Vicente, 2014. "Is Vote Buying Effective? Evidence from a Field Experiment in West Africa," Economic Journal, Royal Economic Society, vol. 124(574), pages 356-387, February.
    3. Jan Stoop, 2014. "From the lab to the field: envelopes, dictators and manners," Experimental Economics, Springer;Economic Science Association, vol. 17(2), pages 304-313, June.
    4. Luc Behaghel & Bruno Cr?pon & Marc Gurgand, 2014. "Private and Public Provision of Counseling to Job Seekers: Evidence from a Large Controlled Experiment," American Economic Journal: Applied Economics, American Economic Association, vol. 6(4), pages 142-174, October.
    5. Joshua D. Angrist & Parag A. Pathak & Christopher R. Walters, 2013. "Explaining Charter School Effectiveness," American Economic Journal: Applied Economics, American Economic Association, vol. 5(4), pages 1-27, October.
    6. Karen Macours & Norbert Schady & Renos Vakis, 2012. "Cash Transfers, Behavioral Changes, and Cognitive Development in Early Childhood: Evidence from a Randomized Experiment," American Economic Journal: Applied Economics, American Economic Association, vol. 4(2), pages 247-273, April.
    7. Adrienne M. Lucas & Isaac M. Mbiti, 2014. "Effects of School Quality on Student Achievement: Discontinuity Evidence from Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 6(3), pages 234-263, July.
    8. Roland G. Fryer, 2011. "Financial Incentives and Student Achievement: Evidence from Randomized Trials," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1755-1798.
    9. Stefano DellaVigna & John A. List & Ulrike Malmendier, 2012. "Testing for Altruism and Social Pressure in Charitable Giving," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(1), pages 1-56.
    10. Lori Beaman & Jeremy Magruder, 2012. "Who Gets the Job Referral? Evidence from a Social Networks Experiment," American Economic Review, American Economic Association, vol. 102(7), pages 3574-3593, December.
    11. Jared Barton & Marco Castillo & Ragan Petrie, 2014. "What Persuades Voters? A Field Experiment on Political Campaigning," Economic Journal, Royal Economic Society, vol. 124(574), pages 293-326, February.
    12. Paul Collier & Pedro C. Vicente, 2014. "Votes and Violence: Evidence from a Field Experiment in Nigeria," Economic Journal, Royal Economic Society, vol. 124(574), pages 327-355, February.
    13. repec:hal:pseose:halshs-00942662 is not listed on IDEAS
    14. Robert W. Fairlie & Rebecca A. London, 2012. "The Effects of Home Computers on Educational Outcomes: Evidence from a Field Experiment with Community College Students," Economic Journal, Royal Economic Society, vol. 122(561), pages 727-753, June.
    15. Bertrand, Marianne & Karlan, Dean S. & Mullainathan, Sendhil & Shafir, Eldar & Zinman, Jonathan, 2005. "What's Psychology Worth? A Field Experiment in the Consumer Credit Market," Center Discussion Papers 28441, Yale University, Economic Growth Center.
    16. repec:hal:pseose:halshs-00840901 is not listed on IDEAS
    17. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1593-1660.
    18. Dean Karlan & Robert Osei & Isaac Osei-Akoto & Christopher Udry, 2014. "Agricultural Decisions after Relaxing Credit and Risk Constraints," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 129(2), pages 597-652.
    19. Andries De Grip & Jan Sauermann, 2012. "The Effects of Training on Own and Co‐worker Productivity: Evidence from a Field Experiment," Economic Journal, Royal Economic Society, vol. 122(560), pages 376-399, May.
    20. Francesco Avvisati & Marc Gurgand & Nina Guyon & Eric Maurin, 2014. "Getting Parents Involved: A Field Experiment in Deprived Schools," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(1), pages 57-83.
    21. Gary Charness & Marie-Claire Villeval, 2009. "Cooperation and Competition in Intergenerational Experiments in the Field and the Laboratory," American Economic Review, American Economic Association, vol. 99(3), pages 956-978, June.
    22. Angelucci, Manuela & De Giorgi, Giacomo & Rangel, Marcos A. & Rasul, Imran, 2010. "Family networks and school enrolment: Evidence from a randomized social experiment," Journal of Public Economics, Elsevier, vol. 94(3-4), pages 197-221, April.
    23. Caterina Calsamiglia & Guillaume Haeringer & Flip Klijn, 2010. "Constrained School Choice: An Experimental Study," American Economic Review, American Economic Association, vol. 100(4), pages 1860-1874, September.
    24. Karen Macours & Renos Vakis, 2014. "Changing Households' Investment Behaviour through Social Interactions with Local Leaders: Evidence from a Randomised Transfer Programme," Economic Journal, Royal Economic Society, vol. 0(576), pages 607-633, May.
    25. Alejandro Drexler & Greg Fischer & Antoinette Schoar, 2014. "Keeping It Simple: Financial Literacy and Rules of Thumb," American Economic Journal: Applied Economics, American Economic Association, vol. 6(2), pages 1-31, April.
    26. Roland G. Fryer, Jr., 2014. "Injecting Charter School Best Practices into Traditional Public Schools: Evidence from Field Experiments," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 129(3), pages 1355-1407.
    27. Bruno Crépon & Esther Duflo & Marc Gurgand & Roland Rathelot & Philippe Zamora, 2013. "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(2), pages 531-580.
    28. Schwerdt, Guido & Messer, Dolores & Woessmann, Ludger & Wolter, Stefan C., 2012. "The impact of an adult education voucher program: Evidence from a randomized field experiment," Journal of Public Economics, Elsevier, vol. 96(7-8), pages 569-583.
    29. Esther Duflo & Michael Greenstone & Nicholas Ryan, 2013. "Truth-telling by Third-party Auditors and the Response of Polluting Firms: Experimental Evidence from India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(4), pages 1499-1545.
    30. Jeffrey R. Kling & Sendhil Mullainathan & Eldar Shafir & Lee C. Vermeulen & Marian V. Wrobel, 2012. "Comparison Friction: Experimental Evidence from Medicare Drug Plans," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(1), pages 199-235.
    31. Orazio Attanasio & Abigail Barr & Juan Camilo Cardenas & Garance Genicot & Costas Meghir, 2012. "Risk Pooling, Risk Preferences, and Social Networks," American Economic Journal: Applied Economics, American Economic Association, vol. 4(2), pages 134-167, April.
    32. Vivi Alatas & Abhijit Banerjee & Rema Hanna & Benjamin A. Olken & Julia Tobias, 2012. "Targeting the Poor: Evidence from a Field Experiment in Indonesia," American Economic Review, American Economic Association, vol. 102(4), pages 1206-1240, June.
    33. Glenn W. Harrison & Morten I. Lau & E. Elisabet Rutström, 2007. "Estimating Risk Attitudes in Denmark: A Field Experiment," Scandinavian Journal of Economics, Wiley Blackwell, vol. 109(2), pages 341-368, June.
    34. Carlsson, Fredrik & Johansson-Stenman, Olof & Nam, Pham Khanh, 2014. "Social preferences are stable over long periods of time," Journal of Public Economics, Elsevier, vol. 117(C), pages 104-114.
    35. Stefan Eriksson & Dan-Olof Rooth, 2014. "Do Employers Use Unemployment as a Sorting Criterion When Hiring? Evidence from a Field Experiment," American Economic Review, American Economic Association, vol. 104(3), pages 1014-1039, March.
    36. repec:hal:pseose:halshs-00754593 is not listed on IDEAS
    37. Martina Björkman & Jakob Svensson, 2009. "Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 124(2), pages 735-769.
    38. Philip Oreopoulos & Daniel Lang & Joshua Angrist, 2009. "Incentives and Services for College Achievement: Evidence from a Randomized Trial," American Economic Journal: Applied Economics, American Economic Association, vol. 1(1), pages 136-163, January.
    39. Suresh de Mel & David McKenzie & Christopher Woodruff, 2013. "The Demand for, and Consequences of, Formalization among Informal Firms in Sri Lanka," American Economic Journal: Applied Economics, American Economic Association, vol. 5(2), pages 122-150, April.
    40. Will Dobbie & Roland G. Fryer Jr., 2013. "Getting beneath the Veil of Effective Schools: Evidence from New York City," American Economic Journal: Applied Economics, American Economic Association, vol. 5(4), pages 28-60, October.
    41. Tran, Anh & Zeckhauser, Richard, 2012. "Rank as an inherent incentive: Evidence from a field experiment," Journal of Public Economics, Elsevier, vol. 96(9-10), pages 645-650.
    42. Esther Duflo & Pascaline Dupas & Michael Kremer, 2011. "Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya," American Economic Review, American Economic Association, vol. 101(5), pages 1739-1774, August.
    43. Jonathan Robinson, 2012. "Limited Insurance within the Household: Evidence from a Field Experiment in Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 4(4), pages 140-164, October.
    44. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    45. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    46. Breman, Anna, 2011. "Give more tomorrow: Two field experiments on altruism and intertemporal choice," Journal of Public Economics, Elsevier, vol. 95(11), pages 1349-1357.
    47. Victor Lavy, 2009. "Performance Pay and Teachers' Effort, Productivity, and Grading Ethics," American Economic Review, American Economic Association, vol. 99(5), pages 1979-2011, December.
    48. Robert W. Fairlie & Jonathan Robinson, 2013. "Experimental Evidence on the Effects of Home Computers on Academic Achievement among Schoolchildren," American Economic Journal: Applied Economics, American Economic Association, vol. 5(3), pages 211-240, July.
    49. Li, Tao & Han, Li & Zhang, Linxiu & Rozelle, Scott, 2014. "Encouraging classroom peer interactions: Evidence from Chinese migrant schools," Journal of Public Economics, Elsevier, vol. 111(C), pages 29-45.
    50. Pascaline Dupas & Jonathan Robinson, 2013. "Savings Constraints and Microenterprise Development: Evidence from a Field Experiment in Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 5(1), pages 163-192, January.
    51. Nicholas Bloom & Benn Eifert & Aprajit Mahajan & David McKenzie & John Roberts, 2013. "Does Management Matter? Evidence from India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(1), pages 1-51.
    52. Alan S. Gerber & Dean Karlan & Daniel Bergan, 2009. "Does the Media Matter? A Field Experiment Measuring the Effect of Newspapers on Voting Behavior and Political Opinions," American Economic Journal: Applied Economics, American Economic Association, vol. 1(2), pages 35-52, April.
    53. Katrina Jessoe & David Rapson, 2014. "Knowledge Is (Less) Power: Experimental Evidence from Residential Energy Use," American Economic Review, American Economic Association, vol. 104(4), pages 1417-1438, April.
    54. Gustavo J. Bobonis, 2009. "Is the Allocation of Resources within the Household Efficient? New Evidence from a Randomized Experiment," Journal of Political Economy, University of Chicago Press, vol. 117(3), pages 453-503, June.
    55. Xavier Gine & Jessica Goldberg & Dean Yang, 2012. "Credit Market Consequences of Improved Personal Identification: Field Experimental Evidence from Malawi," American Economic Review, American Economic Association, vol. 102(6), pages 2923-2954, October.
    56. Jishnu Das & Stefan Dercon & James Habyarimana & Pramila Krishnan & Karthik Muralidharan & Venkatesh Sundararaman, 2013. "School Inputs, Household Substitution, and Test Scores," American Economic Journal: Applied Economics, American Economic Association, vol. 5(2), pages 29-57, April.
    57. Jessica Wisdom & Julie S. Downs & George Loewenstein, 2010. "Promoting Healthy Choices: Information versus Convenience," American Economic Journal: Applied Economics, American Economic Association, vol. 2(2), pages 164-178, April.
    58. Banerjee, Abhijit & Bardhan, Pranab & Basu, Kaushik, 2005. "New Directions in Development Economics: Theory or Empirics? A Symposium in Economic and Political Weekly," Working Papers 127128, Cornell University, Department of Applied Economics and Management.
    59. Erica Field, 2009. "Educational Debt Burden and Career Choice: Evidence from a Financial Aid Experiment at NYU Law School," American Economic Journal: Applied Economics, American Economic Association, vol. 1(1), pages 1-21, January.
    60. Carrell, Scott E. & Hoekstra, Mark & West, James E., 2011. "Is poor fitness contagious?," Journal of Public Economics, Elsevier, vol. 95(7), pages 657-663.
    61. Pascaline Dupas, 2011. "Do Teenagers Respond to HIV Risk Information? Evidence from a Field Experiment in Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 3(1), pages 1-34, January.
    62. Sylvain Chassang & Gerard Padro I Miquel & Erik Snowberg, 2012. "Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments," American Economic Review, American Economic Association, vol. 102(4), pages 1279-1309, June.
    63. Erica Field & Rohini Pande & John Papp & Natalia Rigol, 2013. "Does the Classic Microfinance Model Discourage Entrepreneurship among the Poor? Experimental Evidence from India," American Economic Review, American Economic Association, vol. 103(6), pages 2196-2226, October.
    64. Michal Bauer & Julie Chytilova & Jonathan Morduch, 2012. "Behavioral Foundations of Microcredit: Experimental and Survey Evidence from Rural India," American Economic Review, American Economic Association, vol. 102(2), pages 1118-1139, April.
    65. Mueller, Steffen, 2013. "Teacher experience and the class size effect — Experimental evidence," Journal of Public Economics, Elsevier, vol. 98(C), pages 44-52.
    66. M. Fong, Christina & Oberholzer-Gee, Felix, 2011. "Truth in giving: Experimental evidence on the welfare effects of informed giving to the poor," Journal of Public Economics, Elsevier, vol. 95(5-6), pages 436-444, June.
    67. Karlan, Dean & List, John A. & Shafir, Eldar, 2011. "Small matches and charitable giving: Evidence from a natural field experiment," Journal of Public Economics, Elsevier, vol. 95(5), pages 344-350.
    68. David S. Lyle, 2009. "The Effects of Peer Group Heterogeneity on the Production of Human Capital at West Point," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 69-84, October.
    69. Andreas Ravndal Kostol & Magne Mogstad, 2014. "How Financial Incentives Induce Disability Insurance Recipients to Return to Work," American Economic Review, American Economic Association, vol. 104(2), pages 624-655, February.
    70. Sebastian Kube & Michel Andre Marechal & Clemens Puppe, 2012. "The Currency of Reciprocity: Gift Exchange in the Workplace," American Economic Review, American Economic Association, vol. 102(4), pages 1644-1662, June.
    71. Timothy J. Besley & Konrad B. Burchardi & Maitreesh Ghatak, 2012. "Incentives and the De Soto Effect," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(1), pages 237-282.
    72. Raj Chetty & Adam Looney & Kory Kroft, 2009. "Salience and Taxation: Theory and Evidence," American Economic Review, American Economic Association, vol. 99(4), pages 1145-1177, September.
    73. Rafael Di Tella & Ernesto Schargrodsky, 2013. "Criminal Recidivism after Prison and Electronic Monitoring," Journal of Political Economy, University of Chicago Press, vol. 121(1), pages 28-73.
    74. Aaron Sojourner, 2013. "Identification of Peer Effects with Missing Peer Data: Evidence from Project STAR," Economic Journal, Royal Economic Society, vol. 123(569), pages 574-605, June.
    75. Robert Jensen, 2012. "Do Labor Market Opportunities Affect Young Women's Work and Family Decisions? Experimental Evidence from India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(2), pages 753-792.
    76. Huck, Steffen & Rasul, Imran, 2011. "Matched fundraising: Evidence from a natural field experiment," Journal of Public Economics, Elsevier, vol. 95(5-6), pages 351-362, June.
    77. Dean Karlan & Jonathan Zinman, 2009. "Observing Unobservables: Identifying Information Asymmetries With a Consumer Credit Field Experiment," Econometrica, Econometric Society, vol. 77(6), pages 1993-2008, November.
    78. Karthik Muralidharan & Venkatesh Sundararaman, 2010. "The Impact of Diagnostic Feedback to Teachers on Student Learning: Experimental Evidence from India," Economic Journal, Royal Economic Society, vol. 120(546), pages 187-203, August.
    79. McManus, Brian & Bennet, Richard, 2011. "The demand for products linked to public goods: Evidence from an online field experiment," Journal of Public Economics, Elsevier, vol. 95(5-6), pages 403-415, June.
    80. Marianne Bertrand & Dean Karlan & Sendhil Mullainathan & Eldar Shafir & Jonathan Zinman, 2010. "What's Advertising Content Worth? Evidence from a Consumer Credit Marketing Field Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 263-306.
    81. Jonathan Guryan & Kory Kroft & Matthew J. Notowidigdo, 2009. "Peer Effects in the Workplace: Evidence from Random Groupings in Professional Golf Tournaments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 34-68, October.
    82. Alois Stutzer & Lorenz Goette & Michael Zehnder, 2011. "Active Decisions and Prosocial Behaviour: a Field Experiment on Blood Donation," Economic Journal, Royal Economic Society, vol. 121(556), pages 476-493, November.
    83. Michael Kremer & Jessica Leino & Edward Miguel & Alix Peterson Zwane, 2011. "Spring Cleaning: Rural Water Impacts, Valuation, and Property Rights Institutions," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(1), pages 145-205.
    84. Xavier Giné & Dean Karlan & Jonathan Zinman, 2010. "Put Your Money Where Your Butt Is: A Commitment Contract for Smoking Cessation," American Economic Journal: Applied Economics, American Economic Association, vol. 2(4), pages 213-235, October.
    85. Menno Pradhan & Daniel Suryadarma & Amanda Beatty & Maisy Wong & Arya Gaduh & Armida Alisjahbana & Rima Prama Artha, 2014. "Improving Educational Quality through Enhancing Community Participation: Results from a Randomized Field Experiment in Indonesia," American Economic Journal: Applied Economics, American Economic Association, vol. 6(2), pages 105-126, April.
    86. Nava Ashraf & James Berry & Jesse M. Shapiro, 2010. "Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia," American Economic Review, American Economic Association, vol. 100(5), pages 2383-2413, December.
    87. Orazio P. Attanasio & Costas Meghir & Ana Santiago, 2012. "Education Choices in Mexico: Using a Structural Model and a Randomized Experiment to Evaluate PROGRESA," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 79(1), pages 37-66.
    88. Paul Glewwe & Nauman Ilias & Michael Kremer, 2010. "Teacher Incentives," American Economic Journal: Applied Economics, American Economic Association, vol. 2(3), pages 205-227, July.
    89. Gary Charness & Uri Gneezy, 2009. "Incentives to Exercise," Econometrica, Econometric Society, vol. 77(3), pages 909-931, May.
    90. Castillo, Marco & Petrie, Ragan & Wardell, Clarence, 2014. "Fundraising through online social networks: A field experiment on peer-to-peer solicitation," Journal of Public Economics, Elsevier, vol. 114(C), pages 29-35.
    91. Seán M. Muller, 2015. "Causal Interaction and External Validity: Obstacles to the Policy Relevance of Randomized Evaluations," The World Bank Economic Review, World Bank, vol. 29(suppl_1), pages 217-225.
    92. Hongbin Cai & Yuyu Chen & Hanming Fang & Li-An Zhou, 2009. "Microinsurance, Trust and Economic Development: Evidence from a Randomized Natural Field Experiment," PIER Working Paper Archive 09-034, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania.
    93. Jan Stoop & Charles N. Noussair & Daan van Soest, 2012. "From the Lab to the Field: Cooperation among Fishermen," Journal of Political Economy, University of Chicago Press, vol. 120(6), pages 1027-1056.
    94. Sylvie Moulin & Michael Kremer & Paul Glewwe, 2009. "Many Children Left Behind? Textbooks and Test Scores in Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 1(1), pages 112-135, January.
    95. Bennmarker, Helge & Grönqvist, Erik & Öckert, Björn, 2013. "Effects of contracting out employment services: Evidence from a randomized experiment," Journal of Public Economics, Elsevier, vol. 98(C), pages 68-84.
    96. Hongbin Cai & Yuyu Chen & Hanming Fang, 2009. "Observational Learning: Evidence from a Randomized Natural Field Experiment," American Economic Review, American Economic Association, vol. 99(3), pages 864-882, June.
    97. Ashraf, Nava & Bandiera, Oriana & Jack, B. Kelsey, 2014. "No margin, no mission? A field experiment on incentives for public service delivery," Journal of Public Economics, Elsevier, vol. 120(C), pages 1-17.
    98. Telle, Kjetil, 2013. "Monitoring and enforcement of environmental regulations," Journal of Public Economics, Elsevier, vol. 99(C), pages 24-34.
    99. Maarten J. Voors & Eleonora E. M. Nillesen & Philip Verwimp & Erwin H. Bulte & Robert Lensink & Daan P. Van Soest, 2012. "Violent Conflict and Behavior: A Field Experiment in Burundi," American Economic Review, American Economic Association, vol. 102(2), pages 941-964, April.
    100. Pascaline Dupas, 2014. "Short‐Run Subsidies and Long‐Run Adoption of New Health Products: Evidence From a Field Experiment," Econometrica, Econometric Society, vol. 82(1), pages 197-228, January.
    101. Temple, Jonathan R.W., 2010. "Aid and Conditionality," Handbook of Development Economics, in: Dani Rodrik & Mark Rosenzweig (ed.), Handbook of Development Economics, edition 1, volume 5, chapter 0, pages 4415-4523, Elsevier.
    102. Leonardo Bursztyn & Lucas C. Coffman, 2012. "The Schooling Decision: Family Preferences, Intergenerational Conflict, and Moral Hazard in the Brazilian Favelas," Journal of Political Economy, University of Chicago Press, vol. 120(3), pages 359-397.
    103. Paul J. Gertler & Sebastian W. Martinez & Marta Rubio-Codina, 2012. "Investing Cash Transfers to Raise Long-Term Living Standards," American Economic Journal: Applied Economics, American Economic Association, vol. 4(1), pages 164-192, January.
    104. Raj Chetty & Emmanuel Saez, 2013. "Teaching the Tax Code: Earnings Responses to an Experiment with EITC Recipients," American Economic Journal: Applied Economics, American Economic Association, vol. 5(1), pages 1-31, January.
    105. Brian A. Jacob & Jens Ludwig, 2012. "The Effects of Housing Assistance on Labor Supply: Evidence from a Voucher Lottery," American Economic Review, American Economic Association, vol. 102(1), pages 272-304, February.
    106. Habyarimana, James & Jack, William, 2011. "Heckle and Chide: Results of a randomized road safety intervention in Kenya," Journal of Public Economics, Elsevier, vol. 95(11), pages 1438-1446.
    107. Benjamin A. Olken & Junko Onishi & Susan Wong, 2014. "Should Aid Reward Performance? Evidence from a Field Experiment on Health and Education in Indonesia," American Economic Journal: Applied Economics, American Economic Association, vol. 6(4), pages 1-34, October.
    108. Olivier Armantier & Amadou Boly, 2013. "Comparing Corruption in the Laboratory and in the Field in Burkina Faso and in Canada," Economic Journal, Royal Economic Society, vol. 123(12), pages 1168-1187, December.
    109. Esther Duflo & Michael Kremer & Jonathan Robinson, 2011. "Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya," American Economic Review, American Economic Association, vol. 101(6), pages 2350-2390, October.
    110. repec:hal:pseose:halshs-01067926 is not listed on IDEAS
    111. Atila Abdulkadiroğlu & Joshua D. Angrist & Susan M. Dynarski & Thomas J. Kane & Parag A. Pathak, 2011. "Accountability and Flexibility in Public Schools: Evidence from Boston's Charters And Pilots," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(2), pages 699-748.
    112. repec:feb:framed:0087 is not listed on IDEAS
    113. Emily Oster & Rebecca Thornton, 2011. "Menstruation, Sanitary Products, and School Attendance: Evidence from a Randomized Evaluation," American Economic Journal: Applied Economics, American Economic Association, vol. 3(1), pages 91-100, January.
    114. Scott E. Carrell & James E. West, 2010. "Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors," Journal of Political Economy, University of Chicago Press, vol. 118(3), pages 409-432, June.
    115. repec:hal:pseose:halshs-01053534 is not listed on IDEAS
    116. Damon Jones, 2010. "Information, Preferences, and Public Benefit Participation: Experimental Evidence from the Advance EITC and 401(k) Savings," American Economic Journal: Applied Economics, American Economic Association, vol. 2(2), pages 147-163, April.
    117. Cilliers, Jacobus & Dube, Oeindrila & Siddiqi, Bilal, 2015. "The white-man effect: How foreigner presence affects behavior in experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 118(C), pages 397-414.
    118. M. Fong, Christina & Oberholzer-Gee, Felix, 2011. "Truth in giving: Experimental evidence on the welfare effects of informed giving to the poor," Journal of Public Economics, Elsevier, vol. 95(5), pages 436-444.
    119. Suresh de Mel & David McKenzie & Christopher Woodruff, 2009. "Are Women More Credit Constrained? Experimental Evidence on Gender and Microenterprise Returns," American Economic Journal: Applied Economics, American Economic Association, vol. 1(3), pages 1-32, July.
    120. Jobiba Chinkhumba & Susan Godlonton & Rebecca Thornton, 2014. "The Demand for Medical Male Circumcision," American Economic Journal: Applied Economics, American Economic Association, vol. 6(2), pages 152-177, April.
    121. Jessica Cohen & Pascaline Dupas, 2008. "Free Distribution or Cost-Sharing? Evidence from a Malaria Prevention Experiment," NBER Working Papers 14406, National Bureau of Economic Research, Inc.
    122. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    123. Kory Kroft & Fabian Lange & Matthew J. Notowidigdo, 2013. "Duration Dependence and Labor Market Conditions: Evidence from a Field Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(3), pages 1123-1167.
    124. Uri Gneezy & Kenneth L. Leonard & John A. List, 2009. "Gender Differences in Competition: Evidence From a Matrilineal and a Patriarchal Society," Econometrica, Econometric Society, vol. 77(5), pages 1637-1664, September.
    125. Benjamin Feigenberg & Erica Field & Rohini Pande, 2013. "The Economic Returns to Social Interaction: Experimental Evidence from Microfinance," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 80(4), pages 1459-1483.
    126. Christina M. Fong & Erzo F. P. Luttmer, 2009. "What Determines Giving to Hurricane Katrina Victims? Experimental Evidence on Racial Group Loyalty," American Economic Journal: Applied Economics, American Economic Association, vol. 1(2), pages 64-87, April.
    127. Cerqua, Augusto & Pellegrini, Guido, 2014. "Do subsidies to private capital boost firms' growth? A multiple regression discontinuity design approach," Journal of Public Economics, Elsevier, vol. 109(C), pages 114-126.
    128. Nava Ashraf, 2009. "Spousal Control and Intra-household Decision Making: An Experimental Study in the Philippines," American Economic Review, American Economic Association, vol. 99(4), pages 1245-1277, September.
    129. Esther Duflo & Rema Hanna & Stephen P. Ryan, 2012. "Incentives Work: Getting Teachers to Come to School," American Economic Review, American Economic Association, vol. 102(4), pages 1241-1278, June.
    130. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(3), pages 1117-1165.
    131. Katherine Casey & Rachel Glennerster & Edward Miguel, 2012. "Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(4), pages 1755-1812.
    132. Jenny C. Aker & Christopher Ksoll & Travis J. Lybbert, 2012. "Can Mobile Phones Improve Learning? Evidence from a Field Experiment in Niger," American Economic Journal: Applied Economics, American Economic Association, vol. 4(4), pages 94-120, October.
    133. Yan Chen & F. Maxwell Harper & Joseph Konstan & Sherry Xin Li, 2010. "Social Comparisons and Contributions to Online Communities: A Field Experiment on MovieLens," American Economic Review, American Economic Association, vol. 100(4), pages 1358-1398, September.
    134. repec:ecj:econjl:v:122:y:2012:i::p:376-399 is not listed on IDEAS
    135. Beekman, Gonne & Bulte, Erwin & Nillesen, Eleonora, 2014. "Corruption, investments and contributions to public goods: Experimental evidence from rural Liberia," Journal of Public Economics, Elsevier, vol. 115(C), pages 37-47.
    136. Nava Ashraf & Erica Field & Jean Lee, 2014. "Household Bargaining and Excess Fertility: An Experimental Study in Zambia," American Economic Review, American Economic Association, vol. 104(7), pages 2210-2237, July.
    137. Jonathan Levav & Mark Heitmann & Andreas Herrmann & Sheena S. Iyengar, 2010. "Order in Product Customization Decisions: Evidence from Field Experiments," Journal of Political Economy, University of Chicago Press, vol. 118(2), pages 274-299, April.
    138. Martin Ravallion, 2012. "Fighting Poverty One Experiment at a Time: Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty : Review Essay," Journal of Economic Literature, American Economic Association, vol. 50(1), pages 103-114, March.
    139. repec:oup:qjecon:v:129:y:2014:i:2:p:697-752. is not listed on IDEAS
    140. Allcott, Hunt, 2011. "Social norms and energy conservation," Journal of Public Economics, Elsevier, vol. 95(9-10), pages 1082-1095, October.
    141. Jonah E. Rockoff & Douglas O. Staiger & Thomas J. Kane & Eric S. Taylor, 2012. "Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools," American Economic Review, American Economic Association, vol. 102(7), pages 3184-3213, December.
    142. Manuel F. Bagues & Berta Esteve-Volart, 2010. "Can Gender Parity Break the Glass Ceiling? Evidence from a Repeated Randomized Experiment," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 77(4), pages 1301-1328.
    143. Jen Shang & Rachel Croson, 2009. "A Field Experiment in Charitable Contribution: The Impact of Social Information on the Voluntary Provision of Public Goods," Economic Journal, Royal Economic Society, vol. 119(540), pages 1422-1439, October.
    144. Sarah Baird & Craig McIntosh & Berk Özler, 2011. "Cash or Condition? Evidence from a Cash Transfer Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1709-1753.
    145. Karen Macours & Renos Vakis, 2014. "Changing Households' Investment Behaviour through Social Interactions with Local Leaders: Evidence from a Randomised Transfer Programme," Post-Print halshs-01313720, HAL.
    146. Orazio Attanasio & Adriana Kugler & Costas Meghir, 2011. "Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial," American Economic Journal: Applied Economics, American Economic Association, vol. 3(3), pages 188-220, July.
    147. Dana Burde & Leigh L. Linden, 2013. "Bringing Education to Afghan Girls: A Randomized Controlled Trial of Village-Based Schools," American Economic Journal: Applied Economics, American Economic Association, vol. 5(3), pages 27-40, July.
    148. Dehejia Rajeev, 2015. "Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic," Journal of Globalization and Development, De Gruyter, vol. 6(1), pages 47-69, June.
    149. Christopher Blattman & Nathan Fiala & Sebastian Martinez, 2014. "Generating Skilled Self-Employment in Developing Countries: Experimental Evidence from Uganda," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 129(2), pages 697-752.
    150. repec:idb:brikps:publication-detail,7101.html?id=32886 is not listed on IDEAS
    151. Karen Macours & Norbert Schady & Renos Vakis, 2012. "Cash Transfers, Behavioral Changes, and Cognitive Development in Early Childhood: Evidence from a Randomized Experiment," American Economic Journal: Applied Economics, American Economic Association, vol. 4(2), pages 247-273, April.
    152. Henrik Jacobsen Kleven & Martin B. Knudsen & Claus Thustrup Kreiner & Søren Pedersen & Emmanuel Saez, 2011. "Unwilling or Unable to Cheat? Evidence From a Tax Audit Experiment in Denmark," Econometrica, Econometric Society, vol. 79(3), pages 651-692, May.
    153. Joshua Angrist & Victor Lavy, 2009. "The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial," American Economic Review, American Economic Association, vol. 99(4), pages 1384-1414, September.
    154. Jessica Cohen & Pascaline Dupas, 2010. "Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 1-45.
    155. Karthik Muralidharan & Venkatesh Sundararaman, 2011. "Teacher Performance Pay: Experimental Evidence from India," Journal of Political Economy, University of Chicago Press, vol. 119(1), pages 39-77.
    156. Rema Hanna & Sendhil Mullainathan & Joshua Schwartzstein, 2014. "Learning Through Noticing: Theory and Evidence from a Field Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 129(3), pages 1311-1353.
    157. Hunt Allcott & Todd Rogers, 2014. "The Short-Run and Long-Run Effects of Behavioral Interventions: Experimental Evidence from Energy Conservation," American Economic Review, American Economic Association, vol. 104(10), pages 3003-3037, October.
    158. Torun Dewan & Macartan Humphreys & Daniel Rubenson, 2014. "The Elements of Political Persuasion: Content, Charisma and Cue," Economic Journal, Royal Economic Society, vol. 124(574), pages 257-292, February.
    159. Allcott, Hunt, 2011. "Social norms and energy conservation," Journal of Public Economics, Elsevier, vol. 95(9), pages 1082-1095.
    160. Lant Pritchett & Justin Sandefur, 2015. "Learning from Experiments When Context Matters," American Economic Review, American Economic Association, vol. 105(5), pages 471-475, May.
    161. Suresh de Mel & David McKenzie & Christopher Woodruff, 2009. "Returns to Capital in Microenterprises: Evidence from a Field Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 124(1), pages 423-423.
    162. Robert Jensen, 2010. "The (Perceived) Returns to Education and the Demand for Schooling," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(2), pages 515-548.
    163. McManus, Brian & Bennet, Richard, 2011. "The demand for products linked to public goods: Evidence from an online field experiment," Journal of Public Economics, Elsevier, vol. 95(5), pages 403-415.
    164. Thomas Fujiwara & Leonard Wantchekon, 2013. "Can Informed Public Deliberation Overcome Clientelism? Experimental Evidence from Benin," American Economic Journal: Applied Economics, American Economic Association, vol. 5(4), pages 241-255, October.
    165. Amanda Pallais, 2014. "Inefficient Hiring in Entry-Level Labor Markets," American Economic Review, American Economic Association, vol. 104(11), pages 3565-3599, November.
    166. Eric P. Bettinger & Bridget Terry Long & Philip Oreopoulos & Lisa Sanbonmatsu, 2012. "The Role of Application Assistance and Information in College Decisions: Results from the H&R Block Fafsa Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(3), pages 1205-1242.
    167. Erwin Bulte & Gonne Beekman & Salvatore Di Falco & Joseph Hella & Pan Lei, 2014. "Behavioral Responses and the Impact of New Agricultural Technologies: Evidence from a Double-blind Field Experiment in Tanzania," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 96(3), pages 813-830.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Andor, Mark A. & Gerster, Andreas & Peters, Jörg & Schmidt, Christoph M., 2020. "Social Norms and Energy Conservation Beyond the US," Journal of Environmental Economics and Management, Elsevier, vol. 103(C).
    2. Vikram Tyagi & Sophie Webber, 2021. "A rusting gold standard: Failures in an Indonesian RCT, and the implications for poverty reduction," Environment and Planning A, , vol. 53(5), pages 992-1011, August.
    3. Hoffmann, Nimi, 2020. "Involuntary experiments in former colonies: The case for a moratorium," World Development, Elsevier, vol. 127(C).
    4. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    5. Bensch, Gunther & Kluve, Jochen & Stöterau, Jonathan, 2021. "The market-based dissemination of energy-access technologies as a business model for rural entrepreneurs: Evidence from Kenya," Resource and Energy Economics, Elsevier, vol. 66(C).
    6. Jaime DE MELO & Marcelo OLARREAGA, 2017. "Trade Related Institutions and Development," Working Papers P199, FERDI.
    7. Yokoo, Hide-Fumi & 横尾, 英史, 2020. "Ethics of randomized field experiments: Evidence from a randomized survey experiment," Discussion Papers 2020-07, Graduate School of Economics, Hitotsubashi University.
    8. Schilpzand, Annemiek & de Jong, Eelke, 2023. "Do market societies undermine civic morality? An empirical investigation into market societies and civic morality across the globe," Journal of Economic Behavior & Organization, Elsevier, vol. 208(C), pages 39-60.
    9. Michael Rothgang & Bernhard Lageman & Anne-Marie Scholz, 2021. "Why are there so few hard facts about the impact of cluster policies in Germany? A critical review of evaluation studies," Review of Evolutionary Political Economy, Springer, vol. 2(1), pages 105-139, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    2. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    3. Molina Millán, Teresa & Macours, Karen, 2017. "Attrition in Randomized Control Trials: Using Tracking Information to Correct Bias," IZA Discussion Papers 10711, Institute of Labor Economics (IZA).
    4. Belot, Michèle & James, Jonathan, 2016. "Partner selection into policy relevant field experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 123(C), pages 31-56.
    5. Teresa Molina Millán & Karen Macours, 2017. "Attrition in randomized control trials: Using tracking information to correct bias," FEUNL Working Paper Series novaf:wp1702, Universidade Nova de Lisboa, Faculdade de Economia.
    6. Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
    7. Benjamin A. Olken, 2020. "Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 122(3), pages 853-878, July.
    8. Damgaard, Mette Trier & Nielsen, Helena Skyt, 2018. "Nudging in education," Economics of Education Review, Elsevier, vol. 64(C), pages 313-342.
    9. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    10. Roland G. Fryer, Jr, 2016. "The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments," NBER Working Papers 22130, National Bureau of Economic Research, Inc.
    11. Figlio, D. & Karbownik, K. & Salvanes, K.G., 2016. "Education Research and Administrative Data," Handbook of the Economics of Education,, Elsevier.
    12. Pascaline Dupas & Edward Miguel, 2016. "Impacts and Determinants of Health Levels in Low-Income Countries," NBER Working Papers 22235, National Bureau of Economic Research, Inc.
    13. Karthik Muralidharan & Mauricio Romero & Kaspar Wüthrich, 2019. "Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments," NBER Working Papers 26562, National Bureau of Economic Research, Inc.
    14. Young, Alwyn, 2024. "Asymptotically robust permutation-based randomization confidence intervals for parametric OLS regression," European Economic Review, Elsevier, vol. 163(C).
    15. Eduard Marinov, 2019. "The 2019 Nobel Prize in Economics," Economic Thought journal, Bulgarian Academy of Sciences - Economic Research Institute, issue 6, pages 78-116.
    16. Young, Alwyn, 2024. "Asymptotically robust permutation-based randomization confidence intervals for parametric OLS regression," LSE Research Online Documents on Economics 120933, London School of Economics and Political Science, LSE Library.
    17. B Kelsey Jack, "undated". "Market Inefficiencies and the Adoption of Agricultural Technologies in Developing Countries," CID Working Papers 50, Center for International Development at Harvard University.
    18. Harounan Kazianga & Leigh Linden & Ali Protik & Matt Sloan, 2015. "Impact Evaluation of Burkina Faso's BRIGHT Program: Design Report," Mathematica Policy Research Reports c0250cd3f27d448ea70d909c3, Mathematica Policy Research.
    19. Kast, Felipe & Meier, Stephan & Pomeranz, Dina, 2018. "Saving more in groups: Field experimental evidence from Chile," Journal of Development Economics, Elsevier, vol. 133(C), pages 275-294.
    20. Eric Bettinger & Nina Cunha & Guilherme Lichand & Ricardo Madeira, 2020. "Are the effects of informational interventions driven by salience?," ECON - Working Papers 350, Department of Economics - University of Zurich, revised May 2021.

    More about this item

    Keywords

    systematic review; internal validity; external validity; randomized controlled trials;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:rwirep:589. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/rwiesde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.