IDEAS home Printed from https://ideas.repec.org/a/ucp/ecdecc/v58y2010i3p415-447.html
   My bibliography  Save this article

Matching the Gold Standard: Comparing Experimental and Nonexperimental Evaluation Techniques for a Geographically Targeted Program

Author

Listed:
  • Sudhanshu Handa
  • John A. Maluccio

Abstract

We compare nonexperimental impact estimates based on matching methods with those from a randomized evaluation to determine whether the nonexperimental approach can "match" the so-called gold standard. The social experiment we use was carried out to evaluate a geographically targeted conditional cash transfer antipoverty program in Nicaragua. The outcomes we assess include several components of household expenditure and a variety of children's health outcomes, including breast-feeding, vaccinations, and morbidity. We find that using each of the following improves performance of matching for these outcomes: (1) geographically proximate comparison samples, (2) stringent common support requirements, and (3) both geographic- and household-level matching variables. Even for a geographically targeted program, in which the selection is at the geographic, rather than at the individual or household level, and in which it is not possible to find comparison individuals or households in the program locales, matching can perform reasonably well. The results also suggest that the techniques may be more promising for evaluating the more easily measured individual-level binary outcomes than for outcomes that are more difficult to measure, such as expenditure. (c) 2010 by The University of Chicago. All rights reserved.

Suggested Citation

  • Sudhanshu Handa & John A. Maluccio, 2010. "Matching the Gold Standard: Comparing Experimental and Nonexperimental Evaluation Techniques for a Geographically Targeted Program," Economic Development and Cultural Change, University of Chicago Press, vol. 58(3), pages 415-447, April.
    • Handle: RePEc:ucp:ecdecc:v:58:y:2010:i:3:p:415-447
    DOI: 10.1086/650421
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1086/650421
    File Function: link to full text
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1086/650421?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    2. McKenzie, David & Gibson, John & Stillman, Steven, 2006. "How important is selection ? Experimental versus non-experimental measures of the income gains from migration," Policy Research Working Paper Series 3906, The World Bank.
    3. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    4. John A. Maluccio, 2009. "Household targeting in practice: The Nicaraguan Red de Protección Social," Journal of International Development, John Wiley & Sons, Ltd., vol. 21(1), pages 1-23.
    5. Daniel O. Gilligan & John Hoddinott, 2007. "Is There Persistence in the Impact of Emergency Food Aid? Evidence on Consumption, Food Security, and Assets in Rural Ethiopia," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 89(2), pages 225-242.
    6. James Heckman & Salvador Navarro-Lozano, 2004. "Using Matching, Instrumental Variables, and Control Functions to Estimate Economic Choice Models," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 30-57, February.
    7. Friedlander, Daniel & Robins, Philip K, 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods," American Economic Review, American Economic Association, vol. 85(4), pages 923-937, September.
    8. David McKenzie & John Gibson & Steven Stillman, 2010. "How Important Is Selection? Experimental vs. Non-Experimental Measures of the Income Gains from Migration," Journal of the European Economic Association, MIT Press, vol. 8(4), pages 913-945, June.
    9. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2006. "Moving the Goalposts: Addressing Limited Overlap in the Estimation of Average Treatment Effects by Changing the Estimand," NBER Technical Working Papers 0330, National Bureau of Economic Research, Inc.
    10. Todd, Petra E., 2008. "Evaluating Social Programs with Endogenous Program Placement and Selection of the Treated," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 60, pages 3847-3894, Elsevier.
    11. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    12. Alberto Abadie & Guido W. Imbens, 2008. "On the Failure of the Bootstrap for Matching Estimators," Econometrica, Econometric Society, vol. 76(6), pages 1537-1557, November.
    13. Daniel O. Gilligan & John Hoddinott, 2007. "Is There Persistence in the Impact of Emergency Food Aid? Evidence on Consumption, Food Security, and Assets in Rural Ethiopia," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 89(2), pages 225-242.
    14. Maluccio, John A. & Flores, Rafael, 2005. "Impact evaluation of a conditional cash transfer program: the Nicaraguan Red de Protección Social," Research reports 141, International Food Policy Research Institute (IFPRI).
    15. Menno Pradhan & Laura B. Rawlings, 2002. "The Impact and Targeting of Social Infrastructure Investments: Lessons from the Nicaraguan Social Fund," The World Bank Economic Review, World Bank, vol. 16(2), pages 275-295, August.
    16. Barham, Tania & Maluccio, John A., 2009. "Eradicating diseases: The effect of conditional cash transfers on vaccination coverage in rural Nicaragua," Journal of Health Economics, Elsevier, vol. 28(3), pages 611-621, May.
    17. Alberto Abadie & David Drukker & Jane Leber Herr & Guido W. Imbens, 2004. "Implementing matching estimators for average treatment effects in Stata," Stata Journal, StataCorp LP, vol. 4(3), pages 290-311, September.
    18. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    19. de Brauw, Alan & Hoddinott, John, 2011. "Must conditional cash transfer programs be conditioned to be effective? The impact of conditioning transfers on school enrollment in Mexico," Journal of Development Economics, Elsevier, vol. 96(2), pages 359-370, November.
    20. Jalan, Jyotsna & Ravallion, Martin, 2003. "Estimating the Benefit Incidence of an Antipoverty Program by Propensity-Score Matching," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(1), pages 19-30, January.
    21. Godtland, Erin M & Sadoulet, Elisabeth & De Janvry, Alain & Murgai, Rinku & Ortiz, Oscar, 2004. "The Impact of Farmer Field Schools on Knowledge and Productivity: A Study of Potato Farmers in the Peruvian Andes," Economic Development and Cultural Change, University of Chicago Press, vol. 53(1), pages 63-92, October.
    22. Barbara Sianesi, 2004. "An Evaluation of the Swedish System of Active Labor Market Programs in the 1990s," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 133-155, February.
    23. Thomas D. Cook & William R. Shadish & Vivian C. Wong, 2008. "Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 27(4), pages 724-750.
    24. Deaton, A. & Zaidi, S., 1999. "Guidelines for Constructing Consumption Aggregates for Welfare Analysis," Papers 192, Princeton, Woodrow Wilson School - Development Studies.
    25. Roberto Agodini & Mark Dynarski, "undated". "Are Experiments the Only Option? A Look at Dropout Prevention Programs," Mathematica Policy Research Reports 51241adbf9fa4a26add6d54c5, Mathematica Policy Research.
    26. Roberto Agodini & Mark Dynarski, 2004. "Are Experiments the Only Option? A Look at Dropout Prevention Programs," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 180-194, February.
    27. Charles Michalopoulos & Howard S. Bloom & Carolyn J. Hill, 2004. "Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs?," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 156-179, February.
    28. World Bank, 2003. "Nicaragua - Poverty Assessment : Raising Welfare and Reducing Vulnerability," World Bank Publications - Reports 14668, The World Bank Group.
    29. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 65(2), pages 261-294.
    30. David I. Levine & Gary Painter, 2003. "The Schooling Costs of Teenage Out-of-Wedlock Childbearing: Analysis with a Within-School Propensity-Score-Matching Estimator," The Review of Economics and Statistics, MIT Press, vol. 85(4), pages 884-900, November.
    31. Angus Deaton & Salman Zaidi, 2002. "Guidelines for Constructing Consumption Aggregates for Welfare Analysis," World Bank Publications, The World Bank, number 14101, April.
    32. Skoufias, Emmanuel, 2005. "PROGRESA and its impacts on the welfare of rural households in Mexico:," Research reports 139, International Food Policy Research Institute (IFPRI).
    33. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Azizbek Tokhirov, 2018. "Remittances and subjective well-being of the left behinds in Tajikistan," Economics Bulletin, AccessEcon, vol. 38(4), pages 1735-1747.
    2. Robert D. Osei & Monica Lambon‐Quayefio, 2021. "Cash transfers and the supply of labor by poor households: Evidence from the livelihood empowerment against poverty program in Ghana," Review of Development Economics, Wiley Blackwell, vol. 25(3), pages 1293-1304, August.
    3. Rafael Perez Ribas & Fabio Veras Soares & Clarissa Gondim Teixeira & Elydia Silva & Guilherme Issamu Hirata, 2010. "Beyond Cash: Assessing Externality and Behaviour Effects of Non-Experimental Cash Transfers," Working Papers 65, International Policy Centre for Inclusive Growth.
    4. Ferraro, Paul J. & Miranda, Juan José, 2014. "The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 344-365.
    5. Mathenge, Mary K. & Smale, Melinda & Olwande, John, 2012. "The Impact of Maize Hybrids on Income, Poverty, and Inequality among Smallholder Farmers in Kenya," Working Papers 202591, Egerton University, Tegemeo Institute of Agricultural Policy and Development.
    6. Bernd Hardeweg & Lukas Menkhoff & Hermann Waibel, 2013. "Experimentally Validated Survey Evidence on Individual Risk Attitudes in Rural Thailand," Economic Development and Cultural Change, University of Chicago Press, vol. 61(4), pages 859-888.
    7. Vivian C. Wong & Peter M. Steiner & Kylie L. Anglin, 2018. "What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?," Evaluation Review, , vol. 42(2), pages 147-175, April.
    8. Henrik Hansen & Ninja Ritter Klejnstrup & Ole Winckler Andersen, 2011. "A Comparison of Model-based and Design-based Impact Evaluations of Interventions in Developing Countries," IFRO Working Paper 2011/16, University of Copenhagen, Department of Food and Resource Economics.
    9. Hugh Sharma Waddington & Paul Fenton Villar & Jeffrey C. Valentine, 2023. "Can Non-Randomised Studies of Interventions Provide Unbiased Effect Estimates? A Systematic Review of Internal Replication Studies," Evaluation Review, , vol. 47(3), pages 563-593, June.
    10. Richard de Groot & Sudhanshu Handa & Mike Park & Robert D. Osei & Isaac Osei-Akoto & Luigi Peter Ragno & Garima Bhalla, 2015. "Heterogeneous impacts of an unconditioal cash transfer programme on schooling: evidence from the Ghana LEAP programme," Papers inwopa793, Innocenti Working Papers.
    11. Smith, Lisa C. & Khan, Faheem & Frankenberger, Timothy R. & Wadud, A.K.M. Abdul, 2013. "Admissible Evidence in the Court of Development Evaluation? The Impact of CARE’s SHOUHARDO Project on Child Stunting in Bangladesh," World Development, Elsevier, vol. 41(C), pages 196-216.
    12. Matthew Walsh & Santiago Poy & Ianina Tuñón, 2020. "The Impact of Health Conditionalities in Conditional Cash Transfer Programmes: the case of the AUH in Argentina," Revista Desarrollo y Sociedad, Universidad de los Andes,Facultad de Economía, CEDE, vol. 85(4), June.
    13. Bruno Martorano & Sudhanshu Handa & Carolyn Halpern & Harsha Thirumurthy, 2014. "Subjective Well-being, Risk Perceptions and Time Discounting: Evidence from a large-scale cash transfer programme," Papers inwopa717, Innocenti Working Papers.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.
    2. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    3. Ferraro, Paul J. & Miranda, Juan José, 2014. "The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 344-365.
    4. Vivian C. Wong & Peter M. Steiner & Kylie L. Anglin, 2018. "What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?," Evaluation Review, , vol. 42(2), pages 147-175, April.
    5. Carlos A. Flores & Oscar A. Mitnik, 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," Working Papers 2010-10, University of Miami, Department of Economics.
    6. Alberto Abadie & Guido W. Imbens, 2008. "On the Failure of the Bootstrap for Matching Estimators," Econometrica, Econometric Society, vol. 76(6), pages 1537-1557, November.
    7. de Brauw, Alan & Hoddinott, John, 2011. "Must conditional cash transfer programs be conditioned to be effective? The impact of conditioning transfers on school enrollment in Mexico," Journal of Development Economics, Elsevier, vol. 96(2), pages 359-370, November.
    8. Elizabeth Ty Wilde & Robinson Hollister, 2007. "How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 26(3), pages 455-477.
    9. Andrew Dillon, 2011. "Do Differences in the Scale of Irrigation Projects Generate Different Impacts on Poverty and Production?," Journal of Agricultural Economics, Wiley Blackwell, vol. 62(2), pages 474-492, June.
    10. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    11. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    12. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    13. Dillon, Andrew, 2011. "The Effect of Irrigation on Poverty Reduction, Asset Accumulation, and Informal Insurance: Evidence from Northern Mali," World Development, Elsevier, vol. 39(12), pages 2165-2175.
    14. Deschamps-Laporte, Jean-Philippe, 2013. "The impact of extension services on farming households in Western Kenya: A propensity score approach," Working Papers 2013:5, Örebro University, School of Business, revised 10 Jun 2013.
    15. Fortson, Kenneth & Gleason, Philip & Kopa, Emma & Verbitsky-Savitz, Natalya, 2015. "Horseshoes, hand grenades, and treatment effects? Reassessing whether nonexperimental estimators are biased," Economics of Education Review, Elsevier, vol. 44(C), pages 100-113.
    16. Arraiz, Irani & Calero, Carla & Jon, Songqing & Peralta, Alexandra, 2015. "Planting the seeds: The impact of training on mando producers in Haiti," 2015 Conference, August 9-14, 2015, Milan, Italy 212622, International Association of Agricultural Economists.
    17. Arpino, Bruno & Mealli, Fabrizia, 2011. "The specification of the propensity score in multilevel observational studies," Computational Statistics & Data Analysis, Elsevier, vol. 55(4), pages 1770-1780, April.
    18. Arraiz, Irani & Calero, Carla & Jon, Songqing & Peralta, Alexandra, 2015. "Planting the seeds: The impact of training on mando producers in Haiti," 2015 Conference, August 9-14, 2015, Milan, Italy 212622, International Association of Agricultural Economists.
    19. Andrew P. Jaciw, 2016. "Assessing the Accuracy of Generalized Inferences From Comparison Group Studies Using a Within-Study Comparison Approach," Evaluation Review, , vol. 40(3), pages 199-240, June.
    20. Tymon Słoczyński, 2015. "The Oaxaca–Blinder Unexplained Component as a Treatment Effects Estimator," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 77(4), pages 588-604, August.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ucp:ecdecc:v:58:y:2010:i:3:p:415-447. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Journals Division (email available below). General contact details of provider: https://www.journals.uchicago.edu/EDCC .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.