IDEAS home Printed from https://ideas.repec.org/a/oup/wbecrv/v28y2014i3p413-431..html
   My bibliography  Save this article

Can We Trust Shoestring Evaluations?

Author

Listed:
  • Martin Ravallion

Abstract

Many more impact evaluations could be done, and at lower unit cost, if evaluators could avoid the need for baseline data using objective socio-economic surveys and rely instead on retrospective subjective questions on how outcomes have changed, asked post-intervention. But would the results be reliable? This paper tests a rapid-appraisal, "shoestring" method using subjective recall for welfare changes. The recall data were collected at the end of a full-scale evaluation of a large World Bank supported poor-area development program in China. Qualitative recalls on how living standards have changed are found to provide only weak and biased signals of the changes in consumption as measured from contemporaneous surveys. Importantly, the shoestring method was unable to correct for the selective placement of the program favoring poor villages. The results of this case study are not encouraging for future applications of the shoestring method, although similar tests are needed in other settings.

Suggested Citation

  • Martin Ravallion, 2014. "Can We Trust Shoestring Evaluations?," The World Bank Economic Review, World Bank, vol. 28(3), pages 413-431.
    • Handle: RePEc:oup:wbecrv:v:28:y:2014:i:3:p:413-431.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/wber/lht016
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Michael Bamberger, 2009. "Strengthening the evaluation of programme effectiveness through reconstructing baseline data," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 1(1), pages 37-59.
    2. Elizabeth M. King & Jere R. Behrman, 2009. "Timing and Duration of Exposure in Evaluations of Social Programs," The World Bank Research Observer, World Bank, vol. 24(1), pages 55-82, February.
    3. Chen, Shaohua & Mu, Ren & Ravallion, Martin, 2009. "Are there lasting impacts of aid to poor areas?," Journal of Public Economics, Elsevier, vol. 93(3-4), pages 512-528, April.
    4. Ravallion, Martin & Chen, Shaohua, 2005. "Hidden impact? Household saving in response to a poor-area development project," Journal of Public Economics, Elsevier, vol. 89(11-12), pages 2183-2204, December.
    5. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    6. Anirudh Krishna & Daniel Lumonya & Milissa Markiewicz & Firminus Mugumya & Agatha Kafuko & Jonah Wegoye, 2006. "Escaping poverty and becoming poor in 36 villages of Central and Western Uganda," Journal of Development Studies, Taylor & Francis Journals, vol. 42(2), pages 346-370.
    7. Martin Ravallion & Michael Lokshin, 2001. "Identifying Welfare Effects from Subjective Questions," Economica, London School of Economics and Political Science, vol. 68(271), pages 335-357, August.
    8. Chen, Shaohua & Ravallion, Martin, 1996. "Data in transition: Assessing rural living standards in Southern China," China Economic Review, Elsevier, vol. 7(1), pages 23-56.
    9. Martin Ravallion, 2009. "Evaluation in the Practice of Development," The World Bank Research Observer, World Bank, vol. 24(1), pages 29-53, March.
    10. Heike Solga, 2001. "Longitudinal Surveys and the Study of Occupational Mobility: Panel and Retrospective Design in Comparison," Quality & Quantity: International Journal of Methodology, Springer, vol. 35(3), pages 291-309, August.
    11. Berney, L. R. & Blane, D. B., 1997. "Collecting retrospective data: Accuracy of recall after 50 years judged against historical records," Social Science & Medicine, Elsevier, vol. 45(10), pages 1519-1525, November.
    12. Das, Jishnu & Hammer, Jeffrey & Sánchez-Paramo, Carolina, 2012. "The impact of recall periods on reported morbidity and health seeking behavior," Journal of Development Economics, Elsevier, vol. 98(1), pages 76-88.
    13. Beegle, Kathleen & Himelein, Kristen & Ravallion, Martin, 2012. "Frame-of-reference bias in subjective welfare," Journal of Economic Behavior & Organization, Elsevier, vol. 81(2), pages 556-570.
    14. Jalan, Jyotsna & Ravallion, Martin, 1998. "Are there dynamic gains from a poor-area development program?," Journal of Public Economics, Elsevier, vol. 67(1), pages 65-85, January.
    15. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2006. "Moving the Goalposts: Addressing Limited Overlap in the Estimation of Average Treatment Effects by Changing the Estimand," NBER Technical Working Papers 0330, National Bureau of Economic Research, Inc.
    16. Krishna, Anirudh, 2004. "Escaping Poverty and Becoming Poor: Who Gains, Who Loses, and Why?," World Development, Elsevier, vol. 32(1), pages 121-136, January.
    17. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    18. Ravallion, Martin, 2012. "Poor, or just feeling poor ? on using subjective data in measuring poverty," Policy Research Working Paper Series 5968, The World Bank.
    19. Sendhil Mullainathan & Marianne Bertrand, 2001. "Do People Mean What They Say? Implications for Subjective Survey Data," American Economic Review, American Economic Association, vol. 91(2), pages 67-72, May.
    20. repec:bla:econom:v:68:y:2001:i:271:p:335-57 is not listed on IDEAS
    21. Eva Broegaard & Ted Freeman & Carsten Schwensen, 2011. "Experience from a phased mixed-methods approach to impact evaluation of Danida support to rural transport infrastructure in Nicaragua," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 3(1), pages 9-27.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. McKenzie, David, 2016. "Can Business Owners Form Accurate Counterfactuals? Eliciting Treatment and Control Beliefs about Their Outcomes in the Alternat," CEPR Discussion Papers 11280, C.E.P.R. Discussion Papers.
    2. Minten, Bart & Singh, K.M. & Sutradhar, Rajib, 2013. "Branding and agricultural value chains in developing countries: Insights from Bihar (India)," Food Policy, Elsevier, vol. 38(C), pages 23-34.
    3. Sophie Mitra & Michael Palmer & Daniel Mont & Nora Groce, 2016. "Can Households Cope with Health Shocks in Vietnam?," Health Economics, John Wiley & Sons, Ltd., vol. 25(7), pages 888-907, July.
    4. Masamitsu Kurata & Noriatsu Matsui & Yukio Ikemoto & Hiromi Tsuboi, 2018. "In recent years, the Sustainable Development Goals has managed to shepherd the reduction of energy poverty and extension of sustainable energy, making both international objectives. Using two-period d," Economics Bulletin, AccessEcon, vol. 38(2), pages 995-1013.
    5. McKenzie, David, 2020. "If it needs a power calculation, does it matter for poverty reduction?," World Development, Elsevier, vol. 127(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chen, Shaohua & Mu, Ren & Ravallion, Martin, 2009. "Are there lasting impacts of aid to poor areas?," Journal of Public Economics, Elsevier, vol. 93(3-4), pages 512-528, April.
    2. Chen, Shaohua & Mu, Ren & Ravallion, Martin, 2008. "Are there lasting impacts of aid to poor areas ? Evidence from rural China," Policy Research Working Paper Series 4084, The World Bank.
    3. Ahmed Elsayed & Soiliou Daw Namoro & Rania Roushdy, 2022. "Empowering women in conservative settings: evidence from an intervention in rural Egypt," Review of Economics of the Household, Springer, vol. 20(4), pages 1287-1322, December.
    4. Heim, Sven & Hüschelrath, Kai & Schmidt-Dengler, Philipp & Strazzeri, Maurizio, 2017. "The impact of state aid on the survival and financial viability of aided firms," European Economic Review, Elsevier, vol. 100(C), pages 193-214.
    5. Sheahan, Megan & Liu, Yanyan & Narayanan, Sudha & Barrett, Christopher B., 2015. "Disaggregated labor supply implications of guaranteed employment in India," 2016 Annual Meeting, July 31-August 2, Boston, Massachusetts 237345, Agricultural and Applied Economics Association.
    6. Elsayed, Ahmed & Roushdy, Rania, 2017. "Empowering Women under Social Constraints: Evidence from a Field Intervention in Rural Egypt," IZA Discussion Papers 11240, Institute of Labor Economics (IZA).
    7. Alexandra Peralta & Scott M. Swinton & Songqing Jin, 2018. "The Secret to Getting Ahead Is Getting Started: Early Impacts of a Rural Development Project," Sustainability, MDPI, vol. 10(8), pages 1-20, July.
    8. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2006. "Moving the Goalposts: Addressing Limited Overlap in the Estimation of Average Treatment Effects by Changing the Estimand," NBER Technical Working Papers 0330, National Bureau of Economic Research, Inc.
    9. Deininger, Klaus & Liu, Yanyan, 2013. "Economic and Social Impacts of an Innovative Self-Help Group Model in India," World Development, Elsevier, vol. 43(C), pages 149-163.
    10. Cadot, Olivier & Fernandes, Ana M. & Gourdon, Julien & Mattoo, Aaditya, 2015. "Are the benefits of export support durable? Evidence from Tunisia," Journal of International Economics, Elsevier, vol. 97(2), pages 310-324.
    11. Zhao, Zhong, 2008. "Sensitivity of propensity score methods to the specifications," Economics Letters, Elsevier, vol. 98(3), pages 309-319, March.
    12. Suzuki, Aya & Mano, Yukichi & Abebe, Girum, 2018. "Earnings, savings, and job satisfaction in a labor-intensive export sector: Evidence from the cut flower industry in Ethiopia," World Development, Elsevier, vol. 110(C), pages 176-191.
    13. Ravallion, Martin, 2008. "Evaluating Anti-Poverty Programs," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 59, pages 3787-3846, Elsevier.
    14. Jacobus de Hoop & Furio C. Rosati, 2014. "Cash Transfers and Child Labor," The World Bank Research Observer, World Bank, vol. 29(2), pages 202-234.
    15. Ravallion, Martin & Chen, Shaohua, 2005. "Hidden impact? Household saving in response to a poor-area development project," Journal of Public Economics, Elsevier, vol. 89(11-12), pages 2183-2204, December.
    16. Matias Busso & Patrick Kline, 2008. "Do Local Economic Development Programs Work? Evidence from the Federal Empowerment Zone Program," Cowles Foundation Discussion Papers 1639, Cowles Foundation for Research in Economics, Yale University.
    17. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2009. "Dealing with limited overlap in estimation of average treatment effects," Biometrika, Biometrika Trust, vol. 96(1), pages 187-199.
    18. Dehejia Rajeev, 2015. "Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic," Journal of Globalization and Development, De Gruyter, vol. 6(1), pages 47-69, June.
    19. Ravallion, Martin & Himelein, Kristen & Beegle, Kathleen, 2013. "Can subjective questions on economic welfare be trusted ? evidence for three developing countries," Policy Research Working Paper Series 6726, The World Bank.
    20. Christian Volpe Martincus & Jerónimo Carballo & Pablo M. Garcia, 2012. "Public programmes to promote firms’ exports in developing countries: are there heterogeneous effects by size categories?," Applied Economics, Taylor & Francis Journals, vol. 44(4), pages 471-491, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:wbecrv:v:28:y:2014:i:3:p:413-431.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://edirc.repec.org/data/wrldbus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.