IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/21447.html
   My bibliography  Save this paper

Measuring the Measurement Error: A Method to Qualitatively Validate Survey Data

Author

Listed:
  • Christopher Blattman
  • Julian C. Jamison
  • Tricia Koroknay-Palicz
  • Katherine Rodrigues
  • Margaret Sheridan

Abstract

Field experiments rely heavily on self-reported data, but subjects may misreport behaviors, especially sensitive ones such as crime. If treatment influences survey responses, it biases experimental estimates. We develop a validation technique that uses intensive qualitative work to assess survey measurement error. Subjects were assigned to receive cash, therapy, both, or neither. According to survey responses, receiving both treatments dramatically reduced crime and other sensitive behaviors. Local researchers spent several days with a random subsample of subjects following their endline surveys, building trust and seeking verbal confirmation of six behaviors: theft, drug use, homelessness, gambling, and two expenditures. This validation suggests that subjects in the control and cash only groups underreported sensitive behaviors and expenditures in the survey relative to the other treatment arms. We bound survey-based treatment effects estimates, and find the impacts of cash and therapy on crime may be larger than suggested by surveys alone.

Suggested Citation

  • Christopher Blattman & Julian C. Jamison & Tricia Koroknay-Palicz & Katherine Rodrigues & Margaret Sheridan, 2015. "Measuring the Measurement Error: A Method to Qualitatively Validate Survey Data," NBER Working Papers 21447, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:21447
    Note: DEV
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w21447.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. David J. Deming, 2011. "Better Schools, Less Crime?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 2063-2115.
    2. Julian Jamison & Dean Karlan & Pia Raffler, 2013. "Mixed Method Evaluation of a Passive mHealth Sexual Information Testing Service in Uganda," Working Papers 1025, Economic Growth Center, Yale University.
    3. César Martinelli & Susan Wendy Parker, 2009. "Deception and Misreporting in a Social Program," Journal of the European Economic Association, MIT Press, vol. 7(4), pages 886-908, June.
    4. Bullock, Will & Imai, Kosuke & Shapiro, Jacob N., 2011. "Statistical Analysis of Endorsement Experiments: Measuring Support for Militant Groups in Pakistan," Political Analysis, Cambridge University Press, vol. 19(4), pages 363-384.
    5. Baird, Sarah & Özler, Berk, 2012. "Examining the reliability of self-reported data on school participation," Journal of Development Economics, Elsevier, vol. 98(1), pages 89-93.
    6. Dean Karlan & Jonathan Zinman, 2008. "Lying About Borrowing," Journal of the European Economic Association, MIT Press, vol. 6(2-3), pages 510-521, 04-05.
    7. Blair, Graeme & Imai, Kosuke, 2012. "Statistical Analysis of List Experiments," Political Analysis, Cambridge University Press, vol. 20(1), pages 47-77, January.
    8. McKenzie, David, 2012. "Beyond baseline and follow-up: The case for more T in experiments," Journal of Development Economics, Elsevier, vol. 99(2), pages 210-221.
    9. Karlan, Dean S. & Zinman, Jonathan, 2012. "List randomization for sensitive behavior: An application for measuring use of loan proceeds," Journal of Development Economics, Elsevier, vol. 98(1), pages 71-75.
    10. de Mel, Suresh & McKenzie, David J. & Woodruff, Christopher, 2009. "Measuring microenterprise profits: Must we ask how the sausage is made?," Journal of Development Economics, Elsevier, vol. 88(1), pages 19-31, January.
    11. Jerry Hausman, 2001. "Mismeasured Variables in Econometric Analysis: Problems from the Right and Problems from the Left," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 57-67, Fall.
    12. Beegle, Kathleen & De Weerdt, Joachim & Friedman, Jed & Gibson, John, 2012. "Methods of household consumption measurement through surveys: Experimental results from Tanzania," Journal of Development Economics, Elsevier, vol. 98(1), pages 3-18.
    13. Christopher Blattman & Julian C. Jamison & Margaret Sheridan, 2015. "Reducing crime and violence: Experimental evidence from cognitive behavioral therapy in Liberia," NBER Working Papers 21204, National Bureau of Economic Research, Inc.
    14. Bound, John & Brown, Charles & Mathiowetz, Nancy, 2001. "Measurement error in survey data," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 5, chapter 59, pages 3705-3843, Elsevier.
    15. Paluck, Elizabeth Levy & Green, Donald P., 2009. "Deference, Dissent, and Dispute Resolution: An Experimental Intervention Using Mass Media to Change Norms and Behavior in Rwanda," American Political Science Review, Cambridge University Press, vol. 103(4), pages 622-644, November.
    16. anonymous, 1997. "Fed amends consumer leasing act," Financial Update, Federal Reserve Bank of Atlanta, vol. 10(Oct), pages 1-5.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. M. Niaz Asadullah & Elisabetta De Cao & Fathema Zhura Khatoon & Zahra Siddique, 2021. "Measuring gender attitudes using list experiments," Journal of Population Economics, Springer;European Society for Population Economics, vol. 34(2), pages 367-400, April.
    2. Sedlmayr, Richard & Shah, Anuj & Sulaiman, Munshi, 2020. "Cash-plus: Poverty impacts of alternative transfer-based approaches," Journal of Development Economics, Elsevier, vol. 144(C).
    3. Olivia Bertelli & Thomas Calvo & Emmanuelle Lavallée & Marion Mercier & Sandrine Mesplé-Somps, 2023. "Measuring insecurity-related experiences and preferences in a fragile State. A list experiment in Mali," Working Papers DT/2023/01, DIAL (Développement, Institutions et Mondialisation).
    4. Elisabetta de Cao & Clemens Lutz, 2015. "Measuring attitudes regarding female genital mutilation through a list experiment," CSAE Working Paper Series 2015-20, Centre for the Study of African Economies, University of Oxford.
    5. de Nicola, Francesca & Giné, Xavier, 2014. "How accurate are recall data? Evidence from coastal India," Journal of Development Economics, Elsevier, vol. 106(C), pages 52-65.
    6. Serneels, Pieter & Beegle, Kathleen & Dillon, Andrew, 2017. "Do returns to education depend on how and whom you ask?," Economics of Education Review, Elsevier, vol. 60(C), pages 5-19.
    7. De Cao, Elisabetta & Lutz, Clemens, 2014. "Sensitive survey questions," Research Report 14017-EEF, University of Groningen, Research Institute SOM (Systems, Organisations and Management).
    8. Abay, Kibrom A. & Abate, Gashaw T. & Barrett, Christopher B. & Bernard, Tanguy, 2019. "Correlated non-classical measurement errors, ‘Second best’ policy inference, and the inverse size-productivity relationship in agriculture," Journal of Development Economics, Elsevier, vol. 139(C), pages 171-184.
    9. John Gibson & Omoniyi Alimi, 2020. "Measuring poverty with noisy and corrected estimates of annual consumption: Evidence from Nigeria," African Development Review, African Development Bank, vol. 32(1), pages 96-107, March.
    10. Wollburg, Philip & Tiberti, Marco & Zezza, Alberto, 2021. "Recall length and measurement error in agricultural surveys," Food Policy, Elsevier, vol. 100(C).
    11. Lépine, Aurélia & Treibich, Carole & D’Exelle, Ben, 2020. "Nothing but the truth: Consistency and efficiency of the list experiment method for the measurement of sensitive health behaviours," Social Science & Medicine, Elsevier, vol. 266(C).
    12. Carole Treibich & Aurélia Lépine, 2019. "Estimating misreporting in condom use and its determinants among sex workers: Evidence from the list randomisation method," Health Economics, John Wiley & Sons, Ltd., vol. 28(1), pages 144-160, January.
    13. Fafchamps, Marcel & McKenzie, David & Quinn, Simon & Woodruff, Christopher, 2014. "Microenterprise growth and the flypaper effect: Evidence from a randomized experiment in Ghana," Journal of Development Economics, Elsevier, vol. 106(C), pages 211-226.
    14. repec:dgr:rugsom:14017-eef is not listed on IDEAS
    15. Arthi, Vellore & Beegle, Kathleen & De Weerdt, Joachim & Palacios-López, Amparo, 2018. "Not your average job: Measuring farm labor in Tanzania," Journal of Development Economics, Elsevier, vol. 130(C), pages 160-172.
    16. Gutierrez, Emilio & Rubli, Adrian, 2024. "LGBT+ persons and homophobia prevalence across job sectors: Survey evidence from Mexico," Labour Economics, Elsevier, vol. 87(C).
    17. Ahmad, Syedah & Lensink, Robert & Mueller, Annika, 2023. "Religion, social desirability bias and financial inclusion: Evidence from a list experiment on Islamic (micro-)finance," Journal of Behavioral and Experimental Finance, Elsevier, vol. 38(C).
    18. Bulte, Erwin & Lensink, Robert, 2019. "Women's empowerment and domestic abuse: Experimental evidence from Vietnam," European Economic Review, Elsevier, vol. 115(C), pages 172-191.
    19. Bertelli, Olivia & Calvo, Thomas & Lavallée, Emmanuelle & Mercier, Marion & Mesplé-Somps, Sandrine, 2024. "What one thinks, what one says and what one does: male justifications and practices of gender-based violence in Mali," CEPREMAP Working Papers (Docweb) 2406, CEPREMAP.
    20. repec:lic:licosd:41819 is not listed on IDEAS
    21. Antonia Grohmann & Lukas Menkhoff & Helke Seitz, 2022. "The Effect of Personalized Feedback on Small Enterprises’ Finances in Uganda," Economic Development and Cultural Change, University of Chicago Press, vol. 70(3), pages 1197-1227.
    22. Jean-François Maystadt & Philip Verwimp, 2014. "Winners and Losers among a Refugee-Hosting Population," Economic Development and Cultural Change, University of Chicago Press, vol. 62(4), pages 769-809.

    More about this item

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • I32 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Measurement and Analysis of Poverty
    • K4 - Law and Economics - - Legal Procedure, the Legal System, and Illegal Behavior
    • O1 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:21447. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.