IDEAS home Printed from https://ideas.repec.org/a/wly/amposc/v60y2016i3p783-802.html
   My bibliography  Save this article

An Empirical Validation Study of Popular Survey Methodologies for Sensitive Questions

Author

Listed:
  • Bryn Rosenfeld
  • Kosuke Imai
  • Jacob N. Shapiro

Abstract

When studying sensitive issues, including corruption, prejudice, and sexual behavior, researchers have increasingly relied upon indirect questioning techniques to mitigate such known problems of direct survey questions as underreporting and nonresponse. However, there have been surprisingly few empirical validation studies of these indirect techniques because the information required to verify the resulting estimates is often difficult to access. This article reports findings from the first comprehensive validation study of indirect methods. We estimate whether people voted for an anti‐abortion referendum held during the 2011 Mississippi General Election using direct questioning and three popular indirect methods: list experiment, endorsement experiment, and randomized response. We then validate these estimates against the official election outcome. While direct questioning leads to significant underestimation of sensitive votes against the referendum, indirect survey techniques yield estimates much closer to the actual vote count, with endorsement experiment and randomized response yielding the least bias.

Suggested Citation

  • Bryn Rosenfeld & Kosuke Imai & Jacob N. Shapiro, 2016. "An Empirical Validation Study of Popular Survey Methodologies for Sensitive Questions," American Journal of Political Science, John Wiley & Sons, vol. 60(3), pages 783-802, July.
  • Handle: RePEc:wly:amposc:v:60:y:2016:i:3:p:783-802
    DOI: 10.1111/ajps.12205
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/ajps.12205
    Download Restriction: no

    File URL: https://libkey.io/10.1111/ajps.12205?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lai, Yufeng & Boaitey, Albert & Minegishi, Kota, 2022. "Behind the veil: Social desirability bias and animal welfare ballot initiatives," Food Policy, Elsevier, vol. 106(C).
    2. Carroll, Eamonn & Timmons, Shane & McGinnity, Frances, 2023. "Experimental tests of public support for disability policy," Research Series, Economic and Social Research Institute (ESRI), number RS159.
    3. Pier Francesco Perri & Eleni Manoli & Tasos C. Christofides, 2023. "Assessing the effectiveness of indirect questioning techniques by detecting liars," Statistical Papers, Springer, vol. 64(5), pages 1483-1506, October.
    4. Claire Cullen, 2023. "Method Matters: The Underreporting of Intimate Partner Violence," The World Bank Economic Review, World Bank, vol. 37(1), pages 49-73.
    5. Jorge M. Agüero & Veronica Frisancho, 2022. "Measuring Violence against Women with Experimental Methods," Economic Development and Cultural Change, University of Chicago Press, vol. 70(4), pages 1565-1590.
    6. Gregori, Marco & de Jong, M.G. & Pieters, Rik, 2023. "Response aggregation to obtain truthful answers to sensitive questions: Estimating the prevalence of illegal purchases of prescription drugs," Other publications TiSEM e7e9734c-7d78-42e8-9e4d-4, Tilburg University, School of Economics and Management.
    7. Chapkovski, Philipp & Schaub, Max, 2022. "Solid support or secret dissent? A list experiment on preference falsification during the Russian war against Ukraine," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 9(2), pages 1-6.
    8. Garay, Candelaria & Palmer-Rubin, Brian & Poertner, Mathias, 2020. "Organizational and partisan brokerage of social benefits: Social policy linkages in Mexico," World Development, Elsevier, vol. 136(C).
    9. Sylvain Chassang & Christian Zehnder, 2019. "Secure Survey Design in Organizations: Theory and Experiments," Working Papers 2019-22, Princeton University. Economics Department..
    10. Cullen,Claire Alexis, 2020. "Method Matters : Underreporting of Intimate Partner Violence in Nigeria and Rwanda," Policy Research Working Paper Series 9274, The World Bank.
    11. Ó Ceallaigh, Diarmaid & Timmons, Shane & Robertson, Deirdre & Lunn, Pete, 2023. "Problem gambling: A narrative review of important policy-relevant issues," Research Series, Economic and Social Research Institute (ESRI), number SUSTAT119.
    12. Edmund Malesky & Markus Taussig, 2019. "How Do Firms Feel About Participation by Their Peers in the Regulatory Design Process? An Online Survey Experiment Testing the Substantive Change and Spillover Mechanisms," Strategy Science, INFORMS, vol. 4(2), pages 129-150, June.
    13. Sarah Brierley, 2020. "Unprincipled Principals: Co‐opted Bureaucrats and Corruption in Ghana," American Journal of Political Science, John Wiley & Sons, vol. 64(2), pages 209-222, April.
    14. Schutte, Sebastian & Ruhe, Constantin & Linke, Andrew, 2020. "How indiscriminate violence fuels religious conflict: Evidence from Kenya," SocArXiv kngq2, Center for Open Science.
    15. Jiayuan Li & Wim Van den Noortgate, 2022. "A Meta-analysis of the Relative Effectiveness of the Item Count Technique Compared to Direct Questioning," Sociological Methods & Research, , vol. 51(2), pages 760-799, May.
    16. Gutiérrez Fernández, Emilio & Rubli, Adrian, 2023. "Challenges for Measuring the LGBT+ Population and Homophobia in Mexico," IDB Publications (Working Papers) 12707, Inter-American Development Bank.
    17. Kit Rickard & Gerard Toal & Kristin M. Bakke & John O'Loughlin, 2023. "Polling during war: Challenges and lessons from Ukraine," WIDER Working Paper Series wp-2023-144, World Institute for Development Economic Research (UNU-WIDER).
    18. Nillesen, Eleonora & Grimm, Michael & Goedhuys, Micheline & Reitmann, Ann-Kristin & Meysonnat, Aline, 2019. "On the Malleability of Implicit Attitudes Towards Women Empowerment: Evidence from Tunisia," IZA Discussion Papers 12471, Institute of Labor Economics (IZA).
    19. Gutierrez, Emilio & Rubli, Adrian, 2024. "LGBT+ persons and homophobia prevalence across job sectors: Survey evidence from Mexico," Labour Economics, Elsevier, vol. 87(C).
    20. Laura Boudreau & Sylvain Chassang & Ada González-Torre & Rachel Heath, 2023. "Monitoring Harassment in Organizations," Working Papers 2022-19, Princeton University. Economics Department..
    21. Andrew Bell, 2022. "Combatant socialization and norms of restraint: Examining officer training at the US Military Academy and Army ROTC," Journal of Peace Research, Peace Research Institute Oslo, vol. 59(2), pages 180-196, March.
    22. Bertelli, Olivia & Calvo, Thomas & Lavallée, Emmanuelle & Mercier, Marion & Mesplé-Somps, Sandrine, 2024. "What one thinks, what one says and what one does: male justifications and practices of gender-based violence in Mali," CEPREMAP Working Papers (Docweb) 2406, CEPREMAP.
    23. Olivia Bertelli & Thomas Calvo & Emmanuelle Lavallée & Marion Mercier & Sandrine Mesplé-Somps, 2023. "Measuring insecurity-related experiences and preferences in a fragile State. A list experiment in Mali," Working Papers DT/2023/01, DIAL (Développement, Institutions et Mondialisation).
    24. Sylvain Chassang & Christian Zehnder, 2019. "Secure Survey Design in Organizations: Theory and Experiments," NBER Working Papers 25918, National Bureau of Economic Research, Inc.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:amposc:v:60:y:2016:i:3:p:783-802. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1540-5907 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.