IDEAS home Printed from https://ideas.repec.org/a/cup/polals/v27y2019i04p572-589_00.html
   My bibliography  Save this article

A Note on Dropping Experimental Subjects who Fail a Manipulation Check

Author

Listed:
  • Aronow, Peter M.
  • Baron, Jonathon
  • Pinson, Lauren

Abstract

Dropping subjects based on the results of a manipulation check following treatment assignment is common practice across the social sciences, presumably to restrict estimates to a subpopulation of subjects who understand the experimental prompt. We show that this practice can lead to serious bias and argue for a focus on what is revealed without discarding subjects. Generalizing results developed in Zhang and Rubin (2003) and Lee (2009) to the case of multiple treatments, we provide sharp bounds for potential outcomes among those who would pass a manipulation check regardless of treatment assignment. These bounds may have large or infinite width, implying that this inferential target is often out of reach. As an application, we replicate Press, Sagan, and Valentino (2013) with a design that does not drop subjects that failed the manipulation check and show that the findings are likely stronger than originally reported. We conclude with suggestions for practice, namely alterations to the experimental design.

Suggested Citation

  • Aronow, Peter M. & Baron, Jonathon & Pinson, Lauren, 2019. "A Note on Dropping Experimental Subjects who Fail a Manipulation Check," Political Analysis, Cambridge University Press, vol. 27(4), pages 572-589, October.
    • Handle: RePEc:cup:polals:v:27:y:2019:i:04:p:572-589_00
    as

    Download full text from publisher

    File URL: https://www.cambridge.org/core/product/identifier/S1047198719000056/type/journal_article
    File Function: link to article abstract page
    Download Restriction: no
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Olbrich, Lukas & Sakshaug, Joseph W. & Lewandowski, Eric, 2024. "Evaluating methods to prevent and detect inattentive respondents in web surveys," SocArXiv py9gz, Center for Open Science.
    2. Steven J. Kachelmeier & Dan Rimkus & Jaime J. Schmidt & Kristen Valentine, 2020. "The Forewarning Effect of Critical Audit Matter Disclosures Involving Measurement Uncertainty," Contemporary Accounting Research, John Wiley & Sons, vol. 37(4), pages 2186-2212, December.
    3. Ariane Wenger & Michael Stauffacher & Irina Dallo, 2021. "Public perception and acceptance of negative emission technologies – framing effects in Switzerland," Climatic Change, Springer, vol. 167(3), pages 1-20, August.
    4. Kenneth J. Meier & Seung-ho An, 2020. "Sector bias in public programs: US nonprofit hospitals," Journal of Behavioral Public Administration, Center for Experimental and Behavioral Public Administration, vol. 3(1).
    5. Jaroslaw Kantorowicz, 2023. "Testing public reaction to constitutional fiscal rules violations," Constitutional Political Economy, Springer, vol. 34(4), pages 483-509, December.
    6. Devorah Manekin & Reed M. Wood, 2020. "Framing the Narrative: Female Fighters, External Audience Attitudes, and Transnational Support for Armed Rebellions," Journal of Conflict Resolution, Peace Science Society (International), vol. 64(9), pages 1638-1665, October.
    7. Daniel Silverman & Daniel Kent & Christopher Gelpi, 2022. "Putting Terror in Its Place: An Experiment on Mitigating Fears of Terrorism among the American Public," Journal of Conflict Resolution, Peace Science Society (International), vol. 66(2), pages 191-216, February.
    8. Elena Kantorowicz‐Reznichenko & Jarosław Kantorowicz & Keren Weinshall, 2022. "Ideological bias in constitutional judgments: Experimental analysis and potential solutions," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 19(3), pages 716-757, September.
    9. Maxime Walder & Oliver Strijbis, 2022. "Negative Party Identification and the Use of Party Cues in the Direct Democratic Context," Politics and Governance, Cogitatio Press, vol. 10(4), pages 325-335.
    10. Jimin Pyo & Michael G. Maxfield, 2021. "Cognitive Effects of Inattentive Responding in an MTurk Sample," Social Science Quarterly, Southwestern Social Science Association, vol. 102(4), pages 2020-2039, July.
    11. Diament, Sean M. & Kaya, Ayse & Magenheim, Ellen B., 2022. "Frames that matter: Increasing the willingness to get the Covid-19 vaccines," Social Science & Medicine, Elsevier, vol. 292(C).
    12. Lim, Sijeong & Dolsak, Nives & Prakash, Aseem & Tanaka, Seiki, 2022. "Distributional concerns and public opinion: EV subsidies in the U.S. and Japan," Energy Policy, Elsevier, vol. 164(C).
    13. Yang, Xisi & Thøgersen, John, 2022. "When people are green and greedy: A new perspective of recycling rewards and crowding-out in Germany, the USA and China," Journal of Business Research, Elsevier, vol. 144(C), pages 217-235.
    14. Azusa Uji & Sijeong Lim & Jaehyun Song, 2024. "From plastic to peace: Overcoming public antipathy through environmental cooperation," Journal of Peace Research, Peace Research Institute Oslo, vol. 61(2), pages 279-293, March.
    15. Nicole Wu, 2023. "“Restrict foreigners, not robots”: Partisan responses to automation threat," Economics and Politics, Wiley Blackwell, vol. 35(2), pages 505-528, July.
    16. Herkes, Feie J. & Zouridis, Stavros, 2023. "The legitimacy of land use decisions by public authorities in the Netherlands: Results from a survey experiment," Land Use Policy, Elsevier, vol. 134(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cup:polals:v:27:y:2019:i:04:p:572-589_00. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Kirk Stebbing (email available below). General contact details of provider: https://www.cambridge.org/pan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.