IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v5y2015i2p2158244015584617.html
   My bibliography  Save this article

It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky†Tasks

Author

Listed:
  • David J. Hauser
  • Norbert Schwarz

Abstract

Instructional manipulation checks (IMCs) have become popular tools for identifying inattentive participants in online studies. IMCs function by attempting to trick inattentive participants into responding incorrectly. However, from a conversational perspective, question characteristics are part of the researcher’s contribution to the conversation, and IMCs may teach participants that there is “more than meets the eye,†prompting systematic thinking on subsequent tricky-seeming questions in an attempt to avoid being tricked. In two online studies, participants responded to a simple task either before or after completing an IMC. As expected, answering an IMC prior to the task improved performance on items that benefit from increased systematic thinking—namely, the Cognitive Reflection Test (Study 1), and a probabilistic reasoning task (Study 2). We conclude that IMCs change attention rather than merely measure attention and discuss implications for their use in online studies.

Suggested Citation

  • David J. Hauser & Norbert Schwarz, 2015. "It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky†Tasks," SAGE Open, , vol. 5(2), pages 21582440155, April.
    • Handle: RePEc:sae:sagope:v:5:y:2015:i:2:p:2158244015584617
    DOI: 10.1177/2158244015584617
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/2158244015584617
    Download Restriction: no
    File URL: https://libkey.io/10.1177/2158244015584617?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    2. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    3. Adam J. Berinsky & Michele F. Margolis & Michael W. Sances, 2014. "Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys," American Journal of Political Science, John Wiley & Sons, vol. 58(3), pages 739-753, July.
    4. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Olbrich, Lukas & Sakshaug, Joseph W. & Lewandowski, Eric, 2024. "Evaluating methods to prevent and detect inattentive respondents in web surveys," SocArXiv py9gz, Center for Open Science.
    2. Eugene Y. Chan & Jack Lin, 2022. "Political ideology and psychological reactance: how serious should climate change be?," Climatic Change, Springer, vol. 172(1), pages 1-22, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Del Ponte, Alessandro & Li, Lianjun & Ang, Lina & Lim, Noah & Seow, Wei Jie, 2024. "Evaluating SoJump.com as a tool for online behavioral research in China," Journal of Behavioral and Experimental Finance, Elsevier, vol. 41(C).
    2. Insoo Cho & Peter F. Orazem, 2021. "How endogenous risk preferences and sample selection affect analysis of firm survival," Small Business Economics, Springer, vol. 56(4), pages 1309-1332, April.
    3. David J. Cooper & Krista Saral & Marie Claire Villeval, 2021. "Why Join a Team?," Management Science, INFORMS, vol. 67(11), pages 6980-6997, November.
    4. Zakaria Babutsidze & Nobuyuki Hanaki & Adam Zylbersztejn, 2019. "Digital Communication and Swift Trust," Post-Print halshs-02409314, HAL.
    5. Francesco Capozza & Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2021. "Studying Information Acquisition in the Field: A Practical Guide and Review," CEBI working paper series 21-15, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    6. Chavez, Daniel E. & Palma, Marco A. & Nayga, Rodolfo M. & Mjelde, James W., 2020. "Product availability in discrete choice experiments with private goods," Journal of choice modelling, Elsevier, vol. 36(C).
    7. Prokudina, Elena & Renneboog, Luc & Tobler, Philippe, 2015. "Does Confidence Predict Out-of-Domain Effort?," Other publications TiSEM 2a1830fc-471f-45a3-b9b0-a, Tilburg University, School of Economics and Management.
    8. Charles Noussair & Steven J.Tucker & Yilong Xu, 2014. "A Futures Market Reduces Bubbles but Allows Greater Profit for More Sophisticated Traders," Working Papers in Economics 14/12, University of Waikato.
    9. Francis Bloch & Bhaskar Dutta & Stéphane Robin & Min Zhu, 2016. "The formation of partnerships in social networks," Post-Print halshs-01421347, HAL.
    10. Besedes, Tibor & Deck, Cary & Quintanar, Sarah & Sarangi, Sudipta & Shor, Mikhael, 2011. "Free-Riding and Performance in Collaborative and Non-Collaborative Groups," MPRA Paper 33948, University Library of Munich, Germany.
    11. Brañas-Garza, Pablo & Jorrat, Diego & Alfonso-Costillo, Antonio & Espín, Antonio M. & Garcia, Teresa & Kovářík, Jaromír, 2020. "Exposure to the Covid-19 pandemic and generosity," MPRA Paper 103389, University Library of Munich, Germany.
    12. Albano, Gian Luigi & Cipollone, Angela & Paolo, Roberto Di & Ponti, Giovanni & Sparro, Marco, 2024. "Scoring rules in experimental procurement," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 108(C).
    13. Goswami, Indranil & Urminsky, Oleg, 2021. "Don’t fear the meter: How longer time limits bias managers to prefer hiring with flat fee compensation," Organizational Behavior and Human Decision Processes, Elsevier, vol. 162(C), pages 42-58.
    14. Marco Angrisani & Marco Cipriani & Antonio Guarino, 2022. "Strategic Sophistication and Trading Profits: An Experiment with Professional Traders," Staff Reports 1044, Federal Reserve Bank of New York.
    15. Zylbersztejn, Adam & Babutsidze, Zakaria & Hanaki, Nobuyuki & Hopfensitz, Astrid, 2024. "How beautiful people see the world: Cooperativeness judgments of and by beautiful people," Journal of Economic Behavior & Organization, Elsevier, vol. 218(C), pages 296-308.
    16. Corgnet, Brice & DeSantis, Mark & Porter, David, 2020. "The distribution of information and the price efficiency of markets," Journal of Economic Dynamics and Control, Elsevier, vol. 110(C).
    17. Anna Louisa Merkel & Johannes Lohse, 2019. "Is fairness intuitive? An experiment accounting for subjective utility differences under time pressure," Experimental Economics, Springer;Economic Science Association, vol. 22(1), pages 24-50, March.
    18. Sébastien Foudi, 2024. "Are risk attitude, impatience, and impulsivity related to the individual discount rate? Evidence from energy-efficient durable goods," Theory and Decision, Springer, vol. 96(4), pages 627-661, June.
    19. Michalis Drouvelis & Julian C. Jamison, 2015. "Selecting public goods institutions: Who likes to punish and reward?," Southern Economic Journal, John Wiley & Sons, vol. 82(2), pages 501-534, October.
    20. Neyse, Levent & Bosworth, Steven & Ring, Patrick & Schmidt, Ulrich, 2016. "Overconfidence, Incentives and Digit Ratio," Open Access Publications from Kiel Institute for the World Economy 130145, Kiel Institute for the World Economy (IfW Kiel).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:5:y:2015:i:2:p:2158244015584617. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.