IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v50y2021i1p238-264.html
   My bibliography  Save this article

Using Instructed Response Items as Attention Checks in Web Surveys: Properties and Implementation

Author

Listed:
  • Tobias Gummer
  • Joss Roßmann
  • Henning Silber

Abstract

Identifying inattentive respondents in self-administered surveys is a challenging goal for survey researchers. Instructed response items (IRIs) provide a measure for inattentiveness in grid questions that is easy to implement. The present article adds to the sparse research on the use and implementation of attention checks by addressing three research objectives. In a first study, we provide evidence that IRIs identify respondents who show an elevated use of straightlining, speeding, item nonresponse, inconsistent answers, and implausible statements throughout a survey. Excluding inattentive respondents, however, did not alter the results of substantive analyses. Our second study suggests that respondents’ inattentiveness partially changes as the context in which they complete the survey changes. In a third study, we present experimental evidence that a mere exposure to an IRI does not negatively or positively affect response behavior within a survey. A critical discussion on using IRI attention checks concludes this article.

Suggested Citation

  • Tobias Gummer & Joss Roßmann & Henning Silber, 2021. "Using Instructed Response Items as Attention Checks in Web Surveys: Properties and Implementation," Sociological Methods & Research, , vol. 50(1), pages 238-264, February.
    • Handle: RePEc:sae:somere:v:50:y:2021:i:1:p:238-264
    DOI: 10.1177/0049124118769083
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0049124118769083
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0049124118769083?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Vera Toepoel & Corrie Vis & Marcel Das & Arthur van Soest, 2009. "Design of Web Questionnaires," Sociological Methods & Research, , vol. 37(3), pages 371-392, February.
    2. Adam J. Berinsky & Michele F. Margolis & Michael W. Sances, 2014. "Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys," American Journal of Political Science, John Wiley & Sons, vol. 58(3), pages 739-753, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Olsacher, Alexandra & Bade, Celina & Ehlers, Jan & Fehring, Leonard, 2023. "How to effectively communicate health information on social media depending on the audience's personality traits: An experimental study in the context of organ donation in Germany," Social Science & Medicine, Elsevier, vol. 335(C).
    2. Olbrich, Lukas & Sakshaug, Joseph W. & Lewandowski, Eric, 2024. "Evaluating methods to prevent and detect inattentive respondents in web surveys," SocArXiv py9gz, Center for Open Science.
    3. L. Maxim Laurijssen & Barbara Wisse & Stacey Sanders & Ed Sleebos, 2024. "How to Neutralize Primary Psychopathic Leaders’ Damaging Impact: Rules, Sanctions, and Transparency," Journal of Business Ethics, Springer, vol. 189(2), pages 365-383, January.
    4. Panchalingam, Thadchaigeni & Howard, Gregory & Allen Klaiber, H. & Roe, Brian E., 2023. "Food choice behavior of adolescents under parent-child interaction in the context of US school lunch programs," Food Policy, Elsevier, vol. 115(C).
    5. Veronika Hannus & Johannes Sauer, 2021. "Understanding Farmers’ Intention to Use a Sustainability Standard: The Role of Economic Rewards, Knowledge, and Ease of Use," Sustainability, MDPI, vol. 13(19), pages 1-21, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lindhjem, Henrik & Navrud, Ståle, 2011. "Using Internet in Stated Preference Surveys: A Review and Comparison of Survey Modes," International Review of Environmental and Resource Economics, now publishers, vol. 5(4), pages 309-351, September.
    2. Francis Wong, 2024. "Taxing Homeowners Who Won’t Borrow," CESifo Working Paper Series 11185, CESifo.
    3. Janne Tukiainen & Sebastian Blesse & Albrecht Bohne & Leonardo M. Giuffrida & Jan Jäässkeläinen & Ari Luukinen & Antti Sieppi, 2021. "What Are the Priorities of Bureaucrats? Evidence from Conjoint Experiments with Procurement Officials," EconPol Working Paper 63, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    4. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    5. Jimin Pyo & Michael G. Maxfield, 2021. "Cognitive Effects of Inattentive Responding in an MTurk Sample," Social Science Quarterly, Southwestern Social Science Association, vol. 102(4), pages 2020-2039, July.
    6. Lilith Burgstaller & Annabelle Doerr & Sarah Necker, 2023. "Do Household Tax Credits Increase the Demand for Legally Provided Services?," CESifo Working Paper Series 10211, CESifo.
    7. Christopher Roth & Johannes Wohlfart, 2020. "How Do Expectations about the Macroeconomy Affect Personal Expectations and Behavior?," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 731-748, October.
    8. Alessandro Nai, 2020. "The Trump Paradox: How Cues from a Disliked Source Foster Resistance to Persuasion," Politics and Governance, Cogitatio Press, vol. 8(1), pages 122-132.
    9. Paape, Vanessa Isabel, 2022. "The Hidden Influence of Cognitive Processing Style on Consumers' Intention to Adopt Innovative Products," Junior Management Science (JUMS), Junior Management Science e. V., vol. 7(1), pages 112-133.
    10. Milton Mayfield & Jacqueline Mayfield, 2021. "Sound and Safe: The Role of Leader Motivating Language and Follower Self-Leadership in Feelings of Psychological Safety," Administrative Sciences, MDPI, vol. 11(2), pages 1-30, May.
    11. Alexis Grigorieff & Christopher Roth & Diego Ubfal, 2020. "Does Information Change Attitudes Toward Immigrants?," Demography, Springer;Population Association of America (PAA), vol. 57(3), pages 1117-1143, June.
    12. Bart Buelens & Jan A. van den Brakel, 2015. "Measurement Error Calibration in Mixed-mode Sample Surveys," Sociological Methods & Research, , vol. 44(3), pages 391-426, August.
    13. Nicole Maestas & Kathleen J. Mullen & David Powell & Till von Wachter & Jeffrey B. Wenger, 2023. "The Value of Working Conditions in the United States and the Implications for the Structure of Wages," American Economic Review, American Economic Association, vol. 113(7), pages 2007-2047, July.
    14. Haotian Cheng & Dayton M. Lambert & Karen L. DeLong & Kimberly L. Jensen, 2022. "Inattention, availability bias, and attribute premium estimation for a biobased product," Agricultural Economics, International Association of Agricultural Economists, vol. 53(2), pages 274-288, March.
    15. Chatpong Tangmanee & Phattharaphong Niruttinanon, 2019. "Web Survey’s Completion Rates: Effects of Forced Responses, Question Display Styles, and Subjects’ Attitude," International Journal of Research in Business and Social Science (2147-4478), Center for the Strategic Studies in Business and Finance, vol. 8(1), pages 20-29, January.
    16. Riccardo Vecchio & Gerarda Caso & Luigi Cembalo & Massimiliano Borrello, 2020. "Is respondents? inattention in online surveys a major issue for research?," Economia agro-alimentare, FrancoAngeli Editore, vol. 22(1), pages 1-18.
    17. Fehr Ernst & Epper Thomas & Senn Julien, 2020. "Social preferences and redistributive politics," ECON - Working Papers 339, Department of Economics - University of Zurich, revised Aug 2023.
    18. Joni Hersch & Jennifer Bennett Shinall, 2020. "When equitable is not equal: experimental evidence on the division of marital assets in divorce," Review of Economics of the Household, Springer, vol. 18(3), pages 655-682, September.
    19. Anna Schwarz & Philipp Warum, 2023. "Don't Stop Believin'. Heterogeneous Updating of Intergenerational Mobility Perceptions across Income Groups," WIFO Working Papers 665, WIFO.
    20. Aljosha Henkel & Ernst Fehr & Julien Senn & Thomas Epper, 2024. "Beliefs about inequality and the nature of support for redistribution," ECON - Working Papers 442, Department of Economics - University of Zurich.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:50:y:2021:i:1:p:238-264. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.