IDEAS home Printed from https://ideas.repec.org/a/bla/socsci/v102y2021i4p2020-2039.html
   My bibliography  Save this article

Cognitive Effects of Inattentive Responding in an MTurk Sample

Author

Listed:
  • Jimin Pyo
  • Michael G. Maxfield

Abstract

Objective Mechanical Turk (MTurk) has become an increasingly valuable sampling tool for social scientists. At the same time, concern has emerged about the quality of responses obtained from MTurk workers. This article demonstrates several screening practices for identifying problematic or inattentive MTurk participants. The study also examines how inattentive responding affects psychometric properties of measures and relationships between substantive variables. Method A sample of 498 MTurk workers participated in an experimental study for developing and validating scales of public confidence in the criminal justice system. Multiple screening procedures were used to identify inattentive participants and their potential effects on study measures and hypothesis testing. Results Results reveal adverse impacts on psychometric properties of measures as well as inflated observed relationships between substantive variables. Conclusion Screeners used in this study work well in identifying attention problems among MTurk participants. The present study expands guidance for improving quality of responses from MTurk participants for social science research.

Suggested Citation

  • Jimin Pyo & Michael G. Maxfield, 2021. "Cognitive Effects of Inattentive Responding in an MTurk Sample," Social Science Quarterly, Southwestern Social Science Association, vol. 102(4), pages 2020-2039, July.
    • Handle: RePEc:bla:socsci:v:102:y:2021:i:4:p:2020-2039
    DOI: 10.1111/ssqu.12954
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/ssqu.12954
    Download Restriction: no
    File URL: https://libkey.io/10.1111/ssqu.12954?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Adam J. Berinsky & Michele F. Margolis & Michael W. Sances, 2014. "Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys," American Journal of Political Science, John Wiley & Sons, vol. 58(3), pages 739-753, July.
    2. Jacob M. Montgomery & Brendan Nyhan & Michelle Torres, 2018. "How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It," American Journal of Political Science, John Wiley & Sons, vol. 62(3), pages 760-775, July.
    3. Alvarez, R. Michael & Atkeson, Lonna Rae & Levin, Ines & Li, Yimeng, 2019. "Paying Attention to Inattentive Survey Respondents," Political Analysis, Cambridge University Press, vol. 27(2), pages 145-162, April.
    4. Clifford, Scott & Jerit, Jennifer, 2014. "Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies," Journal of Experimental Political Science, Cambridge University Press, vol. 1(2), pages 120-131, January.
    5. Aronow, Peter M. & Baron, Jonathon & Pinson, Lauren, 2019. "A Note on Dropping Experimental Subjects who Fail a Manipulation Check," Political Analysis, Cambridge University Press, vol. 27(4), pages 572-589, October.
    6. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    7. Yasmina Okan & Eric R. Stone & Wändi Bruine de Bruin, 2018. "Designing Graphs that Promote Both Risk Understanding and Behavior Change," Risk Analysis, John Wiley & Sons, vol. 38(5), pages 929-946, May.
    8. John V. Kane & Jason Barabas, 2019. "No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments," American Journal of Political Science, John Wiley & Sons, vol. 63(1), pages 234-249, January.
    9. repec:cup:judgdm:v:8:y:2013:i:3:p:236-249 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rachel E. Dalafave & W. Kip Viscusi, 2023. "The locus of dread for mass shooting risks: Distinguishing alarmist risk beliefs from risk preferences," Journal of Risk and Uncertainty, Springer, vol. 66(2), pages 109-139, April.
    2. Itskovich, Eran & Factor, Roni, 2024. "An experimental-causal-chain design to explore three mechanisms linking economic inequality and crime," Journal of Criminal Justice, Elsevier, vol. 92(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Del Ponte, Alessandro & Li, Lianjun & Ang, Lina & Lim, Noah & Seow, Wei Jie, 2024. "Evaluating SoJump.com as a tool for online behavioral research in China," Journal of Behavioral and Experimental Finance, Elsevier, vol. 41(C).
    2. Nicole Wu, 2023. "“Restrict foreigners, not robots”: Partisan responses to automation threat," Economics and Politics, Wiley Blackwell, vol. 35(2), pages 505-528, July.
    3. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    4. Gordon Pennycook & David G. Rand, 2022. "Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    5. Vecchio, Riccardo & Caso, Gerarda & Cembalo, Luigi & Borrello, Massimiliano, 2020. "Is respondents’ inattention in online surveys a major issue for research?," Economia agro-alimentare / Food Economy, Italian Society of Agri-food Economics/Società Italiana di Economia Agro-Alimentare (SIEA), vol. 22(1), March.
    6. Lude, Maximilian & Prügl, Reinhard, 2021. "Experimental studies in family business research," Journal of Family Business Strategy, Elsevier, vol. 12(1).
    7. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    8. Le Maux, Benoît & Necker, Sarah, 2023. "Honesty nudges: Effect varies with content but not with timing," Journal of Economic Behavior & Organization, Elsevier, vol. 207(C), pages 433-456.
    9. Nicolas Jacquemet & Alexander G James & Stéphane Luchini & James J Murphy & Jason F Shogren, 2021. "Do truth-telling oaths improve honesty in crowd-working?," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-18, January.
    10. Pierluigi Conzo & Andrea Gallice & Juan S. Morales & Margaret Samahita & Laura K. Taylor, 2021. "Can Hearts Change Minds? Social media Endorsements and Policy Preferences," Carlo Alberto Notebooks 641, Collegio Carlo Alberto.
    11. Felix Bader & Bastian Baumeister & Roger Berger & Marc Keuschnigg, 2021. "On the Transportability of Laboratory Results," Sociological Methods & Research, , vol. 50(3), pages 1452-1481, August.
    12. Yang, Xisi & Thøgersen, John, 2022. "When people are green and greedy: A new perspective of recycling rewards and crowding-out in Germany, the USA and China," Journal of Business Research, Elsevier, vol. 144(C), pages 217-235.
    13. Maxime Walder & Oliver Strijbis, 2022. "Negative Party Identification and the Use of Party Cues in the Direct Democratic Context," Politics and Governance, Cogitatio Press, vol. 10(4), pages 325-335.
    14. Huseyn Abdulla & James D. Abbey & Michael Ketzenberg, 2022. "How consumers value retailer's return policy leniency levers: An empirical investigation," Production and Operations Management, Production and Operations Management Society, vol. 31(4), pages 1719-1733, April.
    15. Olbrich, Lukas & Sakshaug, Joseph W. & Lewandowski, Eric, 2024. "Evaluating methods to prevent and detect inattentive respondents in web surveys," SocArXiv py9gz, Center for Open Science.
    16. Riccardo Vecchio & Gerarda Caso & Luigi Cembalo & Massimiliano Borrello, 2020. "Is respondents? inattention in online surveys a major issue for research?," Economia agro-alimentare, FrancoAngeli Editore, vol. 22(1), pages 1-18.
    17. Diament, Sean M. & Kaya, Ayse & Magenheim, Ellen B., 2022. "Frames that matter: Increasing the willingness to get the Covid-19 vaccines," Social Science & Medicine, Elsevier, vol. 292(C).
    18. Alempaki, Despoina & Starmer, Chris & Tufano, Fabio, 2019. "On the priming of risk preferences: The role of fear and general affect," Journal of Economic Psychology, Elsevier, vol. 75(PA).
    19. Bansak, Kirk & Paulson, Elisabeth, 2023. "Public Opinion on Fairness and Efficiency for Algorithmic and Human Decision-Makers," OSF Preprints pghmx, Center for Open Science.
    20. Ariane Wenger & Michael Stauffacher & Irina Dallo, 2021. "Public perception and acceptance of negative emission technologies – framing effects in Switzerland," Climatic Change, Springer, vol. 167(3), pages 1-20, August.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:socsci:v:102:y:2021:i:4:p:2020-2039. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0038-4941 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.