IDEAS home Printed from https://ideas.repec.org/p/swe/wpaper/2015-20a.html
   My bibliography  Save this paper

MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk

Author

Listed:
  • Jeanette A.M.J. Deetlefs

    (School of Marketing, UNSW Business School, UNSW)

  • Mathew Chylinski

    (School of Marketing, UNSW Business School, UNSW)

  • Andreas Ortmann

    (School of Economics, UNSW Business School, UNSW)

Abstract

Widely accepted as a low-cost, fast-turnaround solution with acceptable validity, Amazon’s Mechanical Turk (MTurk) is increasingly being used to source participants for academic studies (Berinsky et al. 2012; Bohannon 2011; Chandler et al. 2014; Mason and Suri 2012). Yet two commonly raised concerns remain: the presence of quasi-professional respondents, or “Super-Turkers”, and the presence of “Spammers”, those that compromise quality while optimising their pay rate. We isolate the influence on research results of experienced subjects (Super-Turkers), and of unreliable subjects (Spammers), jointly and separately. Jointly including these subjects produces very similar results to jointly excluding them, yet effect sizes decrease disproportionately to their sample representation. Furthermore, separately including experienced subjects in research results is shown to be as problematic as inclusion of unreliable subjects, although the noise introduced by these subjects is divergent and measure dependent. Hence removing only one of these types of respondents can be even more damaging to the reliability of results, than including both.

Suggested Citation

  • Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    • Handle: RePEc:swe:wpaper:2015-20a
    as

    Download full text from publisher

    File URL: http://research.economics.unsw.edu.au/RePEc/papers/2015-20.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Uri Gneezy & Jan Potters, 1997. "An Experiment on Risk Taking and Evaluation Periods," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 631-645.
    2. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    3. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    4. van Rooij, Maarten & Lusardi, Annamaria & Alessie, Rob, 2011. "Financial literacy and stock market participation," Journal of Financial Economics, Elsevier, vol. 101(2), pages 449-472, August.
    5. Todd L. Cherry & Peter Frykblom & Jason F. Shogren, 2002. "Hardnose the Dictator," American Economic Review, American Economic Association, vol. 92(4), pages 1218-1221, September.
    6. Charness, Gary & Karni, Edi & Levin, Dan, 2010. "On the conjunction fallacy in probability judgment: New experimental evidence regarding Linda," Games and Economic Behavior, Elsevier, vol. 68(2), pages 551-556, March.
    7. Lusardi, Annamaria & Mitchell, Olivia S., 2007. "Baby Boomer retirement security: The roles of planning, financial literacy, and housing wealth," Journal of Monetary Economics, Elsevier, vol. 54(1), pages 205-224, January.
    8. Kaufman, Carol Felker & Lane, Paul M & Lindquist, Jay D, 1991. "Exploring More Than 24 Hours a Day: A Preliminary Investigation of Polychronic Time Use," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 18(3), pages 392-401, December.
    9. Hazel Bateman & Christine Eckert & John Geweke & Jordan Louviere & Susan Thorp & Stephen Satchell, 2012. "Financial Competence and Expectations Formation: Evidence from Australia," The Economic Record, The Economic Society of Australia, vol. 88(280), pages 39-63, March.
    10. Annamaria Lusardi & Olivia S. Mitchell, 2017. "How Ordinary Consumers Make Complex Economic Decisions: Financial Literacy and Retirement Readiness," Quarterly Journal of Finance (QJF), World Scientific Publishing Co. Pte. Ltd., vol. 7(03), pages 1-31, September.
    11. Reilly, Michael D, 1982. "Working Wives and Convenience Consumption," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 8(4), pages 407-418, March.
    12. Catherine C. Eckel & Philip J. Grossman, 2008. "Forecasting Risk Attitudes: An Experimental Study Using Actual and Forecast Gamble Choices," Monash Economics Working Papers archive-01, Monash University, Department of Economics.
    13. David G. Rand & Alexander Peysakhovich & Gordon T. Kraft-Todd & George E. Newman & Owen Wurzbacher & Martin A. Nowak & Joshua D. Greene, 2014. "Social heuristics shape intuitive cooperation," Nature Communications, Nature, vol. 5(1), pages 1-12, May.
    14. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    15. Charness, Gary & Gneezy, Uri & Imas, Alex, 2013. "Experimental methods: Eliciting risk preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 87(C), pages 43-51.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Antonio A. Arechar & Gordon T. Kraft-Todd & David G. Rand, 2017. "Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 1-11, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    2. Wladislaw Mill & Cornelius Schneider, 2023. "The Bright Side of Tax Evasion," CESifo Working Paper Series 10615, CESifo.
    3. Ranganathan, Kavitha & Lejarraga, Tomás, 2021. "Elicitation of risk preferences through satisficing," Journal of Behavioral and Experimental Finance, Elsevier, vol. 32(C).
    4. Noviarini, Jelita & Coleman, Andrew & Roberts, Helen & Whiting, Rosalind H., 2023. "Financial literacy and retirees' resource allocation decisions in New Zealand," Pacific-Basin Finance Journal, Elsevier, vol. 79(C).
    5. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    6. repec:cup:judgdm:v:16:y:2021:i:5:p:1290-1313 is not listed on IDEAS
    7. Brañas-Garza, Pablo & Estepa-Mohedano, Lorenzo & Jorrat, Diego & Orozco, Victor & Rascón-Ramírez, Ericka, 2021. "To pay or not to pay: Measuring risk preferences in lab and field," Judgment and Decision Making, Cambridge University Press, vol. 16(5), pages 1290-1313, September.
    8. Collier, Trevor & Cotten, Stephen & Roush, Justin, 2022. "Using pandemic behavior to test the external validity of laboratory measurements of risk aversion and guilt," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 101(C).
    9. Leonard Hoeft & Michael Kurschilgen & Wladislaw Mill & Simone Vannuccini, 2022. "Norms as Obligations," Munich Papers in Political Economy 22, Munich School of Politics and Public Policy and the School of Management at the Technical University of Munich.
    10. Christian Ehm & Christine Laudenbach & Martin Weber, 2018. "Focusing on volatility information instead of portfolio weights as an aid to investor decisions," Experimental Economics, Springer;Economic Science Association, vol. 21(2), pages 457-480, June.
    11. Füllbrunn, Sascha & Luhan, Wolfgang J., 2015. "Am I my Peer's Keeper? Social Responsibility in Financial Decision Making," Ruhr Economic Papers 551, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    12. Sandro Ambuehl & B. Douglas Bernheim & Annamaria Lusardi, 2022. "Evaluating Deliberative Competence: A Simple Method with an Application to Financial Choice," American Economic Review, American Economic Association, vol. 112(11), pages 3584-3626, November.
    13. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," IZA Discussion Papers 15478, Institute of Labor Economics (IZA).
    14. Gangwar, Rachna & Singh, Ritvik, 2018. "Analyzing Factors Affecting Financial Literacy and its Impact on Investment Behavior among Adults in India," MPRA Paper 89452, University Library of Munich, Germany.
    15. Grohmann, Antonia, 2018. "Financial literacy and financial behavior: Evidence from the emerging Asian middle class," Pacific-Basin Finance Journal, Elsevier, vol. 48(C), pages 129-143.
    16. Lusardi, Annamaria & Mitchell, Olivia S., 2011. "Financial literacy around the world: an overview," Journal of Pension Economics and Finance, Cambridge University Press, vol. 10(4), pages 497-508, October.
    17. repec:cup:judgdm:v:14:y:2019:i:3:p:234-279 is not listed on IDEAS
    18. Annamaria Lusardi & Olivia S. Mitchell & Vilsa Curto, 2009. "Financial Literacy among the Young: Evidence and Implications for Consumer Policy," CeRP Working Papers 91, Center for Research on Pensions and Welfare Policies, Turin (Italy).
    19. Lefgren, Lars J. & Sims, David P. & Stoddard, Olga B., 2016. "Effort, luck, and voting for redistribution," Journal of Public Economics, Elsevier, vol. 143(C), pages 89-97.
    20. Joanne W. Hsu, 2016. "Aging and Strategic Learning: The Impact of Spousal Incentives on Financial Literacy," Journal of Human Resources, University of Wisconsin Press, vol. 51(4), pages 1036-1067.
    21. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," GLO Discussion Paper Series 1157, Global Labor Organization (GLO).
    22. Shin, Su Hyun & Kim, Kyoung Tae & Seay, Martin, 2020. "Sources of information and portfolio allocation," Journal of Economic Psychology, Elsevier, vol. 76(C).

    More about this item

    Keywords

    data collection; experimentation; field experiment; internet; Mechanical Turk;
    All these keywords.

    JEL classification:

    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D80 - Microeconomics - - Information, Knowledge, and Uncertainty - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:swe:wpaper:2015-20a. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Hongyi Li (email available below). General contact details of provider: https://edirc.repec.org/data/senswau.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.