IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0121595.html
   My bibliography  Save this article

Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment

Author

Listed:
  • Christoph Bartneck
  • Andreas Duenser
  • Elena Moltchanova
  • Karolina Zawieska

Abstract

Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon’s Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies.

Suggested Citation

  • Christoph Bartneck & Andreas Duenser & Elena Moltchanova & Karolina Zawieska, 2015. "Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-23, April.
    • Handle: RePEc:plo:pone00:0121595
    DOI: 10.1371/journal.pone.0121595
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0121595
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0121595&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0121595?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    2. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    3. Rajagopal, 2014. "The Human Factors," Palgrave Macmillan Books, in: Architecting Enterprise, chapter 9, pages 225-249, Palgrave Macmillan.
    4. David J. Spiegelhalter & Nicola G. Best & Bradley P. Carlin & Angelika Van Der Linde, 2002. "Bayesian measures of model complexity and fit," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(4), pages 583-639, October.
    5. Alan Agresti & David B. Hitchcock, 2005. "Bayesian inference for categorical data analysis," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 14(3), pages 297-330, December.
    6. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Laura Biziou-van-Pol & Jana Haenen & Arianna Novaro & Andrés Occhipinti & Valerio Capraro, 2015. "Does telling white lies signal pro-social preferences?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(6), pages 538-548, November.
    2. Debra A. Hoffmann & Jenna M. Marx & Jacob M. Burmeister & Dara R. Musher-Eizenman, 2018. "Friday Night Is Pizza Night: A Comparison of Children’s Dietary Intake and Maternal Perceptions and Feeding Goals on Weekdays and Weekends," IJERPH, MDPI, vol. 15(4), pages 1-12, April.
    3. Oliver Hahl, 2016. "Turning Back the Clock in Baseball: The Increased Prominence of Extrinsic Rewards and Demand for Authenticity," Organization Science, INFORMS, vol. 27(4), pages 929-953, August.
    4. Casella, Alessandra & Sanchez, Luis, 2019. "Storable Votes and Quadratic Voting. An Experiment on Four California Propositions," CEPR Discussion Papers 13479, C.E.P.R. Discussion Papers.
    5. Scott, Carol F. & Bay-Cheng, Laina Y. & Nochajski, Thomas H. & Lorraine Collins, R., 2024. "Emerging adults’ social media engagement & alcohol misuse: A multidimensional, person-centered analysis of risk," Children and Youth Services Review, Elsevier, vol. 159(C).
    6. Oliver Hahl & Jaekyung Ha, 2020. "Committed Diversification: Why Authenticity Insulates Against Penalties for Diversification," Organization Science, INFORMS, vol. 31(1), pages 1-22, January.
    7. repec:cup:judgdm:v:10:y:2015:i:6:p:538-548 is not listed on IDEAS
    8. Collin Weigel & Laura A. Paul & Paul J. Ferraro & Kent D. Messer, 2021. "Challenges in Recruiting U.S. Farmers for Policy‐Relevant Economic Field Experiments," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(2), pages 556-572, June.
    9. Young Kim & Hyojung Park, 2017. "Is there Still a PR Problem Online? Exploring the Effects of Different Sources and Crisis Response Strategies in Online Crisis Communication Via Social Media," Corporate Reputation Review, Palgrave Macmillan, vol. 20(1), pages 76-104, February.
    10. Tasci, Asli D.A., 2018. "Testing the cross-brand and cross-market validity of a consumer-based brand equity (CBBE) model for destination brands," Tourism Management, Elsevier, vol. 65(C), pages 143-159.
    11. Hongsha Wang & Qihui Chen & Bhagyashree Katare, 2023. "Nudging Chinese consumers to embrace sustainable milk consumption: How should information be provided?," Agribusiness, John Wiley & Sons, Ltd., vol. 39(S1), pages 1512-1534, December.
    12. Kamryn S. Morris & Eleanor K. Seaton & Masumi Iida & Sarah Lindstrom Johnson, 2020. "Racial Discrimination Stress, School Belonging, and School Racial Composition on Academic Attitudes and Beliefs among Black Youth," Social Sciences, MDPI, vol. 9(11), pages 1-20, October.
    13. Ksenia Koroleva & Jasminko Novak, 2020. "How to Engage with Sustainability Issues We Rarely Experience? A Gamification Model for Collective Awareness Platforms in Water-Related Sustainability," Sustainability, MDPI, vol. 12(2), pages 1-24, January.
    14. Feenstra, Sanne & Jordan, Jennifer & Walter, Frank & Stoker, Janka I., 2020. "Antecedents of leaders' power sharing: The roles of power instability and distrust," Organizational Behavior and Human Decision Processes, Elsevier, vol. 157(C), pages 115-128.
    15. Paul Delfabbro & Neophytos Georgiou & Catia Malvaso & Daniel King, 2020. "Is Self-Reported Propensity for Everyday Illusions of Control Higher in Gamblers and Is It Associated With Gambling-Specific Erroneous Beliefs?," SAGE Open, , vol. 10(1), pages 21582440198, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    2. Masha Shunko & Julie Niederhoff & Yaroslav Rosokha, 2018. "Humans Are Not Machines: The Behavioral Impact of Queueing Design on Service Time," Management Science, INFORMS, vol. 64(1), pages 453-473, January.
    3. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.
    4. Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    5. Haas, Nicholas & Hassan, Mazen & Mansour, Sarah & Morton, Rebecca B., 2021. "Polarizing information and support for reform," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 883-901.
    6. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, Institute of Labor Economics (IZA).
    7. Atalay, Kadir & Bakhtiar, Fayzan & Cheung, Stephen & Slonim, Robert, 2014. "Savings and prize-linked savings accounts," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 86-106.
    8. Azzam, Tarek & Harman, Elena, 2016. "Crowdsourcing for quantifying transcripts: An exploratory study," Evaluation and Program Planning, Elsevier, vol. 54(C), pages 63-73.
    9. Dato, Simon & Feess, Eberhard & Nieken, Petra, 2019. "Lying and reciprocity," Games and Economic Behavior, Elsevier, vol. 118(C), pages 193-218.
    10. Wladislaw Mill & Cornelius Schneider, 2023. "The Bright Side of Tax Evasion," CESifo Working Paper Series 10615, CESifo.
    11. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    12. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    13. Lefgren, Lars J. & Sims, David P. & Stoddard, Olga B., 2016. "Effort, luck, and voting for redistribution," Journal of Public Economics, Elsevier, vol. 143(C), pages 89-97.
    14. Jochen Becker & Josip Medjedovic & Christoph Merkle, 2019. "The Effect of CEO Extraversion on Analyst Forecasts: Stereotypes and Similarity Bias," The Financial Review, Eastern Finance Association, vol. 54(1), pages 133-164, February.
    15. Bidhan L. Parmar & Adrian Keevil & Andrew C. Wicks, 2019. "People and Profits: The Impact of Corporate Objectives on Employees’ Need Satisfaction at Work," Journal of Business Ethics, Springer, vol. 154(1), pages 13-33, January.
    16. Alexsandros Cavgias & Raphael Corbi, Luis Meloni, Lucas M. Novaes, 2019. "EDITED DEMOCRACY: Media Manipulation and the News Coverage of Presidential Debates," Working Papers, Department of Economics 2019_17, University of São Paulo (FEA-USP).
    17. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," GLO Discussion Paper Series 1157, Global Labor Organization (GLO).
    18. Ola Andersson & Jim Ingebretsen Carlson & Erik Wengström, 2021. "Differences Attract: An Experimental Study of Focusing in Economic Choice," The Economic Journal, Royal Economic Society, vol. 131(639), pages 2671-2692.
    19. Tim Straub & Henner Gimpel & Florian Teschner & Christof Weinhardt, 2015. "How (not) to Incent Crowd Workers," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 57(3), pages 167-179, June.
    20. Nicolas Jacquemet & Alexander G James & Stéphane Luchini & James J Murphy & Jason F Shogren, 2021. "Do truth-telling oaths improve honesty in crowd-working?," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-18, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0121595. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.